S/08 · Services
Infrastructure & DevOps
Built to stay up when things go down.
Kubernetes clusters and the pipelines around them: spread across regions, ready for disaster, monitored around the clock and updated without downtime.
- Runs on
- Your servers, cloud, or both
- Designed for
- High availability & recovery
- Delivered as
- Infrastructure as code
- 01
Deploys are scary and happen late at night.
- 02
One server going down takes the business with it.
- 03
You have backups but have never restored one.
- 04
Your cloud bill grows faster than your traffic.
- D/01
Kubernetes clusters
We build clusters on your own servers, a cloud provider, or both, with the same layout everywhere: three control-plane nodes, separate worker pools, ingress, storage and certificates. Every setting is written down, so anyone on your team can rebuild the cluster, not only the person who set it up.
Included
- Three control-plane nodes and separate worker pools
- Ingress, TLS certificates and DNS handled automatically
- Persistent storage with scheduled snapshots
- Access by role, with no shared admin account
kubectl get nodes Node Role Zone Status Version cp-1 control-plane zone-a Ready v1.31.4 cp-2 control-plane zone-b Ready v1.31.4 cp-3 control-plane zone-c Ready v1.31.4 worker-1 worker zone-a Ready v1.31.4 worker-2 worker zone-b Ready v1.31.4 worker-3 worker zone-c Ready v1.31.4 - D/02
High availability
Every part that matters runs at least twice, in separate zones, so losing a server, a rack or a whole zone is an alert rather than an outage. We agree the targets with you first: how much downtime is acceptable, and how much data you can afford to lose.
Included
- Services spread across zones, with no single point of failure
- Database replication with automatic failover
- Health checks that take broken instances out of rotation
- Availability targets agreed up front and then measured
docs/topology.mmd - D/03
Disaster recovery
Backups only count if they restore. We back up your data and cluster state, copy it off-site, and then restore it on a schedule, timing every step. You get a recovery time that has been measured, and a list of what to fix when a drill falls short.
Included
- Encrypted backups, copied to a second location
- Point-in-time recovery for the database
- Scheduled restore drills, with timings recorded
- A step-by-step runbook for the worst day
runbooks/restore-drill.md - Cluster state restored from Velero to a clean cluster14 min
- PostgreSQL restored to 09:42, two minutes before the faultRPO 2 min
- Off-site copy opened with the recovery keyverified
- DNS switched to the standby zonemanual step
- Uploaded files restored in 38 minutes, against a 30-minute targetover target
- D/04
CI/CD pipelines
Every change goes through the same path: tests, a container build, a security scan, then a staged roll-out that stops on its own if errors climb. Deploying becomes a routine part of the day, and going back takes one command.
Included
- Tests and image builds on every merge request
- Image scanning before anything reaches production
- Staged roll-outs through Argo CD, with automatic rollback
- Separate staging and production environments
pipeline #1842 · main $ git push origin main
✓ test go test ./... 212 passed 48s
✓ build registry.example.com/booking:3f9c2a1
✓ scan trivy: 0 critical, 0 high
✓ deploy staging · rollout complete 1m12s
✓ canary production 10% · errors 0.02%
✓ promote production 100% · argocd synced
- D/05
Monitoring & alerts
We collect metrics and logs in one place and turn them into dashboards and a short list of alerts that matter. Each alert says what broke, who owns it and where the runbook is, so the person woken at night can act instead of guess.
Included
- Prometheus metrics and Grafana dashboards for every service
- Central logs in Loki, searchable by request
- Alerts routed by severity and owner, with runbook links
- Service-level objectives with an error budget
grafana · booking · SLO Availability · 30 days
99.97%
target 99.9%
p95 latency
184 ms
under 300 ms
Error budget left
70%
13 of 43 min used
Alerts this week
3
all acknowledged
- D/06
Infrastructure as code
Servers, networks, clusters and the apps on them are described in a repository and applied by tools, not by hand. Any environment can be rebuilt from scratch, every change is reviewed like code, and nothing depends on one person’s memory.
Included
- Terraform for servers, networks and DNS
- Helm charts and Argo CD for everything in the cluster
- Secrets encrypted in the repository, never in plain text
- A readme that takes a new engineer from zero to deployed
infra/ - infra/
- README.mdstart here
- terraform/
- network.tfsubnets, firewall
- cluster.tfnodes, zones
- dns.tfexample.com
- k8s/
- apps.yamlapp of apps
- values.yamlHelm
- secrets/SOPS-encrypted
- restore-drill.mdrecovery
- 01
Assess
What runs where, what it costs and what happens when each piece fails.
- 02
Design
Topology, failover and recovery targets agreed before anything is built.
- 03
Migrate
Services moved over one at a time, with a way back at every step.
- 04
Drill & hand over
We break it on purpose, prove recovery works, then hand over the runbooks.
Most infrastructure engagements take six to ten weeks, depending on how many services move and where they run. Your services keep running on the old setup until each one has been moved and checked on the new one.
- Week 1
We map what runs where, what it costs, and what happens when each part fails.
- Week 2
Topology, recovery targets and the order of migration, agreed with your team.
- Weeks 3–4
The cluster, pipelines and monitoring built as code, staging first.
- Weeks 5–8
Services moved one at a time, each with a tested way back.
- Weeks 9–10
A failure drill and a restore drill, then the runbooks handed to your team.
Infrastructure as code
The repository that builds every server, network and cluster, in your own Git.
Pipelines
Build, test and deploy pipelines that run without us.
Runbooks
Step-by-step guides for deploys, failures and restores, proven in a drill.
Dashboards and alerts
Monitoring that tells your team what is wrong and who owns it.
Credentials and access
Every account, key and certificate in your name, with our access removed at the end.
A follow-up drill
We come back after three months and run a restore drill with your team.
- Kubernetes
- Helm
- Argo CD
- Terraform
- Prometheus
- Grafana
- Loki
- Velero
Do we need Kubernetes?
Not always. For a few services a simpler setup is better, and we will tell you so.
Can it run on local or isolated infrastructure?
Yes. We design for local data centres, private mirrors and networks with limited international connectivity.
What about updates?
Rolling updates and staged roll-outs, so the cluster and your apps stay current without downtime.
How long does it take?
Six to ten weeks for most setups. A review of your current infrastructure, with a prioritised list of fixes, takes about two.
Will we depend on you afterwards?
No. Everything is in your repository and documented, and your team runs the drills with us before we leave. Ongoing support is available if you want it, as a separate choice.
Tell us what you are building.
Every project starts with a consultation: you bring the idea and everything you know, we turn it into a design you can build on.
Book a consultation