DevOps lessons from 7 production deployments.

Over the past year we shipped seven production systems on AWS. Payment platforms, e-commerce sites, AI agents, customer portals. Different teams, different scales, different failure modes. The systems are unrelated. The lessons are the same.

Most of these are not in the docs. They are the kind of thing you learn at 2 a.m. on a Saturday when an alert goes off and the runbook is wrong.

1. The health check is the first feature, not the last

Every ECS service needs a real health check. Not a TCP check. Not "did the process start." A real HTTP endpoint that confirms the service can reach its database, its cache, and its message queue. And that endpoint cannot make slow calls or it will time out and flap.

We learned this when an ALB target group started failing because the health check was hitting an endpoint that called Sentry on every request. Sentry was slow that day. The service was healthy. The health check timed out. The service got killed. The replacement also got killed. The whole cluster went red over a Sentry hiccup.

The fix was a dedicated /api/health endpoint that does one cheap thing and returns. We add it on day one now. It does not go in any router with middleware. It does not log. It does not authenticate. It just answers.

2. Desired count is a number that drifts

ECS service desired count can quietly become zero. Maybe someone scaled it down to debug. Maybe a deployment failed in a way that left the count at zero. Either way, your next deployment will look like it succeeded and ship nothing, because zero is a valid count and ECS will dutifully roll out zero new tasks.

We added a check to every deployment script that reads the current desired count and forces it back up if it is zero. It is one more step that prevents one more silent failure. The deployment is not done until the service has running tasks.

3. Build for the platform you actually run on

The build environment matters more than people admit. We build on ARM64 because Graviton is cheap and fast. The Mac dev machines are ARM64. The CI runners are ARM64. So when we shipped a service that had a glibc dependency in a multi-arch Docker image, it ran fine locally and crashed on the production Alpine image.

The fix was libc6-compat in both the builder and the production stage of the Dockerfile. The lesson is that "it works on my machine" is even more dangerous when your machine is the same architecture as production but a different OS family. Test the actual artifact, not the source code.

4. The first deploy of any new app needs node_modules attention

We use Docker volumes for node_modules in development to keep the host filesystem clean. That is a normal pattern. The gotcha is that adding a new dependency on the host does not update the volume. You have to npm install inside the container or rebuild the image.

Twice now we have shipped a new SDK locally, watched it work, and then watched the staging container crash because the SDK was not installed in the volume. Now we have a checklist: any new dependency triggers a container rebuild before we even think about shipping.

5. Webhooks need raw bodies

Stripe, Sentry, Twilio, Shopify, anyone who signs their webhooks with HMAC, needs the raw request body to verify the signature. If your framework parses JSON before the verifier runs, the bytes you hash will not match the bytes the sender hashed. The signature fails. The webhook gets rejected. The customer wonders why their subscription did not activate.

NestJS needs rawBody: true in the factory. Express needs the raw body middleware mounted before express.json. Whatever your framework, the rule is the same: HMAC verification requires byte-exact parity. You cannot reconstruct it from a parsed object. Lock this down before the first webhook ships.

6. Backups that are not tested are not backups

Daily snapshots are easy. Restoring from a snapshot under stress is not. We had a database we backed up religiously for six months. The first time we tried to restore one in a real incident, the schema version did not match the running app, the backup was missing the last hour of data, and the restore took 40 minutes. None of that was a problem we discovered during normal operations.

Now we do quarterly fire drills. Pick a snapshot. Restore it to a throwaway environment. Confirm the app boots against it. Time it. Document what broke. The drill takes a couple of hours. The confidence it produces is the only thing that lets you sleep during a real incident.

7. The deploy is the last step, not a debugging tool

The single biggest source of production fires we see is using the deploy as a way to find out if the change works. "Push it to staging and we will see." That is not engineering. That is gambling.

Before deploy: tests pass locally, types check, lint clean, security review done, eval harness for AI features green, schema migration dry-run clean, and a senior engineer has read the diff. After all of that, then deploy. Deploys are boring when the work that precedes them is not.

The corollary is that deploys are always last. They are not a phase. They are not parallel with QA. They are the consequence of the work being done. If you find yourself debugging in production, something earlier in the pipeline failed and nobody noticed.

Boring is the goal

The pattern across all seven deployments is that the systems we are proudest of are the ones nobody notices. No 2 a.m. alerts. No customer-facing bugs that needed a hotfix. No long incident retrospectives. The site went up. The site stayed up. The customers got what they paid for.

DevOps maturity is not how impressive your dashboards look. It is how rarely you have to look at them.