When Blue-Green Goes Wrong
It was 2 AM on a Tuesday when our primary payment service went dark. We’d just executed what should have been a routine blue-green deployment to our Kubernetes cluster, switching traffic from the old version to the new one with a simple kubectl patch. The health checks were green, the metrics looked normal, and then our Slack channels exploded with alerts. Customers couldn’t complete purchases, and every minute cost us real money.
The issue wasn’t with Kubernetes itself, but with our understanding of how database connection pooling behaved during the cutover. Our new pods were healthy according to our readiness probes, but they hadn’t properly warmed their connection pools. When traffic hit, the database couldn’t handle the sudden spike of new connections from 40 fresh pods simultaneously. This taught me that production deployment strategies aren’t just about moving code around. They’re about understanding how your entire system behaves under real load.
Rolling Updates: The Default That’s Usually Wrong
Kubernetes ships with rolling updates as the default deployment strategy, and for most applications, this creates more problems than it solves. The standard configuration replaces 25% of your pods at a time, which sounds reasonable until you consider what happens to your database connections, in-memory caches, and established TCP connections during this process. I’ve watched teams struggle with mysterious 503 errors during deployments because their load balancer was routing traffic to pods that were terminating but hadn’t finished their graceful shutdown.
The real issue with rolling updates is that extended deployment window. If you’re running 20 replicas and replacing 5 at a time, you have a 10-15 minute period where your cluster is in a mixed state. This gets messy when you’re deploying schema changes or when your application expects all instances to be running the same version. I learned this the hard way during a deployment that included a message queue format change. Old consumers couldn’t process messages from new producers, creating a growing backlog that took hours to clear.
When rolling updates do work well, it’s usually for stateless services with minimal startup time and no expectations of version consistency across instances. But even then, I prefer being more deliberate about the process. Setting maxUnavailable to 0 and maxSurge to a specific number gives you better control over resource usage and deployment timing.
Blue-Green: Simple in Theory, Complex in Practice
Blue-green deployments promise zero downtime by maintaining two identical environments and switching traffic between them. In Kubernetes, this typically means managing two separate deployments and updating a service selector to point traffic from one to the other. The appeal is obvious: instant rollback, clear separation of versions, and no mixed-state periods.
But blue-green requires doubling your resource allocation during deployments, which can be expensive and sometimes impossible depending on your cluster capacity. More importantly, it assumes your application is truly stateless. I once implemented blue-green for a service that wrote to a shared database, only to discover that the brief period where both versions were live created duplicate processing of certain events. The financial reconciliation team was not amused.
That database connection issue I mentioned earlier also applies here. Even with proper readiness probes, you need to account for connection pool warmup, cache population, and any other initialization your application requires. I now build in a soak period where the green environment gets a small percentage of traffic before the full cutover. This isn’t traditional blue-green, but it’s more reliable in practice.
Canary Deployments: The Graduate-Level Approach
Canary deployments represent the most sophisticated approach, gradually shifting traffic from the old version to the new one while monitoring key metrics. This requires proper observability infrastructure and clear success criteria, but it’s the only strategy that scales to truly critical systems. The complexity lies not in the Kubernetes configuration, but in defining what makes a successful deployment.
I typically start with 5% traffic to the new version for 10 minutes, monitoring error rates, latency percentiles, and business metrics like conversion rates. If everything looks good, I move to 25%, then 50%, then 100%. The key insight is that each stage must have specific success criteria. “It looks fine” isn’t good enough when you’re dealing with production traffic. You need automated checks that can halt the deployment if things go wrong.
Building effective canary deployments requires investment in tooling. You need a way to split traffic (service mesh, ingress controller with weighted routing, or application-level feature flags), comprehensive metrics collection, and ideally automated promotion based on those metrics. Tools like Flagger or Argo Rollouts can help, but they’re not magic. They still require you to define what success looks like for your specific application.
The Human Element: Runbooks and Recovery
The best deployment strategy is worthless without proper runbooks and recovery procedures. Every production deployment should have a clear rollback plan that doesn’t require the person who initiated the deployment to be available. I’ve seen too many late-night incidents escalate because the only person who understood the deployment process was unreachable.
Your rollback procedure should be tested regularly, not just documented. I schedule monthly “chaos days” where we intentionally break things and practice our recovery procedures. This isn’t just about the technical steps, but about communication protocols, escalation paths, and decision-making under pressure. The best technical solution means nothing if your team can’t execute it reliably at 3 AM.
Consider also the business impact of your deployment windows. That $50K incident I mentioned happened because we deployed during peak traffic hours in our primary market. Now we align deployment schedules with business impact, deploying critical services during low-traffic periods and having dedicated on-call coverage during planned changes. This might seem obvious, but it’s surprising how often technical teams operate in isolation from business context.
The choice between deployment strategies isn’t just technical. It’s about understanding your system’s failure modes, your team’s capabilities, and your business’s tolerance for risk. What deployment patterns have you found most reliable in your production environments?