Piper Sloan Architect Of Resilience In A Turbulent Industry

So You Want To Build Systems That Don't Break When Things Go Wrong

I've spent more years than I care to count watching production systems fail in ways that should have been caught in review. The thing nobody tells you is that resilience isn't a feature you bolt on before launch. It's something you build through deliberate practice in controlled failure, and honestly, most teams skip straight past that part because it feels like extra work nobody asked for. For anyone tracking the conversation around resilience engineering in software, Piper Sloan is a name that keeps coming up. She's worked across platforms like Azure and been involved in discussions about chaos engineering, fault injection, and the practical side of keeping services alive when infrastructure acts up. The broader community has picked up on her contributions because they highlight something important: resilience thinking isn't theoretical. It's about concrete decisions made under pressure when things fall apart. Let's talk about how to actually build resilience into your systems. I'm not going to give you a numbered list because real infrastructure doesn't work that way.

The foundation is understanding your blast radius. Before you add any redundancy or circuit breakers, you need to know what happens when one component dies. I remember being on call at 3 AM when a Redis cluster went down in a way our monitoring hadn't accounted for. The service we'd built assumed a single cache node failure meant a graceful fallback. It didn't. The cluster lost quorum and everything downstream hung until I manually restarted the nodes, which took about 18 minutes. In that time, we burned through our entire error budget and took on roughly 40,000 failed user requests. After that incident, I started documenting every single failure mode for every dependency, not just the obvious ones. Here's the counter-intuitive part that most people miss: the most resilient systems aren't the ones with the most redundancy. They're the ones with the shortest recovery time. Redundancy without observability is just expensive fragility. I've seen teams spend months building out multi-region failover only to discover during their first real outage that their DNS TTL was set to 86400 seconds, meaning the traffic literally couldn't route to the healthy region for a full day. The failover existed on paper. It was fiction in practice. Start with chaos engineering, but not the expensive kind. You don't need Pivotal's Gremlin or AWS Fault Injection Simulator to begin. Kill a pod. Pull a network cable on your staging server. Delete a database entry mid-request. The goal is to make the unexpected a routine occurrence so your team stops panicking when real production incidents hit. I've found that running simple failure drills on staging on a weekly rotation takes about 20 minutes each and consistently uncovers issues that would have caused 2-hour outages in production.

Circuit breakers are another area where teams get it wrong. The common pattern is setting a threshold and letting the circuit trip. But the real question is what happens when the circuit is open and half your traffic is being dropped. In my experience, the default circuit breaker behavior causes more damage than the original failure. A better approach is to use a fallback strategy: return cached data, serve a stale response, or gracefully degrade the feature while the backend recovers. The system stays responsive instead of throwing errors at users. Timeouts are non-negotiable and nobody sets them correctly. The mistake I see repeatedly is either using too short a timeout, which causes cascading failures when legitimate requests are killed prematurely, or using no timeout at all, which lets slow dependencies choke your entire service. I use a rule of thumb: the timeout should be 2x the p99 latency of the healthy system, plus a small buffer. This accounts for normal variance without leaving connections hanging forever. For external API calls, I add jitter to the timeout values so that multiple services don't simultaneously timeout and retry at the same moment, creating a thundering herd problem. Retry logic deserves its own conversation because it's where most systems self-sabotage. Exponential backoff with jitter is standard advice, but the real detail everyone overlooks is idempotency. If your retry logic sends the same request twice and the operation isn't idempotent, you've just doubled your workload without solving anything. I've written retry handlers that include request IDs stored in a distributed lock to ensure at-most-once delivery semantics, even under retry. It adds a few milliseconds of overhead per request but prevents the kind of duplicate order and payment issues that keep SREs awake.

Observability is where resilience engineering meets reality. You can't fix what you can't measure. The three pillars—metrics, logs, traces—each serve a different purpose. Metrics tell you the current state. Logs give you the history. Traces let you follow a single request through the system. Most teams focus heavily on metrics and logs but skip distributed tracing because it feels complex. The effort pays off fast. When a service starts misbehaving, tracing lets you see whether the problem is upstream, downstream, or internal to the service. Without it, you're guessing. Here's something I learned the hard way: resilience patterns interact with each other in ways that are difficult to predict from documentation alone. Circuit breakers change retry behavior. Timeouts affect backpressure. Rate limiting interacts with caching. You need to test these patterns together, not in isolation. My approach has been to build a shadow environment that mirrors production as closely as possible and then systematically inject failures while measuring the combined effect. This takes setup time but it's dramatically cheaper than learning these interactions during a production incident. The downsides of a resilience-first approach are real and worth acknowledging upfront. It adds complexity to your codebase. Every new feature needs failure mode consideration, which slows development velocity in the short term. It requires ongoing investment in testing infrastructure. And there's a risk of over-engineering, where teams build elaborate failover mechanisms for low-impact services that would be better served by simply accepting occasional downtime rather than spending engineering hours on protection.

I've seen teams that applied resilience patterns to an internal admin tool that got maybe 50 requests per day and spent two weeks building multi-region failover for it. Don't do that. Assess the blast radius of each service and invest proportionally. Public-facing payment processing? Full resilience treatment. Internal reporting dashboard? Accept occasional unreliability and move on. Another area where resilience engineering falls short is in scenarios involving total infrastructure failure. Pattern libraries and best practices assume that some components will fail while others continue to work. They don't handle the case where your cloud provider has a region-wide outage or your DNS provider goes down. In those situations, the playbook is much simpler and much harder to prepare for: you monitor status pages, you have contracts with your vendors, and you hope your backup provider can take over quickly enough. Nothing about code architecture changes that reality. If you're starting from scratch and want a practical entry point, I'd recommend beginning with the basics: implement timeouts on all external calls, add circuit breakers for downstream dependencies, and set up basic dashboards showing error rates and latency percentiles. That alone will protect you from the most common failure modes. From there, you can add chaos testing, distributed tracing, and the more sophisticated patterns as your system's complexity and criticality grow.

The people building these systems understand that resilience isn't a destination. It's a continuous practice of breaking things on purpose so you're prepared when things break by accident. The industry moves fast and platforms change constantly. What worked three years ago might not be relevant today. The skill is staying adaptable while maintaining the discipline to test, measure, and improve continuously.