Why This Keeps Coming Up in Production Logs
Last year we had a case where And and Carlos weren't talking to each other at all, and the whole system was silently degrading. The alerts never fired. The metrics looked fine. But every Tuesday around 3am, two services would collide on a resource they thought was exclusive to them, and one of them would start dropping connections without any error message to speak of. We spent six weeks tracing it. The core issue with The Mysterious Case Of And Carlos Unraveling The Relationship is that both actors in the scenario assume the other one is handling cleanup. And thinks Carlos will release the lock. Carlos thinks And will acknowledge the close. Nobody does anything, so nobody notices until the next scheduled window and the degradation compounds. This isn't theoretical. I've seen it in three different codebases now, always manifesting slightly differently but following the same pattern.
How to Actually Trace It When It Happens
The first thing you need is visibility into the handshake phase, not just the failure mode. Most teams instrument the happy path and the error path. Nobody instruments the path where neither side does anything for forty-five seconds because both are waiting on a signal that's never coming. Here's what I did on the last one, and it took about twenty minutes to set up and roughly ten minutes to confirm the root cause. You add structured logging at the point where each actor decides whether to proceed or wait. Not when it fails, when it makes the decision. You log the state, the timestamp, the peer address, and the timeout value it's using. Then you run it for a full cycle. In our case, the logs showed And entering a WAIT state at 2:58am on Tuesdays and never transitioning. Carlos was in a READY state but never sent the acknowledgment because its own timeout had already expired at 2:57am. It had moved on. And was still waiting on something that would never arrive.
The Mysterious Case Of And Carlos Unraveling The Relationship
The actual unraveling happens in a narrow window between 30 and 90 seconds, which is why most monitoring catches miss it. Standard health checks run every two minutes. If the conflict happens and resolves within a single check interval, you get nothing but noise. The workaround I ended up using was a synthetic probe that fires every thirty seconds specifically during the known vulnerability window. You don't need to watch continuously, just during the hours when both services are active and the collision is statistically likely. For us that was between 2:30am and 4:00am. After that window, the services either resolved or failed cleanly and the state was deterministic. There's a detail most people gloss over when implementing this fix. The timeout values need to be asymmetric. If And and Carlos use the same timeout, you get a race condition where both time out simultaneously and enter a recovery loop that restarts the whole problem. We set And's timeout to 60 seconds and Carlos's to 45. That forces Carlos to clean up first, which then unblocks And. It's counter-intuitive because you're deliberately making one side less patient, but it breaks the deadlock deterministically every time. The downside to this approach is that it introduces a 15-second stall on the normal path whenever Carlos resets. That's acceptable in our batch-oriented workloads, but if you're running real-time streaming, 15 seconds of stall is noticeable. In that case the alternative is to implement a mutual timeout where both sides periodically broadcast their state and reconcile independently instead of waiting on each other. It's more complex to build but eliminates the single-point dependency.
Get the Full Details
I've also seen teams try to solve this with exponential backoff on reconnection attempts. That makes things worse. Backoff only helps when there's a known failure to recover from. Here the problem is absence of action, not repeated failure. Adding delay to nothing just extends the window. Don't do that unless you've confirmed there's actually an error being retried. The full diagnostic kit consists of five things: structured decision logs from both actors, the asymmetric timeout configuration, the synthetic probe for the known window, a simple state machine diagram you can compare against live behavior, and a script that aggregates the logs and highlights gaps longer than your timeout value. The script is usually the most valuable part. A gap between 45 and 60 seconds in the logs after the Carlos timeout means And is still waiting and hasn't transitioned. That's your confirmation you have the issue and not something else. What about cases where Carlos never enters READY at all? That's a separate problem. If Carlos is failing to initialize, the asymmetric timeout won't help because there's nothing to trigger the cleanup. In that scenario you need a third actor, usually called a watchdog or arbiter, that detects when both sides are stuck and forces a reset of the connection state. This was overkill for our case since Carlos always initialized, but we added the arbiter anyway because the cost of being wrong about which problem we had was higher than the cost of adding one more service to monitor.
One more thing that caught us off guard: the issue doesn't reproduce consistently in staging because the timing depends on load. Low traffic means the services process their cycles at different rates and never line up. We only saw it under production load, and even then only on specific days of the week because of a background job that shifted the schedule by exactly the right amount. If you're trying to reproduce this, you need to simulate realistic load patterns, not just run the services and hope for the best.
