The Infrastructure Nobody Notices Until It Breaks
St. Paul operates on a grid of invisible clocks. Most people never think about it because it works. The light turns green when you expect it. The train departs on a schedule that actually matches the timetable. Blood banks rotate their stock rotations without crossing paths with a single calendar error. Behind all of that is something called network time infrastructure and every few years something goes wrong in a way that nobody expects. Here is what that actually looks like under the hood. The city uses a hierarchy of time sources. At the top you have stratum 1 servers, usually GPS or Galileo disciplined oscillators sitting in server rooms or on rooftops with sky-facing antennas. Those feed stratum 2 servers, which are the machines your network devices talk to every day. From there the time ripples down through stratum 3, 4, and everything else until it reaches the desk clock on your phone. The protocol doing most of the heavy lifting is NTP, Network Time Protocol, specifically version 4 as defined in RFC 5905. The city also runs PTP, Precision Time Protocol, for the facilities that need sub-microsecond accuracy. That means the transit signal system, certain hospital equipment, and the emergency communications backbone. If you are not in one of those categories, PTP is overkill and will just cause more problems than it solves.
How It Actually Works In Practice
I spent about three years managing time sync for a mid-size municipal network and the first thing you learn is that the documentation always lies by omission. The manual says NTP will synchronize in under ten seconds. It does not mention the three-hour window where your client will sit at stratum 16 and refuse to pull from any server because its local oscillator is drifting more than five hundred milliseconds from the reference. You have to force it. The command is ntpq -c peercmd peer your-server maxpoll 10 minpoll 10 and then wait. Patience is the actual skill here. One edge case I still think about involved the Ramsey Road corridor. We had a fiber cut that rerouted traffic through a temporary microwave link with about forty milliseconds of asymmetrical latency. NTP was reporting normal synchronization but every system that depended on it was drifting by roughly sixty seconds per day because the round-trip time calculations were garbage. The workaround was simple but not obvious if you are just reading the RFCs. I disabled the default Klipper algorithm and switched to a basic linear filter with a hard cap on acceptable dispersion. The command looked like this: server 10.0.0.1 iburst maxdist 0.0512 minpoll 4 maxpoll 6
That maxdist value of 0.0512 seconds is fifteen times tighter than the default. It forced the daemon to reject measurements that looked suspicious and rely more on the local oscillator in the meantime. The tradeoff was slightly higher short-term jitter but nothing a human operator would notice. The long-term drift dropped to under two milliseconds per day. That was the difference between the transit dispatch system staying correct and it slowly desynchronizing until someone at 3 AM noticed the timing files had drifted by four minutes.
Get the Full Details
Where People Get It Wrong
The most common mistake is assuming that having NTP installed means you have time sync. A lot of organizations install the daemon, point it at a public pool, and consider the problem solved. Public pool servers are shared resources and they are often overloaded. If you are running something that matters, you should be using the government pool or your own upstream. The USNO maintains a list at time.nist.gov and the North American pool runs at pool.ntp.org but neither of those is appropriate for production infrastructure. You want a direct upstream, preferably your own stratum 2 server feeding your LAN. Another counter-intuitive thing: more servers is not better. NTP was designed to pick the best subset from a group of twelve. When you give it twelve servers that are all slightly wrong in different directions, the algorithm does its job but the convergence time stretches out. Eight well-chosen servers from different strata and different upstream providers will always outperform twelve random ones. I have seen networks where removing four servers actually improved stability because the outliers got filtered out faster. PTP has its own trap. People assume it eliminates NTP entirely. It does not. PTP handles the distribution within a LAN at microsecond precision. NTP still anchors the PTP grandmaster to UTC. If you skip the NTP upstream, your entire PTP domain is free-floating and nobody will know until an audit flag shows your timestamps don't match the legal time standard for the jurisdiction. In Minnesota that standard is maintained by the National Institute of Standards and Technology and it matters for compliance purposes in healthcare and transportation.
What You Need To Actually Deploy This
If you are setting up time sync for a facility in the St. Paul area or anywhere in the metro, here is the practical stack. Get a dual-mode GPS/GNSS receiver that feeds a NIST-traceable PPS signal into a dedicated server. Ubuntu Server with chrony is simpler than ntpd for most use cases and handles bad networks better because it uses a different calculation model. The configuration lives in /etc/chrony/chrony.conf and the critical line is refclock SHM 0 offset 0.5 delay 0.2 poll 3 adjusted for your specific hardware. Then set at least four upstream NTP servers from different provider pools. For PTP deployments you need a hardware timestamping capable switch and a PTP grandmaster with a PPS input. The Grandstream and Meinberg models are what I have used. Configure the switch with ptp profile 128a for the IEEE 1588 standard and verify with ptpv2-grandmaster status that the holdover oscillator is engaged when the GPS signal drops. It should be. If it is not, your PTP domain will collapse within seconds of a satellite outage. Monitoring is where most people fail. Set up chronyc tracking or ntpq -p output logging every sixty seconds. The drift rate column is the one that tells you if your local oscillator is degrading. If it climbs past two hundred milliseconds per second you need to replace the hardware oscillator or the GPS disciplining circuit. Ignoring that number leads to the kind of slow drift that creates compliance failures months later.
What This System Does Not Handle Well
Time sync infrastructure is fragile in ways that are easy to underestimate. A cable construction crew cutting fiber on Rice Street does not just take down your internet. They take down the upstream NTP path and your grandmaster's PTP discipline source at the same time. You will fall back to holdover mode and your local oscillators will keep you accurate for a while but not forever. A rubidium oscillator holds within a few nanoseconds per day. A cheap TCXO drifts by microseconds. Know what you are running. Weather also matters more than you would think. Heavy snow load on a rooftop GPS antenna can detune it just enough to increase measurement noise. I have seen a forty percent jump in dispersion after a winter storm because the radome was ice-loaded. The system did not fail but the quality of the time samples dropped enough that downstream clients started seeing slightly worse accuracy. Worth checking the antenna physically after major weather events. The biggest limitation is that none of this prevents human error in scheduling. A time sync system can be perfectly synchronized and your operations team can still miss a window because the change ticket said 2 AM and someone read it as 2 PM. That is not a time sync problem. That is a process problem. But it happens constantly in municipal environments where shift handoffs are informal and documentation is stale.
There is no clean software solution for the handoff issue. The best approach is redundant time display panels in operations rooms showing UTC and local simultaneously, plus automated ticketing that timestamps everything in UTC and converts only at display time. It is mundane and unglamorous but it catches the errors that actually matter.
Bottom Line
Time synchronization in a city like St. Paul is a layered system of GPS receivers, NTP daemons, PTP grandmasters, and a lot of monitoring scripts that run unattended. It works until a fiber cut, a bad config change, or a drifting oscillator reveals that nobody was actually watching the drift column. The systems that stay reliable are the ones where someone checks chronyc tracking output every week and has a documented failover path for when the GNSS signal disappears. Everything else is just hope dressed up as infrastructure.
