What Netflix Live Teaches Us About Building Reliable Systems
How does Netflix stream live events without crashing?
What If You Didn’t Have to Choose Between AI and Traditional SAST?
Presented by Checkmarx
Checkmarx One’s new Hybrid SAST engine combines deterministic rules with AI analysis to improve the quality of security findings before they reach developers. In Checkmarx’s published benchmark, the Hybrid engine achieved 3x the industry’s average fidelity score while reducing false positives by 60%.
How Netflix Streams Live Events Without Crashing
Why is live streaming so much harder than video on demand?
It’s not just because more people are watching.
It’s because live streaming removes the one thing on-demand systems rely on most: time.
With on-demand video, the content is already finished. Netflix has time to encode it, package it, test it, copy it to servers near viewers, and demand is spread out as people press play at different times.
Live has none of those luxuries.
The video exists only seconds before it has to reach the audience. Millions of viewers may arrive at the same time. A missing segment cannot be quietly fixed later. And if something goes wrong, every affected device may try to reconnect at once.
That changes the engineering problem.
Netflix isn’t just delivering video anymore. It’s building a system that has to survive unreliable venues, fluctuating networks, mismatched devices, regional traffic spikes, missing segments, retry storms, and operators making decisions while the event is still happening.
This is the story of how that system is built, and the more general lesson hiding inside it: at this scale, reliability comes from making failure predictable, contained, and recoverable.
Live vs on-demand streaming
On-demand streaming is hard, but it has one enormous advantage: the video is finished before anyone presses play.
Live streaming works differently.
The video is created while the audience is watching. A camera captures the event, production systems process the signal, the feed reaches the cloud, encoders create streamable versions, packagers prepare them for devices, origins expose the segments, the CDN distributes them, and players request the right piece at the right time; all running in seconds behind the actual live event.
When the benefits of on-demand streaming disappears, different failures start to matter:
No pre-computation → Encoding happens in real time, so there’s no batch window to hide mistakes.
Synchronized demand → Everyone arrives near the start, so traffic spikes vertically instead of ramping up.
No do-overs → A bad or missing segment reaches the audience immediately, so correctness has to happen the first time.
Large blast radius → One overloaded component can hit millions of viewers at once.
The hard part is that every layer has to work at the same time.
If the feed breaks, the cloud pipeline cannot recreate missing video.
If the origin cannot publish segments fast enough, the CDN has nothing useful to serve.
If players retry too aggressively, recovery traffic can become worse than the original failure.
So Netflix’s live architecture starts with a simple assumption: something will fail while the stream is running.
The system has to be ready before that happens.
Getting the signal to the cloud
The first hard problem isn’t the cloud. It’s the venue.
Live events happen in stadiums, theaters, arenas, studios, and temporary production sites. Some have strong connectivity and broadcast infrastructure. Many don’t. And if every venue becomes a custom networking project, every live event becomes a new integration risk.
For its first live comedy special, Netflix used two dedicated internet circuits from the venue and sent a high-quality mezzanine feed to the cloud.
It worked. But it exposed the real risk: every new venue could become its own custom integration.
So Netflix moved toward a hub-and-spoke model.
Instead of sending every production feed directly from the venue into the cloud, feeds first go to a well-connected broadcast facility. From there, they move into the cloud live pipeline.
The venue becomes the spoke.
The broadcast facility becomes the hub.
That creates a stable boundary before the core system.
And that is the first useful pattern: when the outside world is messy, do not let every variation leak into your architecture. Put a predictable layer in front of it.
Designing for the failure that will happen mid-stream
Once the signal reaches the cloud, the next problem is failure during the stream.
On-demand video can recover from small network issues by retrying a stable file. Live streaming can’t rely on that.
If a packet drops during a title fight, comedy special, or live game, millions of viewers may see the glitch at roughly the same time. So the live pipeline has to assume something will fail while the event is still running, and keep going anyway.
Netflix handles this by sending each live feed into two separate cloud regions. Each region also receives the feed over two different managed network paths, instead of relying on the open internet.
That gives Netflix four independent feed paths into the live pipeline.
At the packet level, Netflix uses SMPTE 2022-7, a broadcast standard for seamless protection switching. In simpler terms, the system receives duplicate packet streams and picks the best packets in real time. If one path drops packets, the other path can fill the gaps before viewers notice.
But the important idea is not simply “add more copies.”
The important idea is independence.
Two backups are not useful if they share the same failure mode.
Same cloud region → one regional outage can affect both copies.
Same network path → one carrier or routing issue can affect both copies.
Same storage backend → one datastore problem can affect both copies.
Same deployment mistake → one bad rollout can affect both copies.
Netflix is not just duplicating the stream. It’s reducing shared failure modes across regions, network paths, and feed candidates.
That distinction matters.
Not more copies. More independent copies.
The stream still has to work for everyone
Getting a reliable feed into the cloud is only the first part of the problem.
Now Netflix has to turn that feed into something viewers can actually watch.
A viewer on fiber with a 4K TV shouldn’t get the same stream as someone on a weak mobile connection. This is where adaptive bitrate streaming comes in: the player switches between quality levels as network conditions change.
Netflix tunes its bitrate ladder based on resolution, frame rate, and content complexity. Fast-motion events, like WWE, need higher bitrates because motion is harder to compress cleanly.
But quality is not the only constraint.
Netflix also has to support a large device ecosystem. Different devices may need different video, audio, language, and caption formats. Older devices may not support newer formats.
That device compatibility problem is easy to underestimate.
Netflix cannot assume viewers will upgrade their hardware for a live event. The system has to work on the devices people already use.
That is one reason Netflix uses HTTPS-based live streaming with 2-second segments. HTTPS is widely supported and fits the delivery infrastructure Netflix already runs.
The tradeoff is latency.
UDP-based protocols can deliver lower latency, but they are harder to support across every device and network.
Netflix chooses reach and stability over the lowest possible delay.
That is a recurring pattern in their architecture: the best technical choice is not always the fastest one.
Sometimes it’s the one that works reliably across the messy reality of production.
The better move is not always a new stack
A common mistake in new architecture is rebuilding too much.
Netflix didn’t build a separate live playback stack from scratch. It built custom live components only where it had to, then connected them to systems that already worked for on-demand.
The live packager is the best example.
Netflix built its own packager so live streams could integrate with existing playback systems, device compatibility work, captions, and Digital Rights Management.
That avoided years of repeated compatibility work across Netflix’s device ecosystem.
This is the quieter architectural decision, but it is one of the most useful.
Instead of building a parallel live stack, Netflix used the packager as an adapter into proven infrastructure.
New requirements often don’t need a new stack beside the old one. They need the adapter that lets them reuse the stack that already works.
The packager does one more quiet but important thing: it generates segment templates for Netflix’s PlayAPI backend. A segment template lets the player calculate the next segment URL from the current time, instead of repeatedly asking the backend what comes next.
That sounds small, but at live-event scale it matters.
If millions of devices constantly ask the control plane for the next segment, the backend becomes a bottleneck. Segment templates remove that coordination by letting each device calculate the answer locally.
At scale, client behavior becomes infrastructure behavior.
Static storage is not enough for live video
Static storage sounds like a perfectly reasonable origin; at first.
The packager writes segments into storage. The CDN reads from storage. With two regions cross-publishing, the CDN even has multiple candidates to choose from.
But live streaming puts pressure on storage that generic object stores weren’t built for.
Netflix’s early live pipeline used static storage buckets in AWS, with one bucket in each region. Packagers published segments to both buckets, giving the CDN multiple segment candidates.
That improved redundancy, but it did not solve the deeper problem.
Generic storage does not understand media quality.
A segment can exist and still be bad. It might be too short, miss frames, or have timing issues. A storage bucket can confirm the object exists, but it can’t decide whether one candidate is better than another.
So Netflix built a custom Live Origin.
The Live Origin sits between the cloud live pipelines and Open Connect, Netflix’s CDN. It acts as a broker that decides which content reaches the CDN and, eventually, client devices. Because it understands multiple pipelines and regions, it can choose valid segment candidates instead of treating every stored object as equal.
That’s the important difference.
The origin stops being passive storage and becomes a media-aware decision point.
The CDN absorbs synchronized demand
Once the live segments exist, Netflix still has to deliver them to viewers around the world.
This is where Open Connect helps.
Netflix uses Open Connect, its own content delivery network, for live delivery. Open Connect servers are placed close to viewers at over 6,000 locations and connect back to AWS locations through a dedicated Open Connect backbone network.
This matters because live traffic is synchronized.
For a major event, viewers don’t trickle in throughout the day; they cluster around the same start time, in the same regions, on the same title. If every device had to reach back to a handful of cloud origins, the origin layer would be overwhelmed almost instantly.
Open Connect gives Netflix an enormous delivery surface close to viewers.
It also lets live reuse the same distribution layer as on-demand; which improves infrastructure usage and makes catch-up viewing simpler, since past live segments can sit on the same servers that already serve recorded content.
Again, the trick is not rebuilding everything.
The better approach is to use proven architecture where the new workload fits.
The architecture is a chain of reliability decisions
Netflix’s live system is not built around one clever trick.
It’s a chain of decisions that make the system easier to recover when something goes wrong.
The pattern isn’t “move live streaming to the cloud.” It’s: stabilize messy inputs, duplicate paths independently, reuse mature systems, and cut unnecessary coordination.
Notice the pattern behind every decision so far: each layer is designed around the assumption that something will go wrong.
A pipeline that works theoretically is not the same as a system that survives a live audience. The rest of the reliability story is about the moment something breaks; during the event, in front of everyone, with no time to think.
Testing before the audience arrives
Live systems are hard to test because the real traffic often does not exist until the event begins.
For on-demand streaming, Netflix could rely on canary releases and A/B tests using normal viewer traffic.
Live streaming changed that.
So Netflix built internal test streams, synthetic load tests, and automated failure injection.
Its synthetic load testing can generate up to 100,000 playback starts per second by replaying previous live traffic or simulating Netflix devices against cloud and CDN endpoints.
But load is only half the story.
Netflix also injects failures such as missing segments, corrupted video, cloud region outages, network drops, and server timeouts. The goal isn’t only to prove the system works. It’s to prove it fails in predictable ways.
That lesson applies to any distributed system.
Do not only test whether your system can handle peak load.
Test what happens when parts of it fail.
A system that degrades gracefully is much easier to operate than one that tries to preserve every feature until it collapses.
Graceful degradation means choosing what matters
When demand exceeds predictions, Netflix does not treat every request equally.
The live stream is the priority.
Other features may improve the experience, but they are not worth sacrificing playback.
So Netflix uses load shedding: intentionally delaying or rejecting lower-priority work so the critical service stays up. That’s graceful degradation in practice, the system doesn’t pretend everything is fine. It decides what matters most.
Live playback → protect this first because it is the core user experience.
Prefetching → shed this early because it improves performance but is not essential.
Personalization → scale it back because the event can still play without every customized feature.
Video quality → lower it if needed because a lower-resolution stream is better than no stream.
You’ll see this pattern appear well beyond streaming.
When capacity is limited, do not try to preserve every feature.
Decide what matters most, protect it, and let less critical functionality degrade gracefully.
A slightly lighter experience beats a broken one.
Retry storms can turn recovery into the outage
Even a well-provisioned system can meet moments it did not fully predict.
Live streaming has a particularly nasty one: retry storms.
Netflix has seen traffic spike by 10× when a disruption of just 30 seconds causes millions of devices to reconnect at the same time.
That is the dangerous part.
The original problem may be small.
But if every client retries at once, the recovery attempt becomes a second, larger outage.
To prevent this, Netflix studied how devices retry and added server-guided backoff, prioritized load shedding at the Cloud Edge Gateway, and traffic rebalancing across regions.
Server-guided backoff means the server tells clients when to retry, instead of letting every device decide on its own.
It worked because Netflix treated clients as part of the distributed system. If clients retry aggressively, the backend pays for it. If they retry in a controlled way, the system gets the breathing room to recover.
That’s why exponential backoff, jitter, idempotent operations, and priority-aware request rejection aren’t just implementation details. They’re core reliability mechanisms.
Live systems are operated, not just deployed
The final layer is human.
Live events compress decision-making time.
If something breaks during an on-demand launch, teams can often roll back, investigate, and communicate over a comfortable window. If something breaks during a live fight, comedy special, or game, the event keeps moving. The system needs clear ownership before the failure happens.
So Netflix’s live operations include in-person launch rooms for large events, engineering owners for critical systems, early-warning metrics, runbooks, and Game Day exercises. The runbooks extend beyond engineering into customer service, production, communications, social, and executive escalation.
Netflix also built Control Room, a UI for live operations. Operators use it to monitor and control the redundant dual-region architecture, start encoding pipelines, hit failover controls, redirect traffic to a healthy region within seconds, and move viewers out of a stream when an event ends.
This is where many architectures break down.
They focus on components but not control.
A reliable system needs both.
The takeaway
The interesting thing about Netflix’s live architecture is what it does not rely on.
It does not assume every venue will be perfect.
It does not assume every network path will stay healthy.
It does not assume every segment candidate is equally good.
And it does not assume humans can improvise their way through a major outage while millions of people are watching.
Instead, Netflix makes failure cheaper.
Independent paths reduce shared blast radius. Interchangeable segments let the system switch over without negotiation. Degradation ladders help it bend instead of snap. Rehearsed operations turn a crisis into a practiced plan, not an improvisation.
Live streaming is a harder problem than video on demand not because it’s harder to send, but because you lose the ability to retry, to pre-compute, and to spread the load. Once those cushions are gone, “reliable” can’t mean “never breaks.”
So it has to mean something else.
Netflix didn’t spend all their effort trying to prevent failure. They spent it making failure predictable, contained, and recoverable.
Because at this scale, the difference between a great system and a viral outage is how gracefully it handles the moment something goes wrong.
👋 If you found this useful → Like + Restack to help others learn system design.