How Discord Fans Out One Message to a Million Users
A case study in fanout, bottlenecks, and the engineering decisions behind Discord's scale.
🚀 Case Studies Are Here
You voted, and System Design Case Studies earned your top vote. Today's issue is the first one. Thanks for telling us what to write next. This is us delivering.
How Discord Fans Out One Message to a Million Users
Press enter, and your message shows up for everyone in the server, instantly. It feels like the simplest thing an app can do.
It’s one of the hardest.
The moment a server has millions of people online, that single message stops being a database write. It becomes a fanout problem: one event has to reach every relevant online client without slowing the whole system down.
Behind that instant feeling, Discord has to check permissions, find active sessions, route the event, and push updates over WebSockets fast enough that the server still feels alive.
And “eventually delivered” doesn’t count. In a real-time system, users expect the app to react now.
So the real question is deceptively simple:
How do you turn one event into millions of updates without making one process carry the whole load?
Editor’s note: Based on publicly shared engineering details from Discord. Thanks to their engineering team for making this public. Full sources are linked in the References section below.
How one message reaches everyone in a server
Discord’s real-time backend is built on publish/subscribe, or pub/sub.
Instead of one service calling another and waiting for a reply, a process publishes an event, and every interested listener handles it independently.
Two pieces matter here.
Each connected user has a session process: the server-side version of their active connection, usually talking to the app over a WebSocket, a long-lived link between client and backend.
Each server (called a guild internally) has its own process that acts as the routing hub for everything happening inside it. Send a message, join a voice channel, trigger any visible event, and the guild decides which sessions should hear about it.
At a high level, the path looks like this:
Client action → A user sends a message, joins a channel, or performs another visible action.
Guild process → The guild receives the event and decides who should know.
Permission checks → The system filters out users who should not see it.
Session process → Each recipient’s session receives the update.
WebSocket delivery → The session sends the update to the user’s client.
Now notice where the cost hides.
The guild doesn’t blindly blast every event to everyone. First it has to work out who’s connected, who’s allowed to see the event, and which sessions should get it.
Each check is small.
But it happens once per session, every time.
Why large servers become a scaling problem
“Once per session, every time” sounds harmless. It isn’t.
As more people join a server, two things grow at once. More users means more events. And every event now has to reach more connected users. Those two factors multiply, so the total work grows far faster than the headcount.
This is the hidden pressure behind fanout systems.
Every new user is not just another connection. They are another possible sender and another possible recipient.
Eventually the guild process spends so much time broadcasting that it has little left to process new events. That’s the moment users start to feel the system slow down.
Messages arrive late. Reactions lag. The server feels less real-time.
Which raises the obvious question: where, exactly, is all that time going?
Measuring before optimizing
It’s tempting to guess.
Big system, real-time delivery, so surely the fix is more machines, more parallelism, a smarter broadcast.
Discord resisted that. Before changing anything, its engineers wanted to know where the guild process was actually spending its time and memory.
The simplest tool was a stack trace.
A stack trace shows what a running process is doing at a specific moment. By collecting thousands of samples over a few minutes, the team could see which operations showed up again and again.
Then they added more precise instrumentation inside the guild process itself.
They measured:
How often each operation ran
How long each operation took
How much memory it used
Measuring large data structures directly was too expensive, so they built a small library that estimated memory usage through sampling.
None of this made Discord faster by itself.
But it answered the most important question before any optimization:
What is actually slowing the system down?
That matters because large systems usually have more than one possible bottleneck. Guessing can lead to expensive fixes in the wrong place.
Once Discord had measurements, the first big win became clear.
The fastest fanout work is the work you never do.
The cheapest fix: doing less work
Originally, every connected user received every event they had permission to see. But most users belong to many servers at once. They may be online, but not actively looking at a particular server.
So Discord introduced passive sessions.
If a user was not actively looking at a server, the system stopped sending them updates for that server. That meant:
No permission checks
No fanout
No network traffic
At least not until the user became active again.
This worked better than expected. Discord found that around 90% of user connections in large servers were passive, which reduced fanout work by roughly the same amount.
What makes this powerful isn’t the number. It’s the change in the concern.
Instead of asking:
“How do we send one message to everyone faster?”
Discord asked:
“Who actually needs this message right now?”
Those are very different questions.
The first question usually leads to more machines, more parallelism, and more complexity.
The second question removes unnecessary work.
This approach will not work for every system. Multiplayer games, emergency alerts, and some collaborative tools may need to update everyone immediately.
But when the product experience allows it, the cheapest optimization is often to stop doing work that does not need to happen.
Passive sessions helped a lot.
But they did not remove the deeper problem: the guild process was still central.
Splitting the load with relays
Passive sessions reduced fanout work, but each guild was still managed by a single process.
And a single process can only do so much work before it becomes the bottleneck.
So Discord introduced relays.
A relay is an intermediate process that sits between the guild and user sessions. Instead of one guild process managing every connected user directly, each relay takes responsibility for a subset of sessions.
The guild no longer has to fan out every event by itself.
The work can be split:
Guild process → Handles the main guild logic.
Relay processes → Handle permission checks and fanout for subsets of sessions.
Session processes → Deliver updates to users over WebSockets.
Splitting the work wasn’t as clean as it sounds. The team had to decide which operations stayed in the guild, which moved to relays, and which had to live in both.
Then a problem showed up.
The first version of relays kept a full copy of the server’s member list in memory.
That made development easier. Every relay had the data it needed.
But convenience became the next bottleneck.
Duplicating millions of members across many relays wasted huge amounts of memory. Starting a new relay also meant copying the entire member list before it could begin processing.
The fix was to give each relay only the member data it actually used.
It’s a lesson distributed systems eventually learn: a full copy of shared state is the simplest thing to build first, but rarely survives at scale.
Keeping the guild responsive
Throughput is how much work a system can handle.
Responsiveness is how quickly important work finishes.
That difference matters in real-time systems.
A guild process might have enough total capacity, but if one expensive operation blocks the main loop, users still feel lag.
Discord hit exactly this with operations that scan large member lists. An @everyone mention is the classic case: the system has to work out which members are allowed to see it. At massive scale, that scan can take several seconds.
And the scan’s cost wasn’t even the real problem. The problem was that the guild still had to handle normal events while it ran. Messages, joins, reactions, and other updates cannot wait behind one slow operation.
So Discord used ETS and worker processes. ETS (Erlang Term Storage) is an in-memory store that multiple Erlang and Elixir processes can read safely. The guild moved large member data into ETS, kept recent changes close by, and handed the expensive scans to workers:
ETS member storage → large member data, readable by many processes.
Recent changes in heap → fresh updates stay close to the guild.
Worker processes → slow reads run outside the guild’s main event loop.
Smaller guild heap → garbage collection gets cheaper too.
This kept the guild responsive.
Instead of blocking on a slow scan, the guild could hand the work to a worker and keep processing new events.
Large state makes routine operations risky
Discord also used workers during guild handoffs.
A handoff is when a guild process moves from one machine to another, usually for maintenance or a deploy. The system has to start a new guild process, move its state, reconnect sessions, and absorb anything that queued up mid-move.
For huge guilds, that state can run to multiple gigabytes. If the old process pauses while the new guild process starts up, users can see minutes of delay.
That turns an operational task into a product problem.
A deploy is no longer “just restart the process” when that process owns gigabytes of hot data.
So Discord pushed most of the transfer onto a worker. The worker copies the large member data while the old guild kept handling normal events.
At scale, the things engineers do to maintain the system can directly affect the user experience if they block the processes responsible for real-time delivery.
By this point, Discord had reduced unnecessary work, split fanout across relays, reduced copied state, and protected the guild process from slow scans and handoffs.
But one more problem remained.
Sometimes doing less work in one place creates more work somewhere else.
When doing less work backfires
Not every optimization worked on the first try.
Discord had already built Manifold in an earlier phase of scaling. The next idea was to offload more fanout work to a separate sender process, so the guild process had even less to do.
On paper, this looked like an obvious win.
The guild would do less work. Network slowdowns would move away from the main event loop. Fanout would become less blocking.
But when Discord enabled Manifold Offload, performance got worse.
The guild process was doing less work, yet it became busier.
Tracing showed the extra load was coming from garbage collection. The offload pattern repeatedly triggered full garbage collections, forcing the runtime to scan a huge heap just to free a small amount of memory.
That is the kind of bottleneck that is easy to miss.
Once Discord found the cause, the fix was simple. They tuned a BEAM process flag to raise the garbage collection threshold, and the offload worked as intended.
The lesson is that an optimization does not exist in isolation.
Reducing work in one part of the system can quietly create more work somewhere else.
The same pattern appears with connection pools, thread schedulers, caches, garbage collectors, and other shared runtime resources.
At small scale, those details may feel invisible.
At large scale, they become the bottleneck.
The bigger lesson
Discord’s fanout problem started with a simple expectation:
When someone sends a message, everyone who needs to see it should see it now.
But at massive scale, that simple expectation turns into a chain of constraints.
First, the guild process has to find the right recipients.
Then it has to avoid sending updates to users who do not need them yet.
Then it has to split the remaining work across relays.
Then it has to avoid copying too much state.
Then it has to protect the main event loop from expensive scans, handoffs, and runtime side effects.
Each fix made the system better.
But each fix also revealed the next bottleneck.
That is the real pattern behind Discord’s scaling story. The system did not become faster because one clever optimization solved everything. It became faster because the team kept asking a narrower question:
What work is still sitting on the critical path, and does it really need to be there?
Sometimes the answer was to remove work.
Sometimes it was to split work.
Sometimes it was to move work into a worker.
Sometimes it was to tune the runtime so the optimization could actually help.
That is what makes this case study useful beyond Discord.
Large real-time systems do not survive by broadcasting harder. They survive by making the central path smaller, cheaper, and less blocking.
So the lesson is not just “use relays” or “use ETS” or “optimize garbage collection.”
The lesson is to follow the pressure.
Measure where the system is slowing down. Remove the work that does not need to happen. Split the work that cannot be removed. Keep large state away from the hot path. And watch for hidden costs that only appear after the obvious bottleneck is gone.
That is how one message can reach a million users without forcing one process to carry the whole system.
References:
👋 If you found this useful → Like + Restack to help others learn system design.