Build AI Apps with MongoDB

Presented by MongoDB

Looking to stay ahead of the curve on AI? MongoDB AI Learning Hub has the technical training pathways and tools you need to level up your AI app-building game. Explore practical guides, tutorials, and quick starts for all skill levels, from foundational concepts like understanding AI tool stacks to advanced implementations using RAG, Atlas Vector Search, and LLM optimization.

Explore the AI Hub

Vector Databases Clearly Explained

Your users type a question in plain English.

Your search returns results that match the words but miss the meaning.

You add more keywords to the query, tune the weights, and it still feels wrong. Your search engine is treating meaning like a keyword problem.

And that’s the key difference: traditional databases search for matching values, while vector databases search for nearby meaning.

That distinction is the architectural shift behind modern AI-powered search.

What a vector database actually stores

A vector is just an ordered list of numbers, like [0.12, -0.45, 0.89, ...] stretched across hundreds or thousands of dimensions. An embedding is a vector produced by a machine learning model so that similar things end up close together.

For example, the sentences “reset my password” and “I forgot my login” may look different as text, but a good embedding model places them near each other because they mean similar things.

A regular database can store vectors, but a vector database is built to search them quickly, filter them safely, and rank them by similarity at scale.

A vector database stores those embeddings alongside useful context:

• ID → The stable reference back to the original item.

• Vector → The embedding used for similarity search.

• Metadata → The fields used for filtering, permissions, and ranking.

• Source content → The original text, image, document, or a pointer to where it lives.

That metadata matters because real searches rarely ask, “Find anything similar.” They ask, “Find similar support articles for this customer, in English, updated this year, that this user can access.”

So the vector database is not just matching meaning. It is combining similarity, filtering, and ranking into one retrieval path.

How vector search works

Traditional databases are great at exact lookups. You ask for user_id = 123, and the database finds that row.

Vector search is different.

You give the database a query vector, and it looks for nearby vectors. “Nearby” depends on the distance metric, which is the math rule used to compare vectors.

Common metrics include:

• Cosine similarity → Compares vector direction, not size. Common for text embeddings because similar sentences can point the same way even with different magnitudes.

• Euclidean distance → Measures straight-line distance between vectors. The smaller the distance, the more similar the vectors.

• Dot product → Measures direction and magnitude together. Useful when the embedding model treats larger vector values as meaningful.

Use the metric your embedding model expects.

If the model was trained for cosine similarity and you search with the wrong metric, relevance can drop even if the database is working correctly.

The embedding model defines the map. The vector database only helps you navigate it.

Exact vs approximate search

The simplest vector search is exact nearest-neighbor search.

The database takes your query vector, compares it against every stored vector, ranks the results, and returns the closest matches. It is accurate because nothing gets skipped.

But accuracy has a cost.

As the dataset grows, comparing against every vector becomes too slow and too expensive. Searching 10,000 vectors is manageable. Searching 100 million vectors this way is not.

That is why production vector databases usually use approximate nearest-neighbor search, or ANN. ANN does not promise to always find the mathematically closest results. Instead, it finds results that are close enough, much faster.

For example, a support chatbot does not always need the mathematically closest document. It needs a document that answers the user’s question quickly and reliably.

Most production vector databases use ANN as their default. The important nuance is that “ANN versus exact” is not a once-per-database choice; it can be a per-query decision.

Some systems fall back to exact search automatically when filters make the candidate set small enough. The database is not changing its philosophy. It is choosing the cheaper path for the query in front of it.

Approximate search works because the database builds an index ahead of time. The index gives the database a faster path through the vector space, so it does not need to scan everything for every query.

How vector indexes speed up search

A vector database does not always compare your query against every stored vector.

That would be accurate, but slow. As the dataset grows, the database needs a shortcut: an index, which is a data structure that narrows the search space so similar vectors can be found faster.

Different index types make different tradeoffs between speed, recall, and memory.

Recall means “how often the search returns the true nearest results.” Higher recall usually means better relevance, but it often costs more latency or memory.

In practice, HNSW is the default workhorse because its speed-to-recall balance is strong for many production systems.

But it is not always the right answer.

When memory cost becomes the bottleneck, PQ or IVF-PQ can be a better fit because they shrink the vector footprint.

Choose based on your bottleneck: use HNSW when latency matters most; use PQ when memory cost matters most.

Production tradeoffs

Vector databases look simple in demos.

You embed the data, store the vectors, run a search, and return the closest results. But production systems have more moving parts.

New records may not be searchable immediately. Indexes may build in the background. Replicas may lag behind writes. Metadata filters may also change the best search path for each query.

That is why a vector database should usually be treated as a retrieval index, not the main source of truth.

Keep the canonical data in your primary database. Store vectors as a derived view of that data.

This gives you room to rebuild indexes, change embedding models, recover from lag, and compare approximate results against an exact baseline.

A strong production setup tracks:

• Model version → Embeddings from different models may not behave the same, so avoid mixing them without a migration plan.

• Write freshness → New or updated records may take time to appear in search, so the app should not assume instant visibility.

• Filter behavior → Metadata filters can speed up retrieval, but they can also make some queries slower depending on the index and candidate set.

• Recall and latency → Fast results do not help if the database misses the documents the user actually needs.

• Privacy risk → Embeddings are derived from data, but they can still reveal information about the original text, behavior, or documents.

Keep the source of truth in your primary database, and use vectors as a searchable layer on top.

When not to use a Vector Database

A vector database is not the answer to every search problem.

Do not reach for one when exact structure matters more than similarity.

• Exact transactional lookup → Use a relational or key-value database because IDs, constraints, and writes matter more than semantic closeness.

• Range queries → Use an ordered index because vector search organizes by similarity, not by business order.

• Small static datasets → Use brute-force search or a simple library because a full database may add unnecessary operational weight.

• Poor embeddings → Fix the model or data pipeline first because no index can rescue bad vector geometry.

• Strict fresh reads → Use a system with stronger consistency guarantees because some vector indexes update asynchronously.

Vector databases shine when “close enough in meaning” is the right question. They struggle when the real question is “exactly this,” “strictly ordered,” or “immediately consistent.”

Wrapping up

Vector databases change what search can mean.

You no longer need users to type the exact words that appear in your data. You can search by meaning, similarity, behavior, or context.

But meaning-based search is only as good as the system around it. The embedding model shapes what “similar” means. The index controls the speed and accuracy tradeoff. The production setup determines whether results stay fresh, reliable, and safe.

Use vector databases when “nearby” is the right question. Use something else when “exact,” “ordered,” or “immediately consistent” matters more.

👋 If you found this useful → Like + Restack to help others learn system design.

What If You Didn’t Have to Choose Between AI and Traditional SAST?

Presented by Checkmarx

Checkmarx One’s new Hybrid SAST engine combines deterministic rules with AI analysis to improve the quality of security findings before they reach developers. In Checkmarx’s published benchmark, the Hybrid engine achieved 3x the industry’s average fidelity score while reducing false positives by 60%.

See the Results

How Netflix Streams Live Events Without Crashing

Why is live streaming so much harder than video on demand?

It’s not just because more people are watching.

It’s because live streaming removes the one thing on-demand systems rely on most: time.

With on-demand video, the content is already finished. Netflix has time to encode it, package it, test it, copy it to servers near viewers, and demand is spread out as people press play at different times.

Live has none of those luxuries.

The video exists only seconds before it has to reach the audience. Millions of viewers may arrive at the same time. A missing segment cannot be quietly fixed later. And if something goes wrong, every affected device may try to reconnect at once.

That changes the engineering problem.

Netflix isn’t just delivering video anymore. It’s building a system that has to survive unreliable venues, fluctuating networks, mismatched devices, regional traffic spikes, missing segments, retry storms, and operators making decisions while the event is still happening.

This is the story of how that system is built, and the more general lesson hiding inside it: at this scale, reliability comes from making failure predictable, contained, and recoverable.

Live vs on-demand streaming

On-demand streaming is hard, but it has one enormous advantage: the video is finished before anyone presses play.

Live streaming works differently.

The video is created while the audience is watching. A camera captures the event, production systems process the signal, the feed reaches the cloud, encoders create streamable versions, packagers prepare them for devices, origins expose the segments, the CDN distributes them, and players request the right piece at the right time; all running in seconds behind the actual live event.

When the benefits of on-demand streaming disappears, different failures start to matter:

• No pre-computation → Encoding happens in real time, so there’s no batch window to hide mistakes.

• Synchronized demand → Everyone arrives near the start, so traffic spikes vertically instead of ramping up.

• No do-overs → A bad or missing segment reaches the audience immediately, so correctness has to happen the first time.

• Large blast radius → One overloaded component can hit millions of viewers at once.

The hard part is that every layer has to work at the same time.

If the feed breaks, the cloud pipeline cannot recreate missing video.

If the origin cannot publish segments fast enough, the CDN has nothing useful to serve.

If players retry too aggressively, recovery traffic can become worse than the original failure.

So Netflix’s live architecture starts with a simple assumption: something will fail while the stream is running.

The system has to be ready before that happens.

Getting the signal to the cloud

The first hard problem isn’t the cloud. It’s the venue.

Live events happen in stadiums, theaters, arenas, studios, and temporary production sites. Some have strong connectivity and broadcast infrastructure. Many don’t. And if every venue becomes a custom networking project, every live event becomes a new integration risk.

For its first live comedy special, Netflix used two dedicated internet circuits from the venue and sent a high-quality mezzanine feed to the cloud.

It worked. But it exposed the real risk: every new venue could become its own custom integration.

So Netflix moved toward a hub-and-spoke model.

Instead of sending every production feed directly from the venue into the cloud, feeds first go to a well-connected broadcast facility. From there, they move into the cloud live pipeline.

The venue becomes the spoke.

The broadcast facility becomes the hub.

That creates a stable boundary before the core system.

And that is the first useful pattern: when the outside world is messy, do not let every variation leak into your architecture. Put a predictable layer in front of it.

Designing for the failure that will happen mid-stream

Once the signal reaches the cloud, the next problem is failure during the stream.

On-demand video can recover from small network issues by retrying a stable file. Live streaming can’t rely on that.

If a packet drops during a title fight, comedy special, or live game, millions of viewers may see the glitch at roughly the same time. So the live pipeline has to assume something will fail while the event is still running, and keep going anyway.

Netflix handles this by sending each live feed into two separate cloud regions. Each region also receives the feed over two different managed network paths, instead of relying on the open internet.

That gives Netflix four independent feed paths into the live pipeline.

At the packet level, Netflix uses SMPTE 2022-7, a broadcast standard for seamless protection switching. In simpler terms, the system receives duplicate packet streams and picks the best packets in real time. If one path drops packets, the other path can fill the gaps before viewers notice.

But the important idea is not simply “add more copies.”

The important idea is independence.

Two backups are not useful if they share the same failure mode.

• Same cloud region → one regional outage can affect both copies.

• Same network path → one carrier or routing issue can affect both copies.

• Same storage backend → one datastore problem can affect both copies.

• Same deployment mistake → one bad rollout can affect both copies.

Netflix is not just duplicating the stream. It’s reducing shared failure modes across regions, network paths, and feed candidates.

That distinction matters.

Not more copies. More independent copies.

The stream still has to work for everyone

Getting a reliable feed into the cloud is only the first part of the problem.

Now Netflix has to turn that feed into something viewers can actually watch.

A viewer on fiber with a 4K TV shouldn’t get the same stream as someone on a weak mobile connection. This is where adaptive bitrate streaming comes in: the player switches between quality levels as network conditions change.

Netflix tunes its bitrate ladder based on resolution, frame rate, and content complexity. Fast-motion events, like WWE, need higher bitrates because motion is harder to compress cleanly.

But quality is not the only constraint.

Netflix also has to support a large device ecosystem. Different devices may need different video, audio, language, and caption formats. Older devices may not support newer formats.

That device compatibility problem is easy to underestimate.

Netflix cannot assume viewers will upgrade their hardware for a live event. The system has to work on the devices people already use.

That is one reason Netflix uses HTTPS-based live streaming with 2-second segments. HTTPS is widely supported and fits the delivery infrastructure Netflix already runs.

The tradeoff is latency.

UDP-based protocols can deliver lower latency, but they are harder to support across every device and network.

Netflix chooses reach and stability over the lowest possible delay.

That is a recurring pattern in their architecture: the best technical choice is not always the fastest one.

Sometimes it’s the one that works reliably across the messy reality of production.

The better move is not always a new stack

A common mistake in new architecture is rebuilding too much.

Netflix didn’t build a separate live playback stack from scratch. It built custom live components only where it had to, then connected them to systems that already worked for on-demand.

The live packager is the best example.

Netflix built its own packager so live streams could integrate with existing playback systems, device compatibility work, captions, and Digital Rights Management.

That avoided years of repeated compatibility work across Netflix’s device ecosystem.

This is the quieter architectural decision, but it is one of the most useful.

Instead of building a parallel live stack, Netflix used the packager as an adapter into proven infrastructure.

New requirements often don’t need a new stack beside the old one. They need the adapter that lets them reuse the stack that already works.

The packager does one more quiet but important thing: it generates segment templates for Netflix’s PlayAPI backend. A segment template lets the player calculate the next segment URL from the current time, instead of repeatedly asking the backend what comes next.

That sounds small, but at live-event scale it matters.

If millions of devices constantly ask the control plane for the next segment, the backend becomes a bottleneck. Segment templates remove that coordination by letting each device calculate the answer locally.

At scale, client behavior becomes infrastructure behavior.

Static storage is not enough for live video

Static storage sounds like a perfectly reasonable origin; at first.

The packager writes segments into storage. The CDN reads from storage. With two regions cross-publishing, the CDN even has multiple candidates to choose from.

But live streaming puts pressure on storage that generic object stores weren’t built for.

Netflix’s early live pipeline used static storage buckets in AWS, with one bucket in each region. Packagers published segments to both buckets, giving the CDN multiple segment candidates.

That improved redundancy, but it did not solve the deeper problem.

Generic storage does not understand media quality.

A segment can exist and still be bad. It might be too short, miss frames, or have timing issues. A storage bucket can confirm the object exists, but it can’t decide whether one candidate is better than another.

So Netflix built a custom Live Origin.

The Live Origin sits between the cloud live pipelines and Open Connect, Netflix’s CDN. It acts as a broker that decides which content reaches the CDN and, eventually, client devices. Because it understands multiple pipelines and regions, it can choose valid segment candidates instead of treating every stored object as equal.

That’s the important difference.

The origin stops being passive storage and becomes a media-aware decision point.

The CDN absorbs synchronized demand

Once the live segments exist, Netflix still has to deliver them to viewers around the world.

This is where Open Connect helps.

Netflix uses Open Connect, its own content delivery network, for live delivery. Open Connect servers are placed close to viewers at over 6,000 locations and connect back to AWS locations through a dedicated Open Connect backbone network.

This matters because live traffic is synchronized.

For a major event, viewers don’t trickle in throughout the day; they cluster around the same start time, in the same regions, on the same title. If every device had to reach back to a handful of cloud origins, the origin layer would be overwhelmed almost instantly.

Open Connect gives Netflix an enormous delivery surface close to viewers.

It also lets live reuse the same distribution layer as on-demand; which improves infrastructure usage and makes catch-up viewing simpler, since past live segments can sit on the same servers that already serve recorded content.

Again, the trick is not rebuilding everything.

The better approach is to use proven architecture where the new workload fits.

The architecture is a chain of reliability decisions

Netflix’s live system is not built around one clever trick.

It’s a chain of decisions that make the system easier to recover when something goes wrong.

The pattern isn’t “move live streaming to the cloud.” It’s: stabilize messy inputs, duplicate paths independently, reuse mature systems, and cut unnecessary coordination.

Notice the pattern behind every decision so far: each layer is designed around the assumption that something will go wrong.

A pipeline that works theoretically is not the same as a system that survives a live audience. The rest of the reliability story is about the moment something breaks; during the event, in front of everyone, with no time to think.

Testing before the audience arrives

Live systems are hard to test because the real traffic often does not exist until the event begins.

For on-demand streaming, Netflix could rely on canary releases and A/B tests using normal viewer traffic.

Live streaming changed that.

So Netflix built internal test streams, synthetic load tests, and automated failure injection.

Its synthetic load testing can generate up to 100,000 playback starts per second by replaying previous live traffic or simulating Netflix devices against cloud and CDN endpoints.

But load is only half the story.

Netflix also injects failures such as missing segments, corrupted video, cloud region outages, network drops, and server timeouts. The goal isn’t only to prove the system works. It’s to prove it fails in predictable ways.

That lesson applies to any distributed system.

Do not only test whether your system can handle peak load.

Test what happens when parts of it fail.

A system that degrades gracefully is much easier to operate than one that tries to preserve every feature until it collapses.

Graceful degradation means choosing what matters

When demand exceeds predictions, Netflix does not treat every request equally.

The live stream is the priority.

Other features may improve the experience, but they are not worth sacrificing playback.

So Netflix uses load shedding: intentionally delaying or rejecting lower-priority work so the critical service stays up. That’s graceful degradation in practice, the system doesn’t pretend everything is fine. It decides what matters most.

• Live playback → protect this first because it is the core user experience.

• Prefetching → shed this early because it improves performance but is not essential.

• Personalization → scale it back because the event can still play without every customized feature.

• Video quality → lower it if needed because a lower-resolution stream is better than no stream.

  

You’ll see this pattern appear well beyond streaming.

When capacity is limited, do not try to preserve every feature.

Decide what matters most, protect it, and let less critical functionality degrade gracefully.

A slightly lighter experience beats a broken one.

Retry storms can turn recovery into the outage

Even a well-provisioned system can meet moments it did not fully predict.

Live streaming has a particularly nasty one: retry storms.

Netflix has seen traffic spike by 10× when a disruption of just 30 seconds causes millions of devices to reconnect at the same time.

That is the dangerous part.

The original problem may be small.

But if every client retries at once, the recovery attempt becomes a second, larger outage.

To prevent this, Netflix studied how devices retry and added server-guided backoff, prioritized load shedding at the Cloud Edge Gateway, and traffic rebalancing across regions.

Server-guided backoff means the server tells clients when to retry, instead of letting every device decide on its own.

It worked because Netflix treated clients as part of the distributed system. If clients retry aggressively, the backend pays for it. If they retry in a controlled way, the system gets the breathing room to recover.

That’s why exponential backoff, jitter, idempotent operations, and priority-aware request rejection aren’t just implementation details. They’re core reliability mechanisms.

Live systems are operated, not just deployed

The final layer is human.

Live events compress decision-making time.

If something breaks during an on-demand launch, teams can often roll back, investigate, and communicate over a comfortable window. If something breaks during a live fight, comedy special, or game, the event keeps moving. The system needs clear ownership before the failure happens.

So Netflix’s live operations include in-person launch rooms for large events, engineering owners for critical systems, early-warning metrics, runbooks, and Game Day exercises. The runbooks extend beyond engineering into customer service, production, communications, social, and executive escalation.

Netflix also built Control Room, a UI for live operations. Operators use it to monitor and control the redundant dual-region architecture, start encoding pipelines, hit failover controls, redirect traffic to a healthy region within seconds, and move viewers out of a stream when an event ends.

This is where many architectures break down.

They focus on components but not control.

A reliable system needs both.

The takeaway

The interesting thing about Netflix’s live architecture is what it does not rely on.

It does not assume every venue will be perfect.

It does not assume every network path will stay healthy.

It does not assume every segment candidate is equally good.

And it does not assume humans can improvise their way through a major outage while millions of people are watching.

Instead, Netflix makes failure cheaper.

Independent paths reduce shared blast radius. Interchangeable segments let the system switch over without negotiation. Degradation ladders help it bend instead of snap. Rehearsed operations turn a crisis into a practiced plan, not an improvisation.

Live streaming is a harder problem than video on demand not because it’s harder to send, but because you lose the ability to retry, to pre-compute, and to spread the load. Once those cushions are gone, “reliable” can’t mean “never breaks.”

So it has to mean something else.

Netflix didn’t spend all their effort trying to prevent failure. They spent it making failure predictable, contained, and recoverable.

Because at this scale, the difference between a great system and a viral outage is how gracefully it handles the moment something goes wrong.

👋 If you found this useful → Like + Restack to help others learn system design.

Photorealistic Faces for AI Agents. 44% More Usage.

Presented by Anam

Anam gives AI agents a photorealistic face, increasing usage by 44% on average. For many use cases, a face is a far more natural way to interact with AI than chat or voice , think sales agents, recruitment screens, and interactive learning. $5,000 for the best demo built with Cara-4, July 14–26.

Start Building for Free

RAG Clearly Explained

Your model can explain concepts from years ago but struggle with last month’s product launch or your company’s internal docs. The problem isn't the model. It's that its knowledge is frozen in its training data.

As teams started connecting LLMs to internal knowledge bases, support documentation, and constantly changing business data, keeping answers up to date became a new challenge.

Retrieval-Augmented Generation (RAG) solves this by retrieving relevant information at query time and providing it to the model as context. That keeps answers current without retraining.

What RAG actually is

RAG is a system pattern for giving an LLM the right context before it answers.

Instead of asking the model to rely only on what it learned during training, the application first retrieves relevant information from external sources. Then it passes that information to the model so the answer is grounded in real information.

That distinction matters because RAG is often confused with related ideas:

• RAG vs fine-tuning → Fine-tuning changes the model’s weights. RAG keeps the model the same and brings in fresh information at runtime.

• RAG vs plain search → Search returns documents. RAG uses those documents to generate a clear answer.

• RAG vs vector search → Vector search is one retrieval method. RAG can also use keyword search, SQL queries, graph traversal, or other data access patterns.

The retrieval step can be simple or advanced. The model might retrieve once before answering, or retrieve again as the answer develops.

The core idea is simple: find the right information first, then let the model answer with that information in context.

How RAG works

A RAG system has two flows: one for preparing knowledge and one for answering questions.

The preparation flow happens before any user query. Documents are extracted, cleaned, split into chunks, enriched with metadata, and stored in a searchable index.

A chunk is a small piece of content, such as a paragraph, section, or code block. Instead of searching an entire document, the system searches these smaller pieces, making it easier to find the exact information a user is looking for.

When a user asks a question, the query flow begins. The system retrieves relevant chunks, filters them, optionally reranks them, and passes the best information to the model as context.

Here is a typical request flow using Basic RAG and Vector Retrieval:

RAG is not a single model call. It is an information pipeline that finds the right information before the model starts generating.

Main types of RAG

Not all RAG systems work the same way. The biggest differences are how information is retrieved and how much reasoning happens before an answer is generated.

For most teams, Basic RAG is the best place to start. It is easier to build, observe, and debug.

Agentic RAG becomes useful when a question requires multiple lookups or reasoning steps. For example: “Compare this customer’s contract with our refund policy and draft a response.”

That request may require several document lookups, comparisons, and reasoning steps before an answer can be generated.

Retrieval methods

Many teams hear “RAG” and immediately think “vector databases.” In reality, vector search is just one way to retrieve information.

Hybrid retrieval is often the safest default because real questions usually need both. A user might include an exact invoice ID while asking a broader policy question. Keyword search catches the ID. Vector search catches the meaning.

These choices are independent. For example, a system might use Basic RAG with Hybrid Retrieval, or Agentic RAG with Graph Retrieval. One defines how the system operates. The other defines how it finds information.

What usually breaks

A RAG system is only as good as the information it retrieves. Even a powerful model struggles when the context is incomplete, irrelevant, or outdated.

The most common problems are:

• Poor chunking → The information is split across chunks, so only part of the information gets retrieved.

• Weak metadata → The system cannot filter by tenant, date, permissions, or other important attributes.

• Poor retrieval quality → Relevant information exists, but the retriever fails to find it.

• Too much retrieval → Irrelevant chunks fill the prompt and bury the useful information.

• Stale indexes → Outdated or deleted content continues to appear in answers.

• Unsafe retrieval → Untrusted content injects instructions that interfere with the model.

That last one is especially important.

Retrieved documents are external input and shouldn’t always be trusted. The model should follow system instructions, not instructions hidden inside a webpage, PDF, email, or document.

When not to use RAG

RAG works best when you need current information, private data, source citations, or knowledge that changes frequently.

But it is not always the right tool.

• Behavior problems → If you need a specific writing style, output format, or classification behavior, fine-tuning is often a better fit because the challenge is behavior, not missing information.

• Simple lookups → If a user enters an order ID and needs a delivery date, a direct database query is simpler, faster, and more reliable.

• Small knowledge sources → If the entire document fits comfortably in the model’s context window, retrieval may add unnecessary complexity. Passing the document directly to the model is often enough.

Not every AI problem is a retrieval problem. RAG shines when information is missing, changing, or hard to find.

Wrapping up

RAG makes a model more useful without changing its weights. Instead, it changes what the model considers before it answers.

That sounds simple, but it is hard to ship well. Chunking, retrieval quality, permissions, and freshness all need to work before the model generates anything.

The difference between a useful answer and a hallucination often comes down to one thing: whether the right information was retrieved before the model started generating.

👋 If you found this useful → Like + Restack to help others learn system design.

🚀 Case Studies Are Here

You voted, and System Design Case Studies earned your top vote. Today's issue is the first one. Thanks for telling us what to write next. This is us delivering.

How Discord Fans Out One Message to a Million Users

Press enter, and your message shows up for everyone in the server, instantly. It feels like the simplest thing an app can do.

It’s one of the hardest.

The moment a server has millions of people online, that single message stops being a database write. It becomes a fanout problem: one event has to reach every relevant online client without slowing the whole system down.

Behind that instant feeling, Discord has to check permissions, find active sessions, route the event, and push updates over WebSockets fast enough that the server still feels alive.

And “eventually delivered” doesn’t count. In a real-time system, users expect the app to react now.

So the real question is deceptively simple:

How do you turn one event into millions of updates without making one process carry the whole load?

Editor’s note: Based on publicly shared engineering details from Discord. Thanks to their engineering team for making this public. Full sources are linked in the References section below.

How one message reaches everyone in a server

Discord’s real-time backend is built on publish/subscribe, or pub/sub.

Instead of one service calling another and waiting for a reply, a process publishes an event, and every interested listener handles it independently.

Two pieces matter here.

Each connected user has a session process: the server-side version of their active connection, usually talking to the app over a WebSocket, a long-lived link between client and backend.

Each server (called a guild internally) has its own process that acts as the routing hub for everything happening inside it. Send a message, join a voice channel, trigger any visible event, and the guild decides which sessions should hear about it.

At a high level, the path looks like this:

• Client action → A user sends a message, joins a channel, or performs another visible action.

• Guild process → The guild receives the event and decides who should know.

• Permission checks → The system filters out users who should not see it.

• Session process → Each recipient’s session receives the update.

• WebSocket delivery → The session sends the update to the user’s client.

  

Now notice where the cost hides.

The guild doesn’t blindly blast every event to everyone. First it has to work out who’s connected, who’s allowed to see the event, and which sessions should get it.

Each check is small.

But it happens once per session, every time.

Why large servers become a scaling problem

“Once per session, every time” sounds harmless. It isn’t.

As more people join a server, two things grow at once. More users means more events. And every event now has to reach more connected users. Those two factors multiply, so the total work grows far faster than the headcount.

This is the hidden pressure behind fanout systems.

Every new user is not just another connection. They are another possible sender and another possible recipient.

Eventually the guild process spends so much time broadcasting that it has little left to process new events. That’s the moment users start to feel the system slow down.

Messages arrive late. Reactions lag. The server feels less real-time.

Which raises the obvious question: where, exactly, is all that time going?

Measuring before optimizing

It’s tempting to guess.

Big system, real-time delivery, so surely the fix is more machines, more parallelism, a smarter broadcast.

Discord resisted that. Before changing anything, its engineers wanted to know where the guild process was actually spending its time and memory.

The simplest tool was a stack trace.

A stack trace shows what a running process is doing at a specific moment. By collecting thousands of samples over a few minutes, the team could see which operations showed up again and again.

Then they added more precise instrumentation inside the guild process itself.

They measured:

• How often each operation ran

• How long each operation took

• How much memory it used

Measuring large data structures directly was too expensive, so they built a small library that estimated memory usage through sampling.

None of this made Discord faster by itself.

But it answered the most important question before any optimization:

What is actually slowing the system down?

That matters because large systems usually have more than one possible bottleneck. Guessing can lead to expensive fixes in the wrong place.

Once Discord had measurements, the first big win became clear.

The fastest fanout work is the work you never do.

The cheapest fix: doing less work

Originally, every connected user received every event they had permission to see. But most users belong to many servers at once. They may be online, but not actively looking at a particular server.

So Discord introduced passive sessions.

If a user was not actively looking at a server, the system stopped sending them updates for that server. That meant:

• No permission checks

• No fanout

• No network traffic

At least not until the user became active again.

This worked better than expected. Discord found that around 90% of user connections in large servers were passive, which reduced fanout work by roughly the same amount.

What makes this powerful isn’t the number. It’s the change in the concern.

Instead of asking:

“How do we send one message to everyone faster?”

Discord asked:

“Who actually needs this message right now?”

Those are very different questions.

The first question usually leads to more machines, more parallelism, and more complexity.

The second question removes unnecessary work.

This approach will not work for every system. Multiplayer games, emergency alerts, and some collaborative tools may need to update everyone immediately.

But when the product experience allows it, the cheapest optimization is often to stop doing work that does not need to happen.

Passive sessions helped a lot.

But they did not remove the deeper problem: the guild process was still central.

Splitting the load with relays

Passive sessions reduced fanout work, but each guild was still managed by a single process.

And a single process can only do so much work before it becomes the bottleneck.

So Discord introduced relays.

A relay is an intermediate process that sits between the guild and user sessions. Instead of one guild process managing every connected user directly, each relay takes responsibility for a subset of sessions.

The guild no longer has to fan out every event by itself.

The work can be split:

• Guild process → Handles the main guild logic.

• Relay processes → Handle permission checks and fanout for subsets of sessions.

• Session processes → Deliver updates to users over WebSockets.

  

Splitting the work wasn’t as clean as it sounds. The team had to decide which operations stayed in the guild, which moved to relays, and which had to live in both.

Then a problem showed up.

The first version of relays kept a full copy of the server’s member list in memory.

That made development easier. Every relay had the data it needed.

But convenience became the next bottleneck.

Duplicating millions of members across many relays wasted huge amounts of memory. Starting a new relay also meant copying the entire member list before it could begin processing.

The fix was to give each relay only the member data it actually used.

It’s a lesson distributed systems eventually learn: a full copy of shared state is the simplest thing to build first, but rarely survives at scale.

Keeping the guild responsive

Throughput is how much work a system can handle.

Responsiveness is how quickly important work finishes.

That difference matters in real-time systems.

A guild process might have enough total capacity, but if one expensive operation blocks the main loop, users still feel lag.

Discord hit exactly this with operations that scan large member lists. An @everyone mention is the classic case: the system has to work out which members are allowed to see it. At massive scale, that scan can take several seconds.

And the scan’s cost wasn’t even the real problem. The problem was that the guild still had to handle normal events while it ran. Messages, joins, reactions, and other updates cannot wait behind one slow operation.

So Discord used ETS and worker processes. ETS (Erlang Term Storage) is an in-memory store that multiple Erlang and Elixir processes can read safely. The guild moved large member data into ETS, kept recent changes close by, and handed the expensive scans to workers:

• ETS member storage → large member data, readable by many processes.

• Recent changes in heap → fresh updates stay close to the guild.

• Worker processes → slow reads run outside the guild’s main event loop.

• Smaller guild heap → garbage collection gets cheaper too.

  

This kept the guild responsive.

Instead of blocking on a slow scan, the guild could hand the work to a worker and keep processing new events.

Large state makes routine operations risky

Discord also used workers during guild handoffs.

A handoff is when a guild process moves from one machine to another, usually for maintenance or a deploy. The system has to start a new guild process, move its state, reconnect sessions, and absorb anything that queued up mid-move.

For huge guilds, that state can run to multiple gigabytes. If the old process pauses while the new guild process starts up, users can see minutes of delay.

That turns an operational task into a product problem.

A deploy is no longer “just restart the process” when that process owns gigabytes of hot data.

So Discord pushed most of the transfer onto a worker. The worker copies the large member data while the old guild kept handling normal events.

At scale, the things engineers do to maintain the system can directly affect the user experience if they block the processes responsible for real-time delivery.

By this point, Discord had reduced unnecessary work, split fanout across relays, reduced copied state, and protected the guild process from slow scans and handoffs.

But one more problem remained.

Sometimes doing less work in one place creates more work somewhere else.

When doing less work backfires

Not every optimization worked on the first try.

Discord had already built Manifold in an earlier phase of scaling. The next idea was to offload more fanout work to a separate sender process, so the guild process had even less to do.

On paper, this looked like an obvious win.

The guild would do less work. Network slowdowns would move away from the main event loop. Fanout would become less blocking.

But when Discord enabled Manifold Offload, performance got worse.

The guild process was doing less work, yet it became busier.

Tracing showed the extra load was coming from garbage collection. The offload pattern repeatedly triggered full garbage collections, forcing the runtime to scan a huge heap just to free a small amount of memory.

That is the kind of bottleneck that is easy to miss.

Once Discord found the cause, the fix was simple. They tuned a BEAM process flag to raise the garbage collection threshold, and the offload worked as intended.

The lesson is that an optimization does not exist in isolation.

Reducing work in one part of the system can quietly create more work somewhere else.

The same pattern appears with connection pools, thread schedulers, caches, garbage collectors, and other shared runtime resources.

At small scale, those details may feel invisible.

At large scale, they become the bottleneck.

The bigger lesson

Discord’s fanout problem started with a simple expectation:

When someone sends a message, everyone who needs to see it should see it now.

But at massive scale, that simple expectation turns into a chain of constraints.

First, the guild process has to find the right recipients.

Then it has to avoid sending updates to users who do not need them yet.

Then it has to split the remaining work across relays.

Then it has to avoid copying too much state.

Then it has to protect the main event loop from expensive scans, handoffs, and runtime side effects.

Each fix made the system better.

But each fix also revealed the next bottleneck.

That is the real pattern behind Discord’s scaling story. The system did not become faster because one clever optimization solved everything. It became faster because the team kept asking a narrower question:

What work is still sitting on the critical path, and does it really need to be there?

Sometimes the answer was to remove work.

Sometimes it was to split work.

Sometimes it was to move work into a worker.

Sometimes it was to tune the runtime so the optimization could actually help.

That is what makes this case study useful beyond Discord.

Large real-time systems do not survive by broadcasting harder. They survive by making the central path smaller, cheaper, and less blocking.

So the lesson is not just “use relays” or “use ETS” or “optimize garbage collection.”

The lesson is to follow the pressure.

Measure where the system is slowing down. Remove the work that does not need to happen. Split the work that cannot be removed. Keep large state away from the hot path. And watch for hidden costs that only appear after the obvious bottleneck is gone.

That is how one message can reach a million users without forcing one process to carry the whole system.

References:

• Maxjourney: Pushing Discord’s Limits with a Million+ Online Users in a Single Server

• How Discord Scaled Elixir to 5,000,000 Concurrent Users

👋 If you found this useful → Like + Restack to help others learn system design.

What Happens When 81% of Your Code Is AI-Generated?

Presented by Checkmarx

Organizations with more than 81% AI-generated code reported shipping vulnerable code 3.4x more often than organizations with low AI adoption. That’s just one of several findings from Checkmarx’s Future of Application Security in the Era of AI Report. See what the data reveals.

Download the Full Report

LLMs: The Essential Guide

LLMs are not just smarter search engines.

They are distributed systems built from many moving parts: the model itself, training pipelines, retrieval systems, inference infrastructure, safety layers, and the product experience around them.

Most engineers only see the API layer. That works fine until something goes wrong.

And when LLM systems fail, the failures are often hard to diagnose. Hallucinations, latency spikes, context loss, rising inference costs, inconsistent outputs; the problem is usually somewhere beneath the response itself.

Understanding how these systems work changes how you debug them, scale them, and design around their limitations. From tokenization and transformers to retrieval, fine-tuning, inference, and guardrails, every layer shapes the final answer you see on screen.

The four layers

An LLM system has four practical layers:

Most mistakes happen when teams blur these layers.

You do not fix missing company knowledge with a clever prompt. You do not fix hallucinations by blindly fine-tuning. You do not fix latency by only picking a “smarter” model. Each layer has its own job.

Transformers

Transformers are the neural network architecture behind modern LLMs.

Their core mechanism is attention: a way for tokens to look at other tokens in the sequence and decide what matters most.

Instead of reading text strictly one word at a time, Transformers build relationships across the entire context window. In a sentence like “The server crashed because it ran out of memory,” attention helps the model understand that “it” refers to the server.

Different Transformer architectures fit different jobs:

• Encoder-only → Good for classification, embeddings, and semantic search.

• Encoder-decoder → Good for transformations like translation and summarization.

• Decoder-only → Good for chat, code generation, and next-token prediction.

Most modern chat models are decoder-only because they generate text one token at a time.

Tokenization

Tokenization is how text becomes something a model can process.

LLMs do not read raw text directly. They read tokens: small pieces of text created by a tokenizer. A single word may become multiple tokens, and symbols, code, or emojis can expand even further.

Tokens matter because they shape almost every system constraint:

• Cost → Most providers bill by input and output tokens.

• Context window → A model’s working memory is measured in tokens, not words.

• Latency → More tokens usually mean more processing time.

• Retrieval chunking → Documents must be split by token limits, not page count.

This becomes especially important with code, logs, multilingual text, and large prompts. Two inputs with the same word count can produce very different token counts.

Embeddings

Embeddings turn text into vectors which are lists of numbers that represent meaning.

If two pieces of text mean similar things, their vectors should land close together. That makes embeddings the foundation of semantic search and retrieval systems.

A support platform might embed every help article. When a user asks, “Why was my payment declined?”, the system searches for nearby vectors and retrieves related articles about billing failures, fraud checks, or declined cards.

This is the backbone of retrieval-augmented generation (RAG), where external knowledge is fetched before the LLM generates a response.

There is no universally best embedding model. Different models optimize for different retrieval patterns. Some work better for semantic similarity, others for clustering or reranking. So the important part is testing them against your actual data and queries.

Attention, context windows, and long-context methods

Attention lets the model decide which earlier tokens matter most.

But attention becomes expensive as inputs grow because tokens may need to compare against many other tokens. That is one reason long-context systems are difficult to scale efficiently.

A context window is the amount of text a model can consider in one request. Larger windows help with long documents, codebases, and multi-step workflows, but more context is not always better.

A giant prompt full of loosely related information can behave like a desk buried under papers. The answer may exist somewhere in the pile, but finding the right signal becomes harder as the noise grows.

Good long-context systems usually follow a few principles:

• Keep instructions clear → Signal matters more than volume.

• Retrieve only relevant information → Smaller context is often more effective.

• Summarize stale state → Compaction keeps the prompt focused.

Long context works best when many related facts are needed at the same time. Otherwise, retrieval is usually faster and cheaper.

Scaling laws

Scaling laws describe how model performance changes as parameters, data, and compute increase.

A common misconception is that bigger models automatically create better systems. Size helps, but only when the model is trained with enough high-quality data and served efficiently. A poorly trained large model can still underperform a smaller, better-balanced one.

Model selection is not a popularity contest. You need to balance several tradeoffs:

• Quality → Does it solve the actual task well?

• Latency → Can it respond fast enough for users?

• Cost → Does unit economics make sense?

• Reliability → Does it behave predictably under edge cases?

• Grounding → Does it use the right source material correctly?

The best model is usually not the largest one. It is the one that delivers enough quality at the right cost, latency, and risk level.

Pretraining, fine-tuning, and instruction tuning

LLMs are trained in stages, with each stage solving a different problem.

• Pretraining → The model learns general language patterns and broad knowledge from massive datasets.

• Fine-tuning → The pretrained model is adapted for a narrower domain, behavior, or task.

• Instruction tuning → The model learns how to follow prompts and respond in a more conversational, task-oriented way.

  

A common mistake is treating fine-tuning as the solution for everything.

Fine-tuning works best for stable behavior, like tone, formatting, or repeated workflows. A customer support assistant, for example, may need consistent escalation rules and response style.

But fast-changing information usually does not belong in the model weights. Product pricing, internal docs, or policy updates are usually better handled through retrieval systems like RAG.

In practice, fine-tuning shapes behavior. Retrieval supplies fresh knowledge.

Alignment

Alignment is the process of shaping model behavior toward goals like helpfulness, safety, and policy compliance.

The three main approaches are:

• RLHF (Reinforcement Learning from Human Feedback) → Human reviewers rank model outputs, and the model learns to prefer responses people rate more highly.

• DPO (Direct Preference Optimisation) → A simpler approach that learns directly from preference comparisons without a full reinforcement learning loop.

• Constitutional AI → The model critiques and revises its own responses using a defined set of rules or principles.

Alignment is a balancing act, not a safety switch.

A model can become more helpful while also becoming more willing to follow harmful instructions. It can become safer while also refusing harmless requests too often. A polished response can still contain incorrect information.

That is why alignment alone is not enough. Reliable LLM systems still need retrieval, permissions, evaluation, and monitoring around the model itself.

Prompt engineering

Prompt engineering is the design of instructions, examples, and context given to the model.

It is often the fastest way to improve an LLM system because changing a prompt is far cheaper than retraining a model. A good prompt can define the model’s role, constrain the output format, provide examples, and tell the model how to use retrieved information.

But prompts become fragile when they grow without structure.

• Model updates can change behavior → A prompt that works today may drift tomorrow.

• Examples can overfit → The model may mimic patterns instead of understanding intent.

• Complexity accumulates → Large prompts can turn into brittle mini-applications.

Treat prompts like production code. Version them, test them, and monitor them because prompt behavior directly affects system behavior.

Reasoning patterns

Reasoning techniques help LLMs handle tasks with multiple steps, dependencies, or external information needs.

• Chain-of-thought prompting → Encourages the model to reason step by step.

• Self-consistency → Samples multiple reasoning paths and selects the most consistent answer.

• ReAct → Combines reasoning with tool use, letting the model think, act, observe results, and continue.

These approaches work especially well for math, coding, research, and agent workflows where planning and evidence gathering matter more than fast responses.

But reasoning comes with tradeoffs.

Extra reasoning uses more tokens. Tool calls add latency. Multiple reasoning attempts increase cost even further.

Use reasoning patterns when correctness matters more than speed. For simpler tasks like classification, routing, or formatting, they usually add overhead without much benefit.

Retrieval-augmented generation (RAG)

Retrieval-augmented generation (RAG) connects an LLM to external knowledge sources.

The flow is simple: a user asks a question, the system retrieves relevant documents, and the model generates an answer using that retrieved context.

RAG matters because business knowledge changes far faster than model training. Instead of retraining the model every time information changes, you update the data source.

RAG works especially well when:

• Facts change frequently → Policies, pricing, tickets, and docs evolve constantly.

• Provenance matters → Users need to know where answers came from.

• Knowledge is private → Internal documents should not require retraining a foundation model.

• Updates must stay cheap → Reindexing documents is far easier than retraining models.

But RAG introduces its own failure modes.

The retriever may miss the right document. Chunking may separate information from its context. Ranking may surface misleading evidence. The model may even ignore the retrieved material entirely.

That is why grounding matters. Grounding means tying responses to verified source material instead of letting the model freely improvise.

Hallucination, evaluation, and safety

A hallucination is an unsupported or misleading output generated by the model.

But hallucinations are not caused by one single problem. The model may lack knowledge, retrieval may fail, the prompt may be unclear, the source material may be wrong, or a tool may return stale data.

That is why hallucinations cannot be solved with one technique. Reliable systems reduce them through layers of validation:

• Retrieval evaluation → Did the system fetch the right evidence?

• Grounding checks → Is the answer actually supported by the source material?

• Safety testing → Does the system resist harmful or manipulative inputs?

• Monitoring → Do real users expose failures missed during testing?

Benchmarks help, but they are only proxies. A model can perform well offline and still fail in production.

The real test is whether the system answers your users correctly, safely, and consistently under real-world conditions.

Compression and optimization

Efficiency is what turns an impressive model into something financially and operationally viable.

LLM inference is expensive because every generated token depends on the one before it. As outputs grow, latency and compute costs grow with them.

That is why production systems rely on techniques that reduce memory use and improve serving efficiency:

• Quantization → Uses lower-precision weights to reduce memory and compute costs.

• Pruning → Remove less useful parts of the model or computation.

• Distillation → Trains a smaller model to mimic a larger one.

• FlashAttention → Makes attention faster and more memory-efficient.

• Speculative decoding → Uses a smaller model to draft tokens before a larger model verifies them.

• PagedAttention → Improves memory management during inference serving.

  

In practice, the biggest gains often come from simpler optimizations first: shorter prompts, capped outputs, cached context, batching, and smaller fallback models. You usually do not need a larger GPU cluster as the first solution.

Deployment patterns

Deployment determines where model inference happens, and that decision affects latency, privacy, reliability, and cost.

There are three common deployment patterns:

Edge deployment runs the model on-device or close to the user. Cloud deployment sends requests to centralized model infrastructure. Hybrid systems split workloads between both.

Choose edge when responsiveness and data locality matter most. Choose cloud when model capability and tooling matter more. Choose hybrid when you need both and can cleanly separate the workflow.

Privacy and data governance

Privacy and governance shape how data enters, moves through, and leaves an LLM system.

This is not just a compliance problem. It directly affects architecture.

A privacy-sensitive system may require local processing, encrypted logs, restricted tool access, and short retention windows. Regulated workflows may need human approval before the model can send messages, update records, or trigger actions.

Good governance usually comes down to a few core controls:

• Classify inputs → Know which data is sensitive or regulated.

• Separate instructions from retrieved data → External content should not override system rules.

• Limit tool permissions → Give the model only the access it actually needs.

• Control retention → Store logs only as long as necessary.

• Review vendor policies → Understand how providers handle training and data storage.

The more autonomy a model has, the more important the guardrails become.

Multimodality

Multimodal models work with more than just text. They can process images, audio, video, documents, and combinations of different inputs together.

That expands what LLM systems can do: document analysis, visual search, voice assistants, meeting summaries, accessibility tools, and more natural interfaces.

But multimodality also increases the failure surface.

A model may misread an image, miss small text in a screenshot, mistranscribe audio, or incorrectly connect information across formats. A system that performs well in text may still struggle with visual grounding or audio accuracy.

That is why each modality needs separate evaluation:

• Text → Does the model reason correctly?

• Images → Does it identify visual details reliably?

• Audio → Does transcription preserve meaning?

• Combined inputs → Does it connect the right information across formats?

Evaluate each modality separately then evaluate the combined system, and keep modality-specific fallbacks where accuracy matters.

Observability and monitoring

Traditional monitoring tells you whether a request succeeded. LLM observability needs much deeper visibility.

You need visibility into the entire execution path: which prompt was used, what documents were retrieved, how many tokens were consumed, which tools were called, what safety checks fired, and whether the answer actually helped the user.

That usually means tracing several layers at once:

• Prompt version → Which instructions generated the output?

• Token usage → Did costs spike because prompts or outputs grew?

• Retrieval results → Was the right evidence returned?

• Tool calls → Did the model use the correct tools safely?

• Latency → Was time spent in retrieval, generation, or external tools?

• Safety events → Did the system detect prompt injection or sensitive data risks?

• User outcomes → Did the response actually solve the problem?

Without observability, debugging LLM systems becomes guesswork. With it, failures become traceable.

Wrapping up

LLMs make more sense when you treat them as systems, not just models.

The difficult part is not generating text. It is making the entire system reliable, grounded, observable, fast, and safe under real-world conditions.

That is why successful LLM products are not built from prompts alone. They come from layers working together: retrieval, evaluation, guardrails, serving infrastructure, observability, governance, and careful system design.

The model produces the response.

The system around it determines whether that response can be trusted.

Related LUC reading

• MCP Clearly Explained

👋 If you found this useful → Like + Restack to help others learn system design.

🚀 What's Next for LUC

We're building LUC into a world-class resource for learning system design.

To get there, we’re introducing several new content series over the coming months:

✓ System Design Case Studies

Real-world analyses of how systems at companies like Netflix, Uber, Anthropic, and Amazon are designed.

✓ System Design Deep Dive Series

Deeper, more extensive breakdowns than our regular articles.

✓ AI Engineering Breakdowns

Clear, high-signal breakdowns of AI engineering concepts that can be consumed in ~5 minutes.

✓ AI Engineering Deep Dive Series

Deeper, more extensive breakdowns than our regular articles of AI engineering concepts.

✓ Learning Tracks

Curated learning paths that connect related concepts into structured journeys across system design, AI engineering, and software engineering.

We’d love your input.

We’ll use the results to help prioritize what we build next. Thanks for being part of the LUC community. We’re excited to share what’s coming next!

What MCP Actually Is and Why It Matters

How many custom integrations does your AI stack have?

One for the database, one for Slack, one for the internal knowledge base; each written from scratch, each maintained separately.

Add a new model and the count multiplies again.

Model Context Protocol (MCP) is Anthropic’s answer to that problem.

Rather than building a new connector for every model-tool pairing, MCP defines a single standard interface so any compliant AI host can speak to any compliant server. Add a new tool once, every model that supports MCP can use it immediately.

Understanding how MCP works and where it fits changes how you design AI systems.

What is MCP?

Model Context Protocol (MCP) is a standard way for an AI application to talk to external services.

Instead of inventing a new integration pattern for each tool, MCP defines a common protocol for discovering capabilities, requesting data, and invoking actions.

But it doesn’t stop at message formats.

MCP also defines how a session is established, what features are allowed, and how both sides stay compatible over time.

MCP is structured around three core roles:

• Host → The AI application the user interacts with, such as a chat app or coding assistant

• Client → The connector inside that host that manages one server connection

• Server → The external process or service that exposes capabilities to the host

That separation is what makes MCP scale.

The host decides when to ask for help. The client handles the protocol details. The server owns the actual side effects, such as reading data, calling an API, or writing to a system.

How MCP sessions actually work

Under the hood, MCP is a capability protocol with a negotiated lifecycle.

Every connection starts with a handshake where the client and server agree on:

• Protocol version → Ensures both sides speak the same language

• Capabilities → Defines what features are available in this session

  

After that, all interactions are constrained by what was agreed upfront. No hidden features. No assumptions.

This is what makes MCP different from typical APIs. It’s not just “send a request, get a response.” It’s a structured session where both sides explicitly agree on what’s possible before anything happens.

That extra layer is what turns MCP from a simple integration pattern into something that can scale across many tools, models, and environments.

What servers expose

MCP servers expose capabilities in three distinct ways. Each one exists for a different job, and mixing them leads to messy systems.

The important distinction is between tools and resources.

If you blur that line, your system becomes harder to reason about.

Use this rule:

• Context-heavy → Use a resource because you’re supplying data

• Action-oriented → Use a tool because you’re performing something

If a “tool” mostly returns data with no real side effect, it’s probably a resource in disguise.

Keeping this separation clean makes your system easier to test, easier to secure, and easier to evolve.

The benefits of MCP

MCP solves a scaling problem that shows up fast in real AI systems.

Without a shared protocol, every integration becomes custom work. Your chat app talks to one tool one way, your IDE talks to another tool another way, and your internal agent framework invents a third pattern.

That fragmentation slows teams down because every product repeats the same connector logic.

MCP changes that in a few concrete ways:

• Standardized integration surface → New tools do not require a brand-new communication pattern each time.

• Better separation of concerns → Product logic stays in the host, protocol handling stays in the client, and domain actions stay in the server.

• Safer extensibility → Teams can add capabilities gradually because features are negotiated instead of assumed.

• Easier composition → One host can connect to multiple servers without turning every connection into a one-off project.

This is especially useful in products where the model needs live context.

A coding assistant may need repository data, documentation, terminal access, and issue tracker actions in the same session. MCP gives those capabilities a shared protocol so the host can compose them without stitching together a fresh protocol per tool.

The tradeoffs

MCP simplifies integration, but it does not remove the hard parts of building safe and reliable systems.

The most common mistake is treating the protocol as the safety layer.

It isn’t.

MCP gives you structure. You still decide what gets exposed, what requires user consent, and how risky actions are controlled. If those decisions are weak, the protocol won’t save you.

That means core distributed-systems discipline still applies:

• Check inputs carefully → Don’t assume tool inputs are safe; AI can make mistakes or send bad data

• Limit what tools can do → Give each tool only the access it really needs to avoid misuse.

• Don’t trust descriptions → Just because a tool says it does something doesn’t make it safe.

• Make things easy to track → Use IDs, logs, and progress updates so you can see what’s happening when things get complicated.

• Use structured data → Systems are more reliable when they pass clear, well-defined data instead of free-form text.

There is also a system design tradeoff.

MCP standardises communication, but it does not orchestrate behavior.

• No model strategy → You still choose which model to use and when

• No context ranking → You decide what data is relevant and how it’s selected

• No tool arbitration → You define how conflicts between tools are resolved

• No workflow control → You design how multi-step tasks are executed

MCP removes the need to rebuild integrations, but it leaves system design decisions exactly where they belong; with you.

When MCP makes sense (and when it doesn’t)

MCP is designed for systems that need to scale across multiple tools, data sources, and workflows.

It works best when your AI application needs to:

• Connect to multiple services → Each capability can be exposed and managed independently

• Compose workflows → Outputs from one tool feed into another across a session

• Evolve over time → New tools can be added without rewriting integrations

• Support agent-like behavior → The model dynamically discovers and uses capabilities

In these systems, MCP reduces long-term complexity because integrations become reusable and consistent.

On the other hand, it’s likely overkill when a single, simple API call is all you need.

If your setup is just one model calling one tool with a fixed setup, a direct API call is easier to build, easier to monitor, and cheaper to run.

MCP brings extra layers: like managing versions, handling communication rules, and running supporting infrastructure; which only really make sense once your system starts getting more complex.

Wrapping up

As systems grow, the hardest part isn’t adding intelligence.

It’s making everything work together without breaking.

What starts as a simple integration quickly turns into a web of connectors, assumptions, and edge cases. Over time, that complexity becomes the bottleneck.

MCP addresses that at the foundation level.

It gives you a consistent way to connect systems so you can focus on what those systems should do, not how they talk to each other.

And that shift matters.

Because the teams that move fastest aren’t the ones writing the most integrations.
They’re the ones who can reuse, extend, and evolve what they’ve already built.

That’s the real advantage MCP offers.

👋 If you found this useful → Like + Restack to help others learn system design.

A Practical Hub for Building AI Systems

Presented by Oracle

Building production AI systems means combining far more than just models. Retrieval pipelines, memory layers, agent orchestration, evaluation workflows, observability, and infrastructure all need to work together. Oracle’s AI Developer Hub brings these concepts into one place through practical architectures, implementation guides, courses, and real engineering examples for modern AI applications.

Explore the AI Hub

REST, GraphQL, or gRPC? Choosing the Right Tool for the Job

Your mobile app over-fetches because REST returns the entire resource, even when the screen only needs three fields. Your internal services waste CPU serializing JSON nobody reads. Your front-end team rewrites the same data-fetching glue for every page because the API never lets them ask for exactly what they need.

These look like separate performance problems, but they usually come from the same root cause: using the wrong API style for the interaction.

REST, GraphQL, and gRPC solve different communication problems.

The mistake is treating them as interchangeable.

An API style is really a tradeoff between flexibility, simplicity, efficiency, and control. Once you understand which tradeoff your system actually needs, the decision becomes much clearer.

The core mental model

Before comparing performance, tooling, or syntax, focus on the interaction pattern each API style is designed around.

• REST → Organizes communication around resources. A resource is a stable thing like a user, product, or order, exposed through URLs that clients fetch, create, update, or delete over HTTP.

• GraphQL → Organizes communication around queries. Clients describe the exact shape of data they want, and the server resolves only those fields.

• gRPC → Organizes communication around methods. Services expose strongly typed procedures that other services call directly over HTTP/2 using compact binary messages.

That distinction matters more than the syntax itself.

If your system naturally revolves around stable entities like products, accounts, or invoices, REST usually feels predictable and easy to reason about. The API behaves like a catalog where every resource has a clear address and lifecycle.

If different clients constantly need different views of the same data, GraphQL reduces the coordination overhead. A mobile app might need five fields while a dashboard needs fifty. Instead of creating endless custom endpoints, clients ask for exactly what they need.

If your biggest bottleneck is service-to-service communication, gRPC optimizes for efficiency. Binary payloads are smaller, contracts are strict, and HTTP/2 keeps connections open instead of repeatedly starting new ones. It behaves less like browsing web pages and more like calling functions across the network.

Once you identify whether your system is resource-driven, query-driven, or method-driven, the tradeoffs become much easier to evaluate.

REST: the default for clear resources and public APIs

REST works best when your API benefits from standard web behavior.

A product catalog is a good example. Products, categories, prices, and images are stable resources with clear identities. Browsers, CDNs, proxies, and HTTP caches already know how to handle them efficiently.

That simplicity is REST’s biggest advantage. The URL identifies the resource, the HTTP method describes the action, and the status code explains the result.

REST gives you:

• Browser reach → Standard HTTP works everywhere without special tooling

• Cacheability → GET requests work naturally with browsers, proxies, and CDNs

• Simplicity → Most developers already understand URLs, headers, and status codes

• Public API friendliness → Requests are easy to inspect, test, and debug

The tradeoff is composition.

If a single screen needs products, reviews, recommendations, inventory, and shipping data, REST can turn into multiple round trips. You can create larger endpoints, but clients often receive far more data than they actually use.

Use REST when stability, simplicity, and web infrastructure matter more than fine-grained data control.

GraphQL: the choice for client-shaped data

GraphQL solves a different problem: different clients often need different views of the same data.

A mobile app may only need a product name and price. An admin dashboard may also need inventory, supplier details, and audit history. With REST, you either create many specialized endpoints or return oversized responses that waste bandwidth.

GraphQL lets clients request the exact shape they need.

A view can ask for:

• Product name

• Price

• Inventory status

• First three reviews

Nothing more.

The server validates every query against a schema, which is a typed definition of what clients are allowed to request. That schema becomes a shared contract between frontend and backend teams.

GraphQL gives you:

• Flexible field selection → Clients avoid over-fetching and under-fetching

• Strong schema → Teams can discover data and generate types automatically

• Better composition → One query can gather all data needed for a screen

• Frontend independence → Web, mobile, and internal tools can request different shapes

The tradeoff shifts to the server.

Resolvers (responsible for fetching each field) can accidentally trigger large numbers of database calls. A small query can create expensive backend work if you do not control query depth, batching, and caching carefully. Authorization also becomes more complex because many operations flow through a single endpoint.

Use GraphQL when client data needs vary frequently because it reduces endpoint sprawl while keeping the schema consistent.

gRPC: the go-to for efficient service-to-service calls

gRPC works best when both the client and server are controlled by your organization.

Instead of exposing resources, gRPC exposes services and methods defined in a .proto contract file. Code generation then creates typed clients and servers in different languages, so network calls feel more like local function calls.

That model fits distributed systems well.

A checkout service may need to call pricing, inventory, fraud, shipping, and tax services in milliseconds. Those calls need low overhead, strong contracts, retries, deadlines, and sometimes real-time streaming.

gRPC gives you:

• Strong contracts → .proto files define services and message formats clearly

• Efficient payloads → Binary messages are smaller and faster than JSON

• HTTP/2 support → Multiplexing and persistent connections reduce network overhead

• Streaming support → Services can stream data in one or both directions

• Better service controls → Deadlines, retries, and health checks fit internal systems naturally

The tradeoff is browser compatibility.

Browsers work naturally with plain HTTP and JSON, but gRPC usually needs gRPC-Web and a proxy layer at the edge. That complexity is often acceptable internally, but it creates friction for public-facing APIs.

Use gRPC for internal service communication because controlled environments benefit most from typed contracts and efficient transport.

Side-by-side comparison

When not to use each

Every API style has cases where it becomes the wrong abstraction.

Avoid REST when clients constantly need data from many related resources in a single screen. Multiple round trips add latency quickly, especially on mobile networks. Teams often respond by building oversized aggregation endpoints, which slowly weakens REST’s clean resource model.

Avoid GraphQL when your API is small, stable, and public-facing. Schema governance, query optimization, resolver efficiency, and access control all add operational overhead. If a handful of REST endpoints already solves the problem cleanly, GraphQL can introduce unnecessary complexity.

Avoid gRPC when browsers or third parties are your primary consumers. gRPC-Web works, but it requires extra proxy infrastructure and specialized tooling. For public web APIs, REST or GraphQL is usually the simpler choice.

The pattern most systems end up using

Most production systems do not choose a single API style. They choose boundaries.

A common pattern is:

• REST or GraphQL at the edge → Where browsers, mobile apps, and third parties interact

• gRPC internally → Where services benefit from fast transport, typed contracts, retries, and streaming

For example, a product catalog may expose a REST API that CDNs cache aggressively. A mobile app may query a GraphQL layer to fetch only the fields needed for a screen. Behind that layer, internal services communicate through gRPC using compact protobuf messages and deadline-aware calls.

The architecture works because each technology sits where its strengths matter most.

The trap isn’t choosing the wrong one. It’s choosing one and using it everywhere. Most systems benefit from treating these as complementary layers, not competing philosophies.

👋 If you found this useful → Like + Restack to help others learn system design.

AI Developer Hub: Patterns, Architectures, and Guides in One Repo

Presented by Oracle

Oracle AI Developer Hub is a curated collection of real-world examples, architectures, and guides for building AI-powered applications. From RAG pipelines to agent-based systems, and everything in between. Star it to keep it bookmarked, this repo will keep growing, and you’ll want it on hand as you build.

Explore and Star the Repo

Service Discovery in Distributed Systems

In distributed systems, services appear, disappear, and relocate constantly.

If you don’t have a way to track them, your calls will end up in the void.

Service discovery solves this problem by always pointing clients to a healthy, live instance.

The problem

A payments service that worked yesterday on one IP might be gone today. If your application still points to the old address, every request ends in failure.

Now add dozens of microservices, each with multiple replicas, and you have a system where connections break whenever the environment shifts.

Static approaches like hard-coding addresses or juggling config files might hold up in a small setup, but they collapse under the churn of microservices.

What’s needed is a way to always know which instances are alive and how to reach them.

That’s the role of service discovery. Instead of binding calls to fixed addresses, it keeps a live registry of running instances and resolves a simple name like inventory-service into a healthy endpoint at runtime.

The benefits

• Dynamic scaling → New instances register automatically, so traffic flows without manual updates.

• Fault tolerance → Dead or unhealthy instances drop out, keeping requests routed to live services.

• Load distribution → Requests spread across instances, avoiding overload on a single node.

• Location transparency → Clients call by name, not by IP, decoupling deployments from connectivity.

• Polyglot support → Any language or framework can participate, as long as it can query the registry.

• Faster iteration → Teams deploy services independently without reconfiguring every other client.

• Operational consistency → A single naming and lookup system replaces scattered configs.

Client-side discovery

In client-side discovery, the service making the call is responsible for finding a healthy instance of the target service.

The client queries the service registry, receives a list of available endpoints, and then applies its own load balancing logic to pick an endpoint. Once selected, the client sends the request directly, with no middle layer in between.

The benefits

• Lower latency → Requests go straight from client to service without an extra network hop.

• Fine-grained control → Clients can choose their own load-balancing strategy or routing rules.

• Simpler infrastructure → No central router or proxy to deploy and maintain.

• Predictable performance → Each service handles its own lookups, avoiding potential bottlenecks in a shared routing tier.

The tradeoffs

• Added complexity in clients → Every service must include discovery logic or a sidecar, often across multiple languages.

• Higher maintenance burden → Polyglot systems require consistent libraries or tooling in each stack.

• Tighter coupling → Clients become aware of registry details, making it harder to swap out technologies later.

• Risk of duplicated logic → Without careful standardization, different teams may implement discovery inconsistently.

Client-side discovery shines when you value direct connections and low latency, and when your teams can standardize on a shared library or sidecar pattern. But if simplicity in clients or language diversity is a priority, the tradeoffs can outweigh the benefits.

Server-side discovery

In server-side discovery, the client doesn’t talk to the registry at all.

Instead, it sends every request to a stable endpoint; usually a load balancer, reverse proxy, or gateway. That component then queries the registry, picks a healthy instance, and forwards the request on the client’s behalf.

From the client’s perspective, it’s always calling the same address, regardless of where services are actually running.

The benefits

• Simpler clients → Every client calls the same stable endpoint with no extra logic required.

• Easier polyglot support → One routing tier works for all languages and frameworks.

• Centralized control → Load-balancing, retries, and security policies can be managed in one place.

• Operational consistency → A single router can log, monitor, and enforce rules across all requests.

The tradeoffs

• Extra network hop → Each request passes through a proxy or load balancer, adding some latency.

• Infrastructure dependency → The router or proxy must be highly available, since it becomes part of the critical path.

• Operational overhead → Teams need to run, scale, and monitor an additional component.

• Less flexibility in clients → Custom routing strategies are harder, since decision-making happens elsewhere.

Server-side discovery works best when your system has many different kinds of clients, or when you want to centralize complexity in the platform instead of scattering it across services. The tradeoff is accepting one more hop in exchange for simpler and more uniform clients.

Recap

Service discovery may run quietly in the background, but it’s one of the foundations of modern distributed systems.

Without it, scaling, deploying, or even just keeping services connected becomes a fragile mess of broken addresses and manual fixes.

Even if you get service discovery “for free,” knowing the patterns and tradeoffs makes you better equipped to debug failures, scale confidently, and design services that can evolve.

Think of service discovery not as an add-on, but as a design principle that allows distributed systems to operate as a cohesive whole instead of disconnected pieces.

👋 If you found this useful → Like + Restack to help others learn system design.

Build Kafka, Redis, Git, & Claude Code From Scratch

Presented by CodeCrafters

Want to become a better software engineer?

Trusted by engineers at Google, Anthropic, and Vercel, CodeCrafters helps you rebuild technologies like Kafka, Redis, Git, and Claude Code from scratch so you can deeply understand how real systems work.

We’ve partnered with CodeCrafters to give LUC readers an exclusive 40% lifetime discount.

💡 Many engineers expense platforms like this through their team’s learning budget.

Start Building & Unlock 40% Off

Docker vs Kubernetes: The Real Difference That Matters

Docker and Kubernetes are often framed as competitors. That framing leads to the wrong decisions.

One is about packaging and running software. The other is about managing it at scale.

Treating them as rivals is what leads teams to swap one out for the other at exactly the wrong time; either staying on Docker long past the point where orchestration would help, or jumping to Kubernetes before the complexity is justified.

The gap between them is more specific than it first appears, and closing it starts with understanding what each tool is actually responsible for.

What Docker actually solves

Docker is a container platform. It solves a very specific problem:
“How do I package and run my application reliably anywhere?”

It does this through a few core components:

• Images → Immutable blueprints of your app

• Containers → Running instances of those images

• Daemon → The engine that manages everything

• Docker Client → The command-line interface (CLI) tool that you use to interact with the Docker daemon

• Compose → Defines multi-container apps on one machine

  

The Docker client sends instructions to the Docker daemon, which manages images, containers, networks, and volumes. Docker Compose sits one layer above, letting you define multi-container apps in a YAML file and start them with a single command.

Docker Swarm is Docker’s built-in clustering mode. Manager nodes maintain cluster state using a Raft consensus protocol; worker nodes execute containers.

Where Docker works best

• Local development

• CI pipelines

• Small production deployments

• Single-server applications

What you gain

• Fast setup → Run apps with one command

• Predictability → Same container everywhere

• Low operational overhead → Fewer moving parts

Where it breaks

• Multi-node scaling → Limited native orchestration compared to Kubernetes

• Self-healing → Containers don’t restart intelligently across hosts

• Complex deployments → Rolling updates and traffic routing are manual

Docker focuses on building images, running containers, and managing them on a host.

What Kubernetes actually solves

Kubernetes is an orchestration system. It answers a different question:
“How do I run thousands of containers reliably across many machines?”

It introduces a control loop: you declare what you want, and the system keeps correcting reality until it matches.

Here are the key components:

• Pod → Smallest deployable unit

• Control plane → API server, scheduler, controllers

• Nodes → Machines that run workloads

• Services → Stable endpoints for dynamic pods

  

Kubernetes is architecturally richer. Its control plane includes an API server, a scheduler, a controller manager, and etcd (a distributed key-value store that holds all cluster state). Worker nodes run a kubelet (which ensures Pods stay running), kube-proxy (which handles network routing), and a container runtime like containerd.

Where Kubernetes works best

• Multi-service architectures

• High-availability systems

• Large-scale platforms

• Teams needing standardized deployment workflows

What you gain

• Self-healing → Failed containers restart automatically

• Autoscaling → Add/remove instances based on demand

• Rolling updates → Deploy without downtime

• Service discovery → Built-in networking abstractions

What it costs

• Operational complexity → Many components to manage

• Learning curve → Concepts like Pods, Services, RBAC

• Infrastructure overhead → Control plane + networking + storage

Kubernetes supports clusters with thousands of nodes, but that power comes with significant complexity.

The comparison at a glance

How they actually work together

The real-world setup is not “Docker vs Kubernetes.” It’s:

• Docker builds the image

• A registry stores it

• Kubernetes runs and manages it

This separation matters because it keeps concerns clean:

• Build once (Docker)

• Run anywhere (Kubernetes)

Even though Kubernetes removed direct Docker runtime support, it still runs Docker-built images through OCI-compatible runtimes.

How to decide

When Docker is enough

You don’t need Kubernetes just because it exists.

Use Docker alone when:

• You run on a single server

• Your system has few services

• Downtime during deploys is acceptable

• Your team prefers simplicity over automation

Because every added abstraction has a cost. If you don’t need orchestration, Kubernetes becomes overhead, not value.

When Kubernetes is worth it

Kubernetes becomes the right choice when coordination starts hurting.

Use it when:

• You deploy across multiple machines

• You need zero-downtime deployments

• Traffic fluctuates and needs autoscaling

• You run many services with independent lifecycles

Because Kubernetes solves coordination problems, not just runtime problems.

Where teams go wrong

Most mistakes come from using the right tool at the wrong stage.

1. Starting with Kubernetes too early

• Adds complexity before it solves real problems

• Slows development and onboarding

2. Scaling Docker beyond its limits

• Manual scripts replace missing orchestration

• Reliability becomes fragile

3. Treating them as interchangeable

• Leads to poor architecture decisions

The ideal mindset is this: Use the simplest tool that solves your current problem.

Final takeaway

Start simple. Stay simple as long as you can.

Then, when scaling turns into coordination problems, reach for Kubernetes.

Docker is where most systems should begin. Kubernetes is where many systems eventually need to go.

The transition is a response to real complexity.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

AI Code, Architectures, and Patterns in One Repo

Presented by Oracle

Oracle AI Developer Hub is a curated collection of real-world examples, architectures, and guides for building AI-powered applications. From RAG pipelines to agent-based systems, it brings together practical patterns, code samples, and best practices to help individuals and teams build production-grade AI solutions. If you find it useful, give it a star.

Explore and Star the Repo

How OAuth Actually Works

OAuth is not login. It is permission.

That confusion causes a surprising number of design mistakes.

Teams pass around access tokens as if they prove identity, store them in risky places, or keep old flows alive because they “still work.” The result is a system that looks modern on paper but leaks security in the seams.

At its core, OAuth lets one application get limited access to another service without handing over the user’s password. Used well, it narrows access. Used poorly, it creates a false sense of security.

What OAuth actually solves

OAuth exists because “just give the app your password” is a bad system boundary. The moment a third-party app sees user credentials, you lose separation between identity, consent, and API access.

Every OAuth transaction involves four roles:

• Resource owner → The user who owns the data and decides what to share.

• Client → The application requesting access (a web app, mobile app, or background service).

• Authorization server → Issues tokens after verifying the user’s identity and consent.

• Resource server → The API that holds the protected data and validates tokens on every request.

The separation between the authorization server (which authenticates) and the resource server (which serves data) is intentional.

It lets companies run a single sign-on service across dozens of APIs without coupling their identity infrastructure to every service.

A typical flow looks like this:

1. The client redirects the user to the authorization server

2. The user authenticates and consents

3. The server sends an authorization code back to the client

4. The client exchanges that code for an access token over a secure back-channel.

5. That access token is then sent with every API request.

The user’s credentials never leave the authorization server.

Three tokens, three different jobs

OAuth’s token model trips up a lot of developers because the three token types look similar but serve completely different purposes.

• Access token → Short-lived credential sent with every API call. The resource server validates it on each request.

• Refresh token → Long-lived credential used only at the token endpoint to get a new access token without re-prompting the user. Never send this to an API. If it leaks, an attacker can silently maintain access long after the original session should have expired.

• ID token → A signed JWT that carries authentication claims; who the user is, when they authenticated, which identity provider issued it.

That last one is a common gotcha.

An ID token is not an OAuth token.

It belongs to OpenID Connect (OIDC), which adds an identity layer on top of OAuth. So if your real goal is “log the user in and know who they are,” OAuth is not the answer. OIDC is.

Treating an access token as a login token (i.e. as proof of user identity) is one of the most common implementation mistakes.

Where OAuth systems usually break

Under the hood, OAuth uses bearer tokens. Bearer tokens are especially sensitive because whoever possesses them can use them.

There’s no built-in proof of identity or cryptographic binding; the API simply trusts whoever presents the token. As a result, token leakage is effectively credential theft.

Leaks can occur through logs, query strings, browser storage, referrer headers, or weak transport. That’s why modern guidance focuses on limiting both exposure and impact: use HTTPS everywhere, keep access tokens short-lived, store them carefully, rotate refresh tokens on every use, and strictly validate redirect URIs.

For higher-risk environments, you can go further with sender-constrained tokens, which bind a token to a specific client instead of treating it as transferable:

• mTLS → ties the token to the client’s TLS certificate

• DPoP → binds the token to a client-controlled key pair

These add complexity, but they close the gap that bearer tokens leave open.

What to do for web, mobile, and SPAs

For traditional server-rendered web apps, keep it simple: handle the OAuth flow on the server, store tokens there, and give the browser a hardened session cookie instead.

That way, the browser never directly touches OAuth tokens, which significantly reduces exposure to XSS attacks.

For mobile and desktop apps, assume they’re public clients:

• Use Authorization Code + PKCE (a flow where the app gets a one-time code after login and proves it initiated the request)

• Open the system browser (not an embedded web view)

• Store refresh tokens in secure platform storage like Keychain or Keystore.

For single-page apps, a Backend-for-Frontend (BFF) is often the safest route. The browser talks to your backend, the backend handles OAuth, and the browser just holds a session cookie.

If a BFF isn’t practical:

• Fall back to Authorization Code + PKCE (get a one-time code after login and prove the request came from your app)

• Validate the state parameter (to prevent CSRF attacks)

• Keep tokens in memory rather than persistent storage like localStorage

• Lock down CORS as tightly as possible

  

Wrapping up

OAuth gives applications permission to act, not identity to impersonate.

Every design decision in the framework (scoped tokens, short lifetimes, separate authorization and resource servers) exists to keep that boundary clean.

When something goes wrong with an OAuth implementation, it almost always traces back to that boundary getting blurred: a token meant for one API accepted by another, an access token used to prove identity, a scope granted that was never actually needed.

Keep the boundary clear, and the rest of OAuth becomes straightforward.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Postmortems Without the Reconstruction Work

Presented by incident.io

Writing postmortems is traditionally slow, painful, and often incomplete. Sometimes they don’t get written at all. That’s not just a process problem, it’s an experience problem. incident.io flips this by generating a draft from real incident data, so teams start with context and focus on refining what matters. It turns postmortems into a structured, collaborative workflow that’s actually easy to complete.

Read the Article

Infrastructure as Code Clearly Explained

Can you rebuild your entire production environment from scratch today?

Not approximately. Not “close enough.” Exactly as-is.

If the answer is no, your infrastructure depends on tribal knowledge, console history, or luck.

Infrastructure as Code (IaC) replaces all three with something far more reliable: versioned, reviewable definitions that can be applied again and again without surprises.

What Infrastructure as Code actually means

Infrastructure as Code (IaC) is the practice of defining infrastructure in machine-readable files instead of configuring it manually in consoles or through one-off scripts.

There are two closely related disciplines:

• Provisioning IaC → Defines cloud resources such as networks, IAM roles, compute instances, and managed services.

• Configuration IaC → Defines the desired state of machines and services, such as installed packages, files, and system settings.

Provisioning creates the building.
Configuration furnishes and maintains it.

Both aim at the same outcome: desired state and convergence. You describe what should exist, and a tool calculates the steps required to make reality match that description.

Infrastructure stops being something you “set up” and becomes something you declare, review, and reapply safely.

Why it matters in practice

The benefits of IaC flow directly from treating infrastructure like code.

• Reproducibility → Every environment gets built from the same source, eliminating the “works on my machine” problem at the infrastructure level.

• Auditability → Every change goes through version control and peer review, so you always know who changed what, when, and why.

• Faster, safer delivery → Automated pipelines with preview steps replace manual deploys, so teams ship more often with less risk.

• Scalable governance → Policy checks run before anything is provisioned, so compliance rules are enforced automatically rather than audited after the fact.

  

Without IaC, every environment eventually diverges in small, invisible ways. That divergence is the root cause of a wide range of production incidents.

Choosing a tool

The tool choice depends on your context more than on which tool is “best.” Here’s a practical breakdown:

The common pattern in large organizations is to use Terraform or CloudFormation to provision the infrastructure; and Ansible for OS hardening, package installs, and app configuration.

The IaC workflow that actually works

Effective IaC is less about which tool you use and more about the workflow controls around it.

Most failures don’t come from syntax; they come from skipping validation, review, or safe rollout.

Here’s what a robust pipeline typically looks like:

1. Developer writes and pushes IaC to Git → Infrastructure is defined as code (networks, databases, IAM, etc.), making Git the single source of truth.

2. CI runs validation checks → Formatting, syntax validation, linting, and dependency checks ensure the code is correct and consistent.

3. Security & policy checks run → Misconfiguration scanners, policy-as-code rules, and security tools catch risky permissions or unsafe defaults early.

4. A preview (plan) is generated → The system compares desired vs current state and shows exactly what will be created, updated, or deleted before anything changes.

5. Peer review and approval → Engineers review both the code and the plan to catch issues like accidental deletions, cost spikes, or overly broad access.

6. Apply via a deployment pipeline → A trusted CI/CD runner applies the changes, ensuring consistent execution and controlled credentials.

7. Environment-specific config is injected → The same code is reused across dev, staging, and prod, with differences handled through variables and separate state.

8. Post-deploy checks run → Smoke tests, connectivity checks, and monitoring validation confirm the system actually works after deployment.

9. Environment rollout → Changes move from dev → staging → production with increasing levels of approval to reduce risk.

10. State is updated for future runs → The IaC tool records what exists so future changes can safely build on top of it.

11. Continuous drift detection → Scheduled checks alert when real infrastructure diverges from what’s defined in code.

A simple way to think about it: you declare what you want, the pipeline proves it’s safe, and automation makes it real; consistently, every time.

The security risks you can’t ignore

IaC security failures tend to be systemic: fast pipelines amplify mistakes quickly.

These are the three most dominant risk categories.

1. Secrets exposure

Secrets exposure in IaC happens when sensitive data (like passwords or API keys) is stored or leaked through code, state files, logs, or pipelines.

This can lead to serious risks.

Prevent it by using secret managers, encrypting state, avoiding hardcoding, and enforcing least-privilege access.

2. Over-privileged automation

CI/CD pipelines often get broad permissions “just to make it work.”

Instead, the right model separates preview roles from apply roles, scope permissions per environment, and enforces least privilege.

3. Supply chain integrity

Compromised or untrusted external modules, templates, or dependencies are a quiet attack surface.

If these components are tampered with, they can introduce vulnerabilities or malicious code into your infrastructure.

Without pinning and verification, your infrastructure depends on unverified external code.

When not to use full IaC automation

IaC is almost always worth adopting, but the level of automation should match your maturity.

Full automated apply-on-merge makes sense when you have strong test coverage, good observability, and practiced runbooks. Without those foundations, it moves fast in the wrong direction.

For small prototypes or rarely-changed legacy systems, basic version-controlled templates with manual apply still deliver most of the value: reproducibility, auditability, and the ability to review changes before they land.

The pitfalls teams hit first

A few failure patterns show up reliably when teams adopt IaC without the right habits.

• State mishandling → Storing Terraform state locally, or in source control, leads to conflicts and leaks. Use a remote backend with locking from day one.

• Configuration drift → An urgent fix applied via the console creates a gap between reality and the definition. Enforce a norm: no manual changes without a follow-up pull request, and run drift detection on a schedule.

• Unsafe changes → A small template edit can silently force a resource replacement. Preview gates and policy-as-code checks catch this before it reaches production.

• Skipping validation → Treating IaC as “just YAML” and skipping linting and formatting in CI creates a debt that compounds fast.

Recap

IaC is not a tool you adopt; it’s a discipline you commit to.

The tools (Terraform, CloudFormation, Pulumi, Ansible) are expressions of a single idea: infrastructure should be reproducible, reviewable, and automatable by default.

Start with version control and validation. Add preview gates before you automate applies. Lock down secrets and pipeline permissions early.

When IaC is done well, provisioning an environment feels like merging a pull request; not like defusing a bomb.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Just a Computer. On the Internet. Instantly.

Presented by exe.dev

Sometimes you just need a machine on the internet. One that keeps running when your laptop sleeps, runs your cron jobs, and is accessible when others need it. Exe.dev gives you exactly that. A real VM with persistent disk, built-in HTTPS, and secure access out of the box. No auth setup, no config files, no infrastructure overhead. Spin one up with a single command and start building. It stays live and ready to share.

Try it Out

Redis Clearly Explained

Your database is fast. Your queries are optimized. Your indexes are in place. And yet, under real load, the system still slows down.

The problem often isn’t the database itself. It’s that you’re asking it to answer the same questions, over and over, thousands of times a second.

This is where Redis starts to matter.

It shows up at the edge: handling sessions, caching responses, tracking limits, moving messages. Quietly taking pressure off everything else.

How Redis works under the hood

Redis stores everything in RAM as key–value pairs, but the values are not limited to plain strings. It supports a rich set of data structures:

• Strings → Simple values; used for counters, flags, and cached results.

• Hashes → Field-value maps; ideal for user profiles and session data.

• Lists → Ordered sequences; used for queues and recent activity feeds.

• Sets / Sorted Sets → Unique element collections with optional scoring; used for leaderboards and tag systems.

Each structure comes with optimized commands. Instead of fetching data and processing it in your app, Redis lets you operate directly on the data.

That’s the difference between:

• “Get everything, then sort it”
  vs

• “Ask Redis for the top 10 directly”

Execution model

Redis uses a single-threaded event loop, meaning one command executes at a time.

This sounds limiting, but it avoids locking and keeps operations atomic. In practice, it handles massive throughput because the work per operation is small and predictable.

Persistence (optional but important)

Even though Redis is in-memory, it can persist data:

• RDB snapshots → Periodic full saves

• AOF logs → Append every write operation

You choose how durable you want it to be. Faster setups risk losing recent data; safer setups add overhead.

Where Redis actually shines

Redis is not a general-purpose database replacement. It’s a performance layer that solves specific problems extremely well.

1. Caching

The most common use case is caching expensive reads.

The application checks Redis first; on a hit, data returns in microseconds; on a miss, the app queries the primary database and populates Redis for next time.

The primary database stays as the source of truth, Redis handles the hot path.

This works especially well for expensive query results, API responses, rendered HTML fragments, or any data that is read far more often than it changes.

• Faster responses → Microseconds instead of milliseconds

• Lower DB load → Fewer repeated queries

• Better scalability → Handles spikes smoothly

This pattern alone can reduce database traffic dramatically.

2. Session storage

User sessions need fast reads and writes across every request.

Redis fits because:

• Low latency → No delay on every request

• Time-to-live support → Sessions expire automatically

• High throughput → Handles large user bases

Instead of storing sessions in a database, Redis keeps them close to the application.

3. Rate limiting

Redis makes rate limiting simple through atomic operations.

Each request increments a counter per user, there’s a set expiration per counter (e.g. 10 minutes), requests are rejected after a threshold is reached.

Because these are atomic in-memory operations, they add almost no overhead to the request path. This is why Redis is often used behind API gateways.

4. Pub/Sub and queues

Redis has a built-in publish/subscribe mechanism: one service publishes to a channel; all subscribers receive the message in real time.

For more durable workloads, Redis Lists support simple FIFO queues, and Redis Streams add consumer groups and persistence for cases where message durability matters.

It’s simple, fast, and good enough for many workloads before you need heavier tools.

Benefits vs trade-offs

Where Redis fits in the stack

Redis complements a primary database.

In microservices architectures, Redis plays a second role as a shared fast datastore: a central session store that all frontends hit, a rate limiter that all API gateways respect, or a lightweight message broker that connects services without the overhead of a full queue system like Kafka or RabbitMQ.

Cloud providers offer managed Redis services (AWS ElastiCache, Azure Cache for Redis, and Google Cloud Memorystore) that handle provisioning, scaling, and failover automatically. This makes slotting Redis into an existing stack straightforward, without managing infrastructure directly.

When not to use Redis

Redis is not the right tool when:

• Your dataset is large and cold → RAM is expensive per GB. Storing a 10TB archive in Redis would cost a fortune. Use a disk-based store and cache only the hot subset in Redis.

• You need complex queries → “Find all users from Region A who purchased Product B in the last 30 days” requires a relational database. Redis retrieves by key; it does not scan or join.

• You need strong durability → If losing even a few seconds of writes is unacceptable (financial transactions, medical records), Redis’s persistence model needs careful configuration, and a fully ACID-compliant database is the safer option.

• Your workload is stateless and simple → For systems that don’t cache, don’t need sessions, and don’t need messaging, adding Redis introduces operational complexity without a proportional benefit.

The takeaway

Redis is best understood as a performance multiplier, not a database replacement.

It doesn’t try to solve everything. It solves a narrow set of problems extremely well: caching, sessions, rate limiting, lightweight messaging.

And that’s exactly why it’s everywhere.

Used correctly, Redis turns slow paths into fast ones and fragile systems into responsive ones.

Used blindly, it becomes an expensive, leaky abstraction.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Agent Memory: Building Memory-Aware Agents

Presented by Oracle

Free short course on DeepLearning. By the end, you’ll have assembled a fully stateful Memory Aware Agent that loads prior context at startup, assembles relevant context, state, tools, and outputs and improves across sessions.

Start Building

How Database Indexing Really Works

Slow queries have a reflex fix: add an index.

It works often enough that teams keep doing it; until write latency creeps up, storage balloons, and the query planner starts making strange choices.

That’s the index tax. Real, measurable, and entirely predictable once you understand what an index actually costs.

The goal isn’t to index everything. It’s to index the right things, and know when to stop.

What an index really does

At the simplest level, an index maps a search key to where the row lives, much like a book index maps a chapter to page numbers.

The goal is to cut down how much of the table the engine must look through to answer a query.

That helps in a few common cases:

• Point lookups → WHERE id = 42 becomes a targeted search instead of a full scan.

• Range queries → BETWEEN, <, and > work well when keys stay in order.

• Joins → An index on join keys helps the engine find matching rows faster.

• Ordering → A tree index can return rows in sorted order without an extra sort step.

• Uniqueness → Unique indexes help enforce “this value must appear only once.”

The catch is that every INSERT, UPDATE, and DELETE must also update the index. So every helpful index also comes with a write tax. Add too many, and your database starts carrying extra baggage on every row change.

Index types

Not all indexes work the same way. Each structure solves a different query pattern.

Most relational databases use B-tree style indexes by default, often implemented internally as B+ trees.

B-trees are the safe first choice for most OLTP queries:

• Equality filters → Great for exact matches.

• Range filters → Great for time windows and ordered values.

• Ordered reads → Helpful for ORDER BY.

• General-purpose use → Good default when your data is sortable and your workload is mixed.

B-trees are the workhorse, while other index families exist for more specialized conditions and data shapes.

How the optimizer decides

This is what catches most engineers: just because the index exists, doesn’t mean it’ll be used.

The optimizer decides to use an index when its cost model estimates that the index path is cheaper than reading the table directly.

That decision depends heavily on statistics: row counts, value distribution, selectivity, and sometimes histograms. If those statistics are stale, the optimizer can make a bad decision.

This is why keeping statistics fresh matters.

PostgreSQL uses ANALYZE to populate pg_statistic. MySQL manages histograms via ANALYZE TABLE. If statistics are stale, the optimizer can pick a costly plan even when a good index exists.

The optimizer assigns a cost to each candidate plan and picks the cheapest. In PostgreSQL, you can nudge this by tuning random_page_cost relative to seq_page_cost; lowering the ratio makes the planner more willing to use index scans.

When not to add an index

You shouldn’t add an index just because a column appears in a WHERE clause once. Indexes pay off when they serve a repeated access pattern better than their write and storage cost.

Avoid or remove an index when:

• It is rarely used → Unused indexes still slow writes.

• It duplicates another index → Same leading columns often mean overlapping value.

• The predicate is not selective → If most rows match, scanning may still be cheaper.

• Writes dominate the workload → More indexes means more maintenance on every change.

• The query prevents index use → Wrapping a column in a function can block index use unless you build the matching expression index.

There is also a physical side to this.

Indexes can fragment, bloat, and lose page density over time, especially under heavy updates. That’s why rebuild, reorganize, or reindex operations exist across major databases. An index can be conceptually correct and still operationally unhealthy.

The right mental model

The best indexing strategy is not “index everything.”

It is “match the structure to the predicate, then verify with real plans.”

That means:

• Start with the query pattern → Index for how you filter, join, and sort.

• Start with B-trees → Switch only when the data shape clearly demands another family.

• Keep stats fresh → The optimizer can only choose well with up-to-date statistics.

• Prefer fewer, sharper indexes → Each one should earn its keep.

• Re-check over time → Workloads shift, and yesterday’s useful index becomes tomorrow’s write penalty.

Indexes are best understood as a trade: you spend extra work on writes so reads can skip unnecessary work. Once you see that trade clearly, index tuning stops feeling like superstition and starts feeling like engineering.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox:

Write Software: Private, Persistent, Share with a Link

Presented by exe.dev

Exe.dev gives you instant VMs over SSH. Spin up a full VM in seconds over SSH with built-in HTTPS exposure, persistent storage, and zero cloud configuration. No networking setup, no rebuilding environments, no YAML. Optimized for side projects, internal tools, and AI agents. Start a new sandbox, systemd and all, in a second.

Try it Out

Circuit Breakers: How to Stop Small Failures from Spreading

Retries make your system more resilient. Until they make it worse.

When a dependency is genuinely down, retrying doesn’t help.

That is how a small outage turns into a bigger one: threads pile up, retries multiply, queues grow, and healthy parts of the system start failing just because they are waiting on something unhealthy.

A circuit breaker exists to stop that spread.

It fails fast on purpose, because a quick “no” is often safer than a slow collapse.

What a circuit breaker actually is

A circuit breaker is a proxy that sits between your code and a dependency like an API, database, or downstream service.

It monitors every call for failures, timeouts, and slow responses. When things look bad enough, it stops forwarding requests entirely and starts failing fast instead of letting threads pile up waiting for a service that won’t recover on its own.

Most health-based circuit breakers use three states:

• Closed → Requests flow normally, and the breaker tracks failures, timeouts, or slow calls.

• Open → Requests fail immediately instead of touching the unhealthy dependency.

• Half-open → A small number of trial calls test whether the dependency has recovered.

  

What triggers a trip

Not all failures look the same, so not all circuit breakers trip for the same reason.

Some open after a run of consecutive failures. That works well for low-traffic calls where even a few failures are meaningful.

Some open on failure rate. Instead of asking “Did five calls fail in a row?” they ask “Is too large a share of recent traffic failing?” That works better for high-volume services because it filters out noise.

Some open on slow-call rate. This is important because a dependency does not need to be fully down to be dangerous. A slow database can burn your connection pool long before it starts returning errors.

And some proxy or mesh-level breakers focus on capacity rather than health. They trip on too many active requests, pending requests, retries, or connections. These don’t ask whether the dependency is broken. They ask whether the system can safely take more load.

Here is the key difference:

That is why teams often use both. One protects against unhealthy behavior. The other protects against unhealthy volume.

Why circuit breakers help so much

The biggest benefit is not speed. It is containment.

When a dependency is clearly struggling, the breaker stops your service from wasting resources on calls that are likely to fail anyway.

That gives the rest of your system room to breathe.

It also makes failure easier to shape.

• Fail fast → Users get a quick error or fallback instead of waiting through long timeouts.

• Protect capacity → Threads, sockets, and pools stay available for work that still has a chance to succeed.

• Reduce blast radius → One broken service is less likely to drag healthy services down with it.

• Support graceful degradation → You can return cached data, defaults, or partial results instead of total failure.

  

This is why circuit breakers often pair well with fallbacks. If a recommendation service is down, your app can still show popular items. If a profile service is slow, you can still render most of the page.

The user sees degraded behavior, not a frozen system.

Observability is non-negotiable

A circuit breaker without monitoring is a black box that silently shapes your availability.

At minimum, track:

• State transitions (Closed → Open → Half-Open)

• Failure rate and slow-call rate gauges

• Not-permitted calls, requests the breaker rejected without forwarding

• Overflow counters if you’re using Envoy (e.g. upstream_cx_overflow)

Alert on state transitions when they exceed a rate threshold, and alert on sustained not-permitted calls correlated with user-impact SLIs (service level indicators).

One-off blips don’t warrant pages; sustained opens that burn your error budget do.

When not to lean on a circuit breaker

Circuit breakers are powerful, but they are not a substitute for basic resilience hygiene.

Do not use circuit breakers as a bandage for missing timeouts. If your calls can hang indefinitely, a circuit breaker will only react after damage is already done. Always set clear, enforced timeouts first, because they limit how long resources stay blocked.

Separate retries from circuit breakers. They solve different failure patterns. Retries help with brief, transient failures. Circuit breakers help when failure lasts long enough that retrying becomes harmful.

Avoid adding breakers everywhere. Not every dependency needs one. Each breaker introduces configuration, monitoring, and edge cases. Use them where failure is costly, not where it is merely inconvenient.

And be careful with shared breakers. If one breaker covers several independent shards or providers, failures in one can block traffic to the healthy ones too.

Wrapping up

Cascading failure doesn’t start with a catastrophic event.

It starts with one slow service and nothing to stop the pressure from spreading.

A circuit breaker is that stop.

That is the real win: a smaller blast radius when something goes wrong. And in distributed systems, survival often depends less on avoiding failure and more on containing it.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Unblocked: Context That Saves You Time and Tokens

Presented by Unblocked

Stop babysitting your coding agents. Unblocked gives them the organizational knowledge to generate mergeable code without the back and forth. It pulls context from across your engineering stack, resolves conflicts, and cuts the rework cycle by delivering only what agents need for the task at hand.

See how it works

REST APIs Principles Explained Clearly

What makes an API RESTful?

If your first thought is “resources and CRUD,” you’re missing half the picture.

REST is an architectural style with specific constraints, and most APIs only follow a subset. Knowing the full model helps you design better interfaces; and know when to choose something else.

What is REST

REST (Representational State Transfer) is an architectural style for distributed systems. It’s not a protocol, not “just JSON,” and not a synonym for “HTTP API.”

To be truly RESTful, a system follows six constraints:

• Client–server separation → UI and data/service concerns are split so each can evolve independently

• Stateless interactions → Every request carries the full context; the server stores no client session

• Cacheable responses → Responses say whether they can be cached, so intermediaries can safely reuse them

• Uniform interface → Resources have URIs; standard HTTP methods and status codes describe operations. In strict REST, the interface is also driven by hypermedia (HATEOAS), though most APIs don’t fully implement this.

• Layered system → Clients can’t tell if they talk to the origin server, a gateway, or a proxy in the middle

• Code-on-demand (optional) → Servers may send executable code (like JS) to extend client behavior

These rules sound theoretical, but they explain why REST scales, why it’s easy to debug, and why you can throw a CDN or API gateway in front without rewriting your app.

The REST constraints, de-jargoned

Here’s how the core constraints translate into day-to-day API design:

Think of REST as the web’s “grammar” for talking about resources. Once you learn it for one API, others feel familiar because they follow the same structure.

How a REST API actually works

A REST API is basically: resources + URIs + HTTP semantics.

• Resources → Domain nouns like users, orders, products, employees

• URIs → Stable paths: /products, /products/42, /customers/123/orders

• Methods → Map closely to CRUD:

  • GET → Read

  • POST → Create

  • PUT / PATCH → Update

  • DELETE → Remove

Add to that:

• Representations → Usually JSON documents that describe the resource state

• Status codes → Shared language for outcomes (200 OK, 201 Created, 404 Not Found, 500 Internal Server Error)

• Caching → GET endpoints designed to be safely cached at browser, proxy, or CDN layers

For example: in an e-commerce API, GET /products lists items, GET /products/123 shows one product, POST /orders creates an order, and GET /customers/123/orders lists a customer’s history. Internally, the server can switch databases or split into microservices, but as long as those URIs and contracts stay stable, clients don’t care.

Why REST became the default

REST won not because it’s perfect, but because the trade-offs line up well with how most applications work.

• Simple to learn → You can poke at a REST API with a browser or curl because it’s just HTTP

• Interoperable → Any language, framework, or device that can speak HTTP can be a client

• Scalable → Stateless servers are easy to scale horizontally behind a load balancer

• Fast enough → Built-in HTTP caching plus lightweight JSON is plenty for most CRUD-style workloads

• Decoupled → Backend teams can refactor internals or swap infrastructure without breaking clients

• Great tooling → OpenAPI, Postman, language SDK generators, gateways, and observability stacks all “speak REST”

If your domain is resource-centric (products, posts, users, tickets) and most operations are CRUD-ish, REST is usually the path of least resistance.

Where REST starts to creak

REST’s weaknesses show up once your data needs get more complex or more demanding than “list + detail”:

• Over-fetching → Clients get big objects when they only needed a few fields

• Under-fetching → Clients chain multiple calls (user, then orders, then recommendations) to render one screen

• Chattiness → Many small HTTP requests stack up latency, especially on mobile networks

• Loose contracts → JSON shapes are enforced by convention and docs, not by the protocol itself

• Versioning pain → Breaking changes often mean /v2 endpoints or awkward backwards compatibility

• Real-time gaps → REST is request–response; you need WebSockets or SSE for live updates

These aren’t fatal flaws, but they’re signals that you might need something in addition to REST.

REST vs GraphQL vs gRPC: choosing wisely

You don’t have to pick a single style for everything. Use them where they fit best:

A good rule of thumb:

• Start with REST → because it’s simple, well-understood, and easy to expose externally

• Add GraphQL → when specific clients juggle too many REST calls or need tailored payloads

• Use gRPC → inside your backend when performance, streaming, or strong typing is critical

When not to use REST

Avoid pure REST as the main interface when:

• Your main goal is to query complex data, not manage individual resources.

• Your services call each other thousands of times per second and every millisecond and byte counts

• Your clients need live streams of updates, not occasional snapshots

In these cases, REST can still play a supporting role (for management, control, or external access), but it shouldn’t carry the whole workload.

Recap

REST is more than JSON over HTTP. It’s a set of constraints that, when followed, give you APIs that are simple, scalable, cacheable, and widely compatible.

Understand those principles and you can do three important things: design cleaner APIs, recognize when a “REST API” isn’t really RESTful, and know when to reach for GraphQL or gRPC instead.

That’s the difference between copying patterns and making deliberate architectural choices.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Build a Multi Agent RAG system with A2A Protocol and LangChain

Presented by Oracle

Most multi-agent systems hit a bottleneck when one agent gets overloaded. The usual fix is scaling the entire pipeline, which adds cost without solving the real problem. With the A2A protocol, agents can discover each other, declare capabilities, and dynamically share workload so you can scale only the bottleneck.

This guide walks you through building a scalable multi-agent RAG system.

Start building

How Single Sign-On Actually Works

You’ve used SSO dozens of times without thinking about it.

You click “Sign in with Google,” land on a familiar login page, and seconds later you’re inside an app that never asked for your password.

It felt instant and frictionless.

But behind that smooth experience is a precise sequence of redirects, tokens, and cryptographic checks played out in an instant.

SSO isn’t magic. It’s a trust protocol; and understanding it changes how you design authentication, debug login failures, and make decisions about identity infrastructure.

The core idea: Centralize identity, decentralize access

At its heart, SSO separates authentication from application logic.

Instead of every app verifying usernames and passwords, one central service does it. That service is called the Identity Provider (IdP). The applications you actually use are called Service Providers (SPs).

The key shift is this: SPs don’t authenticate you. They verify that the IdP authenticated you.

That distinction matters because it defines who owns credentials, who signs tokens, and who enforces trust.

Why browsers make this necessary

Browsers enforce the same-origin policy. That means one website cannot read another website’s cookies.

So if app1.com and app2.com are separate domains, they cannot directly share login sessions.

SSO solves this by introducing a trusted third party: the IdP lives on its own domain (e.g. login.company.com). Each application redirects you there when authentication is required. The IdP becomes the only place where credentials are checked.

Instead of apps sharing cookies, they share trust.

The step-by-step flow

Here’s what happens when you access an SSO-protected app for the first time:

1. User requests a protected app

You visit serviceprovider.com.

The SP checks for a valid session cookie, finds none, and knows it needs to authenticate you.

2. SP redirects to the IdP

Instead of showing a login form, the SP issues a HTTP redirect to the IdP.

That redirect includes:

• A request identifying the SP

• A return URL

• Possibly a signed authentication request

Your browser now talks to the IdP.

3. IdP authenticates the user

If you’re not already logged in at the IdP, it presents a login page.

You enter credentials. The IdP verifies them. It may also enforce:

• Multi-factor authentication

• Device checks

• Risk-based policies

If successful, the IdP creates its own session cookie for you.

4. IdP issues a token

The IdP generates an authentication token. This token contains:

• User identity

• Issuer (the IdP)

• Audience (the SP)

• Expiration timestamp

• Digital signature

The browser is redirected back to the SP with this token attached.

5. SP validates the token

The SP verifies:

• Signature → Was it signed by the trusted IdP?

• Expiry → Is it still valid?

• Audience → Was it intended for this app?

If everything checks out, the SP creates its own session and logs the user in.

At no point did the SP see the password.

Where “single” sign-on actually happens

The magic appears when you visit a second application.

You go to anotherservice.com.

It redirects you to the same IdP.

But now the IdP sees its session cookie. You’re already authenticated.

So it immediately generates a new token for the second app without asking for credentials again.

From your perspective, no login screen appears.

Behind the scenes, the full protocol still runs. It just skips the credential step.

That’s single sign-on.

What makes it secure

The authentication token is the core artifact. Its security rests on three properties:

• Digital signature → The IdP signs the token with its private key. The SP verifies the signature with the IdP’s public key, proving the token wasn’t forged or altered in transit.

• Bounded lifetime → Tokens include an expiry timestamp. A stolen token becomes useless after minutes, limiting the blast radius of interception.

• Audience restriction → Tokens name the intended SP. A token issued for app1.com is rejected by app2.com, preventing token reuse across services.

The SP never sees the user’s password. It only sees the signed assertion.

When SSO becomes a risk

SSO centralizes authentication. That’s powerful, but it concentrates risk.

• If the IdP goes down → every SP login fails

• If the IdP is compromised → every integrated app is exposed

SSO reduces password sprawl but increases dependency on identity infrastructure.

It works best when:

• The app is used by teams or organizations, not just individuals

• Customers expect centralized admin control (onboarding, offboarding, permissions)

• The app handles sensitive data or privileged actions (finance, customer data, infra, admin tooling)

• You’re selling into mid-market or enterprise, where SSO is often a requirement

• Users are part of a broader ecosystem (Google Workspace, Entra, Okta) and need seamless access across tools

• The product is part of a multi-app suite, and reducing login friction improves workflow

It may be overkill for:

• Consumer apps with individual users and no org-level administration

• Early-stage products where few (or no) customers are asking for SSO yet

• Low-risk applications that don’t handle sensitive data or critical operations

• Small internal tools used by a fixed group, especially behind VPN or other trust boundaries

• Apps where identity is not central to the workflow and login happens infrequently

Wrapping up

SSO removes the password from your app.

It doesn’t remove the responsibility.

Token validation, session expiry, single logout, and IdP availability all stay on your plate; and they matter more, not less, because authentication is now shared infrastructure.

The protocol handles the authentication. Everything that follows is still yours to design.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox:

Reproducible Dev Environments That Work Everywhere

Presented by Flox

Flox lets teams define development environments once and run them consistently across local environments, CI pipelines, and production systems. By capturing dependencies, tools, and environment variables in a single manifest, teams can eliminate environment drift and ship software more reliably.

Learn More

Hashing vs Encryption vs Tokenization: How to Pick the Right Tool

Most security vulnerabilities aren’t caused by weak algorithms.

They come from using the right tool in the wrong situation.

Encrypting what should be hashed, or hashing what should be tokenized, doesn’t trigger an alarm; it just quietly erodes your defenses while everything appears to be working fine.

Hashing, encryption, and tokenization all transform sensitive data, but they answer completely different questions. Knowing which question you’re asking is the whole game.

The core question: what are you trying to protect?

Before picking a tool, answer one question:

Do you need to recover the original value?

• If no → hashing is likely the right fit.

• If yes, and the goal is confidentiality → use encryption.

• If yes, but the goal is limiting where sensitive data spreads → use tokenization.

Each technique maps cleanly to a different security objective:

• Hashing → Integrity and verification

• Encryption → Confidentiality with controlled access

• Tokenization → Exposure reduction across systems

They overlap in underlying mechanisms. They do not overlap in purpose.

Hashing

A cryptographic hash function takes input of any size and produces a fixed-length output called a digest. Change a single character, and the output changes completely.

It is intentionally one-way: you cannot feasibly recover the original input from the digest.

Hashing is built for integrity:

• Verifying file downloads

• Detecting message tampering

• Supporting digital signatures

• Storing passwords (with specialized constructions)

But hashing has a subtlety most developers miss: not all hashing is the same.

• Unkeyed hash (e.g. SHA-256) → Confirm that data hasn’t changed, but only through a trusted channel. An attacker who controls both the file and its published digest can swap both.

• Keyed hash (e.g. HMAC) → Use a shared secret to authenticate data against an active attacker. Required when you need integrity and authenticity.

• Password hashing (e.g. Argon2, scrypt, bcrypt) → A specialized case. Deliberately slow and memory-hard to make brute-force attacks expensive after a database breach. Never use SHA-256 for passwords.

The most common failure here is using a fast, unkeyed hash as a stand-in for authentication.

It isn’t.

What hashing cannot do:

• It cannot provide confidentiality.

• It cannot restore data.

• It does not automatically provide authentication unless keyed.

If you need the original value later, hashing is the wrong tool.

Encryption

Encryption transforms plaintext into ciphertext using a key. Only someone with the right key can reverse it.

Unlike hashing, encryption is reversible by design.

This makes it the right tool for confidentiality:

• Data in transit (TLS)

• Data at rest (disk/database encryption)

• Secure messaging

• Secure storage

Modern encryption comes in two main shapes:

• Symmetric encryption (AES) → The same key encrypts and decrypts. Fast and designed for bulk data.

• Asymmetric encryption (RSA, ECC) → Public key encrypts, private key decrypts. Slower, used primarily for key exchange and signatures, not bulk data.

Real systems almost always use hybrid encryption: symmetric encryption for the data itself, asymmetric encryption to securely exchange the symmetric key. TLS works exactly this way.

There’s one critical nuance: confidentiality is not the same as integrity.

Today, encryption without integrity is considered incomplete. Authenticated Encryption with Associated Data (AEAD), such as AES-GCM, protects both:

• Confidentiality → Attackers can’t read it.

• Integrity/authenticity → Attackers can’t modify it undetected .

Older designs that are encrypted without authentication can be manipulated by an attacker who never sees the plaintext. This is why AEAD is now strongly preferred.

What encryption optimizes for:

• Controlled access to original data

• Strong confidentiality guarantees

• Regulatory recognition (GDPR, HIPAA references encryption explicitly)

But encryption introduces a heavy dependency:

Key management.

Key lifecycle, rotation, storage (often HSM-backed), and cryptoperiod policies become first-class concerns.

Encryption protects content. It does not reduce how widely that content spreads.

Tokenization

Tokenization replaces a sensitive value with a surrogate “token”. The token carries no exploitable meaning on its own.

For example, in payment systems a PAN (Primary Account Number) becomes a token and downstream systems store the token, not the card number.

The difference from encryption is architectural:

• Encryption protects data.

• Tokenization limits exposure of data across systems.

Encryption is the answer to “how do we hide this data?”. Tokenization is the answer to “how do we stop this data from spreading?”.

There are three main implementations:

• Vault-based tokenization → A random token is generated and stored alongside the original value in a protected database (the vault). De-tokenization means querying the vault. Simple to reason about, but the vault becomes your highest-value attack target.

• Vaultless (cryptographic) tokenization → The token is generated using format-preserving encryption. No vault needed, but key management becomes critical.

• Irreversible tokenization → Tokens are never designed to be reversed. Used when you only need to validate or reference a record, not recover the original value.

Tokenization earns its place in regulated environments because it limits where sensitive data lives.

Side-by-side comparison

Hashing, encryption, and tokenization all transform data, but they protect it in very different ways.

When not to use each

Knowing when not to reach for a tool is as important as knowing when to use it.

Don’t use hashing when:

• You need to retrieve the original value. A hash is a one-way door.

• You’re protecting a password with a fast algorithm like SHA-256. Use bcrypt, scrypt, or Argon2 instead, because speed is the enemy of password security.

• You’re treating an unkeyed hash as a MAC. It isn’t, and extension attacks will prove it.

Don’t use encryption when:

• You never need to read the original value back. Hashing is simpler and removes the key-management burden.

• You’re trying to limit data spread across services. Encrypting a PAN and sending the ciphertext everywhere just moves the problem; tokenization removes it.

Don’t use tokenization when:

• Your system is small and doesn’t have a serious compliance or data-minimization requirement. Running a vault adds availability dependencies and operational overhead that may not be justified.

• You need the original value to be queryable for range scans or analytics joins. Tokenization scatters values in ways that break ordering and aggregation.

Recap

Hashing, encryption, and tokenization aren’t variations of the same idea; they’re answers to different questions.

Hash when you need to verify. Encrypt when you need to retrieve. Tokenize when you need to contain.

Get the question wrong and the technique doesn’t fail loudly; it just quietly protects the wrong thing.

The algorithm is rarely the problem. The choice usually is.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox:

The Aiven Free Tier Competition

Presented by Aiven

The best way to learn is by doing. Build with Aiven’s free tier services like Kafka and PostgreSQL. No credit cards, no expiring trials, just real, managed instances to help you get your project off the ground. Build, share, and win $1,000!

Start building

Synchronous vs Asynchronous Communication

Why does placing an order feel instant in one product and slow in another?

The difference is often invisible to users but fundamental to the system.

It comes down to how much of the checkout flow runs synchronously, waiting for every dependency to confirm success, versus how much runs asynchronously, allowing non-critical work to complete after the order is accepted.

The only difference that matters: do you wait?

Imagine clicking “Purchase”.

Behind that button, the system fans out to payments, inventory, shipping, email, analytics, and fraud checks.

In a synchronous design, the checkout flow waits.

The order service calls each dependency and blocks until every one responds. If payment is slow or the email service hiccups, the user stares at a spinner. Nothing moves forward until the last response comes back.

In an asynchronous design, the order is accepted first.

The system records the order, responds to the user, and emits messages for the rest of the work. Payment confirmation, receipts, analytics, and fulfillment happen independently, often through a queue or event bus, without holding the checkout flow hostage.

That’s the real distinction.

Synchronous communication means sending a request and waiting for a reply before continuing.

Asynchronous communication means sending a message and moving on, trusting the system to finish the work later.

Synchronous communication: wait for the response

Synchronous calls feel clean because the control flow stays linear:

• Inline response → you get a result (or error) right away, in the same execution path.

• Known outcome on return → the caller knows whether it succeeded as soon as the call finishes.

• User-facing fit → logins, reads, validations, and short operations match the “user is waiting” model.

But synchronous has temporal coupling (dependency on being up at the same time). If the downstream service slows down or fails, the caller waits, times out, or fails too; often turning one slow component into a platform-wide incident.

It also stacks latency. A single call can be fast, but a chain of calls is serial: A waits for B, then B waits for C, and the user waits for all of it. Under load, threads pile up waiting and throughput collapses.

When not to use synchronous communication

Asynchronous communication: send and continue

Asynchronous systems usually introduce a message broker (a system that stores and routes messages) and communicate via queues or topics. Producers publish; consumers process when they can.

This changes the system’s structure:

• Decoupling & resilience → messages can wait when a consumer is slow or offline, reducing cascading failures.

• Throughput & scalability → producers don’t block, and you can scale consumers horizontally to chew through backlog.

• Fan-out by default → one “order placed” event can trigger email, analytics, fraud checks, and inventory updates without the producer calling each service directly.

The trade is eventual consistency: the system acknowledges the request now, but the final outcome happens later. How long it takes depends on how many messages are waiting and how fast consumers process them.

Async also raises the engineering bar:

• More moving parts → brokers, persistence, acknowledgements, retries.

• Idempotency required → consumers must safely handle duplicates (idempotent means “running it twice produces the same effect as once”).

• Harder failure handling → errors may surface later, sometimes via a dead-letter queue (messages that repeatedly fail and need inspection).

When not to use asynchronous communication

How to choose in practice

Most real systems don’t pick one style.

They place synchronous and asynchronous communication deliberately.

Use synchronous communication when the system cannot move forward without a definitive answer.

This includes login checks, payment authorization, inventory validation, and critical reads. In these cases, waiting is the point because correctness matters more than throughput. Failures should surface immediately and stop the flow.

Use asynchronous communication when the work does not need to block the user.

Long-running tasks, bursty workloads, and fan-out behavior belong here. Queuing the work keeps the system responsive and prevents one slow dependency from stalling everything else.

This is how systems gain autonomy and absorb partial failure.

Most mature architectures follow a simple pattern:

• Sync at the edge → confirm the user’s action quickly and clearly.

• Async behind the edge → process side effects like notifications, analytics, and downstream updates without holding the request open.

That split keeps the user experience fast while letting the system scale and recover under load.

One important caveat: “async code” is not the same as asynchronous communication.

If a service must wait for a response before it can proceed, the interaction is still synchronous at the architecture level; even if it uses promises, callbacks, or non-blocking I/O.

Wrapping up

Go back to the moment the user clicks “Purchase.”

Whether that action feels instant or slow is a design choice.

Payment authorization, inventory checks, and order creation run synchronously because the system needs a definitive answer.

Waiting there is intentional. That’s the critical path.

Everything else (emails, analytics, fraud checks, fulfillment) doesn’t need to block the user. Those steps run asynchronously so the order can complete even when downstream systems are slow or unavailable.

That’s the real trade-off.

Synchronous communication gives you immediate certainty but shared failure. Asynchronous communication gives you resilience and throughput by letting work finish later.

Well-designed systems use both.

If a user flow feels slow, the fix isn’t always faster processing.

Often, it’s deciding what truly needs an answer now; and what can wait.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox:

Why Your AI Agent Has Amnesia

Presented by Oracle

Many AI agents struggle with memory. They lose context, forget prior steps, and produce inconsistent outputs across sessions. This guide explores why it happens and how to fix it using agent memory patterns. A great read!

Read the article

How Databases Keep Passwords Safe

“Hash it and you’re done” isn’t a strategy. It’s how breaches turn into headlines.

Every system is one misconfiguration, forgotten backup, or zero-day exploit away from a database leak.

When that happens, the only thing standing between you and a wave of account takeovers is how those passwords were stored.

Password security isn’t about keeping data secret; it’s about making stolen data useless.

That’s why modern storage relies on one-way hashing, unique salts, and slow, memory-hard algorithms that make every brute-force attempt painfully expensive.

Real security doesn’t assume safety, it assumes compromise and plans for it.

How secure storage works

When a database breach happens, it’s already too late to protect the data itself.

What still matters is how much that data can be used against you.

If your system stores plaintext passwords, attackers can log in immediately. Even “hashed” passwords aren’t safe if you used fast, general-purpose algorithms like MD5 or SHA-1.

GPUs can brute-force at billions of guesses per second.

To prevent that, password storage must slow attackers down.

Modern password storage relies on layered defenses. Each step (hashing, salting, stretching, and sometimes peppering) closes a different gap that attackers exploit.

• Hashing → Converts a password into a one-way digest that can’t be reversed. The server never stores the password itself, only its hash.

  • Use password-specific algorithms like bcrypt, scrypt, or Argon2id, not general-purpose hashes like SHA-256. Fast hashes are designed for data integrity, not for resisting brute force. A proper password KDF (Key Derivation Function) intentionally slows verification to make offline guessing expensive.

• Salting → Adds a random, per-user value to each password before hashing.

  • Salts ensure that identical passwords never produce the same hash and stop rainbow table lookups. They don’t need to be secret, but they must be unique and unpredictable (16 bytes or more).

• Key stretching → Increases the cost of each hash operation through extra iterations or memory hardness.

  • This throttles brute-force attacks: where a fast hash might take microseconds, a stretched one takes hundreds of milliseconds. On your servers, that’s manageable. For an attacker, multiplied by millions of guesses, it’s economically pointless.

• Peppering (optional) → Adds a server-side secret, stored outside the database (like in an HSM or key vault).

  • Even if the database leaks, the attacker can’t verify guesses without the pepper. It strengthens security but complicates rotation; use it only if your infrastructure can protect and rotate secrets safely.

The goal isn’t to make passwords uncrackable, but to make cracking so slow and individualized that attackers move on long before they succeed.

How passwords are stored and verified

When a user creates an account or logs in later, the system never handles their password in plain text.

Both flows (signup and login) rely on the same hashing process but apply it at different points.

1. Signup → Creating and storing the hash

When a user signs up or resets their password, the system never stores the password itself. Instead, it transforms it into a secure representation:

1. Generate a salt → Create a random, unique salt for this user (usually 16 bytes or more).

2. Hash the password → Run the password and salt through a password hashing function such as Argon2id, bcrypt, or scrypt. The function also takes cost parameters (iterations, memory, or parallelism) that control how slow the process is.

3. Store the result → Save the resulting hash, along with the salt and parameters, in the database.

   

The original password is discarded immediately. Only the salted hash and configuration is stored.

2. Login → Verifying the password

When the user logs in later, the system must check the password without knowing the original one:

1. Retrieve stored values → Fetch the user’s stored hash, salt, and parameters.

2. Re-hash the input → Run the entered password through the same function using the retrieved salt and parameters.

3. Compare securely → Use a constant-time comparison to check whether the new hash matches the stored one. Unlike normal string checks that exit early on mismatches, constant-time comparisons take the same amount of time no matter how similar the inputs are. This prevents timing attacks, where an attacker measures tiny response-time differences to guess parts of the correct hash one byte at a time.

4. Rehash if needed → If parameters are outdated (for example, fewer iterations, lower memory, or an outdated algorithm), re-hash and update the stored entry after a successful login.

5. Authorize access → If hashes match, issue a session token or JWT for the user.

   

At no point does the system store, transmit, or recover the plaintext password. The entire process proves identity by reproducing the same one-way result, not by revealing the secret itself.

What not to do

Most password breaches don’t happen because teams ignored security, they happen because they trusted weak defaults or outdated practices.

Here’s what to avoid if you want your password storage to hold up under real attacks.

• Store plaintext passwords → The worst-case scenario. A single database dump means instant access to every account. There’s no recovery from this.

• Use general-purpose hashes (MD5, SHA-1, SHA-256) → These functions are designed for speed, not security. They are trivial to brute-force at scale with modern GPUs and ASICs.

• Skip or reuse salts → Without a unique, random salt per password, identical passwords produce identical hashes, enabling rainbow table attacks and cross-user correlation.

• Set low cost factors → bcrypt cost=8 or PBKDF2 with a few thousand iterations might have been fine a decade ago. Today, it’s too cheap to crack. Always tune cost to current hardware performance.

• Roll out your own algorithm → Security through creativity is not security. Established KDFs like Argon2id, bcrypt, or scrypt have been battle-tested and peer-reviewed; your custom hashing function has not.

• Forget edge behaviors → Logging plaintext passwords during debugging or including them in error traces can silently undo all your protections.

• Expose hints or partial passwords → Anything that helps an attacker narrow the search space reduces the work of cracking. Even “helpful” password hints expose predictable details.

Recap

Good password storage doesn’t depend on secrecy; it depends on math, cost, and discipline.

Hashing, salting, stretching, and (optionally) peppering turn every password into a unique, expensive problem for attackers to solve.

The goal isn’t to make breaches impossible, but to make them irrelevant by ensuring stolen data is useless.

Use proven KDFs like Argon2id or bcrypt, tune them regularly, and avoid shortcuts that trade speed for safety.

When done right, a database leak becomes an inconvenience; not a disaster.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox:

Build AI Apps with MongoDB

Presented by MongoDB

Looking to stay ahead of the curve on AI? MongoDB AI Learning Hub has the technical training pathways and tools you need to level up your AI app-building game. Explore practical guides, tutorials, and quick starts for all skill levels, from foundational concepts like understanding AI tool stacks to advanced implementations using RAG, MongoDB Vector Search, and LLM optimization.

Check it out (it's free)

CAP Theorem Clearly Explained

“Make it consistent, highly available, and fault-tolerant” sounds like a reasonable requirement.

It isn’t.

In any distributed system, a network failure forces a choice that no amount of engineering can avoid: either your nodes agree on the truth and some requests fail, or every request gets an answer and some of those answers are wrong.

Both are valid designs, and CAP Theorem is the lens that makes that tradeoff explicit so you choose deliberately instead of by accident.

What is CAP Theorem?

CAP Theorem states that a distributed data store cannot simultaneously guarantee all three of the following properties at the same time:

• Consistency (C) → Every read returns the most recent write, or an error; never stale data.

• Availability (A) → Every request receives a response, even if some nodes are out of date.

• Partition Tolerance (P) → The system keeps operating despite network failures that split nodes into isolated groups.

The common summary is “pick two of three.”

It’s catchy, but misleading.

CAP is not about permanently giving something up. It’s about what happens when the network splits.

A clearer way to look at it: when a partition occurs, you must choose between consistency (all nodes agree on the same data) and availability (every request receives a response).

During normal operation, when nodes can communicate reliably, you can have both.

The tradeoff only appears the moment the network breaks.

CP vs AP: two different failure philosophies

Systems that lean toward Consistency + Partition Tolerance (CP) will refuse or delay operations when they can’t guarantee a single, up-to-date view of the data. They’d rather return an error than return something wrong.

Systems that lean toward Availability + Partition Tolerance (AP) will keep serving requests even when nodes disagree, accepting that some responses may be stale. They’d rather serve something than serve nothing.

How a partition actually plays out

A partition occurs when there’s a network split and nodes can’t reliably talk to each other.

Imagine a five-node cluster. A network failure divides it into two groups: three nodes on one side, two on the other.

CP behavior

CP systems preserve a single source of truth by refusing to make progress without enough agreement.

• Quorums/consensus → Writes (and sometimes reads) require enough nodes to agree.

• Leader-based replication → Only one leader can accept writes; the other side of a split becomes read-only or errors out.

• User experience → Some users see failures/timeouts, but you avoid split-brain state.

The three-node side has a majority, so it continues processing requests.

The two-node side does not, so it stops accepting writes. It may become read-only or fully unavailable.

When the network heals, the minority side syncs from the majority. There is no data conflict because only one side was allowed to move forward.

The result is clean, but some users saw errors or timeouts during the split.

AP behavior

An AP system keeps accepting operations on each side of the partition and repairs the mismatch later.

• Eventual consistency → Replicas converge after communication returns.

• Conflict resolution → You need a resolution plan: last-write-wins, merge, keep multiple versions, or domain-specific reconciliation.

• User experience → The system stays up, but different users may see different answers temporarily.

Each side believes it has the latest state. When the network reconnects, the system detects divergent updates; for example, two users modifying the same record on different sides.

Now reconciliation begins.

The database applies a conflict strategy such as last-write-wins. The service stayed available, but some data required conflict resolution.

Neither outcome is a bug. Both are deliberate design decisions baked into the system you chose.

Common mistakes that cause outages

Most confusion around CAP doesn’t come from the theorem itself.

It comes from how we simplify it.

Here are the mistakes that cause the most architectural regret.

• Treating CAP as a permanent label → A system is not “CP” or “AP” as a fixed identity. During normal operation, well-designed systems provide both consistency and availability because the network is healthy.

• Assuming partition tolerance is optional → You cannot avoid partitions by trusting your infrastructure. You can reduce their frequency with better networking, tighter time synchronization, or private backbone links, but you cannot eliminate its possibility entirely.

• Confusing CAP consistency with ACID consistency → These are different guarantees. ACID consistency means a transaction leaves the database in a valid state according to defined rules. CAP consistency means every read returns the most recent write across nodes.

• Forgetting to design recovery logic → Choosing availability during a partition is only half the decision. “Eventually consistent” requires a reconciliation strategy. You must define how conflicts resolve.

A decision checklist you can actually use

Start by asking one question: what hurts your product more, returning incorrect data, or returning no data at all?

• Returning incorrect data hurts more → Choose CP.

  • Common use cases: payments, balances, inventory counts, permissions, leader election

• Returning no data at all hurts more → Choose AP.

  • Common use cases: feeds, analytics views, caches, activity timelines, search indexes

Most real systems don’t pick one globally; they pick per component.

A common pattern is to have a small CP core for writes that must be correct (orders, balances, permissions), and AP layers that serve and cache data quickly across regions.

The core enforces truth; the edge keeps the system fast.

The worst outcome is not choosing consciously. Systems that ignore CAP don’t escape its constraints; they just encounter them as surprises.

Wrapping up

Every distributed system has already answered the CAP question, either deliberately or by default.

The database you picked, the replication strategy you configured, the consistency level you left unchanged: all of it adds up to a position on the spectrum.

Understanding CAP doesn’t give you new options. It tells you which option you’re already on, and whether it’s the right one.

👋 If you liked this post → Like + Restack + Share to help others learn system design.

Subscribe to get high-signal, clear, and visual system design breakdowns straight to your inbox: