What OpenAI’s Data Agent Teaches Us About Building Reliable AI Agents
How OpenAI built its data agent to work across 70,000 datasets.
Prove You Can Build Production AI
Presented by MongoDB
Go beyond AI tutorials. MongoDB’s hands-on AI Skill Badges teach practical AI engineering skills like vector search, RAG, and agent memory, through project-based learning designed for production AI applications. Earn credentials as you progress and share them on LinkedIn.
How OpenAI Built Context, Memory, and Evals Into Its Data Agent
What makes a data agent trustworthy?
Not just whether it can generate SQL, but whether it can produce an answer that is accurate, secure, and easy to verify.
A query can run without a single error and still return the wrong result. It might use the wrong table, apply an outdated metric definition, or join two datasets in a way that silently doubles the count.
The SQL is valid.
The answer is not.
That was the challenge OpenAI faced while building an internal data agent for more than 3,500 users working across 70,000 datasets and over 600 petabytes of data.
At that scale, generating the query is only one part of the problem. The agent also needs the right context, a way to catch mistakes, permission boundaries it cannot bypass, and enough transparency for users to inspect its work.
So the interesting part is not simply how OpenAI taught a model to query its warehouse.
It’s how every part of the architecture was designed to make the result more trustworthy.
Editor’s note: Based on publicly shared engineering details from OpenAI. Thanks to their engineering team for making this public. Full sources are linked in the References section below.
The query is not the real bottleneck
Every large company accumulates knowledge that never makes it into the schema.
Someone knows which table should be used for revenue reporting.
Another person knows that a particular field became unreliable after a migration.
A Slack thread explains why two dashboards use slightly different definitions of an active user.
The information exists, but it is scattered across people, code, documents, and past analyses.
Without that context, even a strong analyst (or model) can confidently choose the wrong data.
Before writing a useful query, someone still needs to answer several questions:
Which datasets represent the business concept being asked about?
What do the fields and metrics actually mean?
Which tables can be joined safely?
Is the underlying data current?
What exclusions or caveats does the organization normally apply?
And even after choosing the right tables, the query itself can fail in subtle ways.
A many-to-many join can inflate a count. A null-handling mistake can remove valid records. A filter applied too early can change the result.
The goal at OpenAI’s scale is to flip where analysts spend their time: less debugging SQL, more defining metrics, testing assumptions, and making decisions. The agent absorbs the lower-level work.
What a basic data agent gets wrong
A simple analytics assistant follows a straightforward path:
That works when the right table is obvious, the metric is clear, and the first query succeeds.
But real analysis is rarely that tidy.
Take a question like “Why did usage drop after launch?”
Answering it may require the agent to:
Identify the relevant usage events
Confirm the actual launch date
Check whether telemetry changed
Compare different customer segments
Notice incomplete data
Revise the query when an assumption fails
The answer is not contained in one SQL statement.
It emerges through an investigation.
So OpenAI built the agent to own the full analysis, not just one step of it. It discovers datasets, inspects schemas, runs and refines queries, interprets results, and publishes notebooks or reports. Teams across Engineering, Data Science, Go-To-Market, Finance, and Research now use it for questions about launches, business health, and metric accuracy.
But none of it works without accurate grounding. And that’s where the architecture gets interesting.
Six layers of context
A strong model with weak context still gives weak answers. Without reliable grounding, it will misuse an internal term, pick the wrong dataset, or confidently compute the wrong metric.
This is the real question the agent has to answer; not “what does this table contain?” but “which of these five similar tables is the right one for this analysis?”
OpenAI grounds the agent in six context layers. Each one fills a gap the others can’t:
These layers are not six versions of the same information.
Each solves a different failure mode.
The first layers help the agent find and interpret the data.
The later layers help it deal with uncertainty, change, and missing information.
Table usage and human annotations
The first layer gives the agent a map of the warehouse:
Schema metadata → column names and data types
Table lineage → where data comes from and where it flows
Historical queries → the common joins, filters, and table combinations people actually use
But a map only shows where things are.
It does not explain what they mean.
A table called daily_active_users might exclude part of the product, lean on an outdated identity field, or encode an old metric definition. Nothing in the schema tells you that.
So the second layer, human annotations, captures meaning.
Domain experts document each dataset’s purpose, unit of analysis, exclusions, and known caveats. They can capture details that are invisible in a schema, such as:
A field that became unreliable after a migration
A metric that requires a particular filter
A table that should no longer be used for new analyses
A dataset that only covers a specific type of traffic
This turns local knowledge into shared context.
Instead of remaining trapped in someone’s head, it becomes information the agent can retrieve.
But even expert documentation has limits.
It describes what people believe a table does.
The code shows what it actually does.
Codex enrichment
Annotations depend on someone bothering to write them down. The code, by contrast, is already the source of truth; it just isn’t readable as context yet.
That’s the third layer. OpenAI uses Codex to inspect the code behind each dataset.
By crawling the code, Codex identifies a table’s purpose, primary keys, granularity, update behavior, downstream uses, alternative sources, and field transformations. That gives the agent:
Data scope → which traffic or users the table includes
Derivation logic → how raw events become reported metrics
Uniqueness rules → which fields define a row, and where duplicates can appear
Freshness → how often the pipeline updates and what delays to expect
Alternatives → which dataset to use when the default is incomplete
This matters because two tables can look nearly identical while behaving very differently.
The schema alone does not explain those differences.
The production logic does.
But code still only explains the system.
It does not explain what was happening around the system.
Institutional knowledge
Code explains how a number is produced. It can’t explain why the number moved.
The fourth layer pulls in the organizational context that lives outside the warehouse entirely.
Launch plans documented in Google Docs. Incident timelines living in Slack. Metric definitions stored in Notion.
OpenAI ingests this content along with its metadata and permissions, then retrieves it at runtime while enforcing access controls.
This is what connects numbers to real events.
A drop in usage may initially look like a shift in customer behaviour. But an incident report may reveal that telemetry broke after a launch. Without that context, the agent can calculate the drop accurately and still explain it incorrectly.
The warehouse shows what changed.
Institutional knowledge may explain why.
At this point, the agent has a much stronger foundation.
But no amount of preloaded context can anticipate every correction, exception, or future change.
That creates the next problem.
Memory
Some of the most useful analytical rules are discovered only after the agent makes a mistake.
For example, the agent may learn that filtering a particular experiment requires matching a specific value in an experiment gate rather than using a fuzzy string match.
Without memory, the agent could repeat the same mistake every time.
So OpenAI allows it to retain lessons from corrections and feedback.
Future analyses can begin from a stronger baseline instead of rediscovering the same nuance.
But memory creates a new tradeoff.
Persistent memory is still persistent state.
If the system stores every correction indefinitely, an old workaround can survive long after the underlying data model has changed. An uncertain deduction can gradually become treated as a fact.
The same feature that preserves useful knowledge can preserve mistakes just as efficiently.
Therefore, memory needs governance.
A production memory system needs controls for:
Scope → Does the learning apply to one user, one team, or everyone?
Ownership → Who is responsible for keeping it accurate?
Approval → Can an uncertain deduction become permanent?
Editing → How is the memory updated when the system changes?
Auditability → Where did the memory come from?
Conflict handling → What happens when personal and global guidance disagree?
OpenAI scopes memories globally and personally, allows users to edit and confirm them, and surfaces them rather than hiding them inside the system.
The goal is not to remember everything.
It is to retain the right context and keep it accurate.
Still, even well-governed memory eventually becomes incomplete or stale.
When that happens, the agent needs somewhere else to look.
Runtime context
The sixth layer gives the agent access to runtime context.
When retrieved information is insufficient, the agent can inspect live schemas and data. It can also consult systems such as the metadata service, Airflow, and Spark for operational details.
This creates a progressive fallback:
Retrieved context → Precomputed information for fast, common queries
Live warehouse inspection → Current schemas and data when confidence is low
Platform inspection → Orchestration and processing systems when the warehouse isn’t enough
The agent takes the cheapest reliable path first, then digs deeper only when uncertainty rises.
This avoids two bad extremes.
Loading every schema, document, code definition, and query history for every request would be too slow and expensive.
But relying only on precomputed context would eventually produce stale answers.
The progressive fallback gives common requests a fast path without preventing unusual questions from being investigated properly.
That raises another architectural question:
How do you make so much context available without blocking the request path?
Expensive work happens before the request
OpenAI separates preparation from execution.
A daily pipeline combines table usage, human annotations, and Codex enrichment into a normalized format. It then produces embeddings that can be retrieved semantically.
At request time, retrieval-augmented generation brings in only the context relevant to the question.
The agent consults live systems only when it needs something fresher or more specific.
This creates two paths:
Offline path → Enrich, normalize, and index stable context ahead of time
Online path → Retrieve a small context set and inspect live systems only when necessary
This keeps the interactive path focused on work that depends on the user’s actual question.
This pattern applies well beyond data agents:
Move stable, repeatable work out of the request path.
Keep the live path for work that depends on the current request or the latest system state.
That is a common system design tradeoff: more preparation upfront usually means lower latency and more predictable costs at runtime.
But a fast system can still fail silently.
So the next challenge is making sure the agent can detect when its own actions did not work.
Build a loop, not a one-shot generator
A basic query generator assumes its first answer is correct. OpenAI’s agent assumes it might not be.
If a join returns zero rows, the agent doesn’t conclude that no data exists. It checks whether the join keys are wrong, a filter stripped the records, or the tables cover different time periods, then adjusts the query and tries again.
So the agent does not stop at the first result.
It works in a loop:
Act → Run the query
Observe → Inspect the result
Validate → Check whether it makes sense
Adjust → Change the approach when necessary
Because the agent keeps context across each step, users can redirect the analysis, change the scope, or ask follow-up questions without starting over.
The same pattern appears in many engineering systems:
Deployments check rollout health before increasing traffic.
Payment systems verify transaction state before retrying a charge.
Data pipelines inspect incomplete output before rerunning a job.
Reconciliation systems compare expected and observed state before applying a fix.
The useful pattern is not merely “try again.”
It is “observe what happened, then decide what the next action should be.”
A closed loop turns failure into information the system can actually use.
Repeated patterns become workflows
At the beginning, flexibility is valuable.
OpenAI cannot predict every question employees will ask, so the agent needs room to explore different datasets and analytical paths.
Over time, however, repeated patterns emerge.
Teams run the same reports. They validate the same tables. They apply the same filters. They follow the same sequence when investigating particular metrics.
OpenAI turned these repeated patterns into reusable workflows that preserve the right context, steps, and best practices so the same analysis produces consistent, reproducible results.
This same approach works well when building almost any product:
Start with a flexible system.
Observe how people actually use it.
Identify the repeated behavior.
Standardize the common paths.
Preserve flexibility for the unusual use cases.
Building every workflow in advance would require guessing how people will use the system.
Leaving everything open-ended forever would force the agent to repeatedly rediscover the same process.
A strong design needs both.
But even a standardized system can get worse over time.
Models change. Prompts change. Retrieval changes. New context may improve one type of analysis while breaking another.
Without evaluation, those regressions can remain invisible.
Evaluating correctness
An agent that changes over time can improve; but it can also regress.
And without a systematic feedback loop, that drift stays invisible until it produces a wrong answer in production.
So OpenAI evaluates the agent against curated question-and-answer pairs.
Each question covers an important analytical pattern and is paired with a manually authored “golden” SQL query that produces the expected result.
The agent generates its own query, runs it, and compares the output against that benchmark.
But OpenAI doesn’t require the SQL to match exactly.
Two queries can use different joins, aliases, or subqueries and still return exactly the same correct answer. So OpenAI evaluates both the query and the resulting data, then uses an Evals grader to score the response and explain any differences.
The useful lesson here is to test the thing users actually rely on. In this case, that is not the shape of the SQL. It is whether the final answer is correct and can stand up to scrutiny.
The same distinction applies well beyond data agents:
Code generation → test behavior and correctness, not exact source text
Document extraction → test the extracted fields, not the model’s phrasing
Search systems → test whether useful evidence appears, not whether rankings match one fixed list
Support agents → test policy compliance and issue resolution, not response similarity alone
Meaning is the target. Implementation is only one signal.
Make correctness visible
The agent summarizes its assumptions and execution steps alongside the answer. It also links users to the underlying query results.
That lets users inspect the raw data and verify the analysis.
This matters because observability is not only an operator concern in an agentic system.
It is part of the product interface.
A black-box answer may sound confident, but users cannot tell where it came from.
An inspectable answer lets them see:
Which assumptions were made
Which data was used
What actions the agent performed
Where the analysis may have gone wrong
When the agent is correct, this builds confidence.
When it is wrong, it gives users somewhere to begin the correction.
Security inherits existing boundaries
There’s one last way an agent like this can go wrong, and it has nothing to do with correctness.
A conversational interface sitting on top of all your data is a tempting shortcut around permissions.
The safest approach is not to build a second authorization model for the agent.
The agent acts as an interface over existing data platforms and passes through the requesting user’s permissions.
You can only query tables you were already allowed to access. If a dataset is restricted, the agent explains the limitation or finds an approved alternative. That keeps the conversational layer from quietly becoming a privilege-escalation path.
It also avoids maintaining two systems that may gradually disagree about who can access what.
This is an important rule for any system: new interfaces should inherit existing access controls wherever possible.
Every separate permission model creates another place for security rules to diverge.
The lesson beyond OpenAI
OpenAI’s agent is built around its own warehouse, tools, and organizational knowledge.
But the patterns underneath it apply to almost any data-heavy and agentic systems:
Offline preparation → Move expensive work like enrichment, code analysis, and embedding generation out of the request path
Progressive fallback → Use fast retrieved context first, then inspect live systems when that context is missing or stale
Managed memory → Give persistent knowledge clear scope, ownership, editing, and audit controls
Closed-loop reasoning → Act, observe, validate, and adjust instead of assuming the first attempt worked
Outcome-based evaluation → Test whether the result is correct, not whether the implementation matches one expected form
Inherited authorization → Reuse existing permission boundaries instead of creating a separate access model
These decisions all support the same goal: trust.
Context improves accuracy. Validation catches mistakes. Evaluations expose regressions. Existing permissions prevent the agent from exceeding the user’s access.
Trust in an agentic system isn’t a feature you add at the end. It’s what every architectural decision is quietly building toward.
References:
Related LUC reading
👋 If you found this useful → Like + Restack to help others learn AI engineering.