Can an LLM replace your consumer data? We built an open-source clean room and ran the test

Short answer: No, an LLM cannot replace your consumer data.

LLMs can guess demographics like gender accurately from public name/birth-record patterns alone, but they can't reliably infer attributes like age for people whose profile differs from the population average. In a locked-down test against real customer data, Faraday matched an LLM on gender (98% vs. 97.6%) but beat it by about 6x on age accuracy (1.9 years mean error vs. 11.5).

The takeaway: some attributes are guessable from public data; others are only knowable through identity-resolved, first-party data.

Every data company is getting the same question right now: if large language models are this capable, do customers still need a consumer data platform? Couldn't a marketer point a frontier model at their customer list and have it fill in the demographics?

There's a lot of magical thinking about what a model can do with a customer list, and the honest answer is: some of it, cheaply, and some of it not at all. We didn't want to settle it with a hot take, so we built an experiment we could stand behind—fair, reproducible, and secure—and put Faraday head-to-head with a frontier LLM, Anthropic's Claude (Opus 4.8), on the same real customer file, scored against ground truth we could actually verify.

It's easy to picture the wrong experiment here. The LLM wasn't eyeballing names and guessing that "Loretta sounds about 60, Liam sounds about 5." Inside the sandbox we gave it real public reference data: a century of Social Security birth records and Census age-by-ZIP tables. It used those to write its own Bayesian statistical model, then ran that over every record.

The short version

Scored per person against held-out first-party truth (n = 150; a naive "always guess the majority gender" baseline is 70.7%):

	Faraday (individual matches)	Faraday (all records)	LLM
Gender accuracy	97.6%	94.0%	98.0%
Gender coverage	82%	100%	100%
Age mean error	1.9 yrs	2.9 yrs	11.5 yrs
Age within ±5 yrs	95.9%	88.7%	28.0%

The two Faraday columns: individual matches are records resolved to a specific person (a high share for this client); all records adds everyone else, where Faraday falls back to household/area-level values. So the LLM's 100% coverage is measured against a Faraday number that also covers 100%—not just the easy-to-match subset.

At a glance

Gender is guessable from public patterns alone — the LLM matched Faraday's accuracy, at full coverage, for pennies.
Age is not — the LLM missed by over a decade on average; Faraday was off by under two years.
The LLM's cost advantage only holds because it wrote a script once instead of reasoning row by row — that distinction, not "AI is cheap," is the actual finding.

Gender is guessable

A first name plus a century of public birth records is enough: the LLM matched Faraday, at full coverage, for pennies. For attributes genuinely encoded in public patterns, modern AI is excellent and cheap—and third-party data adds little to the raw guess itself.

Age is not—it's knowable only from real data

The model missed by more than a decade on average and skewed systematically young; only about a quarter of its estimates landed within five years of the truth, against Faraday's ~96%.

Cost cuts both ways

Claude wrote the inference method once — the Bayesian script that estimates age and gender from public data — and that took $1.23 total for this whole 150-person test. After that, you're not paying for a model anymore; you're just running a normal Python script on your own machine, which costs next to nothing no matter how many records you feed it. The alternative most people picture when they hear "run AI on your customer list" skips that write-once step entirely: it means asking a model to reason through every customer individually, which means paying LLM prices per record, every single time you run it.

	Write-once script (what we did)	Row-by-row reasoning (the approach to avoid)
Cost to build the method	$1.23, once	—
Cost to run on 150 records	Negligible	~$1.50–$4.50
Cost to run on 1M records	Negligible	$10,000–$30,000
Cost to run again next month	Negligible	Full price, again

The rest of this post is how we set it up, how we secured it against cheating, what we found, and what we deliberately did not do.

The question, stated precisely

"Can an LLM replace consumer data" is too vague to test. We narrowed it to a specific, falsifiable question about customer insights:

"Given a client's own first-party customer list, how accurately can an LLM infer per-person demographics like age and gender compared to an identity-resolved data platform — and does it even need third-party data to do it?"

We took a depth-first approach—prove one clean, end-to-end comparison before broadening—so this test pits Faraday against an LLM working alone: the client's first-party data plus publicly available data, with no access to Faraday, scored head-to-head. Two design commitments carried through the whole thing: score at the row level (per person against ground truth, never trusting an aggregate the model could hallucinate), and track token cost as a first-class metric, because "how well" is only half the question and "at what cost" is the other half.

What we actually tested

For the pilot we chose a single client: a real consumer brand whose customer base skews notably older than the U.S. average (this detail matters later). We drew a reproducible random sample of 150 customers and scored two attributes for which the business had trustworthy first-party ground truth:

Age, derived from the client's own first-party date of birth.
Gender, from the client's own first-party gender field.

The test rested on two design choices. First, leakage control: we stripped every field that trivially encodes an answer—date of birth, gender, and even the honorific/title field (Mr./Mrs.)—so neither arm (Faraday or the LLM) had a shortcut to the truth. Second, we held out the ground-truth values of both arms entirely.

And to be clear about fairness, because it's a natural question: both arms received the identical file, full PII included—name, complete mailing address, email, and phone. We handicapped neither. As you'll see, the LLM chose to build its method from first name and ZIP alone (and ignored email, phone, and street); that was its own methodological call, not a limit we imposed—and Faraday was certainly never restricted to those fields. Faraday resolves identity on the whole record.

The clean room: how we made cheating impossible

The single biggest risk to a test like this is the model getting the real answer instead of calculating it — for example, by looking a person up online rather than inferring their details from patterns. That would make the LLM look brilliant while actually measuring nothing: we'd be testing whether it can find an answer, not whether it can figure one out. So instead of just telling the model not to do that, we made it structurally impossible — the model has no path to the open web at all, so there's nothing to peek at in the first place. That requirement drove an isolated environment we're calling the clean room, and the security properties are worth spelling out because they're the same properties we care about when handling any customer's data.

The clean room is a two-container setup on an isolated host:

The workbench (where the model runs) sits on a Docker network configured with no route to the internet at all. It can reach exactly one thing: a local proxy.
The proxy is an allow-list egress gateway. It permits outbound HTTPS to a short list of hosts and denies everything else. For this test that list was the model's API endpoint and the U.S. Census Bureau API—nothing more. No general web, no people-search sites, no social networks.

We verified the boundary before every run: a request to the Census API succeeded; requests to Google and to a people-search site were refused at the proxy; and a deliberate attempt to bypass the proxy failed because the workbench has no DNS or route to the outside world. Every allowed and denied request is written to an audit log, so the environment doesn't just prevent lookups—it proves none happened.

Blocking the open web was the point, not a limitation. This test isolates what a model can infer from public data from what it could retrieve by looking individuals up—two very different questions with very different cost and compliance profiles. The second—an LLM with open-web access—is a distinct experiment we're keen to run next (see below); it isn't what this one measures. And even inside these bounds the model wasn't handed a single table: it decided which public data to use, pulling in the Social Security birth records and querying the Census Bureau's API on its own initiative.

And because the whole environment is just a handful of small config files, we've open-sourced it: the clean room, the model's inference pipeline, the exact operator prompts, and the scoring code are available on GitHub. You can read precisely what it permits—and reproduce the test on your own customer list.

Equally important is what the clean room does not contain. It has no Faraday source code, no credentials, no database access, and no connection to the Faraday Identity Graph or any of our proprietary data or tooling. The only inputs mounted into it were the client's stripped customer file (read-only) and a public dataset (below). The client's PII never left this controlled, isolated environment, and Faraday's internals were never exposed to the model. Fair test, and a responsible one.

The operator constraint mattered too. The person driving the LLM was informed and technical, but under a strict rule: guide the method the way a sharp outside analyst would, and never inject Faraday's "secret sauce" or steer the model toward answers we already knew. In practice the operator's inputs were narrow: naming the allowed public data sources up front, approving the scale-up from 15 records to 150, and asking verification questions—never target numbers, never "the answer is X." (The one substantive piece of client knowledge we fed in later—"this base skews older"—gets its own section below, precisely because it's the exception.) Every intervention was logged.

Arm A — Faraday

Faraday's arm is our normal path: resolve each record to a real person via the Faraday Identity Graph, then attach enriched attributes. Because match confidence varies, we tracked the precision of each enrichment and reported two views: individual-level matches (a specific person was resolved) versus records that fell back to area-level imputation when no confident individual match existed. That distinction is central to reading the results honestly—individual-level is Faraday's true per-person capability; imputation is closer to inference.

Arm B — the LLM

Inside the clean room, the operator worked with Claude (Opus tier) to build an inference pipeline. Notably, the model did not reason about customers one at a time—it wrote a deterministic script that it then ran over all 150 records. The signals it used, all public:

SSA national baby-name data (name, sex, and count per birth year, 1880–present), pre-loaded into the environment as read-only files. This gives a strong prior for name → gender (sum counts by sex) and a weaker prior for name → birth cohort (a name's popularity peaks in certain eras).
U.S. Census ACS 5-year data (table B01001, "Sex by Age," by ZIP-code tabulation area) via the allow-listed API, to refine the age prior using each customer's ZIP.

What it built was not a crude heuristic—the method was its own. We specified which public signals it was allowed to use, but not how to combine them. For gender it used P(sex | first name) from the SSA counts. For age it wrote a small Bayesian estimator: a per-person posterior over birth year combining a likelihood (how characteristic the name is of each birth cohort, from SSA) with a prior (the age distribution of that person's ZIP, from the Census table), then took the posterior mean. It added its own refinements—dividing each name's count by that year's total births to isolate naming fashion from raw birth-volume, and correctly anchoring ages across the four-year gap between the 2022 Census vintage and a 2026 "as-of" date—and it reported honest confidence, scoring itself low on names like "William" that are common in every decade and therefore carry little age signal. In short: a genuinely principled statistical pipeline, designed by the model.

It deliberately ignored email, phone, street, and purchase history. We calibrated on the first 15 records so the operator could review the method before scaling, then ran the full 150. And we instrumented everything: prompts, interventions, wall-clock, and token usage, so we could report cost precisely.

Two things let us prove the isolation held. First, midway through, the operator asked the model point-blank whether any of its network requests had been blocked and whether it was building only from the allowed sources. The model didn't just assert—it audited itself: it checked its Census coverage (142 of 143 ZIP codes resolved), traced the single gap to a PO-box-only ZIP that has no Census geography (a legitimate empty result, not a block), and confirmed it had used only the local SSA files and the Census API, with the email and phone columns deliberately untouched. Second, the proxy's audit log independently corroborated it—every request that left the environment went to exactly the two allowed destinations. The clean room didn't just constrain the model; it let us verify, after the fact, precisely what it did and didn't touch.

Cost and privacy, in detail

We put the headline cost comparison up top; two things are worth adding here.

First, where that $1.23 went: it was dominated by output and cached input—roughly 17.8K output tokens, about 6.7K of fresh input, and ~750K input tokens served from prompt cache (billed at a tenth of the input rate)—over about four minutes of model time, tracked from the run's own token accounting.

Second, and bigger than cost: the row-by-row approach is a privacy exposure. Feeding a customer list through a hosted model row by row means sending your entire customer base—names, addresses, and all—to a third party, on every run. And the moment "running AI on a list" involves per-row web or people-search lookups—exactly how many people picture it—you're broadcasting each of your customers out to the open internet. That's a data-governance problem, not a line item. The deterministic approach sidesteps it: the per-record work is a local script that reads only public reference data, so your customers' information is never shipped to a model, or the web, one row at a time.

What it means: guessable vs. knowable

The finding is not "AI wins" or "AI loses." It's a clean line.

Gender is guessable. A first name carries a strong public signal, and a century of birth records makes it a safe inference. The LLM matched Faraday—at full coverage, for pennies. We'll say it plainly: for an attribute genuinely encoded in public patterns, modern AI is excellent and cheap, and third-party data adds little to the raw guess itself.

Age is not guessable—it's knowable only from real data. The LLM was off by more than a decade on average and biased systematically young; only about a quarter of its estimates landed within five years. This isn't a tuning problem, and it isn't for lack of trying: remember, the model built a proper Bayesian estimator, not a lazy guess. A Bayesian posterior mean is, by construction, a shrinkage estimate—it pulls each person toward the population you hand it as a prior. For a customer base that sits well off the population average, like this older-skewing one, that shrinkage shows up as systematic bias: ages dragged roughly a decade too young. No amount of cleverness fixes it, because the information needed to place a specific person simply isn't in a first name and a ZIP. The best any inference engine can do is regress toward an average—the average age of that name, the average age of that neighborhood—and for a base that skews away from the crowd, that average is wrong person by person. Inference pulls everyone toward the mean. Identity resolution pins each individual to who they actually are.

Most of what actually drives targeting, modeling, and spend lives on the knowable side: age, income, wealth, life stage, recent moves, in-market behaviors. Those aren't latent in a name. A cheap guess that's ten years wrong isn't context—it's noise wearing a confidence interval.

And age is not a special case. The same shrinkage governs every attribute worth having—income, wealth, life stage, propensity: the further your customers sit from the average, the harder inference pulls them back toward it. Age is simply where the failure was visible, because we had birthdays to check it against.

Which exposes a deeper catch this experiment happens to illustrate. The only reason we can tell you the LLM nailed gender and missed age is that we held real answers to grade it against. A company running blind inference has none of that—no ground truth, no error bar, no way to notice when the model drifts or when a new segment quietly breaks its assumptions. Knowing an attribute is guessable and knowing a specific guess is right are different problems, and the second is unsolvable without real data and the validation and monitoring built around it. That isn't a footnote to what a data platform provides; it's a large part of the point.

We tried to close the gap—with what a real client knows

A fair objection: our operator was flying half-blind, and a real marketer isn't—they know things about their own customers. So we gave the model the single most relevant thing a client in this position actually knows—our customer base skews older than average—and let it fold that into its prior. Qualitatively only ("older than the general population"), with no number borrowed from the answer key.

It helped, and it helped honestly — the average age error dropped from about eleven years to about seven, and the systematic "too young" bias largely disappeared. To be sure we hadn't just tuned to our own test, we froze that method and re-ran it on a completely fresh, non-overlapping sample — it held there too, about seven years off again, so it wasn't a fluke of the first sample.

But it still wasn't close. Faraday's identity-resolved age was off by about two years; the domain-informed model, by about seven — landing within five years of the truth less than half the time, against Faraday's 96%. Domain knowledge moved where the crowd sits; it corrected the population. It couldn't place the person. Which is the line one more time, now with the strongest help we could fairly give the model: the aggregate is guessable — but the individual is knowable only from resolved data.

What we did not do

We're deliberately clear about the limits of this first test:

One client, one profile. A single, older-skewing customer base. The age result almost certainly depends on how far the base sits from the population mean; a younger or more average base could look different.
Small sample. 150 records. Enough to see the effect clearly, not enough to quote decimal points as gospel.
Two attributes, binary gender. Age and gender only. Gender was scored as binary because the ground truth was binary.
Income deferred. No per-person income truth; skipped rather than faked.
Only the LLM-alone case. We compared Faraday against an LLM with no access to Faraday. We did not run the hybrid case—an LLM allowed to buy Faraday's data—nor an "LLM + open web" variant, which would answer a different and important question: can retrieval beat inference, and at what cost and compliance risk?
One operator, one model. A single informed operator and a single model tier.

What we're testing next

This is the first in a series pitting Faraday against LLMs, and it deliberately tested one narrow thing: demographic enrichment—age and gender. The comparison most teams actually care about is the next one:

Can an LLM replace the models, not just the data? Guessing a demographic is one thing; predicting who will convert, churn, or respond—and what to recommend to them—is another. Whether an LLM can match Faraday's propensity and recommender models—not just its data—is the question we're most eager to answer, and it's the subject of our next test in this series.
Re-run as new models ship. We'll refresh these results with each major new model release—the clean room makes it a fast, repeatable re-run, so the comparison stays current as the frontier moves.
Replicate across customer profiles, especially bases whose age distribution sits near or below the national average, to map where inference holds up and where it breaks.
Stratify by Faraday match status (individually-matched vs. not), so the head-to-head stays explicit even on datasets with lower match rates than this one's.
Add income—with a licensed, authoritative truth source—plus wealth, life stage, and life-event attributes, which we expect to fall firmly on the knowable side.
Run the hybrid case (LLM + Faraday data) to measure how much a model improves when it can buy real enrichment.
Run an "LLM + open web" arm in the same clean room (allow-listing more sources) to quantify the accuracy, cost, and compliance trade-offs of retrieval vs. inference.
Scale N for tighter confidence intervals, and test a cheaper model tier to see how far the cost floor drops.

Why it matters

We're not defensive about AI—we build with it every day, and this test was itself an AI-assisted build. The point of running it in a locked-down, reproducible way is that it replaces opinion with evidence: it tells you exactly which parts of "just point AI at your customer list" are real and which are wishful.

The right mental model isn't "AI replaces your data." It's "AI raises the floor on what's guessable, and makes what's only knowable—the identity-resolved, longitudinal context behind every real person—more valuable than ever." That knowable layer is what the Faraday Identity Graph is built to deliver, across 240M U.S. adults.

We'll keep running these tests and sharing what we find. If there's a comparison you'd like to see, tell us.