A holdout evaluation is the industry-standard method data scientists use to validate predictive models: train on historical data, set aside a portion the model has never seen, then check whether the highest-scoring leads actually converted at higher rates. If they did, the model works.

If you're evaluating predictive data vendors, this is the test worth asking for. With tight budgets and increasing pressure to show ROI, you can't afford to bet on unproven tools or spend months figuring out if a vendor will live up to the promise. You need a way to separate legitimate predictive tools from the noise.

That's what our holdout evaluation does. It's not a methodology we invented — it's an industry standard used to validate [custom predictive data](https://faraday.ai/blog/predictive-lead-scoring-b2c), including [propensity-to-buy scores](https://faraday.ai/blog/propensity-to-buy-machine-learning) used across lead buying, direct mail, and [personalization](/blog/personalization-tactics-that-drive-conversions).

Below we've outlined how it works and what to look for when evaluating whether it's delivering real value.

## What is a holdout evaluation?

A holdout evaluation is a method for validating predictive models by comparing their predictions to real-world outcomes. We **set aside a portion of recent conversion data**—data the model has never seen before—and use it as an independent benchmark, against which predictions can be compared.

This "holdout set" acts as a real-world proving ground. It is untouched until after the predictive model has been trained.

Meanwhile, the rest of your historical data is combined with rich consumer data from the [Faraday Identity Graph (FIG)](/blog/whats-in-faraday-identity-graph) ([which includes over 1,400 data points on 240M+ U.S. adults and their households](https://faraday.ai/features/faraday-identity-graph)). We use this combined data to train the new predictive model.

Training a predictive model simply means finding patterns in historical conversions. In other words, what can we learn from your past customers to predict future outcomes? Once trained, the model generates predictions—scoring leads based on their likelihood to convert and only after these predictions are made do we compare them to the **real conversion rates** in the holdout set.

If the highest-scoring leads actually convert at higher rates than lower-scoring ones, the model is validated as accurate and reliable.

Holdout evaluations ensure predictive models aren't just theoretical—they work in the real world, using real data, to deliver real results.

## How holdout validation works at Faraday

We've streamlined this validation process to be transparent and straightforward:

1. ![customer conversion data](/images/blog/holdout_testing_diag/holdout_testing_diag-01.png)

   #### Step 1: Set aside a holdout group

   We start with your recent [lead conversion data](https://faraday.ai/blog/first-party-vs-third-party-data-marketing). Before any processing occurs, we reserve a portion of your most recent leads as a "holdout set"—data that the model will never see during training. This ensures we have an independent benchmark for evaluating predictions.

2. ![merged with FIG](/images/blog/holdout_testing_diag/holdout_testing_diag-02.png)

   #### Step 2: Enrich with consumer data

   With the holdout set reserved, we then combine remaining data with Faraday's [built-in consumer data](https://faraday.ai/features/faraday-identity-graph) sourced from the Faraday Identity Graph (FIG). This expanded dataset provides a deeper view into key predictive signals, like property and demographic information, life events, buying habits, and more.

3. ![model is trained and predictions begin](/images/blog/holdout_testing_diag/holdout_testing_diag-03.png)

   #### Step 3: Train the model and score leads

   Using this enriched dataset (excluding the holdout set), we train a predictive model to identify patterns in conversion behavior—learning from consumer signals, demographic data, and behavioral patterns.

   Once the model is trained, it scores each lead based on its likelihood to convert, generating a ranked list of opportunities—from high-value leads to those less likely to convert.

4. ![model is validated](/images/blog/holdout_testing_diag/holdout_testing_diag-04.png)

   #### Step 4: Compare predictions to real outcomes

   Finally, we put the model to the test: we compare its predictions against the real outcomes in that untouched holdout set. If the highest-scoring leads don't convert at significantly higher rates, we go back to the drawing board.

## How to know if a holdout validation was successful

A holdout validation succeeds when predicted scores align with actual conversion rates across segments. Look for these three signs:

1. **Clear correlation between predicted probability and actual conversion rates**  
   The model's predicted likelihood to convert should directly align with the actual conversion rates. If the predictions are accurate, high-probability leads should convert at a noticeably higher rate than lower-probability ones.

2. **High-scoring leads convert at significantly higher rates**  
   The top-scoring leads should consistently perform better than those ranked lower. This demonstrates that the model is identifying truly valuable leads.

3. **Consistent performance across different customer segments**  
   The model should show reliable results across various segments—whether you're targeting different demographics, regions, or other customer profiles. If the model works well across a broad set of data, it proves its reliability and scalability.

When a model meets these criteria, it's ready to generate custom predictive data points—whether that means optimizing [lead buying](https://faraday.ai/docs/how-to/use-cases/leads), refining your [direct mail strategy](https://faraday.ai/templates/lead-suppression), or enhancing your email [personalization](https://faraday.ai/templates/personalization) approach.

## Questions you should ask us

We want you to push us on what matters most for your business. Here are key questions to evaluate if we're the right fit:

1. **How does this model's performance compare to the lift you are seeing with similar businesses in our industry?**

   While we can't share client specifics, we can provide insight into broader trends and expected performance benchmarks.

   → **Why this matters:** Understanding industry benchmarks helps you gauge whether custom predictive data is truly a competitive advantage for your business.

2. **How much historical data do we need for reliable predictions?**

   The answer varies by business model and conversion volume. We'll help determine whether you have enough data—or how to supplement it if needed.

   → **Why this matters:** Predictive models work best when trained on strong data. Knowing the threshold for reliability ensures you're making data-driven decisions with confidence.

3. **What happens if the model doesn't perform as expected?**

   We have a structured iteration process to refine models when necessary, ensuring they reach meaningful predictive power.

   → **Why this matters:** No model is perfect on the first run. Understanding the process for continuous improvement ensures long-term success.

4. **How will this integrate with our existing marketing stack?**

   We'll provide technical specs, discuss implementation timelines, and outline the resources needed to get up and running.

   → **Why this matters:** Seamless integration means faster time-to-value and less friction in your workflow.

## Why holdout evaluations work

Holdout evaluations work because they test a model against data it has never seen — making it impossible to game the results. Any predictive data vendor can show you strong in-sample performance. Holdout testing is the check that confirms those patterns hold in the real world.

At the end of the day, three things matter to us:

1. **Confidence before commitment:**  
   You see proof before you commit

2. **Higher marketing efficiency:**
   Your marketing dollars go to the leads most likely to convert

3. **Stronger sales performance:**
   Your sales team focuses on opportunities worth their time

If the validation doesn't show meaningful lift, we'll tell you. If it does, we'll quantify exactly what that means for your business.

Interested in generating value you can trust? [Talk to a Context Consultant](/contact).

## Frequently asked questions

**What's the difference between a holdout evaluation and an A/B test?**

A holdout evaluation validates that a predictive model works—it confirms that high-scoring leads actually convert at higher rates before you commit to anything. An A/B test measures the impact of a decision you've already made (like sending a campaign to scored vs. unscored leads). Faraday runs the holdout evaluation first so you have confidence in the model itself, not just the downstream campaign.

**Do I need a large dataset for holdout testing to be meaningful?**

Not necessarily. The threshold varies by industry and conversion volume. In many cases, a few months of lead and conversion history is enough to build a reliable model. We'll tell you upfront if your data isn't sufficient—and if it's borderline, we can discuss ways to supplement it with data from the [Faraday Identity Graph](https://faraday.ai/features/faraday-identity-graph).

**What happens to my data during the validation process?**

Your first-party data is used solely to train and validate the predictive model—it is never shared with other clients or used to enrich Faraday's broader data assets. The holdout set is kept completely separate from training and is only used as an independent benchmark after predictions have been made.

Faraday uses holdout testing to validate custom predictive data before you commit, ensuring it delivers real, measurable results.

Predictive data generates tangible value, and we can prove it

[Momentum Solar's call center was working hard](https://faraday.ai/blog/solar-call-center-lead-conversion-boost), but still reaching the wrong people. But after replacing their rules-based scoring with a propensity model, they hit the same appointment goal with 33% fewer outbound calls. The leads didn't change. The model just finally knew which ones were worth calling.

Today, almost every marketer has felt the same frustration: you have thousands of customers or leads, and you know some of them are about to buy, some are about to [churn](/blog/churn-prevention-machine-learning), and some are worth ten times more than others — but you can't tell which is which.

Propensity models solve this. They predict how likely a specific person (or household) is to take a specific action — whether that is buy, churn, reactivate, upgrade, apply for a loan — so you can treat each person according to their actual calculated likelihood to perform a behavior, rather than a guess.

This guide covers [what they are](/blog/propensity-models-predict-customer-behavior#what-is-a-propensity-model), [how they work](/blog/propensity-models-predict-customer-behavior#how-propensity-models-work), and [how consumer brands use them](/blog/propensity-models-predict-customer-behavior#use-cases-across-customer-lifecycle) across the full customer lifecycle.

## What is a propensity model?

A propensity model is a machine learning algorithm trained on your historical customer data to predict the probability that a given individual will perform a defined action — any of the behaviors described above, and more.

At Faraday, propensity models are trained on both your first-party outcomes and the [Faraday Identity Graph (FIG)](/blog/whats-in-faraday-identity-graph) — [1,400+ verified consumer data points](https://faraday.ai/catalog) across 240M U.S. adults — which gives the model far more signal to work with than first-party data alone.

The model learns by studying examples — customers who did the thing you care about, and customers who didn't — and identifying the patterns that distinguish the two groups. It then applies those patterns to everyone else in your database and assigns each person a score representing their probability of performing the action.

Those scores are the output. What you do with them is up to you — prioritize your call center queue, build a suppression list, trigger a retention campaign, identify lookalike acquisition audiences. The model is the engine; your stack is the vehicle.

## How consumer brands use propensity models

| Use case                                                                         | What it predicts                                         | Best for                                                            | How teams use it                                                                                                                                                                                                  |
| :------------------------------------------------------------------------------- | :------------------------------------------------------- | :------------------------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Lead scoring**](https://faraday.ai/blog/predictive-lead-scoring-b2c)           | Likelihood to convert                                    | Sales and marketing teams                                           | Rank leads by conversion probability; focus reps on high-probability prospects and move low-probability leads into automated nurture paths or suppress them entirely                                              |
| [**Churn prediction**](https://faraday.ai/blog/churn-prevention-machine-learning) | Likelihood to stop buying                                | Retention teams                                                     | Trigger a [personalized](https://faraday.ai/blog/personalization-tactics-that-drive-conversions) retention offer, check-in, or discount at the moment a customer starts to disengage — before they've already left |
| **Reactivation**                                                                 | Which lapsed customers will respond to a win-back        | CRM and lifecycle teams                                             | Target win-back campaigns precisely instead of spending equally on everyone who hasn't bought in 90 days                                                                                                          |
| **LTV prediction**                                                               | Long-term revenue value per customer                     | Acquisition and retention teams                                     | Prioritize spend toward high-LTV customers; used alongside churn scores, a high-churn/high-LTV customer is worth fighting for — a high-churn/low-LTV one might not be                                             |
| **Next best offer**                                                              | Most relevant offer per customer at each lifecycle stage | Financial services, subscription brands, and deep-catalog retailers | Stack one model per product and surface the most relevant offer at each point in the customer lifecycle                                                                                                           |

## How propensity models work

Faraday's prediction system evaluates multiple algorithms — individual decision trees, logistic regressions, neural networks, and random decision forests — and selects the best fit for each use case at run time. In most cases, the winner is random decision forests.

A random decision forest is made up of dozens of individual decision trees. Each tree is a classifier — a flow chart that maps a series of data splits to a predicted outcome. The forest combines the outputs of all those trees to produce a single probability score that's more accurate and more stable than any individual tree alone.

Three properties make random decision forests particularly well-suited for consumer data:

- **Handling missing data:** Unlike neural networks or logistic regressions, which require complete values to function, random decision forests navigate missing data seamlessly. If a value is absent at one node, the algorithm moves to the next decision path — keeping the prediction accurate even when individual records are incomplete.
- **Identifying collinearity:** Consumer data is inherently noisy, and many attributes are correlated — household income and dwelling type, for example, or age and life-stage indicators. Random decision forests find areas of greatest information gain, identifying which of those interdependent data points gives the model the most leverage over the prediction rather than double-counting correlated signals.
- **Explainability:** Because the model maps discrete decision splits, Faraday can surface [exactly which data points contributed most to each score](https://faraday.ai/blog/faraday-score-explainability) — household signals, life events, spending trajectory, and more. You're not trusting a black box; you're seeing the reasoning.

## The data foundation matters

A propensity model is only as good as the data it trains on. [First-party data](https://faraday.ai/blog/why-first-party-data-and-rfm-is-not-enough) — purchase history, engagement data, CRM records — tells the model what your customers have done with your brand. It doesn't tell the model who those customers are.

That's the gap the [Faraday Identity Graph](/blog/whats-in-faraday-identity-graph) fills. FIG covers 240M U.S. adults across [1,400+ verified consumer data points](https://faraday.ai/catalog) — demographics, financial signals, property data, life events, lifestyle indicators, and more. When Faraday builds a propensity model, it trains on your first-party outcomes enriched with FIG data, which means the model has far more signal to work with than CRM-native scoring tools that are limited to whatever happens to be in your system.

FIG also provides [historical data](https://faraday.ai/blog/fig-v2-historical-data) — full historical timelines for each data point, not just current-state snapshots. This matters for training because customers don't look the same after they convert as they did before. A model trained on current-state data is learning the post-purchase signature, not the pre-purchase one. Historical data lets Faraday train on what a customer looked like just before they took the action you care about — which makes the model dramatically better at finding genuine future prospects.

## Momentum Solar: 33% fewer calls to hit the same goal

[Momentum Solar](https://momentumsolar.com/) — one of the fastest-growing solar installers in the country — had a lead prioritization problem. Their call center was operating on a rules-based scoring system built around a handful of data points: lead source, demographics, homeownership status, location. The rules were subjectively chosen, static, and couldn't account for the full range of signals that actually predict whether a homeowner will book a solar assessment.

The result: reps were spending time on leads that would never convert, while high-probability leads got the same treatment as everyone else.

Momentum partnered with Faraday to replace their rules-based system with a propensity model trained on their historical customer base and enriched with FIG data. Leads were scored dynamically — updated as new customer examples were added and lead behavior was observed — and bucketed into three tiers: green (high probability), yellow (mid), and red (low).

The results from a clean A/B test against their previous approach:

- **\+191% lift in conversion rate** for green-tier leads vs. baseline
- **\+51% lift** for yellow-tier leads vs. baseline
- **33% fewer outbound calls** needed to hit the same appointment goal

The red group converted below baseline — confirming exactly where the call center should and shouldn't spend its time. [Read the full Momentum Solar case study.](https://faraday.ai/blog/solar-call-center-lead-conversion-boost)

## Propensity models across verticals

The same framework applies across industries and use cases. A few examples from Faraday customers:

- **E-commerce:** A leading subscription box brand replaced manual product curation rules with a two-sided propensity model — scoring both customer affinity and product performance — and achieved a 5% lift in revenue per customer, a 3% reduction in churn, and 5x monthly ROI. [See the full story](/blog/monthly-subscription-box-uses-two-sided-recommendations).
- **Financial services:** Advia Credit Union used propensity scoring to identify which members were most likely to apply for an auto loan, built with FIG enrichment and bias mitigation for fair lending compliance. Application rates lifted from 1.19% to 5.18% in 90 days, generating $2.7M in new loans. [See the full story](/blog/advia-credit-union-auto-loans).
- **Home services:** American Standard used propensity scoring to transform their contact center from a top-of-funnel filter into a full-funnel routing optimizer — moving high-fit leads to the front of the queue, mid-tier prospects into the right workflows, and lower-fit records into nurture paths. Contact rate lifted from 5.56% to 20%, and high-score leads converted at 3x the rate of others. [See the full story](/blog/how-we-changed-american-standards-mind).

## Ready to get started?

If you want to talk through how propensity modeling could work for your business, [talk to a Context Consultant](https://faraday.ai/talk-to-sales). Or if you'd rather get started on your own, [try it on buy.faraday.ai](https://buy.faraday.ai).

## FAQ

### What's the difference between a propensity model and a lead score?

A [lead score](/blog/predictive-lead-scoring-b2c) is one type of propensity model — specifically, one that predicts likelihood to convert. But propensity models can predict any defined action: churn, reactivation, loan application, product adoption, and more. Lead scoring is the most common starting point, not the full picture.

### How much data do I need to build a propensity model?

Faraday typically needs a few hundred historical examples of the outcome you want to predict — customers who converted and customers who didn't. Because FIG enriches every record with 1,400+ data points, even a modest first-party dataset can produce a strong model.

### How is this different from the scoring tools built into my CRM?

CRM-native scoring tools are limited to whatever data lives in your system — typically engagement signals like email opens, form fills, and page views. Faraday trains on those first-party signals enriched with external consumer context from FIG, which gives the model far more signal to work with and produces significantly higher predictive lift.

### How do I know if the model is actually working?

Faraday validates every model using [holdout testing](/blog/holdout-testing-with-faraday) before deployment — reserving a portion of your data that the model never sees during training, then measuring how well it predicts outcomes on that holdout group. You can also run ongoing A/B tests in-market to measure lift against your existing approach.

### Can propensity models be used in regulated industries like financial services?

Yes. Faraday builds bias mitigation into the modeling process — rebalancing training data to correct for historical underrepresentation and monitoring scores for fairness across protected classes. For a detailed example, see how Advia Credit Union used propensity scoring with fair lending compliance built in.

Using propensity modeling, Momentum Solar cut outbound calls by 33% and hit the same appointment goal. This is a complete guide to how propensity models work and how consumer brands use them across the full customer lifecycle.

What are propensity models: a complete guide for consumer brands

A high match rate is a vanity metric if you don't distinguish between where someone lives (address-level matching) and who they are (person-level matching). And even a perfect match doesn't guarantee useful enrichment — those are two independent dimensions of precision, and conflating them is one of the most common and costly mistakes teams make.

## The two dimensions of data precision

There are two independent dimensions of precision in any data enrichment process: **match precision** and **enrichment precision**. They are not the same thing, and a strong score on one does not imply a strong score on the other.

- **Match precision** describes how confidently a data provider recognizes the record you sent them. It has two levels: **person-level matching** (the provider has identified a specific individual — e.g., Jane Smith at 123 Main St, Unit 4B) and **address-level matching** (the provider has identified the household or residence, but not the specific person living there).  
    
- **Enrichment precision** describes the quality of the data being appended to that matched record. It also has two levels: **observed data** (the provider has actual, directly measured values for that person or household) and **imputed data** (the provider doesn't have the value directly, so they estimate it).  
    
- **Imputation** is the standard data science technique used when a value is missing. The system may look at everyone else known to live at the same fully qualified address — a spouse, an adult child, a roommate — take the values that are available for them, and average those to produce a best estimate. It's often better than returning nothing, but the resulting data is residence-level, even when the match itself was person-level.

These two dimensions combine into four possible outcomes. The quadrant below shows what each combination is good for:  

[![2x2 grid that describes enrichment precision and match precision.](/images/blog/enrichment-precision-diagram.png)](/images/blog/enrichment-precision-diagram.png)

## What matching precision actually means in practice

Both levels of matching are valid, depending on what you're doing. If you're running a direct mail campaign based on property characteristics — say, targeting homeowners with pools — address-level matching is perfectly fine. You care about the house, not the person. But if you're personalizing outreach based on individual behavior or demographics, person-level matching is what you actually need.

Most providers report a single "match rate" number without distinguishing between these levels. A 95% match rate sounds great — until you realize a significant portion of those matches are address-level, not person-level, and your personalization strategy depends on the latter.

## What enrichment precision actually means in practice

Not all attributes are created equal. Some are naturally person-level — things like age, income, or purchase behavior. These describe an individual. Others are naturally residence-level — like bathroom count, square footage, or home value. These describe a property.

Here's a key insight: more precision isn't always better. If you ask for bathroom count for everyone in your customer base, you're going to get residence-level data — and that's exactly what you want, because bathroom count is fundamentally a property of the home, not the person. Asking for it at the person level wouldn't even make sense. The "right" level of precision depends on what the attribute actually describes.

When everything aligns perfectly, you get a person-level match with observed person-level data for a person-level attribute. You know who Jane is, and you have her actual income. That's the best case.

But data is inherently sparse. Even the best providers don't have every attribute for every person. So when a provider has matched Jane at the person level but doesn't have her income specifically, imputation kicks in — and the value you get back is residence-level, even though the match itself was person-level.

This is the distinction most teams miss: a person-level match does not guarantee person-level enrichment.

## Why this matters for your strategy

If you're buying data to fuel AI models, personalization engines, or predictive analytics, the quality of your outputs depends entirely on understanding what went in. And "quality" here isn't just about volume or match rates — it's about the right kind of precision for your use case.

This connects to a broader problem we see across the industry: teams buy more data thinking it will solve their performance issues, when what they actually need is the right data at the right level of precision. A massive dataset with high match rates but mostly imputed enrichment data isn't necessarily more useful than a smaller dataset with genuine person-level attributes.

Here's a practical framework for thinking about it:

When address-level precision is fine:

* Direct mail campaigns targeting properties (not people)  
* Geographic or neighborhood-level analysis  
* Property-based segmentation (home value tiers, housing type)

When you need person-level precision:

* Individualized outreach or personalization  
* Predictive models based on personal demographics or behavior  
* Lookalike audience building based on individual customer profiles  
* AI agent workflows that need to understand a specific consumer

Questions to ask your data provider:

* What percentage of my matches are person-level vs. address-level?  
* For enriched attributes, how much of the data is directly observed vs. imputed?  
* For the specific attributes I care about, are they naturally person-level or residence-level?  
* How do you handle sparsity for the attributes that matter most to my use case?

## Transparency builds better outcomes

Most data providers don't explain any of this. They give you a match rate, they give you enriched fields, and they let you assume the best. That works fine — until your AI model underperforms and you can't figure out why, or your personalized campaigns feel generic despite having "complete" customer profiles.

At Faraday, we think the answer is transparency. When you understand exactly what level of precision you're getting — for both matching and enrichment — you can make smarter decisions about how to use it. You can weight imputed data differently in your models. You can choose the right attributes for your use case. You can stop over-investing in data volume and start investing in data precision.

Because ultimately, the question isn't "how much data do I have?" It's "do I have the right data, at the right level of precision, for what I'm trying to do?"

Want to understand how Faraday handles matching and enrichment precision for your data? [Grab a time to talk to one of our context consultants](https://faraday.ai/talk-to-sales).

Match rate alone doesn’t tell you whether enriched data is actually useful, teams need to separate match precision, meaning person-level vs. address-level matching, from enrichment precision, meaning observed vs. imputed data. The right level of precision depends on the use case.

Understanding enrichment precision: what your match rate is actually telling you (and what it's not)

**See our complete guide:** [Customer personas explained — how Faraday builds them and what they unlock](/blog/what-are-ai-customer-personas).

---

Marketers have always used [personas](/docs/abstractions/persona_sets) to bring customer data to life. But too often, [persona](/blog/k-means-clustering-customer-personas) outputs end up feeling opaque, confusing, or overwhelming. Blurry averages flatten nuance, biased naming conventions create distraction, and long trait tables? They leave teams unsure what truly drives action.

At Faraday, we’re committed to making data and predictions easier to interpret and act on. Building on our recent [persona naming update](/changelog/2025-10-08-persona-naming-update), we’re now rolling out **salience-based persona interpretation features** right in our existing dashboard—a major step toward clearer, bias-free insights that help marketers move from analysis to strategy with confidence.

## What’s new in persona interpretation

With this release, every [persona](/blog/how-many-personas-do-you-need) in Faraday now includes:

- **Plain-English summaries**
  A one-paragraph description of what defines the group is now found in the definitions tab so teams can immediately understand who they’re looking

[![persona summaries in definition tab](/images/blog/trait-salience/persona-summary.png)](/images/blog/trait-salience/persona-summary.png)

- **Salient traits (or datapoints)**  
  Clear indicators of which characteristics distinguish this persona from the rest of the set are now found in the Analysis tab. In practice, this means you can see which of Faraday’s 1,400+ consumer datapoints most clearly set this persona apart from others — helping you understand what’s unique about them.

In your dashboard, these most distinctive traits are now flagged with a star. ⭐

[![trait salience in the analysis tab](/images/blog/trait-salience/trait-salience.png)](/images/blog/trait-salience/trait-salience.png)

- **Strategic recommendations**  
  Practical ideas for how to engage each persona more effectively, which turn insights into action

[![persona recommendations tab](/images/blog/trait-salience/recommendations.png)](/images/blog/trait-salience/recommendations.png)

Together, these tools connect Faraday’s predictive data to clearer audience segmentation and campaign planning.

## Why interpretation matters

Interpreting personas shouldn’t feel like decoding a spreadsheet. Historically, persona analysis has relied on modes (the most common values) and assumptions That leads to two familiar problems:

1. **Blurry signals** — Minor differences (like 51% vs. 49%) get overstated, while meaningful behavioral patterns are buried.
2. **Too much noise** — Lengthy trait tables provide data, but not direction.

The new interpretation features in Faraday personas cut through that clutter. Each persona now highlights only the _salient_ traits that truly define the group, accompanied by plain-English summaries and strategic recommendations. Combined with our recent move to neutral persona naming, these updates give you a clearer, more trustworthy picture of your audiences.

## Key takeaways

✅ New interpretation features make persona insights clearer and more actionable.  
✅ Salient traits highlight what defines each audience segment.  
✅ Strategic recommendations turn analysis into immediate next steps.  
✅ New and rebuilt personas automatically include these updates.

## Explore our new personas features today

This feature is now live in both the Faraday Dashboard and API. Head to your personas to see the new summaries, salient traits, and recommendations in action.

If you have questions about how to get the most out of enhanced personas, your Faraday account manager is here to help.

## FAQ

**Why are some cells that appear most different not starred?**  
Each cell represents the mode, or the most common value, of the trait for that persona. But modes don’t always tell the full story. Suppose Persona 1 shows 100K+ as its mode for income, while others show 80–100K. Persona 1 appears to be most different, yet it might actually include 51% individuals earning 100K+ and 49% earning under 80K. In that case, the mode alone doesn’t capture what distinguishes that persona’s income.

Instead, the salient labels identify which persona shows the most distinctive pattern within each trait. This process repeats for every trait. Specifically, traits that receive stars are those whose distributions most strongly differentiate that persona from others. This allows the system to highlight what truly distinguishes a persona within each trait, not just what values of traits occur most frequently.

**Do I need to rebuild existing personas to see the new interpretation features?**  
Yes. Existing personas stay as they are, but any new or rebuilt sets include summaries, salient traits, and recommendations automatically.

**Will this affect my API workflows?**  
No. There are no API changes—personas will simply include richer interpretation data when rebuilt.

**Where do I find these updates?**  
In the Faraday Dashboard: summaries in _Definitions_, salient traits in _Analysis_, and recommendations in _Recommendations_.

**Why do some traits have no starred values across any personas?**  
Some traits receive no stars because the data patterns for those traits are too similar. The system applies two rules before assigning a star. First, a value’s salience must exceed a minimum threshold. Second, it must be significantly higher than the salience of that same trait’s values in other personas. Only when both conditions are met does the system star a trait value.

Faraday’s personas now include plain-English summaries, salient traits, and strategic recommendations—making it easier than ever to interpret and act on your customer insights. Building on our recent persona naming update, this release adds a new layer of clarity and context that turns analysis into action.

What's new: Personas get interpretation features for faster insight

We are excited to announce a significant improvement to how Faraday handles real-time identity resolution with **payload fallbacks**—a new strategy for producing consistent, reliable, and useful results for Lookup API users, even when an identity is only partially recognized.

## A challenge: partial matches

Faraday maintains a comprehensive, industry-leading database of U.S. consumer identities and insights, which we use to enrich identity data provided to us by clients. Even then, we sometimes are given identities that we only partially recognize—most commonly, we recognize an address, but not a household, or an address and a last name (household), but not a specific person. In these cases, Faraday would attempt to choose the closest or best match from our system, but would still resolve the request to a single identity.

## Our solution: payload fallbacks

To address this challenge, we've implemented a hierarchical approach to identity resolution that ensures meaningful results even for partially recognized identities.

Now, when our real-time API is queried with unrecognized individuals who have a known address or postal code, we aggregate data from other known individuals at that location to provide a meaningful composite score and set of attributes. Each API response includes a `match_type` field that transparently indicates which resolution level was used:

| Match Type          | Resolution Level | Description                                                                   |
| :------------------ | :--------------- | :---------------------------------------------------------------------------- |
| `address_full_name` | Person           | Exact match to a specific individual                                          |
| `address_last_name` | Household        | Data aggregated from individuals with the same last name at the given address |
| `address_only`      | Address          | Data aggregated from all individuals at the address                           |
| `postcode_only`     | Postcode         | Zipcode-level aggregated data                                                 |

When the `address_last_name` or `address_only` match types are used, we additionally provide a `cohabitants` array (limited to 10 entries) that gives visibility into the individuals used for aggregation. This array helps you understand exactly how the aggregated values were calculated. Match types which do not consider address components (e.g. `email_last_name`) are not subject to aggregation.

**Example:** If you submit an address with no name information, and Faraday knows five individuals at that address, we'll aggregate their scores and traits to provide a representative result.

## Benefits for your business

These enhancements to identity resolution deliver several important benefits:

- More complete data with fewer gaps or null values
- Higher-quality predictions by leveraging household and geographic patterns
- Better handling of address-only or partial identity information
- Transparent methodology that clearly indicates the resolution level used
- Consistent results across all your Faraday deliverables

### Rollout details note

Some payload fallback features will be rolling out gradually by default, such as the `postcode_only` match type. For existing Lookup API users, this match type will only be available after your targets have been updated.

If you’d like access today or want to confirm your status, just reach out to your SAM and we’ll get you squared away!

## In conclusion

Faraday’s enhanced identity resolution ensures you get the most reliable insights, even from partial matches. By aggregating datapoints at the household, address, and postcode levels, we provide more complete data, higher-quality predictions, and greater transparency.

Want to see how we turn partial matches into actual, reliable predictions? Learn how score aggregation enhances modeling accuracy in this [blog](https://faraday.ai/blog/score-aggregation). Or already convinced to put better identity resolution to work for your business? Let’s [talk](https://faraday.ai/talk-to-sales).

Faraday has introduced payload fallbacks, a new identity resolution strategy that ensures more reliable results for Lookup API users, even when an identity is only partially recognized.

Smarter identity resolution through payload fallbacks with Faraday

The real estate industry is undergoing a major shift. In March 2024, the [National Association of Realtors (NAR)](https://www.nar.realtor/) reached a settlement that will change how real estate agent commissions are handled. [Under the new rules](https://www.bankrate.com/real-estate/real-estate-commission-changes/), sellers will no longer be required to offer compensation to buyers’ agents via the [MLS (Multiple Listing Service)](https://www.mls.com/). Instead, buyers and their agents must negotiate compensation directly, rather than relying on a commission included in the home’s sale price. While this change is intended to increase transparency, it also adds complexity for buyers, who must now determine how they will compensate their agent—potentially before viewing homes. This shift could lead to more negotiation hurdles and uncertainty, potentially slowing down transactions for both buyers and agents. This new landscape presents a significant challenge for real estate agents who have historically relied on the established commission structure.

In this evolving landscape, one thing has become clear: data-driven decision-making, particularly through the strategic use of lead scoring, is now essential for real estate agents to thrive. With commissions becoming more negotiable and unpredictable, agents need to focus on the highest-potential clients. Faraday’s custom predictive datapoints and [lead scoring models](https://faraday.ai/templates)—leveraging advanced AI to provide unparalleled accuracy in identifying and prioritizing prospects in this shifting market—help real estate professionals prioritize the right prospects, streamline their pipeline, and navigate the new commission structure with confidence.

### The impact of new commission rules on real estate agents

The most significant change in the new commission rules is the requirement for buyers' agents to have a written agreement with clients about fees before showing any properties. This means buyers will be more directly aware of how much they’re paying for representation, making it crucial for agents to clearly communicate their value.

At the same time, some predict a "price war" will erupt, with buyers' agents competing fiercely for clients in an attempt to offer lower fees. Others fear this change will create confusion and extend the home-buying process, especially for first-time buyers who are already navigating high prices and interest rates.

### How lead scoring helps

In this new environment, agents who can prioritize high-value leads will have a significant edge. Lead scoring can use data to assess which leads are most likely to convert, allowing agents to focus their time and resources on the clients who are most likely to result in a sale. Faraday’s predictive lead scoring models analyze a wide range of signals—from behavioral patterns to buying intent—to give agents a clear picture of which leads are worth pursuing.

With Faraday’s platform, agents can assess leads in real-time, adjusting their strategies quickly to align with changing market conditions. Instead of wasting time on leads that aren’t ready to engage, agents can focus on those who are most likely to convert, maximizing their chances of closing deals while keeping costs in check. Agents using predictive lead scoring have reported a significant increase in conversion rates by focusing their efforts on the most promising leads.

### Addressing the new challenges for buyers and sellers

The new commission structure also increases transparency, but it has its drawbacks. Buyers will be more aware of the costs of hiring an agent, and many may feel the pressure to negotiate commissions themselves. Sellers, meanwhile, could face more complicated negotiations if buyers demand concessions or if they struggle to cover commission fees upfront.

Lead scoring helps agents respond to these challenges by providing a data-driven way to approach both buyers and sellers. For buyers, lead scores can indicate their level of urgency and financial readiness, helping agents tailor their approach to address potential commission concerns proactively. For sellers, understanding buyer agent interest and engagement through lead scoring can inform strategies for offering competitive compensation. Agents can use lead scores to identify clients who are ready to negotiate or those who may need additional support. This allows agents to tailor their approach to meet each client's specific needs, whether that’s offering strategic advice on negotiations or focusing on clients who are already financially prepared.

### Stay ahead in a shifting market

As commission structures continue to evolve, and the world around us changes every day, the real estate industry faces both challenges and opportunities. Agents who embrace data-driven tools like lead scoring will be better equipped to adapt to these changes and thrive in a competitive market. By focusing on high-value leads, real estate professionals can ensure that they’re making the most out of every opportunity—even in uncertain times.

Faraday’s predictive lead scoring tools provide real estate agents with the insights they need to make smarter, more strategic decisions. In a world where commissions are more negotiable than ever, having a clear understanding of lead potential is key to staying ahead.

### Ready to transform your real estate business?

Learn more about how Faraday can help you navigate the new real estate landscape and stay ahead of the competition. [Request a demo](https://faraday.ai/talk-to-sales) to see Faraday in action and discover how our AI-powered lead scoring can revolutionize your business.

Faraday’s custom predictive datapoints help real estate agents adapt to new commission structures by identifying high-value prospects, streamlining their pipeline, and maximizing conversions in an increasingly competitive market.

Navigating new commission structures in real estate: How lead scoring can help agents adapt

One of the most critical aspects of model development is ensuring that our models can generalize well to unseen data (i.e. when the predictions are deployed). While it's relatively straightforward to train a model that performs well on training data, the real challenge is creating models that maintain their performance when faced with new, previously unseen examples. This is where cross-validation (CV) comes into play to validate our models' performance and make more informed decisions during the model development process. Throughout this post, we will review different CV variants and how Faraday uses them to train its model and forecast return on investment (ROI) for our clients.

## What is cross-validation

At its core, cross-validation is a resampling procedure used to evaluate machine learning models on limited data samples. Instead of using a single train-test split, cross-validation provides a framework for multiple evaluations of our model using different portions of the available data.

Cross-validation involves partitioning the available data into complementary subsets, training the model on some subsets (training set), and validating it on the remaining subset (validation set). This process is repeated multiple times, with each data point getting the opportunity to be part of both training and validation sets.

The key principles of cross-validation include:

1. Data Independence: Ensuring that training and validation sets are strictly separated in each iteration
2. Comprehensive Evaluation: Using all available data for both training and validation through multiple iterations
3. Statistical Robustness: Generating multiple performance metrics to assess model stability

### K-Fold cross-validation

K-fold cross-validation is the most common and widely used variant of cross-validation. In this approach, the data is divided into k equal-sized folds, and the following process is repeated k times:

1. One fold is selected as the validation set
2. The remaining k-1 folds are used as the training set
3. One model is trained on the training set
4. Predictions on the validation set are saved

Faraday uses a 5-fold CV to evaluate its models. This means that for any given propensity model, Faraday builds 6 variations of the model (5 for the k-fold validation and one final version used as the deployed model.)

[![K-fold cross-validation diagram](/images/blog/cross-validation-blog/regular_cv.png)](/images/blog/cross-validation-blog/regular_cv.png)

Once all five models are trained, each sample in the data has been scored once. This means that the cross-validation allows us to assign a prediction to each person, allowing us to compute the performance metrics using the reconstructed scores.

Special variants of k-fold cross-validation include:

- **Stratified K-Fold**: Maintains the same class distribution in each fold (useful for imbalanced datasets)
- **Leave-one-out cross-validation**: Sets k equal to the number of data points (useful for very small datasets)
- **Time series cross-validation**: The folds are defined using a rolling window for time-based data.

[![Time series cross-validation](/images/blog/cross-validation-blog/time_cv.png)](/images/blog/cross-validation-blog/time_cv.png)

## The benefits of cross-validation

From the previous section, it is clear that CV implies more computation than training a single model. However, cross-validation offers several crucial advantages that outweigh the added costs and complexity, making it a mandatory step in the training process. Let's explore the key benefits that make this technique so valuable.

### Preventing overfitting

One of the primary benefits of cross-validation is its ability to help prevent overfitting. When a model learns the training data too well, it starts to capture noise and specific patterns that don't generalize to new data. Overfitting leads to poor performance in real-world applications. In the charts below, we compare the effects of three different model architectures on the same dataset:

1. The first model is too simplistic to capture the trend in the data.
2. The third one tries too hard to capture every data point and adds a lot of noise between them.
3. The second model does a good job of capturing the overall trend in the data without necessarily trying to capture small variations in the location of the data points.

[![Underfitting vs overfitting curves](/images/blog/cross-validation-blog/blog_3_curves.png)](/images/blog/cross-validation-blog/blog_3_curves.png)

Cross-validation helps detect overfitting by evaluating the model's performance on multiple subsets of the data. If a model performs significantly better on the training data than on the validation sets, it's a clear indicator of overfitting. This early detection allows data scientists to make necessary adjustments to the model's complexity, feature selection, or hyperparameters before deploying it to production.

If we compute our model performance (ROC, accuracy) on the training data, the third model is the best option. However, when using CV, the second model performs better.

### Computing reliable performance metrics

Traditional train-test splits can sometimes provide misleading performance metrics due to the random nature of the split. Cross-validation addresses this limitation by:

- Providing multiple performance estimates using different data splits
- Enabling the calculation of confidence intervals for performance metrics
- Reducing the variance in performance estimation
- Offering a more robust assessment of model stability

These multiple evaluations give us a clearer picture of how our model is likely to perform in real-world scenarios, helping us make more informed decisions about model selection and deployment.

The content of our outcome report and UI relies entirely on the cross-validation but two charts explicitly depict results per fold. Our “folded ROC” plot shows the ROC curve for each fold. This is useful to ensure the model is stable (i.e. the results across folds are nearly identical).

[![Folded ROC curve](/images/blog/cross-validation-blog/Folded_ROC.png)](/images/blog/cross-validation-blog/Folded_ROC.png)

The lift chart illustrates the volatility of lift, particularly for those with higher scores. This information is crucial for lead generation models, which aim to identify potential customers. Faraday assigns a score to each individual within a specified population, and the client uses only the top X% of scorers.

[![Lift curve](/images/blog/cross-validation-blog/lift_curve.png)](/images/blog/cross-validation-blog/lift_curve.png)

## Technical dive

### Formal definitions

**Dataset definition**

- Let $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ be our dataset where:
- $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$ are feature vectors for sample $i$
- $y_i \in \mathcal{Y}$ is the label for sample $i$
- $n$ is the number of samples
- Define $\mathcal{P}_{XY}$ as the true joint distribution of $(X,Y)$

**Model space**

- Let $\mathcal{F} = \{f_\theta: \mathcal{X} \rightarrow \mathcal{Y} | \theta \in \Theta\}$ be the hypothesis space
- $\theta$ represents the model parameters
- $\Theta$ is the parameter space

**Loss function**

- Define $\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}_{\geq 0}$ as the loss function
- The expected risk is:

```math
R(f) = \mathbb{E}_{(X,Y)\sim\mathcal{P}_{XY}}[\mathcal{L}(f(X),Y)]
```

$R(f)$ is interpreted as the error of the model $f$ on the dataset.

- The empirical risk on dataset $\mathcal{D}$ is the average loss over $\mathcal{D}$:

```math
\hat{R}_{\mathcal{D}}(f) = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(f(x_i),y_i)
```

**K-Fold partitioning**

- Partition indices $\{1,\ldots,n\}$ into $k$ disjoint subsets: $\{\mathcal{I}_1,\ldots,\mathcal{I}_k\}$
- For each fold $j$:
  - $\mathcal{D}_{val}^{(j)} = \{(x_i,y_i) | i \in \mathcal{I}_j\}$ (validation set)
  - $\mathcal{D}_{train}^{(j)} = \{(x_i,y_i) | i \notin \mathcal{I}_j\}$ (training set)

**Cross-validation estimator**  
The k-fold CV estimate of the expected risk is:

```math
CV_{(k)}(f) = \frac{1}{k}\sum_{j=1}^k \hat{R}_{\mathcal{D}_{val}^{(j)}}(f_{\theta_j})
```

where $f_{\theta_j}$ is trained on $\mathcal{D}_{train}^{(j)}$

### Statistical properties

**Bias of CV estimator**  
The bias of the k-fold CV estimator can be expressed as:

```math
Bias(CV_{(k)}) = \mathbb{E}[CV_{(k)}] - R(f^*)
```

where $f^*$ is the true optimal function and $\mathbb{E}[CV_{(k)}]$ is the expected risk over all the folds.

**Variance decomposition**  
The variance can be decomposed into:

```math
Var(CV_{(k)}) = \frac{1}{k^2}\sum_{j=1}^k Var(\hat{R}_{\mathcal{D}_{val}^{(j)}}) + \frac{2}{k^2}\sum_{i<j}Cov(\hat{R}_{\mathcal{D}_{val}^{(i)}}, \hat{R}_{\mathcal{D}_{val}^{(j)}})
```

Where $Var(\hat{R}_{\mathcal{D}_{val}^{(j)}})$ is the variance of the risk estimate over a fold. It does not refer to the variance of the risk itself (which is just a scalar value). Instead, it refers to the variance of the estimated risk based on multiple possible samples of the validation set within that fold.

Therefore, as k increases, the overall variance of the cross-validation risk decreases leading to a less volatile estimation of the model performance post-deployment.

# Holdout test

### What is it?

A "holdout test" can refer to any process where a model is being evaluated on unseen data (e.g. cross-validation above). At Faraday, a holdout test specifically refers to the process of choosing a date in the past $t_0$, fitting a propensity model with data prior to $t_0$, and evaluating the model on data after $t_0$.

On one hand, a holdout test is just a special case of time series cross-validation with 1 fold, but as we will see there are a few nuances to consider when configuring a holdout test and analyzing the results to inform a business decision.

### Configuring a holdout test in the Faraday platform

A typical scenario where a holdout test can be used is lead scoring. We will walk through how to run a lead scoring holdout test in the Faraday platform, but the same principles apply to a holdout test for other use-cases (e.g. predicting churn).

#### What is needed for a holdout test?

To configure a holdout test a user needs lead and conversion records with PII, dates, and a way to distinguish between the two groups. The setup for this can vary, but perhaps the simplest form is a single table that includes the following:

- PII columns
- a `lead_created` date that captures when a record became a lead
- a `conversion_date` that captures when that lead record became a conversion

NOTE: In our working example our conversions are called `tier_1_leads`.

In addition to the raw data, a user needs to have some prior knowledge of the typical attribution window for conversion which we will discuss in more detail when configuring the cohorts.

#### Datasets and events

Given a dataset described above, we need to configure the PII as one or more identity sets and two event streams, one for leads and one for conversions.

Configuring the leads event can be as simple as mapping a single date column  
[![Lead event](/images/blog/cross-validation-blog/lead_event.png)](/images/blog/cross-validation-blog/lead_event.png)

Similarly for the `tier_1_leads` event  
[![Attainment event](/images/blog/cross-validation-blog/attainment_event.png)](/images/blog/cross-validation-blog/attainment_event.png)

At this point our resource graph has a dataset table that emits two event streams  
[![Datasets graph](/images/blog/cross-validation-blog/datasets_graph.png)](/images/blog/cross-validation-blog/datasets_graph.png)

#### Cohorts

Next, we need to configure groups of people (cohorts) that partition the data into a training and validation set (recall this is time series cross-validation with 1 fold).

The inputs to determine this are:  
[![XYZ categories](/images/blog/cross-validation-blog/xyz.png)](/images/blog/cross-validation-blog/xyz.png)

[![XYZ description](/images/blog/cross-validation-blog/xyz_descriptions.png)](/images/blog/cross-validation-blog/xyz_descriptions.png)

The "prior period" is the training set for the holdout test, and the "holdout period" is the validation set. The amount of data available can influence how long/short these time windows are, but in general, we want to have enough data in the prior period to train a model and enough data in the holdout period to be confident in the results. For example, it would be bad to have zero conversions in the holdout period.

The attribution window is the time it takes for a typical lead to convert. Depending on the situation, this could be a few days to months.

The data lag is the time delta between today and the most recent date of any lead or conversion.

Once these have been determined we can create three cohorts

- prior leads: all records that became leads in the prior period
- recent leads: all records that became leads after the prior period
- conversions: all conversions (`tier_1_leads` in the working example) without any date filtering

Prior leads example configuration  
[![Prior lead configuration](/images/blog/cross-validation-blog/prior_leads_config.png)](/images/blog/cross-validation-blog/prior_leads_config.png)

Recent leads example configuration  
[![Recent lead configuration](/images/blog/cross-validation-blog/recent_leads_config.png)](/images/blog/cross-validation-blog/recent_leads_config.png)

Our resource graph now looks like  
[![Cohort graph](/images/blog/cross-validation-blog/cohorts_graph.png)](/images/blog/cross-validation-blog/cohorts_graph.png)

#### Outcome

Once cohorts are configured it is time to build a model and get some output. In Faraday, a propensity model is defined by an "outcome" which takes two cohorts as an input.

[![Outcome configuration](/images/blog/cross-validation-blog/outcome_config.png)](/images/blog/cross-validation-blog/outcome_config.png)

Note that using the prior leads from the prior period implicitly restricts the conversions (tier 1 leads) to the prior period.

#### Pipeline

A pipeline applies an outcome (propensity model) to a group of people specified by a cohort. In this case we want to apply the outcome trained using data from the prior period to score recent leads records. Additionally, we want to have conversion information available in the output to analyze performance.

[![Scope configuration](/images/blog/cross-validation-blog/scope_config.png)](/images/blog/cross-validation-blog/scope_config.png)

Here is the full resource graph  
[![Resource graph](/images/blog/cross-validation-blog/resource_graph.png)](/images/blog/cross-validation-blog/resource_graph.png)

This pipeline produces a table whose rows correspond to the recent lead records and whose columns include:  
[![Payload columns](/images/blog/cross-validation-blog/payload_cols.png)](/images/blog/cross-validation-blog/payload_cols.png)

Any of the `fdy_outcome_` columns can be used together with the `fdy_cohort_*_member` column (which is true when a row is a conversion) to measure model performance.

### Analyzing results

The output from the above pipeline can be used for custom analysis as desired by the user. One example is the automated reporting that Faraday produces to analyze holdout results and potential ROI.

#### Holdout results

First, we look at the internal performance metrics of the model built using the prior period.

[![Training data used](/images/blog/cross-validation-blog/training_data_used.png)](/images/blog/cross-validation-blog/training_data_used.png)

These metrics are based on cross-validation described above and distilled down into a single plot.

[![Model results](/images/blog/cross-validation-blog/model_results.png)](/images/blog/cross-validation-blog/model_results.png)

Next, we look at the pipeline output to see how the model performed on the recent leads.

[![Data used for holdout test](/images/blog/cross-validation-blog/data_used_for_holdout_test.png)](/images/blog/cross-validation-blog/data_used_for_holdout_test.png)

Again, we can produce conversion rates by decile.

[![Holdout test results](/images/blog/cross-validation-blog/holdout_test_results.png)](/images/blog/cross-validation-blog/holdout_test_results.png)

The last step of the holdout analysis is to compare the observed results from the holdout test with the model performance computed at training time via cross-validation.

[![Comparing model report to test results](/images/blog/cross-validation-blog/comparing_test_results.png)](/images/blog/cross-validation-blog/comparing_test_results.png)

If these results are comparable, then it gives the user confidence that the performance metrics observed at training time are reflective of the model performance moving forward.

#### ROI calculation

When additional business context is known, the holdout test results can also be used to estimate ROI. For example

[![Lead rejection ROI calculator](/images/blog/cross-validation-blog/roi_calculator.png)](/images/blog/cross-validation-blog/roi_calculator.png)

## Conclusion

Cross-validation can be used to prevent overfitting and evaluate a model's performance at training time. A holdout test is just a special case of cross-validation in which a single fold is determined by a point in time. Combining these two approaches with business assumptions enables the user to deploy models confidently, with a sense of expected performance and insight into expected ROI.

Faraday uses cross-validation to evaluate model performance, ensuring accuracy and value by training on multiple data subsets and validating on others. This allows us to make informed decisions and deliver reliable ROI for our clients.

How Faraday validates our predictive models

**See our complete guide:** [What is the Faraday Identity Graph?](/blog/whats-in-faraday-identity-graph).

---

As businesses race to integrate AI into their software, many turn to first-party data modeling techniques like [RFM (recency, frequency, monetary value)](<https://en.wikipedia.org/wiki/RFM_(market_research)>) to build predictive models. These models typically rely on user interactions, including website navigation, email engagement, and purchase behavior to anticipate action. But first-party models alone have significant limitations, lacking the depth, speed-to-deployment, and predictive accuracy that businesses need to get ahead in today’s competitive market.

Faraday’s models solve these problems by integrating rich third-party data from the [Faraday Identity Graph (FIG)](https://faraday.ai/blog/unlocking-the-power-fig), out of the box. The FIG enhances our models by integrating high-quality, consented offline insights into your first-party data, creating a more complete and actionable dataset. By enriching your existing data with verified external datapoints—such as demographics, lifestyle indicators, and purchasing behaviors—the FIG fills in gaps, improves predictive accuracy, and speeds up deployment to help businesses reach the right audiences at the right time with the right product.

## Why first-party data alone falls short

First-party data has its advantages, but it also has critical limitations that can hinder AI-driven personalization and prediction.

First-party data is information that businesses collect directly from their customers and audience through owned channels. This includes website activity, purchase history, and email engagement. Because it comes straight from the source, first-party data is highly accurate, privacy-compliant, and essential for personalized marketing and predictive modeling. However, it can be limited in scope, which is why many businesses enhance it with external data for deeper insights.

One major challenge to overcome with first-party data alone is the [cold start problem](https://www.sciencedirect.com/topics/computer-science/cold-start-problem). When a new customer visits your business, there is no past activity to analyze, making it impossible to generate useful predictions without supplemental third-party data. A first-party data-based model will most likely only improve over time as the customer interacts with the brand, but this delay can mean lost opportunities, particularly in industries where real-time data is crucial like financial services.

Even as first-party data accumulates, it often remains noisy and incomplete. Activity data is unpredictable, with various factors influencing when a user engages with your site or makes a purchase, meaning first-party models must quickly adjust to these fluctuations. External factors like age or income level are also essential for understanding customer behavior, but they’re typically not captured in first-party data. Additionally, first-party data only reflects [activity, not affinity](https://faraday.ai/blog/why-first-party-data-and-rfm-is-not-enough). For instance, knowing that someone purchased a pillow tells you little about their broader lifestyle, financial capacity, or long-term interests.

## How Faraday enriches customers’ datasets to power our models

Our data enrichment process uses identity resolution to securely match first-party identifiers—such as email addresses or hashed customer IDs—with the FIG’s extensive dataset before they are fed into our models. This process ensures a seamless connection between your existing data and additional insights, following three key steps:

1. **Secure matching** – Faraday uses identity resolution techniques to match first-party data with offline records, ensuring privacy and compliance along the way.
2. **Data enrichment** – Once linked, Faraday appends high-quality demographic, behavioral, and lifestyle datapoints to each record.
3. **Model integration** – These enriched profiles are seamlessly fed into our AI models, enabling more accurate predictions and personalized customer experiences.

[![Data-appends diagram](/images/blog/first-party-merged-w-fig.png)](/images/blog/first-party-merged-w-fig.png)

## Powering better AI with enriched data

Our models’ ability to access the **Faraday Identity Graph (FIG)** solves numerous problems experienced by first-party-only models. Here are some examples:

### 1\. Solving the cold start problem

With the FIG, even first-time visitors are no longer unknowns. By matching first-party identifiers to offline, third-party data, Faraday provides insights into likely interests, purchasing power, and household composition, allowing our models to make informed predictions from the very first interaction.

### 2\. Boosting model stability and accuracy

AI models trained solely on first-party data must constantly adapt to new patterns in an ever-changing environment. The FIG introduces stable, high-quality datapoints, reducing reliance on transient, high-variance signals like click-through rates. This results in more durable models that perform consistently over time.

### 3\. Enhancing AI with affinity data

Instead of just looking at past transactions, the FIG integrates data that reflects who customers are, not just what they’ve done. This includes household income, education level, homeownership status, and even media consumption habits.

## The best of both worlds

Faraday’s FIG takes you data to the next level. By seamlessly integrating high-quality, consented third-party insights with your existing first-party information, Faraday empowers you to access more accurate and predictive models through our platform. With this rich array of demographic, lifestyle, and behavioral datapoints, you can unlock deeper insights into your customers and make smarter, data-driven decisions. The Faraday Identity Graph goes beyond the limitations of first-party data alone, giving you a clearer, more holistic view of your audience.

Want to see how the FIG can supercharge your AI models? **Let’s talk.**

The Faraday Identity Graph (FIG) allows users to create more accurate, stable, and predictive AI models within our platform with deeper insights into customer behavior and preferences.

Creating better predictive models with the Faraday Identity Graph

A [ping post](https://activeprospect.com/glossary/ping-post/) is a real time lead distribution method where lead sellers send basic lead details (the "**ping**") to potential buyers, who then evaluate the lead based on predefined criteria. If a buyer decides to bid on the lead, they receive the full details (the "**post**") and complete the purchase. This system allows for fast, competitive bidding, commonly used in industries like home services, financial services, and insurance.

While this method offers efficiency, it also introduces challenges. Many businesses end up paying for leads that don’t convert, leading to wasted budget and ineffective outreach efforts. The ability to filter and score leads before purchasing them is crucial to maintaining a high return on investment (ROI) and optimizing marketing spend.

## How we use Ping Posts at Faraday

The challenge with ping-post lead buying is that companies often spend money on leads that don’t convert. At Faraday, we integrate with ActiveProspect’s [LeadConduit](https://activeprospect.com/leadconduit/) to enhance this process with predictive scoring. By applying machine learning at the ping stage, brands can bid only on high-converting prospects, filtering out low-quality leads in real time. This optimization reduces wasted spend and improves overall campaign efficiency.

Predictive scoring works by analyzing historical data, behavioral signals, and demographic datapoints to assess a lead’s likelihood of converting. Thanks to the built-in [Faraday Identity Graph](https://app.faraday.ai/traits) ([which comes preloaded with 1400+ attributes on nearly every US consumer and their households](https://faraday.ai/features/faraday-identity-graph)), our models score each lead against rich third-party data before a bid is placed instead of relying on limited seller-provided information. This creates the opportunity for businesses to make data-driven decisions in real time, ensuring they focus only on high-quality opportunities.

For example, an insurance provider purchasing thousands of leads daily might bid on any lead that meets basic criteria, but many of those leads never convert—driving up acquisition costs. If each lead costs \$10 and converts at a rate of 1 in 5, their effective customer acquisition cost (CAC) would be \$50. But not all leads are equal. By rejecting the bottom 40%—the least likely to convert—they could improve their conversion rate significantly, lowering their CAC to just $30.

In this scenario, instead of wasting budget on low-quality leads, they could acquire more customers for the same spend. With predictive lead scoring, they could make these decisions in real time, ensuring they only invest in leads with real potential.

## Even without TCPA 1:1 consent rule, better targeting still matters

[While the expected TCPA 1:1 consent rule did not go into effect earlier this year](https://faraday.ai/blog/tcpa_1-1_vacated), that doesn’t mean businesses can ignore lead quality or continue to enact spray and pray targeting in 2025\. Just because you’re not being forced to change doesn’t mean you shouldn’t. Rising acquisition costs and increasing competition make it critical to invest in leads that are both high-quality and likely to convert. Faraday’s real-time predictive scoring ensures that brands maximize their ROI while staying ahead of the curve. By leveraging AI-driven insights, lead buyers can refine their targeting strategies and remain competitive in an evolving marketplace.

Additionally, compliance concerns aren’t going away. Even without the immediate pressure of TCPA amendments, consumer privacy laws and regulations continue to evolve, and businesses that proactively refine their lead acquisition strategies will be better prepared for future changes. More rigorous lead assessment doesn’t just improve efficiency—it also reduces the risk of compliance issues and enhances the overall customer experience by ensuring outreach efforts are more relevant and targeted.

## Looking ahead

The lead generation landscape will continue to evolve, and companies that prioritize precision over volume will have the advantage. Rather than buying as many leads as possible and hoping some might convert, businesses need to shift toward smarter, data-driven strategies that optimize every dollar spent.

Faraday’s predictive lead scoring is a powerful tool for staying ahead, allowing businesses to purchase leads with confidence and efficiency. Whether regulatory shifts happen or not, the need for better targeting and smarter decision-making isn’t going away.

Ping posts are a real time lead distribution method where basic lead details are sent to potential buyers, who then decide whether to bid before receiving the full lead information.

So what is a Ping Post anyway?

**See our complete guide:** [Customer personas explained — how Faraday builds them and what they unlock](/blog/what-are-ai-customer-personas).

---

Using machine learning to identify customer personas in your data can help improve your understanding of your customer base, and with the right infrastructure, deliver personalized experiences at a greater scale.

In a nutshell, the process entails using an algorithm to segment your customer data into distinct groups, or personas. Faraday uses a time-tested method called k-means clustering.

## What is k-means clustering?

K-means clustering is an unsupervised clustering algorithm that was first introduced in 1957. An unsupervised algorithm does not require the data to be labeled in order to train the model. This is an important feature of the k-means algorithm because it allows for the discovery of subgroups within the data without any previous assumptions about possible groupings — in this case, personas.

## How does k-means clustering work?

When presented with a data set, the k-means algorithm first introduces random center points around which each datapoint will eventually be clustered (circles, Fig. 1). Then each datapoint is assigned to the group of the center to which they are closest.
This happens by calculating the distance from a datapoint to each cluster center using an equation known as the Euclidean distance measure. For two dimensions, the Euclidean distance is represented by the diagonal line connecting the point to each of the cluster centers. The point is then assigned to the cluster center with the shortest diagonal line (Fig. 2).

Once each point has been assigned to the randomly chosen cluster centers, the centers then "update," moving to the center of the group by recalculating the mean (average) distances to all the points in the group (Fig. 3).

After the cluster centers have updated to their new positions, each point is reassigned to a new group using the Euclidean distance measure again. Most of the points will remain in their original groups, but some points will be assigned to a new group (Fig. 4). So, then the centers must be updated again to be at the center of the new group. This pattern repeats until the centers no longer move.

[![K-means diagram](/images/blog/k-means_diagram.png)](/images/blog/k-means_diagram.png)

## Using k-means clustering for customer personas

All of the above is how k-means clustering worked back in 1957. Since then there have been upgrades allowing us to use something called categorical features. At Faraday, we create persona groups using a mix of these features.

As far as algorithms go, k-means is rather straightforward, but has one major downfall: you have to tell it how many groups to look for, and it will always succeed in finding that many groups. That means if you tell the algorithm to look for 500 groups, it will find them — even if there is no reason for there to be 500 distinct groups within the data.

If you're a marketer, you probably don't want to create 500 unique ads for every campaign. But should you make four? Or six? Or ten?

## How to find the optimal number of personas using k-means clustering

We target the optimal number of groups hidden within the data, rather than guessing and hoping we are right. We’ll share a few methods we use to find the optimal number of personas for our users.

### The elbow method: finding diminishing returns

The most common metric for measuring the success of k-means clustering is called the ratio of sum of squares (rss). At Faraday, we determine the optimal number of personas by comparing the rss for k-means models with two to fourteen personas. We employ the elbow method, which shows the cutoff after which we see diminishing returns for increasing the number of personas. For most clients, the optimal number of distinct personas is usually between four and seven.

**Going beyond the elbow method**

The elbow method is simple, but it has a downside: sometimes, there just isn't a clear single point that is optimal. Maybe three personas could be optimal, maybe six, and we have to intervene based on our client’s business needs to choose one or the other as the optimal number of personas.

Additionally, while reducing variability within each group is an extremely useful way to measure k-means clustering success, it doesn't tell the whole story. We believe that an important aspect of clustered groups is stability, which isn't considered with the elbow method selection process. Our clients expect that their persona groups won't shift every time they convert a lead. This is why we are currently researching a new method for determining the optimal number of personas, which focuses on the stability of the persona group.

### The bootstrap method: focusing on stability

Our first step to assess the stability of k-means clustering involves resampling the client’s data using the bootstrap method. Like k-means clustering, bootstrapping has been around for a while – it was first [introduced in 1979 by Bradley Efron](https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full). The basic idea behind bootstrapping is to take data from the pool we have, with the ability to pick the same datapoint multiple times. This is a method known as sampling with replacement.

The bootstrap method provides us with a powerful tool to measure how much our desired clustering outcome could change with different data. We can get a good sense of how much the personas will change with the addition of new customers. Our goal is to minimize how much influence single customers, or small groups of customers, have on the resulting personas. We want personas that remain relatively unchanged by single customers. In other words, we want more stability.

## Pulling it all together

Once we have the results from bootstrapping, we use the ratio of sum of squares again to measure just how stable the personas are. What we’re looking for is a ratio close to 1, which would indicate that there is no variability within the personas.

The stability metric for determining the optimal number of personas shows promising results for the basic examples in this blog post. At Faraday, we’re actively working on applying this methodology to the more complex persona groupings our clients rely on. Stay tuned for more developments with respect to improving our personas methodology.

Learn how machine learning is used to identify distinct clusters—or personas—in your customer data.