**What happens when you pair world-class lead scoring with the most advanced lead management system in the home services industry?**

In this video, Faraday CEO [Andy Rossmeissl](https://www.linkedin.com/in/rossmeissl/) and [ActiveProspect](https://activeprospect.com/) GM [Ruben Ugarte](https://www.linkedin.com/in/ruben-ugarte-solar-junkie/) dive deep into the lead ecosystem—from [TCPA](/blog/tcpa_1-1_vacated) compliance to real-time rejection, data enrichment to ping/post bidding—and explain how home services brands can unlock ROI across every stage of their funnel.

🎯 **In this video, you’ll learn:**
- Why the “spray and pray” era of lead buying is over
- How real-time scoring and routing help brands do more with less
- What the rise of ping/post means for the future of home services
- How lead underwriting protects your CRM—and your bottom line
- Why treating all leads equally is a rookie mistake


<iframe width="560" height="315" src="https://www.youtube.com/embed/aMxmPlDfqn0?si=4wFhbAXVmJ_8_NBz" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Faraday CEO Andy Rossmeissl sits down with Ruben Ugarte, GM of ActiveProspect, to talk about the evolution of lead buying—why real-time infrastructure is becoming table stakes, how ping/post is reshaping home services, and what marketers can do today to get ahead.

What happens when predictive AI meets real-time lead filtering?

We are excited to announce a significant improvement to how Faraday handles real-time identity resolution with **payload fallbacks**—a new strategy for producing consistent, reliable, and useful results for Lookup API users, even when an identity is only partially recognized.

## A challenge: partial matches

Faraday maintains a comprehensive, industry-leading database of U.S. consumer identities and insights, which we use to enrich identity data provided to us by clients. Even then, we sometimes are given identities that we only partially recognize—most commonly, we recognize an address, but not a household, or an address and a last name (household), but not a specific person. In these cases, Faraday would attempt to choose the closest or best match from our system, but would still resolve the request to a single identity.

## Our solution: payload fallbacks

To address this challenge, we've implemented a hierarchical approach to identity resolution that ensures meaningful results even for partially recognized identities.

Now, when our real-time API is queried with unrecognized individuals who have a known address or postal code, we aggregate data from other known individuals at that location to provide a meaningful composite score and set of attributes. Each API response includes a `match_type` field that transparently indicates which resolution level was used:

| Match Type | Resolution Level | Description |
| :---- | :---- | :---- |
| `address_full_name` | Person | Exact match to a specific individual |
| `address_last_name` | Household | Data aggregated from individuals with the same last name at the given address |
| `address_only` | Address | Data aggregated from all individuals at the address |
| `postcode_only` | Postcode | Zipcode-level aggregated data |

When the `address_last_name` or `address_only` match types are used, we additionally provide a `cohabitants` array (limited to 10 entries) that gives visibility into the individuals used for aggregation. This array helps you understand exactly how the aggregated values were calculated. Match types which do not consider address components (e.g. `email_last_name`) are not subject to aggregation.

**Example:** If you submit an address with no name information, and Faraday knows five individuals at that address, we'll aggregate their scores and traits to provide a representative result.

## Benefits for your business

These enhancements to identity resolution deliver several important benefits:

* More complete data with fewer gaps or null values  
* Higher-quality predictions by leveraging household and geographic patterns  
* Better handling of address-only or partial identity information  
* Transparent methodology that clearly indicates the resolution level used  
* Consistent results across all your Faraday deliverables

### Rollout details note

Some payload fallback features will be rolling out gradually by default, such as the `postcode_only` match type. For existing Lookup API users, this match type will only be available after your targets have been updated.

If you’d like access today or want to confirm your status, just reach out to your SAM and we’ll get you squared away!


## In conclusion

Faraday’s enhanced identity resolution ensures you get the most reliable insights, even from partial matches. By aggregating traits at the household, address, and postcode levels, we provide more complete data, higher-quality predictions, and greater transparency.

Want to see how we turn partial matches into actual, reliable predictions? Learn how score aggregation enhances modeling accuracy in this [blog](https://faraday.ai/blog/score-aggregation). Or already convinced to put better identity resolution to work for your business? Let’s [talk](https://faraday.ai/talk-to-sales).

Faraday has introduced payload fallbacks, a new identity resolution strategy that ensures more reliable results for Lookup API users, even when an identity is only partially recognized.

Smarter identity resolution through payload fallbacks with Faraday

The real estate industry is undergoing a major shift. In March 2024, the [National Association of Realtors (NAR)](https://www.nar.realtor/) reached a settlement that will change how real estate agent commissions are handled. [Under the new rules](https://www.bankrate.com/real-estate/real-estate-commission-changes/), sellers will no longer be required to offer compensation to buyers’ agents via the [MLS (Multiple Listing Service)](https://www.mls.com/). Instead, buyers and their agents must negotiate compensation directly, rather than relying on a commission included in the home’s sale price. While this change is intended to increase transparency, it also adds complexity for buyers, who must now determine how they will compensate their agent—potentially before viewing homes. This shift could lead to more negotiation hurdles and uncertainty, potentially slowing down transactions for both buyers and agents. This new landscape presents a significant challenge for real estate agents who have historically relied on the established commission structure.   

In this evolving landscape, one thing has become clear: data-driven decision-making, particularly through the strategic use of lead scoring, is now essential for real estate agents to thrive. With commissions becoming more negotiable and unpredictable, agents need to focus on the highest-potential clients. Faraday’s [predictive lead scoring](https://faraday.ai/templates) agents—leveraging advanced AI to provide unparalleled accuracy in identifying and prioritizing prospects in this shifting market—help real estate professionals prioritize the right prospects, streamline their pipeline, and navigate the new commission structure with confidence.

### The impact of new commission rules on real estate agents

The most significant change in the new commission rules is the requirement for buyers' agents to have a written agreement with clients about fees before showing any properties. This means buyers will be more directly aware of how much they’re paying for representation, making it crucial for agents to clearly communicate their value.

At the same time, some predict a "price war" will erupt, with buyers' agents competing fiercely for clients in an attempt to offer lower fees. Others fear this change will create confusion and extend the home-buying process, especially for first-time buyers who are already navigating high prices and interest rates.

### How lead scoring helps

In this new environment, agents who can prioritize high-value leads will have a significant edge. Lead scoring can use data to assess which leads are most likely to convert, allowing agents to focus their time and resources on the clients who are most likely to result in a sale. Faraday’s agentic lead scoring models analyze a wide range of signals—from behavioral patterns to buying intent—to give agents a clear picture of which leads are worth pursuing.

With Faraday’s platform, agents can assess leads in real-time, adjusting their strategies quickly to align with changing market conditions. Instead of wasting time on leads that aren’t ready to engage, agents can focus on those who are most likely to convert, maximizing their chances of closing deals while keeping costs in check. Agents using predictive lead scoring have reported a significant increase in conversion rates by focusing their efforts on the most promising leads.   

### Addressing the new challenges for buyers and sellers

The new commission structure also increases transparency, but it has its drawbacks. Buyers will be more aware of the costs of hiring an agent, and many may feel the pressure to negotiate commissions themselves. Sellers, meanwhile, could face more complicated negotiations if buyers demand concessions or if they struggle to cover commission fees upfront.

Lead scoring helps agents respond to these challenges by providing a data-driven way to approach both buyers and sellers. For buyers, lead scores can indicate their level of urgency and financial readiness, helping agents tailor their approach to address potential commission concerns proactively. For sellers, understanding buyer agent interest and engagement through lead scoring can inform strategies for offering competitive compensation. Agents can use lead scores to identify clients who are ready to negotiate or those who may need additional support. This allows agents to tailor their approach to meet each client's specific needs, whether that’s offering strategic advice on negotiations or focusing on clients who are already financially prepared.

### Stay ahead in a shifting market

As commission structures continue to evolve, and the world around us changes every day, the real estate industry faces both challenges and opportunities. Agents who embrace data-driven tools like lead scoring will be better equipped to adapt to these changes and thrive in a competitive market. By focusing on high-value leads, real estate professionals can ensure that they’re making the most out of every opportunity—even in uncertain times.

Faraday’s predictive lead scoring tools provide real estate agents with the insights they need to make smarter, more strategic decisions. In a world where commissions are more negotiable than ever, having a clear understanding of lead potential is key to staying ahead.

### Ready to transform your real estate business?

Learn more about how Faraday can help you navigate the new real estate landscape and stay ahead of the competition. [Request a demo](https://faraday.ai/talk-to-sales) to see Faraday in action and discover how our AI-powered lead scoring can revolutionize your business.

Faraday’s predictive lead scoring helps real estate agents adapt to new commission structures by identifying high-value prospects, streamlining their pipeline, and maximizing conversions in an increasingly competitive market.

Navigating new commission structures in real estate: How lead scoring can help agents adapt

In today's data-driven world, businesses can amass a wealth of first-party information, but this presents only a partial picture of their current and potential customers. Gaps in their data can limit the effectiveness of their marketing strategies, market research, and predictive models. That’s where Faraday’s data appends come in. By enhancing their existing datasets with high-quality third-party attributes, brands can unlock deeper insights to make smarter, more informed decisions.

## What are data appends, and why do they matter?

Data appending is a process by which businesses can purchase and integrate external data into their own datasets to supplement their first-party information. These attributes might include demographic information, behavioral patterns, or even lifestyle indicators, data points that are generally not available through first-party channels alone. By enriching your data with these additional insights, you can gain a fuller picture of your customers and their behaviors. 

But Faraday’s data appends go beyond simple enrichment. Our data is rigorously vetted, privacy-compliant, and built for seamless integration into your existing workflows. Whether refining market research, improving segmentation, or even powering your own predictive models, these appends help businesses make smarter, data-driven decisions. With Faraday, you’re not just adding data—you’re adding intelligence.

## How businesses use data appends

Data appends address various business needs. Here are some common ways companies leverage them:

### Exploratory research

Understanding your market often requires more than just your existing data. Take, for example, a roofing company. They might want to analyze roof sizes across different regions to gauge demand. Instead of sourcing this data themselves (which would be slow and expensive) or purchasing a generic dataset that requires extensive processing, they can use Faraday to buy and append only the relevant geographic and property data, seamlessly. This allows them to conduct targeted research efficiently and make informed decisions—like leveraging predictive targeting with Faraday next.  

### Supplementing existing datasets

Businesses often lack key customer details. Take a retailer that has always relied on email marketing. They would have extensive email lists but likely never collected addresses or phone numbers. If they want to expand into direct mail or call center outreach, they’ll need that missing data. This is where Faraday’s data appends come in, providing the necessary information without the hassle, time commitment, or high costs of sourcing it themselves. 

### Building custom predictive models

 An advanced use case for our data appends is supporting businesses that prefer to develop their own machine learning models. Just as we use this data at Faraday, these companies leverage our high-quality, cleaned, and structured attributes to enhance their predictive outcomes. In this case, Faraday acts as a trusted data provider, delivering the precise insights needed to model a variety of customer attributes—such as likelihood to churn, lifetime value, or purchasing behavior—ready to be seamlessly integrated into customers’ [Python](https://www.python.org/) environments.

## How Faraday’s data appends work

The Faraday Identity Graph (FIG) is the backbone of our data appending services. This robust dataset combines multiple, meticulously cleansed, and feature-engineered data sources into one comprehensive dataset. [With over 1,500 data attributes covering nearly all U.S. adult consumers and their households](https://app.faraday.ai/traits), it offers a deep well of insights into things like demographic trends, property ownership, life events, and financial indicators, just to name a few.

The process to access this data as appends is simple and streamlined:

1. **Select attributes**: Choose the data attributes that align with your business goals (e.g., income levels, household size, location-specific behaviors).  
2. **Match & append**: Faraday will match these attributes to your existing customer records, providing a more complete profile of your audience.  
3. **Integration**: The enriched data is delivered in a clean, structured format that’s easy to integrate with your models and business processes.

[![Data-appends diagram](/images/blog/data-appends-diagram.png)](/images/blog/data-appends-diagram.png)

## Why not just find the data yourself?

Some companies attempt to enhance their marketing efforts by purchasing third-party data directly from various vendors themselves, rather than using packaged and processed services like Faraday’s data appends. However, this approach often presents significant challenges:

**Contracting complexity**: Sourcing offline data can involve lengthy negotiations and large upfront commitments.

**Security risks**: Handling sensitive data requires strict compliance frameworks, such as SOC-2 certification. Unlike many vendors, all Faraday data appends are privacy-compliant, ethically sourced, and adhere to high standards of data protection, ensuring your customers' personal information remains secure while providing actionable insights.

**Regulatory challenges**: Companies storing offline data must navigate consumer privacy laws, including opt-out and deletion requests. Faraday manages these complexities for you, so you can focus on what matters—leveraging new insights to drive revenue.

**Data science overhead**: Raw offline data often requires significant processing, feature selection, and bias mitigation before it can be effectively used. This can result in costly time and resource investments as companies try to integrate third-party data into their models.

Faraday eliminates these challenges by providing ready-to-integrate data that is pre-processed and structured for immediate use. Our data appends come with built-in privacy compliance, so you don’t have to worry about regulatory hurdles or the risk of handling sensitive information. Additionally, the data is feature-engineered and bias-mitigated, saving you the time and resources typically needed to prepare raw data. With Faraday, you can access high-quality, actionable insights without the complexities and overhead associated with managing third-party data directly. This allows your team to focus on refining your marketing strategies and driving better business outcomes.

## How to get started

Getting started with Faraday’s data appends is simple. Once you’ve identified the attributes you need, we’ll work with you to integrate them into your existing systems. Whether you’re looking for a one-time data boost or ongoing data enrichment, our flexible pricing and easy integration make it simple to scale.

To learn more or request a demo, **contact us today**.

Faraday’s data appends enrich your existing customer data with third-party attributes, helping you fill gaps, refine targeting, and make smarter decisions.

Leveraging the power of the Faraday Identity Graph through Data Appends

One of the most critical aspects of model development is ensuring that our models can generalize well to unseen data (i.e. when the predictions are deployed). While it's relatively straightforward to train a model that performs well on training data, the real challenge is creating models that maintain their performance when faced with new, previously unseen examples. This is where cross-validation (CV) comes into play to validate our models' performance and make more informed decisions during the model development process. Throughout this post, we will review different CV variants and how Faraday uses them to train its model and forecast return on investment (ROI) for our clients.

## What is cross-validation

At its core, cross-validation is a resampling procedure used to evaluate machine learning models on limited data samples. Instead of using a single train-test split, cross-validation provides a framework for multiple evaluations of our model using different portions of the available data.

Cross-validation involves partitioning the available data into complementary subsets, training the model on some subsets (training set), and validating it on the remaining subset (validation set). This process is repeated multiple times, with each data point getting the opportunity to be part of both training and validation sets.

The key principles of cross-validation include:

1. Data Independence: Ensuring that training and validation sets are strictly separated in each iteration
2. Comprehensive Evaluation: Using all available data for both training and validation through multiple iterations
3. Statistical Robustness: Generating multiple performance metrics to assess model stability

### K-Fold cross-validation

K-fold cross-validation is the most common and widely used variant of cross-validation. In this approach, the data is divided into k equal-sized folds, and the following process is repeated k times:

1. One fold is selected as the validation set
2. The remaining k-1 folds are used as the training set
3. One model is trained on the training set
4. Predictions on the validation set are saved

Faraday uses a 5-fold CV to evaluate its models. This means that for any given propensity model, Faraday builds 6 variations of the model (5 for the k-fold validation and one final version used as the deployed model.)  

[![K-fold cross-validation diagram](/images/blog/cross-validation-blog/regular_cv.png)](/images/blog/cross-validation-blog/regular_cv.png)


Once all five models are trained, each sample in the data has been scored once. This means that the cross-validation allows us to assign a prediction to each person, allowing us to compute the performance metrics using the reconstructed scores.

Special variants of k-fold cross-validation include:

- **Stratified K-Fold**: Maintains the same class distribution in each fold (useful for imbalanced datasets)
- **Leave-one-out cross-validation**: Sets k equal to the number of data points (useful for very small datasets)
- **Time series cross-validation**: The folds are defined using a rolling window for time-based data.

[![Time series cross-validation](/images/blog/cross-validation-blog/time_cv.png)](/images/blog/cross-validation-blog/time_cv.png)

## The benefits of cross-validation

From the previous section, it is clear that CV implies more computation than training a single model. However, cross-validation offers several crucial advantages that outweigh the added costs and complexity, making it a mandatory step in the training process. Let's explore the key benefits that make this technique so valuable.

### Preventing overfitting

One of the primary benefits of cross-validation is its ability to help prevent overfitting. When a model learns the training data too well, it starts to capture noise and specific patterns that don't generalize to new data. Overfitting leads to poor performance in real-world applications. In the charts below, we compare the effects of three different model architectures on the same dataset:

1. The first model is too simplistic to capture the trend in the data.
2. The third one tries too hard to capture every data point and adds a lot of noise between them.
3. The second model does a good job of capturing the overall trend in the data without necessarily trying to capture small variations in the location of the data points.

[![Underfitting vs overfitting curves](/images/blog/cross-validation-blog/blog_3_curves.png)](/images/blog/cross-validation-blog/blog_3_curves.png)

Cross-validation helps detect overfitting by evaluating the model's performance on multiple subsets of the data. If a model performs significantly better on the training data than on the validation sets, it's a clear indicator of overfitting. This early detection allows data scientists to make necessary adjustments to the model's complexity, feature selection, or hyperparameters before deploying it to production.

If we compute our model performance (ROC, accuracy) on the training data, the third model is the best option. However, when using CV, the second model performs better.

### Computing reliable performance metrics

Traditional train-test splits can sometimes provide misleading performance metrics due to the random nature of the split. Cross-validation addresses this limitation by:

- Providing multiple performance estimates using different data splits
- Enabling the calculation of confidence intervals for performance metrics
- Reducing the variance in performance estimation
- Offering a more robust assessment of model stability

These multiple evaluations give us a clearer picture of how our model is likely to perform in real-world scenarios, helping us make more informed decisions about model selection and deployment.

The content of our outcome report and UI relies entirely on the cross-validation but two charts explicitly depict results per fold. Our “folded ROC” plot shows the ROC curve for each fold. This is useful to ensure the model is stable (i.e. the results across folds are nearly identical).

[![Folded ROC curve](/images/blog/cross-validation-blog/Folded_ROC.png)](/images/blog/cross-validation-blog/Folded_ROC.png)

The lift chart illustrates the volatility of lift, particularly for those with higher scores. This information is crucial for lead generation models, which aim to identify potential customers. Faraday assigns a score to each individual within a specified population, and the client uses only the top X% of scorers.

[![Lift curve](/images/blog/cross-validation-blog/lift_curve.png)](/images/blog/cross-validation-blog/lift_curve.png)

## Technical dive

### Formal definitions

**Dataset definition**  

 - Let $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ be our dataset where:  
  - $x_i \in \mathcal{X} \subseteq \mathbb{R}^d$ are feature vectors for sample $i$  
  - $y_i \in \mathcal{Y}$ is the label for sample $i$  
  - $n$ is the number of samples  
 - Define $\mathcal{P}_{XY}$ as the true joint distribution of $(X,Y)$  


**Model space**   
 - Let $\mathcal{F} = \{f_\theta: \mathcal{X} \rightarrow \mathcal{Y} | \theta \in \Theta\}$ be the hypothesis space  
  - $\theta$ represents the model parameters  
  - $\Theta$ is the parameter space  

**Loss function**  
 - Define $\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}_{\geq 0}$ as the loss function  
 - The expected risk is:  
```math
R(f) = \mathbb{E}_{(X,Y)\sim\mathcal{P}_{XY}}[\mathcal{L}(f(X),Y)]
```
$R(f)$ is interpreted as the error of the model $f$ on the dataset.  
 - The empirical risk on dataset $\mathcal{D}$ is the average loss over $\mathcal{D}$:  
```math
\hat{R}_{\mathcal{D}}(f) = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(f(x_i),y_i)
```
**K-Fold partitioning**  
 - Partition indices $\{1,\ldots,n\}$ into $k$ disjoint subsets: $\{\mathcal{I}_1,\ldots,\mathcal{I}_k\}$  
 - For each fold $j$:  
   - $\mathcal{D}_{val}^{(j)} = \{(x_i,y_i) | i \in \mathcal{I}_j\}$ (validation set)  
   - $\mathcal{D}_{train}^{(j)} = \{(x_i,y_i) | i \notin \mathcal{I}_j\}$ (training set)  


**Cross-validation estimator**  
The k-fold CV estimate of the expected risk is:  
```math
CV_{(k)}(f) = \frac{1}{k}\sum_{j=1}^k \hat{R}_{\mathcal{D}_{val}^{(j)}}(f_{\theta_j})
```
   where $f_{\theta_j}$ is trained on $\mathcal{D}_{train}^{(j)}$  


### Statistical properties

**Bias of CV estimator**  
The bias of the k-fold CV estimator can be expressed as:  
```math
Bias(CV_{(k)}) = \mathbb{E}[CV_{(k)}] - R(f^*)
```
   where $f^*$ is the true optimal function and $\mathbb{E}[CV_{(k)}]$ is the expected risk over all the folds.  

**Variance decomposition**  
The variance can be decomposed into:  
```math
Var(CV_{(k)}) = \frac{1}{k^2}\sum_{j=1}^k Var(\hat{R}_{\mathcal{D}_{val}^{(j)}}) + \frac{2}{k^2}\sum_{i<j}Cov(\hat{R}_{\mathcal{D}_{val}^{(i)}}, \hat{R}_{\mathcal{D}_{val}^{(j)}})
```
Where $Var(\hat{R}_{\mathcal{D}_{val}^{(j)}})$ is the variance of the risk estimate over a fold. It does not refer to the variance of the risk itself (which is just a scalar value). Instead, it refers to the variance of the estimated risk based on multiple possible samples of the validation set within that fold.  

Therefore, as k increases, the overall variance of the cross-validation risk decreases leading to a less volatile estimation of the model performance post-deployment. 

# Holdout test

### What is it?

A "holdout test" can refer to any process where a model is being evaluated on unseen data (e.g. cross-validation above). At Faraday, a holdout test specifically refers to the process of choosing a date in the past $t_0$, fitting a propensity model with data prior to $t_0$, and evaluating the model on data after $t_0$.

On one hand, a holdout test is just a special case of time series cross-validation with 1 fold, but as we will see there are a few nuances to consider when configuring a holdout test and analyzing the results to inform a business decision.

### Configuring a holdout test in the Faraday platform

A typical scenario where a holdout test can be used is lead scoring. We will walk through how to run a lead scoring holdout test in the Faraday platform, but the same principles apply to a holdout test for other use-cases (e.g. predicting churn).

#### What is needed for a holdout test?

To configure a holdout test a user needs lead and conversion records with PII, dates, and a way to distinguish between the two groups. The setup for this can vary, but perhaps the simplest form is a single table that includes the following:

- PII columns
- a `lead_created` date that captures when a record became a lead
- a `conversion_date` that captures when that lead record became a conversion

NOTE: In our working example our conversions are called `tier_1_leads`.

In addition to the raw data, a user needs to have some prior knowledge of the typical attribution window for conversion which we will discuss in more detail when configuring the cohorts.

#### Datasets and events

Given a dataset described above, we need to configure the PII as one or more identity sets and two event streams, one for leads and one for conversions.

Configuring the leads event can be as simple as mapping a single date column  
[![Lead event](/images/blog/cross-validation-blog/lead_event.png)](/images/blog/cross-validation-blog/lead_event.png)

Similarly for the `tier_1_leads` event  
[![Attainment event](/images/blog/cross-validation-blog/attainment_event.png)](/images/blog/cross-validation-blog/attainment_event.png)

At this point our resource graph has a dataset table that emits two event streams  
[![Datasets graph](/images/blog/cross-validation-blog/datasets_graph.png)](/images/blog/cross-validation-blog/datasets_graph.png)


#### Cohorts

Next, we need to configure groups of people (cohorts) that partition the data into a training and validation set (recall this is time series cross-validation with 1 fold).

The inputs to determine this are:  
[![XYZ categories](/images/blog/cross-validation-blog/xyz.png)](/images/blog/cross-validation-blog/xyz.png)

[![XYZ description](/images/blog/cross-validation-blog/xyz_descriptions.png)](/images/blog/cross-validation-blog/xyz_descriptions.png)

The "prior period" is the training set for the holdout test, and the "holdout period" is the validation set. The amount of data available can influence how long/short these time windows are, but in general, we want to have enough data in the prior period to train a model and enough data in the holdout period to be confident in the results. For example, it would be bad to have zero conversions in the holdout period.

The attribution window is the time it takes for a typical lead to convert. Depending on the situation, this could be a few days to months.

The data lag is the time delta between today and the most recent date of any lead or conversion.

Once these have been determined we can create three cohorts

- prior leads: all records that became leads in the prior period
- recent leads: all records that became leads after the prior period
- conversions: all conversions (`tier_1_leads` in the working example) without any date filtering

Prior leads example configuration  
[![Prior lead configuration](/images/blog/cross-validation-blog/prior_leads_config.png)](/images/blog/cross-validation-blog/prior_leads_config.png)

Recent leads example configuration  
[![Recent lead configuration](/images/blog/cross-validation-blog/recent_leads_config.png)](/images/blog/cross-validation-blog/recent_leads_config.png)
 
Our resource graph now looks like  
[![Cohort graph](/images/blog/cross-validation-blog/cohorts_graph.png)](/images/blog/cross-validation-blog/cohorts_graph.png)

#### Outcome

Once cohorts are configured it is time to build a model and get some output. In Faraday, a propensity model is defined by an "outcome" which takes two cohorts as an input.  

[![Outcome configuration](/images/blog/cross-validation-blog/outcome_config.png)](/images/blog/cross-validation-blog/outcome_config.png)

Note that using the prior leads from the prior period implicitly restricts the conversions (tier 1 leads) to the prior period.

#### Pipeline

A pipeline applies an outcome (propensity model) to a group of people specified by a cohort. In this case we want to apply the outcome trained using data from the prior period to score recent leads records. Additionally, we want to have conversion information available in the output to analyze performance.  

[![Scope configuration](/images/blog/cross-validation-blog/scope_config.png)](/images/blog/cross-validation-blog/scope_config.png)

Here is the full resource graph  
[![Resource graph](/images/blog/cross-validation-blog/resource_graph.png)](/images/blog/cross-validation-blog/resource_graph.png)

This pipeline produces a table whose rows correspond to the recent lead records and whose columns include:  
[![Payload columns](/images/blog/cross-validation-blog/payload_cols.png)](/images/blog/cross-validation-blog/payload_cols.png)

Any of the `fdy_outcome_` columns can be used together with the `fdy_cohort_*_member` column (which is true when a row is a conversion) to measure model performance.

### Analyzing results

The output from the above pipeline can be used for custom analysis as desired by the user. One example is the automated reporting that Faraday produces to analyze holdout results and potential ROI.

#### Holdout results

First, we look at the internal performance metrics of the model built using the prior period.  

[![Training data used](/images/blog/cross-validation-blog/training_data_used.png)](/images/blog/cross-validation-blog/training_data_used.png)

These metrics are based on cross-validation described above and distilled down into a single plot.  

[![Model results](/images/blog/cross-validation-blog/model_results.png)](/images/blog/cross-validation-blog/model_results.png)

Next, we look at the pipeline output to see how the model performed on the recent leads.  

[![Data used for holdout test](/images/blog/cross-validation-blog/data_used_for_holdout_test.png)](/images/blog/cross-validation-blog/data_used_for_holdout_test.png)

Again, we can produce conversion rates by decile. 

[![Holdout test results](/images/blog/cross-validation-blog/holdout_test_results.png)](/images/blog/cross-validation-blog/holdout_test_results.png)

The last step of the holdout analysis is to compare the observed results from the holdout test with the model performance computed at training time via cross-validation.  

[![Comparing model report to test results](/images/blog/cross-validation-blog/comparing_test_results.png)](/images/blog/cross-validation-blog/comparing_test_results.png)
 
If these results are comparable, then it gives the user confidence that the performance metrics observed at training time are reflective of the model performance moving forward.

#### ROI calculation

When additional business context is known, the holdout test results can also be used to estimate ROI. For example  

[![Lead rejection ROI calculator](/images/blog/cross-validation-blog/roi_calculator.png)](/images/blog/cross-validation-blog/roi_calculator.png)

## Conclusion

Cross-validation can be used to prevent overfitting and evaluate a model's performance at training time. A holdout test is just a special case of cross-validation in which a single fold is determined by a point in time. Combining these two approaches with business assumptions enables the user to deploy models confidently, with a sense of expected performance and insight into expected ROI.

Faraday uses cross-validation to evaluate model performance, ensuring accuracy and value by training on multiple data subsets and validating on others. This allows us to make informed decisions and deliver reliable ROI for our clients.

How Faraday validates our predictive models

As businesses race to integrate AI into their software, many turn to first-party data modeling techniques like [RFM (recency, frequency, monetary value)](https://en.wikipedia.org/wiki/RFM_$market_research$) to build predictive models. These models typically rely on user interactions, including website navigation, email engagement, and purchase behavior to anticipate action. But first-party models alone have significant limitations, lacking the depth, speed-to-deployment, and predictive accuracy that businesses need to get ahead in today’s competitive market. 

Faraday’s models solve these problems by integrating rich third-party data from the [Faraday Identity Graph (FIG)](https://faraday.ai/blog/unlocking-the-power-fig), out of the box. The FIG enhances our models by integrating high-quality, consented offline insights into your first-party data, creating a more complete and actionable dataset. By enriching your existing data with verified external attributes—such as demographics, lifestyle indicators, and purchasing behaviors—the FIG fills in gaps, improves predictive accuracy, and speeds up deployment to help businesses reach the right audiences at the right time with the right product.

## Why first-party data alone falls short

First-party data has its advantages, but it also has critical limitations that can hinder AI-driven personalization and prediction.

First-party data is information that businesses collect directly from their customers and audience through owned channels. This includes website activity, purchase history, and email engagement. Because it comes straight from the source, first-party data is highly accurate, privacy-compliant, and essential for personalized marketing and predictive modeling. However, it can be limited in scope, which is why many businesses enhance it with external data for deeper insights.

One major challenge to overcome with first-party data alone is the [cold start problem](https://www.sciencedirect.com/topics/computer-science/cold-start-problem). When a new customer visits your business, there is no past activity to analyze, making it impossible to generate useful predictions without supplemental third-party data. A first-party data-based model will most likely only improve over time as the customer interacts with the brand, but this delay can mean lost opportunities, particularly in industries where real-time data is crucial like financial services.

Even as first-party data accumulates, it often remains noisy and incomplete. Activity data is unpredictable, with various factors influencing when a user engages with your site or makes a purchase, meaning first-party models must quickly adjust to these fluctuations. External factors like age or income level are also essential for understanding customer behavior, but they’re typically not captured in first-party data. Additionally, first-party data only reflects [activity, not affinity](https://faraday.ai/blog/why-first-party-data-and-rfm-is-not-enough). For instance, knowing that someone purchased a pillow tells you little about their broader lifestyle, financial capacity, or long-term interests.

## How Faraday enriches customers’ datasets to power our models

Our data enrichment process uses identity resolution to securely match first-party identifiers—such as email addresses or hashed customer IDs—with the FIG’s extensive dataset before they are fed into our models. This process ensures a seamless connection between your existing data and additional insights, following three key steps:

1. **Secure matching** – Faraday uses identity resolution techniques to match first-party data with offline records, ensuring privacy and compliance along the way.  
2. **Data enrichment** – Once linked, Faraday appends high-quality demographic, behavioral, and lifestyle attributes to each record.  
3. **Model integration** – These enriched profiles are seamlessly fed into our AI models, enabling more accurate predictions and personalized customer experiences. 

[![Data-appends diagram](/images/blog/first-party-merged-w-fig.png)](/images/blog/first-party-merged-w-fig.png)


## Powering better AI with enriched data

Our models’ ability to access the **Faraday Identity Graph (FIG)** solves numerous problems experienced by first-party-only models. Here are some examples:

### 1\. Solving the cold start problem

With the FIG, even first-time visitors are no longer unknowns. By matching first-party identifiers to offline, third-party data, Faraday provides insights into likely interests, purchasing power, and household composition, allowing our models to make informed predictions from the very first interaction.

### 2\. Boosting model stability and accuracy

AI models trained solely on first-party data must constantly adapt to new patterns in an ever-changing environment. The FIG introduces stable, high-quality attributes, reducing reliance on transient, high-variance signals like click-through rates. This results in more durable models that perform consistently over time.

### 3\. Enhancing AI with affinity data

Instead of just looking at past transactions, the FIG integrates data that reflects who customers are, not just what they’ve done. This includes household income, education level, homeownership status, and even media consumption habits.

## The best of both worlds

Faraday’s FIG takes you data to the next level. By seamlessly integrating high-quality, consented third-party insights with your existing first-party information, Faraday empowers you to access more accurate and predictive models through our platform. With this rich array of demographic, lifestyle, and behavioral attributes, you can unlock deeper insights into your customers and make smarter, data-driven decisions. The Faraday Identity Graph goes beyond the limitations of first-party data alone, giving you a clearer, more holistic view of your audience.

Want to see how the FIG can supercharge your AI models? **Let’s talk.**

The Faraday Identity Graph (FIG) allows users to create more accurate, stable, and predictive AI models within our platform with deeper insights into customer behavior and preferences.

Creating better predictive models with the Faraday Identity Graph

In the world of predictive analytics, data is everything. The more accurate, comprehensive, and up-to-date your data is, the better your models perform. That’s where the [**Faraday Identity Graph (FIG)**](https://faraday.ai/docs/abstractions/traits) comes in—a massive, carefully curated dataset designed to help businesses make smarter decisions with ease.

## What is the Faraday Identity Graph?

The Faraday Identity Graph is a powerful combination of multiple data sources, meticulously cleansed, de-duplicated, and feature-engineered to create one comprehensive dataset. It consists of [1,500 data attributes](https://faraday.ai/features/faraday-identity-graph) covering virtually all U.S. adult consumers and their households, offering a deep well of insights into demographic trends, property ownership, life events, financial indicators, and more.

And what’s even better is that it comes built-in with our services.

## So where does the data come from?

We partner with **best-in-breed data providers** like [Epsilon](https://www.epsilon.com/us) to compile a rich dataset that encompasses:

* **Demographic information** (age, education, income, etc.)  
* **Property details** (homeownership, property value, etc.)  
* **Life events** (moving, marriage, new child, etc.)  
* **Financial indicators** (credit attributes, equity, etc.)  
* **Buying patterns** (how they spend across dozens of categories)  
* **Shopping behavior** (e.g. in-store vs online)

By combining multiple premium data sources, we eliminate the need for businesses to juggle multiple vendors—saving them time, effort, and money.

## How frequently is the data updated?

The Faraday Identity Graph is updated regularly throughout the year as we receive new data from our vendors. The frequency of updates varies based on data type, but customers can rest assured that the data contained in our graph is as up to date and relevant as possible to ensure the best predictive outcomes.

## Can I buy the data directly?

Yes\! We offer direct data licensing, allowing businesses to purchase cleansed, consolidated, and competitively priced data. If you're looking for a variety of data sources, without the hassle of amalgamation and the time and expense of working with a range of vendors, we should talk—it’s likely we can save you all that time and money by bundling everything you need into one seamless package, and since we already made that package, you could be accessing that data tomorrow\!

## How can businesses use this data?

Our clients use the Faraday Identity Graph in a variety of ways but here are two core use cases:

* **Data appends:** Businesses can purchase specific data attributes to enhance their own internal models and processes.  
* **AI agents:** We can combine our data with your first-party data to build agents that answer key questions, such as:  
  * **Where are your best customers?** Read about our [market opportunity analysis service](https://faraday.ai/docs/abstractions/MOA).  
  * **Who is most likely to buy?** Check out our [lead scoring quick start guide](https://faraday.ai/docs/quickstarts/leads/lead-scoring).  
  * **What are they most likely to purchase?** Dive into our [product recommender](https://faraday.ai/docs/abstractions/recommenders) system, which predicts the best products or services for each individual.

Whether you're looking for raw data or full-service predictive modeling, Faraday makes it easy to harness the power of data-driven decision-making.

## The Faraday advantage

What makes this data even more valuable is access to our team of forward-deployed data science experts. Our team works alongside you to ensure that you are accessing not just the highest quality data, but the right data for your specific needs. With the addition of this service, whether you're looking for raw data or full-service predictive modeling, Faraday makes it easy to harness the power of data-driven decision-making. 

Additionally, Faraday delivers a broad suite of other services like real-time scoring, giving you instant access to dynamic, data-driven insights that enable rapid decision-making and model explainability, ensuring you can trace every prediction back to its data-driven roots—this transparency enhances model trust and guides smarter adjustments. 

What's more, with immediate access to this rich data from day one, you can avoid the cold start problem, quickly deploying predictive models without sacrificing accuracy. Lastly, Faraday’s flexible API integration allows you to seamlessly incorporate our data into your existing systems, providing a scalable solution that evolves alongside your needs and analytics workflows.

## Questions?

Sounds pretty great right? We think so too. If you want to learn more, reach out to talk to our sales reps or even our data team to learn how Faraday’s data can push your business to the next level\!

The Faraday Identity Graph (FIG) is a massive, meticulously curated dataset of 1,500 attributes covering virtually all U.S. adults, enabling businesses to enhance their models, streamline data sourcing, and make smarter decisions with ease.

Unlocking the power of the Faraday Identity Graph (FIG)

If you are reading this, you are probably talking to Faraday about a potential contract and have a healthy amount of skepticism as to whether we can actually do what we say we are going to do.

And here’s the thing: you’re right to be skeptical. With tight budgets and increasing pressure to show ROI, you can't afford to bet on unproven solutions or spend countless cycles figuring out if a tool is really going to live up to the promise. You need a way to separate legitimate predictive tools from the noise.

Every week, we get to talk to dozens of CMOs, CTOs, & CEOs across industries who are trying to discern the same thing: which tools are legitimate and which ones are not. As a leader of a high profile brand, you are inundated with promises of predictive insights and transformation, but you need more than just promises – you need guarantees.

That's where our holdout evaluation comes in. It's not a proprietary methodology we invented – it's an industry-standard approach used by data scientists to validate predictive models.

**What makes us different is simple: we run this validation before you sign any contract.** Because frankly, if we can't prove we'll deliver real results for your business, we don't want to waste your time.

Below we’ve outlined what we are doing “behind the scenes” and how you can evaluate if we will add the value you need. 

## What is a holdout evaluation?

A holdout evaluation is a method for validating predictive models by comparing their predictions to real-world outcomes. We **set aside a portion of recent conversion data**—data the model has never seen before—and use it as an independent benchmark, against which predictions can be compared.

This “holdout set” acts as a real-world proving ground. It is untouched until after the predictive model has been trained. 

Meanwhile, the rest of your historical data is combined with rich third-party insights from the [Faraday Identity Graph](https://faraday.ai/docs/abstractions/traits) ([which includes 1,500 attributes on 240M+ U.S. consumers and their households](https://faraday.ai/features/faraday-identity-graph)). We use this combined data to train the new predictive model.

Training a predictive model simply means finding patterns in conversion behavior. In other words, what can we learn from your previous customers to identify future customer behavior? Once trained, the model generates predictions—scoring leads based on their likelihood to convert and only after these predictions are made do we compare them to the **real conversion rates** in the holdout set.

If the highest-scoring leads actually convert at higher rates than lower-scoring ones, the model is validated as accurate and reliable. 

Holdout evaluations ensure predictive models aren’t just theoretical—they work in the real world, using real data, to deliver real results.

## How holdout validation works at Faraday

We've streamlined this validation process to be transparent and straightforward:

1. ![customer conversion data](/images/blog/holdout_testing_diag/holdout_testing_diag-01.png)

   #### Prepare your data and set aside a holdout group

   We start with your recent [lead conversion data](https://faraday.ai/blog/first-party-vs-third-party-data-marketing). Before any processing occurs, we reserve a portion of your most recent leads as a "holdout set"—data that the model will never see during training. This ensures we have an independent benchmark for evaluating predictions.

2. ![merged with FIG](/images/blog/holdout_testing_diag/holdout_testing_diag-02.png)

   #### Combine data with Faraday Identity Graph

   With the holdout set reserved, we then combine remaining data with Faraday’s [built-in, rich  consumer data](https://faraday.ai/features/faraday-identity-graph) sourced from the Faraday Identity Graph. This expanded dataset provides a deeper view into key predictive signals, like property and demographic information, life events, buying habits, and more.

3. ![model is trained and predictions begin](/images/blog/holdout_testing_diag/holdout_testing_diag-03.png)

   #### Train the model and begin predictions

   Using this enriched dataset (excluding the holdout set), we train a predictive model to identify patterns in conversion behavior—learning from traits, behaviors, and other signals.

   Once the model is trained, it scores each lead based on its likelihood to convert, generating a ranked list of opportunities—from high-value leads to those less likely to convert.

4. ![model is validated](/images/blog/holdout_testing_diag/holdout_testing_diag-04.png)

   #### Compare predictions to reality to validate model

   Finally, we put the model to the test: we compare its predictions against the real outcomes in that untouched holdout set. If the highest-scoring leads don't convert at significantly higher rates, we go back to the drawing board.


## How to know if a holdout validation was successful

A holdout validation is considered successful when the following criteria are met:

1. **Clear correlation between predicted probability and actual conversion rates**  
   The model’s predicted likelihood to convert should directly align with the actual conversion rates. If the predictions are accurate, high-probability leads should convert at a noticeably higher rate than lower-probability ones.

2. **High-scoring leads convert at significantly higher rates**  
   The top-scoring leads should consistently perform better than those ranked lower. This demonstrates that the model is identifying truly valuable leads.

3. **Consistent performance across different customer segments**  
   The model should show reliable results across various segments—whether you’re targeting different demographics, regions, or other customer profiles. If the model works well across a broad set of data, it proves its reliability and scalability.

When a model meets these criteria, it’s ready for deployment—whether that means optimizing [lead buying](https://faraday.ai/docs/how-to/use-cases/leads), refining your [direct mail strategy](https://faraday.ai/templates/lead-suppression), or enhancing your email [personalization](https://faraday.ai/templates/personalization) approach.

## Questions you should ask us

We want you to push us on what matters most for your business. Here are key questions to evaluate if we're the right fit:

1. **How does this model's performance compare to the lift you are seeing with  similar businesses in our industry?**

   While we can’t share client specifics, we can provide insight into broader trends and expected performance benchmarks.

   → **Why this matters:** Understanding industry benchmarks helps you gauge whether predictive modeling is truly a competitive advantage for your business.

2. **How much historical data do we need for reliable predictions?**

   The answer varies by business model and conversion volume. We’ll help determine whether you have enough data—or how to supplement it if needed.

   → **Why this matters:** Predictive models work best when trained on strong data. Knowing the threshold for reliability ensures you're making data-driven decisions with confidence.

3. **What happens if the model doesn't perform as expected?**

   We have a structured iteration process to refine models when necessary, ensuring they reach meaningful predictive power.

   → **Why this matters:** No model is perfect on the first run. Understanding the process for continuous improvement ensures long-term success.

4. **How will this integrate with our existing marketing stack?**

   We’ll provide technical specs, discuss implementation timelines, and outline the resources needed to get up and running.

   → **Why this matters:** Seamless integration means faster time-to-value and less friction in your workflow.


## Why holdout evaluations work

The businesses who win this year will be those who successfully incorporate predictive modeling into their core workflows. It's no longer a nice-to-have – it's a need-to-have. But we know you can't afford to make decisions based on hunches or unproven tools. That's why we're eager to validate our value up front. 

At the end of the day, three things matter to us:

1. **Confidence before commitment:**  
   You see proof before you commit

2. **Higher marketing efficiency:** 
   Your marketing dollars go to the leads most likely to convert

3. **Stronger sales performance:** 
   Your sales team focuses on opportunities worth their time

If we can't demonstrate value, we'll be the first to tell you we're not the right fit. But if we can – and we usually do – we want you to know the exact savings or additional revenue you can expect to see over the course of the next year with Faraday. 

Interested in generating value you can trust? Reach out to learn more about what we can do for your business\!

Faraday uses holdout testing to validate predictive models before you commit, ensuring they deliver real, measurable results.

Predictive modeling generates tangible value, and we can prove it

A [ping post](https://activeprospect.com/glossary/ping-post/) is a real time lead distribution method where lead sellers send basic lead details (the "**ping**") to potential buyers, who then evaluate the lead based on predefined criteria. If a buyer decides to bid on the lead, they receive the full details (the "**post**") and complete the purchase. This system allows for fast, competitive bidding, commonly used in industries like home services, financial services, and insurance.

While this method offers efficiency, it also introduces challenges. Many businesses end up paying for leads that don’t convert, leading to wasted budget and ineffective outreach efforts. The ability to filter and score leads before purchasing them is crucial to maintaining a high return on investment (ROI) and optimizing marketing spend.

## How we use Ping Posts at Faraday

The challenge with ping-post lead buying is that companies often spend money on leads that don’t convert. At Faraday, we integrate with ActiveProspect’s [LeadConduit](https://activeprospect.com/leadconduit/) to enhance this process with predictive scoring. By applying machine learning at the ping stage, brands can bid only on high-converting prospects, filtering out low-quality leads in real time. This optimization reduces wasted spend and improves overall campaign efficiency.

Predictive scoring works by analyzing historical data, behavioral signals, and demographic attributes to assess a lead’s likelihood of converting. Thanks to the built-in  [Faraday Identity Graph](https://app.faraday.ai/traits) ([which comes preloaded with 1500+ attributes on nearly every US consumer and their households](https://faraday.ai/features/faraday-identity-graph)), our models score each lead against rich third-party data before a bid is placed instead of relying on limited seller-provided information. This creates the opportunity for businesses to make data-driven decisions in real time, ensuring they focus only on high-quality opportunities.

For example, an insurance provider purchasing thousands of leads daily might bid on any lead that meets basic criteria, but many of those leads never convert—driving up acquisition costs. If each lead costs \$10 and converts at a rate of 1 in 5, their effective customer acquisition cost (CAC) would be \$50. But not all leads are equal. By rejecting the bottom 40%—the least likely to convert—they could improve their conversion rate significantly, lowering their CAC to just $30.

In this scenario, instead of wasting budget on low-quality leads, they could acquire more customers for the same spend. With predictive lead scoring, they could make these decisions in real time, ensuring they only invest in leads with real potential.

## Even without TCPA 1:1 consent rule, better targeting still matters

[While the expected TCPA 1:1 consent rule did not go into effect earlier this year](https://faraday.ai/blog/tcpa_1-1_vacated), that doesn’t mean businesses can ignore lead quality or continue to enact spray and pray targeting in 2025\. Just because you’re not being forced to change doesn’t mean you shouldn’t. Rising acquisition costs and increasing competition make it critical to invest in leads that are both high-quality and likely to convert. Faraday’s real-time predictive scoring ensures that brands maximize their ROI while staying ahead of the curve. By leveraging AI-driven insights, lead buyers can refine their targeting strategies and remain competitive in an evolving marketplace.

Additionally, compliance concerns aren’t going away. Even without the immediate pressure of TCPA amendments, consumer privacy laws and regulations continue to evolve, and businesses that proactively refine their lead acquisition strategies will be better prepared for future changes. More rigorous lead assessment doesn’t just improve efficiency—it also reduces the risk of compliance issues and enhances the overall customer experience by ensuring outreach efforts are more relevant and targeted.

## Looking ahead

The lead generation landscape will continue to evolve, and companies that prioritize precision over volume will have the advantage. Rather than buying as many leads as possible and hoping some might convert, businesses need to shift toward smarter, data-driven strategies that optimize every dollar spent.

Faraday’s predictive lead scoring is a powerful tool for staying ahead, allowing businesses to purchase leads with confidence and efficiency. Whether regulatory shifts happen or not, the need for better targeting and smarter decision-making isn’t going away.

Ping posts are a real time lead distribution method where basic lead details are sent to potential buyers, who then decide whether to bid before receiving the full lead information.

So what is a Ping Post anyway?

Using machine learning to identify customer personas in your data can help improve your understanding of your customer base, and with the right infrastructure, deliver personalized experiences at a greater scale.

In a nutshell, the process entails using an algorithm to segment your customer data into distinct groups, or personas. Faraday uses a time-tested method called k-means clustering.

## What is k-means clustering?

K-means clustering is an unsupervised clustering algorithm that was first introduced in 1957. An unsupervised algorithm does not require the data to be labeled in order to train the model. This is an important feature of the k-means algorithm because it allows for the discovery of subgroups within the data without any previous assumptions about possible groupings — in this case, personas.

## How does k-means clustering work?

When presented with a data set, the k-means algorithm first introduces random center points around which each data point will eventually be clustered (circles, Fig. 1). Then each data point is assigned to the group of the center to which they are closest.
This happens by calculating the distance from a data point to each cluster center using an equation known as the Euclidean distance measure. For two dimensions, the Euclidean distance is represented by the diagonal line connecting the point to each of the cluster centers. The point is then assigned to the cluster center with the shortest diagonal line (Fig. 2).

Once each point has been assigned to the randomly chosen cluster centers, the centers then "update," moving to the center of the group by recalculating the mean (average) distances to all the points in the group (Fig. 3).

After the cluster centers have updated to their new positions, each point is reassigned to a new group using the Euclidean distance measure again. Most of the points will remain in their original groups, but some points will be assigned to a new group (Fig. 4). So, then the centers must be updated again to be at the center of the new group. This pattern repeats until the centers no longer move.

[![K-means diagram](/images/blog/k-means_diagram.png)](/images/blog/k-means_diagram.png)

## Using k-means clustering for customer personas

All of the above is how k-means clustering worked back in 1957. Since then there have been upgrades allowing us to use something called categorical features. At Faraday, we create persona groups using a mix of these features.

As far as algorithms go, k-means is rather straightforward, but has one major downfall: you have to tell it how many groups to look for, and it will always succeed in finding that many groups. That means if you tell the algorithm to look for 500 groups, it will find them — even if there is no reason for there to be 500 distinct groups within the data.

If you're a marketer, you probably don't want to create 500 unique ads for every campaign. But should you make four? Or six? Or ten?

## How to find the optimal number of personas using k-means clustering

We target the optimal number of groups hidden within the data, rather than guessing and hoping we are right. We’ll share a few methods we use to find the optimal number of personas for our users.

### The elbow method: finding diminishing returns

The most common metric for measuring the success of k-means clustering is called the ratio of sum of squares (rss). At Faraday, we determine the optimal number of personas by comparing the rss for k-means models with two to fourteen personas. We employ the elbow method, which shows the cutoff after which we see diminishing returns for increasing the number of personas. For most clients, the optimal number of distinct personas is usually between four and seven.

**Going beyond the elbow method**

The elbow method is simple, but it has a downside: sometimes, there just isn't a clear single point that is optimal. Maybe three personas could be optimal, maybe six, and we have to intervene based on our client’s business needs to choose one or the other as the optimal number of personas.

Additionally, while reducing variability within each group is an extremely useful way to measure k-means clustering success, it doesn't tell the whole story. We believe that an important aspect of clustered groups is stability, which isn't considered with the elbow method selection process. Our clients expect that their persona groups won't shift every time they convert a lead. This is why we are currently researching a new method for determining the optimal number of personas, which focuses on the stability of the persona group.

### The bootstrap method: focusing on stability

Our first step to assess the stability of k-means clustering involves resampling the client’s data using the bootstrap method. Like k-means clustering, bootstrapping has been around for a while – it was first [introduced in 1979 by Bradley Efron](https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full). The basic idea behind bootstrapping is to take data from the pool we have, with the ability to pick the same data point multiple times. This is a method known as sampling with replacement.

The bootstrap method provides us with a powerful tool to measure how much our desired clustering outcome could change with different data. We can get a good sense of how much the personas will change with the addition of new customers. Our goal is to minimize how much influence single customers, or small groups of customers, have on the resulting personas. We want personas that remain relatively unchanged by single customers. In other words, we want more stability.

## Pulling it all together

Once we have the results from bootstrapping, we use the ratio of sum of squares again to measure just how stable the personas are. What we’re looking for is a ratio close to 1, which would indicate that there is no variability within the personas.

The stability metric for determining the optimal number of personas shows promising results for the basic examples in this blog post. At Faraday, we’re actively working on applying this methodology to the more complex persona groupings our clients rely on. Stay tuned for more developments with respect to improving our personas methodology.

Learn how machine learning is used to identify distinct clusters—or personas—in your customer data.