AI is everywhere. Businesses of all kinds are using it for every use case imaginable. From creating copy for their emails or website, to generating graphics for their blog, to [predicting customer behavior](https://faraday.ai/features/customer-behavior-predictions) and beyond. While using AI for the more innocuous cases like brainstorming ideas or generating art tends to leave you with the set-it-and-forget-it mentality, that’s not the case in the realm of predicting customer behavior. Here, [explainable AI](https://faraday.ai/blog/what-is-explainable-ai-why-is-it-important) is critical in helping non-data scientists understand what data your predictive models used to come to their conclusions, and whether or not you can trust them.

[![Graphic of dynamic prediction tenure scoring](/images/blog/transparent-reporting/transparent-reporting.png)](/images/blog/transparent-reporting/transparent-reporting.png)

For this reason, every model in Faraday comes with at-a-glance performance reporting so that you can quickly assess how strong your predictive model is. Beyond this, you can dig into things like [feature importances and directionality](https://faraday.ai/blog/faraday-model-reports) to see what individual features from the Faraday Identity Graph and [your own data](/blog/first-party-features) had the greatest impact in creating the predictive model, and even where specifically within those features they had such impact. Additionally, reporting on ROC and lift are included.

On top of this at-a-glance reporting, each model comes with a complete, in-depth technical report that digs into the nitty-gritty that data scientists love \- from analysis of training data to model calibration, cross-validation, and more. Further, Faraday strongly believes in [Responsible AI](https://faraday.ai/features/ai-safety-and-responsible-ai), which means that this technical report also includes [score explainability](https://faraday.ai/blog/faraday-score-explainability) and background on how Faraday came to the conclusion of its bias and fairness assessments.

Don’t _just trust us_ (or any other AI that you use\!) \- our reports speak for themselves.

[Sign up free](https://faraday.ai/signup).

From at-a-glance performance reporting to feature importances with directionality to in-depth technical reporting, Faraday helps you understand the models we build.

Transparent predictive model reporting with Faraday

**See our complete guide:** [What is the Faraday Identity Graph?](/blog/whats-in-faraday-identity-graph).

---

## What is identity resolution?

Identity resolution is the process of taking all of the files and databases of people data in your organization and figuring out "who is who" so that you can unify individual records into a single entity. For example, one database might have email, and another database might have street address; how do you connect them? Identity resolution is the answer.

Within identity resolution, there are two broad categories: **known identity** and **anonymous** (unknown) **identity**. Known identity is figuring out the email, phone number, and/or street address of someone. If you have known identity, you can find someone in the real world. Anonymous identity is defining some IP address, device ID, browser fingerprint, or cookie that represents a person without actually knowing who they are in the real world. In either case, you’re trying to get a join key or universal ID that lets you combine data from different sources across time.

How does identity resolution work? In a few different ways:

- You _explicitly_ collect known identifiers like email in web forms or addresses in orders/transactions.
- You _implicitly_ collect anonymous identifiers like IP addresses, cookies, as they browse.
- You stitch together known and anonymous identifiers when people click links in emails, log in, enter billing and shipping data, etc.
- You connect known identifiers with marketing data like age, gender, and household income using third parties like Faraday.
- You de-duplicate and "master" records: e.g. by saying that two different emails belong to the same person
- You upload unknown identifiers to ad networks and they resolve them into users on their platforms.

Finally, it's important to know the terms first- and third-party data. **First-party data** is data that you collect and own, and can include both known and anonymous identifiers. It also includes transaction amounts, order details, product details, etc. **Third-party data**, on the other hand, is data that is owned by others, but is available for specific uses. (You may also hear about “zero-party data,” which is a special kind of first-party data that users explicitly provide, such as by filling out surveys.)

When you unify first and third party data, you can get age, gender, household income, and other traits for a user so that you know both what they bought (first-party data) and who they are (third-party data). You’ll generally get the highest enrichment or match rate by starting with known identity, but it's possible to start with anonymous identity, too.

## Why does identity resolution matter?

If your business has more than just one useful database of information (which is highly likely), identity resolution is mandatory in order to properly understand not only who’s who within your data, but also how they’re engaging with your business. Doing so opens up a world of use cases that can be done efficiently because you’re able to group similar customers and leads together: [personalize their journeys](https://faraday.ai/templates/thematic-personalization), recommend first-and-[next best offer](https://faraday.ai/templates/next-best-offer), plan [retention campaigns](https://faraday.ai/templates/churn-scoring), and more.

For example, transaction data stored in back office systems can be combined with behavior observations in the marketing or product teams, with call logs from customer-facing teams, and so on. To illustrate this latter point, customer [churn prediction](https://faraday.ai/blog/churn-prevention-machine-learning) is often much stronger when it takes into account activity like messaging or calling.

Taking this a step further–when generative AI is available–identity resolution can help take unstructured data from call center transcriptions and combine it with structured data like transaction histories. This allows you to use sentiment analysis or entity extraction to create data attributes (i.e. predictive features) that can be combined with other first and third party features in predictive models, such as [lead scoring](https://faraday.ai/solutions/lead-management).

## Why identity resolution is difficult to implement

Identity resolution is important, but it's hard to implement. To start, the datasets involved need to have PII that can be compared. For example, maybe you have transactions with billing addresses included in one dataset, and views/clicks by email address in another. To resolve the identities in these two datasets, you’d need to purchase third-party email hashes connected to postal addresses so that both have billing address and email data, _then_ you’d need to ensure the data is standardized so that they can be compared.

The process is further complicated when you consider that people aren’t static–they move addresses, they have kids, their habits change–life goes on. You'll need to rent or purchase historical address data to make your graph complete. Your customers and leads change, and your data should be able to change with them.

## What does the identity resolution market look like?

There are a wide variety of identity resolution products on the market. Some are pre-integrated into the systems that use them, such as traditional "packaged" CDPs, purpose-built person-level AI infrastructure like Faraday, and others. These identity resolution systems primarily serve the purpose of the software they're integrated with, and offer the fastest time-to-value. Software in the master data management space like Reltio is dedicated to resolving and de-duping records for use by downstream systems.

There are also data warehouse-native identity resolution packages, such as Hightouch Identity Resolution or AWS Entity Resolution that run directly inside of a customer's cloud account or data warehouse. These provide standard functions for resolving identity that can be applied to any business, though they won't be immediately usable out of the box. TransUnion and other traditional data aggregators offer identity resolution packages on the Snowflake Marketplace that serve to connect first-party data to third-party identifiers and data. Finally, consultants like Deloitte will often build custom identity resolution projects on a per-client basis, if that client has previously purchased third-party data.

Due to competitive pressure from all of the data warehouse-native and standalone identity resolution providers, some software with built-in identity resolution like Faraday now allow data warehouse-native identity resolution too, which can be a good option that combines the low time-to-value with a flexible, proven base for other software and processes to be developed on top of.

## How Faraday’s identity resolution works

Faraday’s advanced identity resolution is built in, which means you don’t need to separately license any data or go through the laborious process of manually unifying it with your own to create distinct profiles of your customers and leads. All parts are included: from data and data unification that’s continuously up-to-date and responsibly sourced, to identity resolution and AI-powered customer predictions. Whether you’re interested in prioritizing your best leads and rejecting leads that won’t end up converting, seeing which customers are going to churn or finding their next best offer, Faraday has you covered from end to end.

To start, Faraday securely ingests your data through a [connection](https://faraday.ai/docs/abstractions/connections) to wherever your data is stored (or via CSV file), after which you describe your data to Faraday via [datasets](https://faraday.ai/docs/abstractions/datasets). In your datasets, you’ll tell Faraday what personally identifying information (PII) fields to look for (column mapping), as well as what [events](https://faraday.ai/docs/abstractions/events) take place in your data, which you’ll use to define your customer prediction objectives. Everything Faraday does is [fully API-enabled](https://faraday.ai/docs/reference), too.

With your data defined, we search the Faraday Identity Graph (FIG) for matching PII through fields like `email`, `first_name`, and `address`. Once a match is found, all data that you have on this individual is unified with and enriched by their data in FIG. From demographics, to financials, property data, and more, 1,400+ robust traits supplement your data on how they interact with your business with how they exist _outside_ of those interactions.

[![Graphic of Faraday identity resolution](/images/blog/identity-resolution/identity-resolution.png)](/images/blog/identity-resolution/identity-resolution.png)

The result is a unified, enriched customer profile that’s embedded directly into your applications and workflows–with customer predictions on top–for all of your customers and leads, ensuring that your teams have all of the information they need to engage effectively.

At the end of the day, identity resolution is critical to unlocking truly data-driven marketing and AI/ML projects. With it, all of your data comes together from its disparate sources to form unique identities for each individual. These identifiers cover their engagement with your business through visits, email addresses, transactions, etc, their marketing highlights like age and income, and any other traits that you have. With identity resolution, you can effectively predict which leads are most likely to convert and which you shouldn’t waste time and money on, which customers are most likely to churn, their buyer persona, what product or service to recommend next, and more.

Faraday handles the ML struggle so that you can focus on your downstream applications and workflows.

[Sign up free](https://faraday.ai/signup).

Faraday unifies your data with rich third-party data and applies advanced identity resolution techniques to reveal panoramic views of your customers and leads.

How Faraday unifies your data with the Faraday Identity Graph for advanced identity resolution

Predictive patterns change over time. When it comes to predicting customer behavior, it’s entirely likely that a person who’s been a lead for say, three months, is going to behave differently than someone who just signed up for your mailing list. The former has been receiving your email drip for a few weeks, is aware of your offerings, and probably has a decent idea of what they’re looking for. The latter is much closer to the top of the funnel, and is just getting to know your business better.

Why does this matter? Well, the person that just subscribed to your mailer might be less likely to convert compared to someone who’s been interacting with your business for 3 months - or they might not be! If the behavior “visited your site 3x” tends to be highly predictive of conversion, that’s only relevant to leads who’ve had the time in the funnel to actually visit your site more than once, so a different model should be used for fresh leads. Additionally, aggregated data is used to capture the situation of a person at the time of prediction. For example, has a new email been sent since we last produced a prediction?

With dynamic prediction, we create predictive models for several tenure bins, or periods of time, that someone has been in the eligibility cohort that you’re using for your prediction (i.e. their funnel stage). Then, we use the best possible model to make predictions on an individual based on that tenure bin. In the below example, you can see two strategies applied (A and B) across four tenure bins (0, 10, 20, and 30 days). In it, strategy A scored higher for the bins of 10 and 20 days, while B scored higher for 0 and 30 days, which means the optimal scoring combination is to use strategy A to score people that have been in the cohort for 0-10 days, strategy B for those in it for 11-20 days and 21-30 days, and A again for those in it for 30+ days.

[![Graphic of dynamic prediction tenure scoring](/images/blog/dynamic-prediction/dynamic-prediction-1.png)](/images/blog/dynamic-prediction/dynamic-prediction-1.png)

To illustrate this point further, below is an example of a pipeline that uses tenure bins to select the best model for each tenure group. In it, you can see the **pipeline population** group is broken down into three tenure bins, each of which has a “best” predictive model that’s used for scoring.

[![Graphic of dynamic prediction pipeline](/images/blog/dynamic-prediction/dynamic-prediction-2.png)](/images/blog/dynamic-prediction/dynamic-prediction-2.png)
_For further reading on Faraday model calibration and probability scores, check out [this blog post](/blog/faraday-probability-scores)._

In short, dynamic prediction enables us to provide scores that update with your data, ensuring accurate, powerful predictions for all stages of the customer lifecycle.

[Sign up free](https://faraday.ai/signup).

Predictive patterns change over time. Faraday uses dynamic predictions to choose the best model for you based on subject tenure.

How Faraday uses dynamic predictions to choose the best model for you

## What is a feature in machine learning?

In machine learning, a feature is a piece of data that’s used to train a predictive model. A feature can come in a variety of different formats, be it a static trait like age, whether or not someone’s a frequent home remodeler, or some value that’s related to the events taking place in your own data.

Once these features have been fed into the machine learning algorithms, predictive models begin learning from the data by looking for patterns to analyze, and the end result is predictions that help your business find your most likely-to-convert leads–and reject the least likely-to-convert ones–find more high-LTV customers, and more.

## Creating first-party features from your data

Once your customer and lead data is ingested through a [connection](https://faraday.ai/docs/abstractions/connections) to your data warehouse or a CSV upload, you’ll point Faraday toward the [events](https://faraday.ai/docs/abstractions/events) and [traits](https://faraday.ai/docs/abstractions/traits) within your data. While doing so, you’ll define when an event took place, its value, and any relevant event properties (e.g. the acquisition channel of the event).

[![Screenshot of an event stream's configuration in Faraday](/images/blog/first-party-features/event-features.png)](/images/blog/first-party-features/event-features.png)

Then, when you’re creating a prediction in Faraday, we’ll use these properties that you define to engineer features to be used alongside the [Faraday Identity Graph’s 1400+ vivid traits](https://faraday.ai/features/faraday-identity-graph). While analyzing the predictions you’ve created–like the likely-to-buy-again outcome below–you’ll find your first-party features with a sparkle next to them.

[![Screenshot of an outcome's feature importance in Faraday](/images/blog/first-party-features/outcome-features.png)](/images/blog/first-party-features/outcome-features.png)

In the above example, a first-party feature was created based on the date the last subscription box was purchased by a customer of this client’s in `last_box_purchased`, and another for the category of that box from an event property in `orders category` (orders being the event stream, category the event property defined within the stream). Lastly, a feature was created for the `first_box_assigned` trait–designating which category of subscription box the customer received on signup (kitchen, camping, etc).

These three features had a major impact on the scores of people in this outcome, and were highly predictive of the behavior the client was predicting for: likelihood of buying again.

You can read more about how individual features impact an outcome’s predictions in the data section of our [outcomes](https://faraday.ai/docs/abstractions/outcomes) docs.

[Sign up free](https://faraday.ai/signup).

Faraday automatically engineers machine learning features based on your first-party data to be used as predictors.

How Faraday creates first-party features from your data

**FOR IMMEDIATE RELEASE**

BURLINGTON, VT — Faraday, the leading infrastructure provider for customer behavior prediction technology, has announced the beta release of its newest prediction: Recommendations.

Users can employ this new prediction to determine which action among several each customer is most likely to take. For example, a retail brand could use Faraday Recommendations to personalize an email campaign, highlighting the “next best offer” each individual customer is most likely to buy. An insurance carrier could use this new technology to predict which insurance line each customer is most likely to adopt next in order to shape a more responsive customer journey.

Recommendation joins Faraday’s other two generally-available predictions—Outcomes, which gauge the probability of each individual exhibiting or not exhibiting a certain behavior, and Personas, which organize a group of customers or other individuals into thematically coherent subgroups—to round out its platform. A fourth, Forecast, which predicts how much an individual will spend with a given brand (also known as lifetime value or LTV), is available in “alpha” with a broader release aimed for later this year.

As with all Faraday predictions, Recommendations are powered by best-in-class AI algorithms combined with panoramic built-in consumer data—over 1,400 attributes illuminating nearly every adult consumer.

“At Faraday we believe that every customer experience should and will be predictive,” said Andy Rossmeissl, the company’s CEO. “Our mission is to predict every customer behavior, responsibly, and our new Recommendations product is a major step in delivering that mission.”

Bespoke Post, a leading e-commerce subscription brand, used an alpha release of Faraday’s Recommendations offering to improve the “add-on” bundling experience at checkout. When this feature was used, it delivered a 10% conversion rate lift compared to a control group.

Meriwest, a large credit union operating in the San Francisco Bay and Silicon Valley areas, also used Recommendations in alpha, this time to predict which financial service each member is likely to adopt next. The credit union plans to use these predictions to personalize email campaigns, with the goal of boosting conversion rates.

At the technical level, Faraday’s Recommendations product employs a variety of time-tested algorithms which work in concert to deliver high-confidence predictions. Building a recommendation “model” involves two stages. First, the system uses matrix factorization to implement a “collaborative filtering“ approach. Next, the system uses findings from the collaborative filtering model to inform a decision tree-based approach (random forest ensemble). This technique allows Faraday to make predictions about existing customers as well as brand-new leads or even unknown prospects.

“As an AI engineering project, this was an exciting challenge,” said Dr. Mike Musty, a Faraday data scientist, who was instrumental in designing the product. “Delivering self-service recommendations at scale required several novel optimizations, including our hybrid algorithmic approach.”

Existing and new Faraday clients looking to take advantage of the new Recommendations product while in beta should contact their account manager or file a support request.

## About Faraday

Faraday is an infrastructure platform that offers unified AI and data for predicting customer behavior. Based in Burlington, Vermont, the company makes billions of predictions every day for thousands of brands and their partners. Learn more at [faraday.ai](https://faraday.ai).

Faraday has announced the beta release of its new Recommendations product, which predicts the most likely action each customer will take, joining their existing Outcomes and Personas predictions. This new feature aims to enhance personalization in customer interactions and has already demonstrated success in improving conversion rates for early adopters.

Press release: Faraday launches Recommendations product into beta

**See our complete guide:** [Personalization tactics that actually drive conversions](/blog/personalization-tactics-that-drive-conversions).

---

[Personalization](/blog/predictive-personalization-machine-learning) is the holy grail of growth teams everywhere. At this point, everyone knows how important it is to speak to their customers and prospects like they’re real people, and not just orders and clickstreams. It’s often easier said than done, though, and brands with a significant amount of data and a large customer base may find themselves struggling to make heads or tails of how to organize it, or even know what data is worth focusing on.

## Challenges of personalization at scale

Doing personalization at scale right can provide incredible value to your brand, and has a major impact on how your customers see and interact with you. A daunting set of challenges is presented to most brands that try to tackle the issue themselves–let’s dive into a few of the most glaring challenges.

### Data, and a lot of it

To know more about–and how to speak to–your customers, you need to have data on them. Through the data you currently have, you’ll likely have some level of insight into the demographic, financial, and transactional background of your customer base as a whole. Though this data is strong, and gives a unique perspective due to it being exclusive to your brand, your customers exist outside of interacting with your brand. A healthy mix of [first-and-third-party data](/blog/first-party-vs-third-party-data-marketing) can provide additional layers of complexity and nuance to how you view your customers–and how you engage with them.

The problem is that you need a lot of strong data to personalize meaningfully, and the more data you collect, the harder it becomes to personalize at all.

### Finding the relevant data

Collecting the data is the first step to creating an impactful strategy for personalization at scale–but what do you do with it? Historically, data-savvy marketers have sifted through the terabytes of data they’ve collected to try and piece together what makes up their ideal customer, and what their key levers are. If your brand is thorough, this is a _ton_ of data to sift through, and the process of finding what’s relevant–on the whole as well as on the individual level–is colossal.

### Personalization at scale

By using machine learning for predictive personalization, the effort involved in scaling personalization across your entire customer base–past, present, and future–isn’t as herculean as it sounds. Machine learning algorithms can segment your customers into bespoke, cohesive personas that you can deploy [anywhere in your stack](/integrations), allowing you to create individualized, personalized campaigns for any given subgroup of people you’re interested in.

### Responsibly-sourced data

Privacy is front-and-center of any data-focused discussion in 2025–as it should be. It began with the (albeit slow-rolling) [death of third-party cookies](/blog/death-of-third-party-cookies-strategies), and will continue to be a primary focus of consumers moving into the future. Privacy matters, and largely, people are uncomfortable with things like cookies–previously critical to a third-party data strategy–tracking them creepily across the web to find out what they’re searching and where they’re visiting, even if the end-result is personalized content.

The data you collect and use should be [responsibly-sourced, opt-in, and permissioned](/features/data). It’s possible to provide a perfectly personalized experience while simultaneously being a privacy-forward brand, but it takes effort.

## Benefits of personalization at scale

With the barriers to true personalization at scale being so high, is the end-result worth it? With companies that personalize collecting [40% more revenue](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying#0) than those that don’t, the answer is a resounding _yes_.

### The right message, at the right time, every time

It’s an adage as old as marketing itself, but it’s always going to be true when a single poorly-personalized, or otherwise irrelevant message could mean the loss of a customer–or potential customer–forever. With personalized content delivered to every individual you’re engaging with, at any scale of your campaign, you’ll always be delivering the right message, at the right time, every time.

### Consumers demand personalized content

With [61% of consumers](https://www.salesforce.com/resources/research-reports/state-of-the-connected-customer/) reporting that they feel that most companies treat them as a number rather than a person, it’s clear that personalized content is near and dear to the modern consumer’s heart. As a dog lover, you wouldn’t want to receive a postcard from a pet food brand focused exclusively on cats, would you? Your customers don’t either.

### Building brand loyalty

[78% of consumers](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying#0) say that they’re more likely to buy again from brands that personalize their content. Loyal customers are also more likely to recommend brands they love to their friends & family, ensuring that, if you focus on keeping your customers happy, they’ll return the favor by shouting your brand’s name from the rooftops. Keeping a strong base of loyal customers is key to a good [churn prevention strategy](/blog/churn-prevention-machine-learning).

## How to achieve personalization at scale

It’s clear that the benefits of personalization outweigh the challenges, but how exactly can it be achieved? There are three key pillars to successful personalization at scale, and they all focus on your customer data.

### Collect good first-party data

Data is the most important feature of propensity to buy and customer predictions. Without a strong enough foundation, the predictive models that your data science team builds won’t have enough data to predict strongly off of, leaving you with questionable-at-best insights. You’ll want a strong mix of first-and-third-party data collected through tactics like web forms, surveys, and loyalty programs, where you can _really_ draw out what makes your customers unique.

### Optimize your data with first-party data enrichment

As good as your own customer data is, your customers have more going on in their lives than you can see through their interactions with your brand. You can buy data from companies like Acxiom or Epsilon, who have been collecting data for years and years, or you can leverage assets like publicly available census data to find more about who your customers really are.

### Use AI for customer segmentation

With your data collected and enriched, you need to segment your customers into unique subgroups, or personas. It can be manually done, but the process is incredibly labor-and-time-intensive, and is prone to human error. Using [AI for customer segmentation](/predict/category/persona_assignments) takes the load off of your team, leaving you ready to personalize campaigns at any scale, create internal, [persona-based talk track alignment](/blog/sealed-lead-optimization), and more.

## How Faraday supports personalization at scale

Faraday lets you easily identify distinct personas and predict customer behavior with machine learning, unlocking powerful insights to shape better personalization strategies at scale.

### Unified, responsible data

Faraday unifies your first party customer data with detailed, opt-in, permissioned, cookieless consumer data, giving you the insights your brand needs while simultaneously being privacy-centric. Hundreds of data points through demographics, psychographics, financials, property information, and more to be leveraged for your customer predictions.

Consumers have adapting needs, and [65% of consumers](https://www.salesforce.com/resources/research-reports/state-of-the-connected-customer/) expect brands to adapt to them. This is a laborious task manually, but with Faraday’s built-in data always staying up-to-date, your brand can adapt alongside your customers.

### Delivering personalized messaging and creative across multiple channels and touchpoints

Faraday takes your data–combined with Faraday’s–and uses AI to segment your customers into unique subgroups, or personas, based on hundreds of data points.

[![Faraday customer personas](/images/blog/personalization_at_scale_guide/personalization-at-scale-guide-personas.png)](/images/blog/personalization_at_scale_guide/personalization-at-scale-guide-personas.png)

With your customer personas predicted, easily kick off campaigns by deploying your entire persona set, or a single persona, to anywhere in your stack. Every individual your brand is interested in, revealed with clear-cut traits like _dog owner_, grouped with people that share similar interests, shopping behaviors, and more.

[![Faraday prediction deployment](/images/blog/personalization_at_scale_guide/personalization-at-scale-guide-deployment.png)](/images/blog/personalization_at_scale_guide/personalization-at-scale-guide-deployment.png)

Personalization at scale has never been easier, or more accessible.

### Predicting the next best product for every individual

Alongside your customer personas, you can optionally predict the propensity, or likelihood, that any action is taken by an individual. For example, take your Emerging Adult Eli customer persona and predict the likelihood that anyone in that persona will [repeat purchase](/predict/category/repeat_purchasers), and, further, predict the next best product for them based on similar shoppers.

### Providing personalized discounts to foster customer loyalty

Personalization extends beyond the copy and creative that’s in your engagement with a given customer–it can include providing strategic discounts to customers that have, for example, made a second or third purchase. This way, your customer predictions guide you toward the people that are most likely to repeat purchase so that you can then turn around and give them _just_ the right discount they need to keep coming back.

**Ready to achieve true personalization at scale? [Create a free account](https://faraday.ai/signup).**

Consumers are demanding personalized content at an increasing rate, but personalization at scale has traditionally been out of reach for many brands. Using a customer insights platform makes true personalization at scale more accessible than ever.

The modern marketer’s guide to personalization at scale

_This article is a followup to our [bias reporting](/blog/faraday-bias-reporting) article, and aims to provide a high-level overview of how Faraday users can mitigate harmful bias._

## What is bias mitigation in machine learning?

Bias mitigation in machine learning encompasses a set of methodologies and techniques devised to diminish and ideally eradicate unjust and discriminatory predictions inherent in the decision-making processes of artificial intelligence (AI) systems. This is a key component of [Responsible AI](/features/responsible-ai).

Bias, often unintentional, may infiltrate these systems as a result of the historical data they are shown, leading to inequitable treatment of specific demographic groups. For example, there is a history of [discrimination](https://www.investopedia.com/the-history-of-lending-discrimination-5076948) against specific minority groups in financial lending, and this history can perpetuate bias if it is not intentionally mitigated.

It is important to note that not all algorithmic bias is harmful. Predictions about, for example, the likelihood of buying products that are exclusively relevant to, say, certain age groups will and should discriminate in favor of those age groups. For the remainder of this article, when we discuss bias, we’re excluding innocuous bias like this and focusing on harmful bias.

Bias mitigation ensures that AI systems render decisions characterized by impartiality and fairness, irrespective of variables such as race, gender, or other sensitive attributes. This undertaking is pivotal in the development of AI systems that uphold principles of fairness and equality for all individuals.

The subsequent sections of this article describe how Faraday’s bias mitigation feature allows users to create fair propensity predictions. For more on bias management in Faraday, read our [bias management announcement](/blog/faraday-bias-management-announcement).

## How to reduce harmful bias in machine learning

When predicting propensity, Faraday can mitigate bias in two broad ways:

1. The first step is to ensure that the predictive algorithm considers every demographic group as equal. In order to do so, Faraday rebalances your historical data before submitting it to the algorithm for pattern discovery.
2. Second, when later making predictions, Faraday will adjust the nominal propensity ranking to ensure that the highest scorers for each demographic are ranked highly overall. This ensures that the predictions are meeting the fairness objective while maximizing predictive lift.

Some Faraday terminology will be used here, and will link to documentation.

### Unbiasing historical data

In order to mitigate bias in a client’s existing customer data, Faraday assigns different “weights” to each customer in the data to fine-tune their influence on the model's learning process. The goal is to address and rectify biases present in the data, ensuring that the model gives equal consideration to all relevant examples.

To understand this process, consider an example: a home mortgage company. Given historical disparities arising from discriminatory laws and government policies (among other reasons), some demographics are less likely to appear in the company’s customer data.

Formally, Faraday will discover this by noticing a difference between the “attainment” (all customers) and “eligible” (regional adult population) [cohorts](/docs/product/cohorts) in one or more dimensions of concern. For example, we could find fewer women than men in the historical customer data, even though the regional population is an even split: a disparity that has no rational basis. Without correction, the algorithm would believe (incorrectly) that there is some intrinsic quality which makes men more likely mortgage customers than women.

To unbias the historical data, Faraday would give higher weights to customers in underrepresented or disadvantaged groups, increasing their influence on the algorithm while it is looking for salient patterns. Building on the mortgage example above, Faraday could give higher weight to the proportionally fewer women customers, evening out their influence versus men. This results in more equitable predictions for individuals belonging to these groups.

By adjusting the weights assigned to different customers, Faraday aims to create a more balanced and fair learning process, ultimately leading to AI models that provide fairer outcomes across diverse demographic groups.

### Unbiasing predictions

When you include a propensity [outcome](/docs/product/outcomes) in your [pipeline](/docs/product/pipelines), Faraday can take additional measures to rectify potential biases in the resulting predictions. After the initial predictions are made, this approach evaluates and adjusts them to ensure fairness and reduce discriminatory outcomes.

Given that the first step eliminates bias in the historical customer data, why is this second step even necessary? When we unbias historical data, we can't alter the algorithm to completely ignore the overrepresented population in favor of the underrepresented one–we’re simply telling it to pay more attention to the latter until it has a roughly equal representation. This weighting can have a cascading effect that creates biases of its own, though.

To take the mortgage example even further, maybe the women customers tend to be older than the non-customers, whereas the men customers tend to be younger than their non-customer counterparts–after the weight adjustment, this can make the trait _age_ mean that an individual is more likely to attain the outcome if they are older, unfairly penalizing those without that characteristic, which is why this step is crucial.

In this step, Faraday looks for differences between the demographic makeup of the pipeline’s eligible population and the subgroup receiving the highest scores. The predictions are then adjusted to align more closely with the desired fairness objectives.

This method allows for the correction of biases that may have been present in the original predictions, promoting a more fair distribution of outcomes across different demographic groups. By incorporating fairness considerations after the initial predictions are generated, prediction this re-ranking contributes to developing machine learning models that exhibit improved fairness and reduced problematic disparity in their decision-making processes.

## Mitigating harmful bias with Faraday

As of the publication of this article (February 2024), Faraday currently allows for bias detection and mitigation for the **age** and **gender** dimensions (these features are often referred to as protected classes or Dimensions Of Concern).

When you define a propensity outcome in Faraday, you can choose how you want the system to handle bias:

- **No mitigation**: The outcome fairness is only reported as part of the outcome dashboard. No further transformation is applied to ensure that protected classes are treated fairly. This is perfectly appropriate in cases where the product or service being sold is intrinsically linked to certain demographic groups.
- **Equality**: This mitigation strategy ensures that demographic patterns are similar between the eligible population and the target group (those with the highest scores). For example if “young adult, women” represent 25% of your eligible population, we ensure that any population targeted from the pipeline will contain 25% of “young adult, women”.

  The chart below depicts the effects of historical data bias mitigation using the “equality” setting with three populations (A, B, C). Population **A is underrepresented** in the customer data (attainment population)—30% are customers vs. 47% in the overall population while **B** and **C are both overrepresented**. The mitigation process ensures that in the target group, population **A** accounts for 47% while **B** and **C** account for their original 14% and 39% respectively.

  [![Equality bias mitigation](/images/blog/faraday_bias_mitigation/equality-mitigation.png)](/images/blog/faraday_bias_mitigation/equality-mitigation.png)

- **Equity**: This mitigation strategy “inverts” bias so that populations that were historically underrepresented are now instead prioritized.

  In the example below, the customer group (attainment cohort) consists of 30% of population **A**, 20% of population **B**, and 50% of population **C**. The equity mitigation is going to promote the underrepresented populations **A** and **B** at the expense of population **C**. The target population will contain 37% of **A**, 47% of **B**, and 16% of **C**.

  [![Equity bias mitigation](/images/blog/faraday_bias_mitigation/equity-mitigation.png)](/images/blog/faraday_bias_mitigation/equity-mitigation.png)

Note that when mitigating bias, Faraday will consider all dimensions simultaneously, leading the system to consider subpopulations like “young adult, women” and “middle-aged men.”

## Does mitigating bias hurt performance?

Because the bias mitigation process necessarily alters the ranking and promotes sub-populations with otherwise lower predicted scores, it can reduce lift.

That said, the impact is often small. The top two graphs below depict the impact of **equality** mitigation on Gender. The lift curve for the un-mitigated model is only slightly better than the one for the mitigated model. The main difference lies in the 20% to 40% score percentile range with lift in that range decreasing from ~300% to ~220%.

The two bottom graphs show the mitigation at work. On the left, the un-mitigated model will create a target group with 2x too many males.

After mitigation, the graph on the right shows only a slight discrepancy between the fair proportion of each gender, within 3%. This is a significant fairness improvement over the initial target population breakdown with only a minor reduction in performance.

[![Bias mitigation comparison](/images/blog/faraday_bias_mitigation/mitigation-comparison.png)](/images/blog/faraday_bias_mitigation/mitigation-comparison.png)

## Why mitigate bias?

Bias isn’t inherently harmful. Earlier in this article, we touched on innocuous bias that occurs when products or services are explicitly designed for and marketed toward certain groups, like a womens’ swimwear brand marketing mostly to women. It’s the _other_, harmful–and often hidden–bias that should be surfaced and mitigated.

With Faraday, bias mitigation is easy. If you find that a predictive outcome you created has bias that you want to mitigate, simply create a new outcome using the same cohorts of people, toggle mitigation on for the sensitive dimension in question, select your mitigation strategy, and save the outcome. Once it finishes building, compare the bias report from the un-mitigated outcome with the report from the mitigated outcome. Alternatively, use the **bias_mitigation** object with the API call to [create](/developers/reference/createoutcome) an outcome.

[![Bias mitigation comparison](/images/blog/faraday_bias_mitigation/bias-mitigation-in-app.png)](/images/blog/faraday_bias_mitigation/bias-mitigation-in-app.png)

It’s in the best interest of anyone developing or using AI to do so responsibly. Whether you’re using [generative AI](https://www.bloomberg.com/graphics/2023-generative-ai-bias/) for email copy and creative, predicting customer behavior like we walked through in this article, or something else entirely, you should be wary of the existence of bias in the data that the algorithms are using, and how it impacts who you market and sell to. By employing responsible, [explainable AI](/blog/what-is-explainable-ai-why-is-it-important), you can build a foundation of trust between not only yourself and your customers, but between your business and your data.

Responsible AI isn't just good business–it’s the right thing to do.

Mitigating harmful bias with Faraday is simple, whether through our developer API or in-app. Learn more about how Faraday mitigates bias in your customer predictions.

How Faraday helps mitigate harmful bias in machine learning

Quantifying bias in predictive models is the first step towards building models with fair treatment of protected groups.

Faraday bias reporting: How we measure & report on bias

One of Faraday’s founding pillars is the use of [Responsible AI](/features/responsible-ai), which includes both preventing direct harmful bias and reporting on possible indirect bias. We’ve been building these features directly into the platform over the past several months, and today we’re going to walk through two new features in Faraday with **bias reporting** and **bias mitigation**.

## Bias reporting

To see the bias report on an outcome, head to [outcomes](https://app.faraday.ai/outcomes) and scroll down to the **bias** section to see the breakdown of bias discovered in the _data_, _power_, _predictions_, and _fairness_ categories, which we'll dive into below. These can also be retrieved via the API using [get outcome analysis](/developers/reference/getoutcomeanalysis).

[![Bias reporting on an outcome](/images/blog/faraday_bias_reporting/outcomes-bias-reporting.png)](/images/blog/faraday_bias_reporting/outcomes-bias-reporting.png)

Faraday uses four categories to report on bias for an outcome:

- **Data**: Measures selection bias in the underlying cohorts used in the outcome. In a Faraday outcome, two labels exist for the purpose of this blog post: _positive_, or the people from the attainment cohort that were previously also in the eligibility cohort, and _candidate_, or the people from the eligibility cohort.
- **Power**: Measures outcome performance on a subpopulation compared to baseline performance–for example, Faraday will compare how well the outcome performs on the subpopulation “Senior, Male” compared to everyone else.
- **Predictions**: Measures proportions of subpopulations being targeted compared to baseline proportions in order to see if the subpopulation is over or under-represented.
- **Fairness**: Measures overall fairness using a variety of common fairness metrics in the literature. For example, Faraday will look at whether or not the subpopulation "Senior Male" is privileged or underprivileged.

Each bias category uses groups of people called subpopulations, which are combinations of people defined in the [understanding bias reporting](/docs/product/outcomes#understanding-bias-reporting) section's **sensitive dimensions** dropdown table. Examples of these are "Teens with gender unknown" and "Senior Females." We're launching with two of these sensitive dimensions in **age** and **gender**, but will be rolling out more over time.

If you discover that one of your outcomes includes harmful bias, you can create a new outcome using the same cohorts, but enable bias mitigation options while creating it to compare results.

## Bias mitigation

When creating a new outcome, you can choose to mitigate bias by using the toggles in the **bias** section. Check out our [outcomes documentation](/docs/product/outcomes#mitigating-bias) for info on different bias mitigation strategies.

[![Bias mitigation in an outcome](/images/blog/faraday_bias_reporting/outcomes-bias-mitigation.png)](/images/blog/faraday_bias_reporting/outcomes-bias-mitigation.png)

<Callout type="warning" title="Taking action on bias">
Mitigating bias isn't something you should set-and-forget, meaning it's not recommended that you toggle both **age** and **gender** on for every single outcome you create. For example, if you're a womens' swimwear brand, your data will skew heavily toward women, and mitigating gender would negatively impact your lift.
</Callout>

Always keep the populations that you're making predictions on in mind when considering bias. In the above example, it's _totally_ fine to ignore the biased skew towards a certain gender or age group because they're your ideal customers.

We're thrilled to continue to drive Responsible AI forward by providing agency for our users to both discover and mitigate bias in their customer predictions. Stay tuned for a deep dive into the science behind how Faraday determines bias.

Learn how Faraday's bias management suite enables you to analyze and mitigate bias found in your customer predictions.

Introducing Faraday bias management

I've always loved Zapier. It really does feel like a superpower to set up a workflow with a zap: the gift that keeps on giving. This section has been on their homepage longer than anything else and I can see why — Zapier users all feel this way.

[![Testimonials from Zapier users](/images/blog/zapier/testimonials.png)](/images/blog/zapier/testimonials.png)

That's why I'm so excited and proud to share that you can now [incorporate Faraday predictions into your zaps](https://zapier.com/apps/faraday/integrations). With this new integration, you can score leads, assign customers to personas, make product recommendations to shoppers, forecast spend, and more, all within your automated Zapier workflows.

Predictions are the key to improving customer engagement throughout the lifecycle—you can personalize outreach, customize offers, prioritize better opportunities—the list goes on.

## Working through an example Zap with Faraday predictions

Let's say we want to score new leads when they arrive in HubSpot, evaluating them for their likelihood to close and assigning them to a persona. This way, you'll be able to tailor your handling of these leads differently according to their predictions.

### Faraday setup

We'll assume you already have basic customer and lead data, along with their respective cohorts, set up in your Faraday account. If you're new to Faraday, check out the [docs](/docs/intro/welcome) and [Faraday Academy](/docs/academy) to get up to speed.

Next, you'll need to set up your two predictions if you don't have them already:

1. **Lead scoring outcome** — [docs](/docs/how-to/predictions/likelihood-to-convert)
2. **Customer personas** — [docs](/docs/how-to/predictions/customer-personas)

Finally, if you don't already have it, create a pipeline to get these predictions deployed:

- **Population** – everybody
- **Payload** – the two predictions above
- **Deployment** – Lookup API

That last bullet is important to get things working right on Zapier.

### Creating your zap with Zapier

I love Zapier's new LLM interface for creating zaps, and it works great here. Just head to your dashboard and type "When a new lead comes into HubSpot, score it using Faraday," like so:

[![Zapier prompt saying "When a new lead comes into HubSpot, score it using Faraday"](/images/blog/zapier/prompt.png)](/images/blog/zapier/prompt.png)

Zapier should automatically calculate your zap template for you—something like this:

[![Zapier template for "Score new leads in HubSpot with Faraday"](/images/blog/zapier/template.png)](/images/blog/zapier/template.png)

Hit that "Try it" button. This should load the template up in the zap editor, which will look something like this:

[![Zapier zap editor for "Score new leads in HubSpot with Faraday"](/images/blog/zapier/editor.png)](/images/blog/zapier/editor.png)

Click the HubSpot box to pull up settings for this trigger step. You'll have to choose your HubSpot account and then pick your lead capture form from the dropdown. It will look like this:

[![Zapier HubSpot settings for "Score new leads in HubSpot with Faraday"](/images/blog/zapier/hubspot.png)](/images/blog/zapier/hubspot.png)

Hit the test button, and it will find a recent lead from your form data. When you hit continue, you'll make your way into the Faraday action step settings.

Choose or add your Faraday account (you'll need your Faraday API key for this), then proceed to the Action section. In the "Select pipeline" dropdown, choose the pipeline you created earlier. If it's not on the list, make sure your Lookup API deployment was properly added to this pipeline in Faraday.

Then, use the form data from HubSpot to fill out as many of the identity fields as you can. In this case, we capture email in our lead form, so I'm using that (actually, the Zapier LLM had that preconfigured for me ✨):

[![Zapier Faraday settings for "Score new leads in HubSpot with Faraday"](/images/blog/zapier/faraday.png)](/images/blog/zapier/faraday.png)

After testing what you've got so far, it's time to store your predictions for this new lead somewhere. You could put them back in HubSpot with an "Update Contact" action, or whatever you'd like. Then publish your zap and you're all set!

**Ready to try Faraday with Zapier? [Create a free account](https://faraday.ai/signup) to get started today.**

Zapier is everybody's favorite workflow automation tool. Now, you can retreive predictions from Faraday in your Zaps.

Product