Regardless of what part of your business you find yourself in, your team has likely discussed implementing artificial intelligence (AI) of some sort in 2024–and it’s no wonder, with [33% of businesses](https://thesocialshepherd.com/blog/ai-statistics) already using AI and another 42% considering adopting it, the benefits of harnessing AI are too good to be ignored.

As leaders, employees, and consumers alike learn more about predictive models, machine learning algorithms, and various other pieces of the AI puzzle, questions naturally arise about how AI really works. Can the predictive models created be trusted? Can they produce biased results? Making complex machine learning algorithms and their results explainable to humans has proven to be a challenging feat for organizations to overcome in a tangible way, though there are significant benefits for those that do.

Put simply, if you’re going to change the way you interact with customers based on what AI tells you, how can you trust what it’s telling you?

## What is explainable AI?

Historically, AI has been plagued by the [black box problem](https://umdearborn.edu/news/ais-mysterious-black-box-problem-explained), in which artificial intelligence reaches purportedly valuable conclusions—finding high-scoring leads, producing compelling generative content, and so on—and the user is left with the option to take it or leave it. No insight was provided into how the AI reached the conclusion it did, and what data it used to do so, thus the term “black box.”

Explainable AI aims to solve this problem by providing much-needed insight into the inner workings of predictive models so that the end-result of a predictive model offers both the sought-after conclusions, as well as how those conclusions were reached. This means that deploying [customer predictions](https://faraday.ai/features/predictions) and other AI-generated content can be done with confidence, rather than a hand-wave.

## Why does explainable AI matter?

[According to IBM](https://www.ibm.com/thought-leadership/institute-business-value/en-us/report/business-trends-2024), 9 in 10 consumers say trust is the most important deciding factor when deciding which brand to adopt. People have never been more keenly aware of the importance of brand trust, and how quickly it can vanish. The need for consumers to trust brands they interact with has resulted in the need for explainable AI, and for the black box to be removed. IBM also reports that general consumer trust in businesses dropped from 48% in 2021 to 20% in 2023, meaning that businesses need to employ frameworks like explainability that help establish and maintain trust in the relationship for consumers to even consider making a purchase. The relationship between trust and explainability extends far beyond direct-to-consumer goods, too.

In [finance](https://faraday.ai/blog/how-to-optimize-ads-for-compliance-in-finance)—a space that has historically struggled with bias—AI can be used to determine whether or not an applicant should receive a loan, which leaves the door open for their machine learning algorithms to use historically biased data to permit or deny a loan. AI must be explainable in order for lenders, or anyone else building or employing it, to consider it for use.

## Benefits of explainable AI

Explainable AI enables businesses that employ AI pipelines to properly interpret and understand its output without requiring data science degrees. This benefit is twofold: first, the importance of understanding the tools you’re working with rather than just taking their word for it can’t be overstated. Knowledge is power, after all. This leads straight into the second, in that it allows you to not only understand why the AI reached the conclusion it did, but then enables you to take that reasoning a step further by analyzing its context. If you’re using machine learning algorithms to find out which of your customers is [most likely to churn](https://faraday.ai/predict/category/churn_scores), you can dig into its explainability to reveal what traits about them helped make that critical decision. Similarly, explainable AI can help [reveal and mitigate bias](https://faraday.ai/blog/faraday-bias-management-announcement) in your data, which helps you stay compliant with local laws and regulations.

Additionally, a foundation of trust is built (and maintained) between the business and its customers, which is a rare commodity in 2024. With existing customers generating [72% of company revenue](https://www.hubspot.com/hubfs/HubSpots%202024%20Sales%20Trends%20Report.pdf), a happy customer base that you’ve nurtured with trust will keep coming back for more–and they’ll refer their friends, too.

## Challenges of explainable AI

With such impactful benefits on the table, why hasn’t explainable AI become mainstream? The simple answer is that it’s difficult. Entire classes of ML algorithms are fundamentally incapable of effectively explaining their inferences, so you have to choose your methodology carefully. Explainability also requires well-prepared and structured attributes that not every organization is lucky enough to have on the shelf. Finally, in a commercial environment where data science resources are both scarce and overburdened, AI explainability never quite seems to make it to the top of the todo list.

## Explainability is built into Faraday

While businesses that are fully committed to building their own AI pipelines need to overcome the hurdles involved in implementing explainable AI, many businesses are instead adopting platforms like Faraday that are purpose-built for using AI–and in Faraday’s specific case, predicting customer behavior.

Faraday has long been a proponent of [responsible](https://faraday.ai/features/responsible-ai), explainable AI. In fact, it’s one of our founding principles. Every [predictive model](https://faraday.ai/blog/faraday-probability-scores) that Faraday users create allows customization of what traits are used, allows removal of protected class data, and comes with a full report that details things like the most important traits that helped build the model. In 2023, we released [score explainability](https://faraday.ai/blog/faraday-score-explainability), a feature that outlines each of the most influential pieces of data within an individual person’s predictions, and how much influence they had.

[![Image of a Faraday score explainability output](/images/blog/explainable_ai/explainability-output.png)](/images/blog/explainable_ai/explainability-output.png)

Let’s use the above graphic to visualize the churn prediction example in the previous section. Jane Madison has a 75% probability of churning, and the explanation provided in the predictive model that scored Jane said that the fact that Jane is a millennial played a major role in why she’s so likely to churn. Meanwhile, John’s children made it less likely that he would churn. Not only do we know that Jane is more likely to churn than John, but we know _why_.

In the end, explainable AI results in tangible benefits for both the businesses employing it and their customers. Businesses are able to understand their own tools, and use that understanding to reveal impactful pieces of insight like _why exactly_ Jane Madison is going to churn. From a consumer perspective, businesses employing explainable AI stand out as trustworthy and worth interacting with–and worth referring their friends to do the same.

With AI being built into many businesses' workflows, concerns are rightfully being raised about its trustworthiness, and emphasis put on the importance of explainable AI.

Why explainable AI is so important for businesses

These days, marketers are looking for every advantage to help them stand out from the crowd. From [personalization](/blog/predictive-personalization-machine-learning) to utilizing AI tools like ChatGPT to lend a hand with copy, consumers are getting more picky about which brands they interact with, and it’s forcing brands to adapt. One such advantage makes use of predictive analytics and machine learning to enable brands to stay one step ahead of their customers and their competition: propensity to buy.

## What is propensity to buy?

It’s likely that either you or someone on your team has found themselves muttering “I wish I knew whether or not this lead is going to be a waste of time,” or “if only I knew this customer was about to churn” at some point recently. Predicting propensity to buy–or propensity, in short–is using a machine learning algorithm like [random decision forest](/blog/propensity-models-predict-customer-behavior) to predict how likely someone is to take an action. And it doesn’t just apply to new customer acquisition, or even just lead conversion. You can predict the propensity for virtually _any_ action, meaning that your sales and marketing tactics become proactive rather than reactive.

## Benefits of propensity to buy

Propensity to buy models can be applied to an incredible variety of use cases that provide real, actionable insights for the modern marketer. Let’s take a look at three.

### Market sizing

Expanding your brand’s reach into a new territory is inherently risky. Manual market sizing analysis can help mitigate the risk, and smaller campaigns can test the waters and see if there are any bites from your ideal customers, but then you’re spending both time–your most valuable resource–and money, on a strategy that may or may not pan out. Sure, brands have been doing that forever–but with propensity to buy models [removing the guesswork in a market sizing analysis](/blog/bottom-up-market-sizing-predictive-analytics), it can be done accurately and with confidence.

### Ad targeting

Similarly, propensity to buy models can be applied across the entire U.S. population, providing [likely buyer scores](/predict/category/likely_buyers) that let you easily filter out the bad fits. By eliminating the bad fits that were never going to buy in the first place, the extra spend that would’ve gone toward showing them ads that weren’t relevant can go toward ones that resonate perfectly with your ideal customers.

### Retention campaigns

The effectiveness of propensity to buy extends beyond acquisition campaigns–they’re equally as important and effective for retention as well, through [propensity to churn models](/blog/churn-prevention-machine-learning). The dream of knowing that a customer’s going to churn before they make that critical decision becomes a reality through actionable churn scores. The higher the score, the more likely they are to churn, giving your team crucial insight and warning to keep them on board through a discount, promo offer, or some other attractive deal.

## How do propensity models work?

Let’s take a look under the hood: how does it work? Propensity models use machine learning algorithms to pore over your customer data to find historical patterns in it. Customers placing orders, churning, [ordering again](/predict/category/repeat_purchasers), and more are compared and analyzed to compare each person to another for any similarities. Often utilizing algorithms like random decision forests or logistic regressions, propensity to buy models–while data-hungry, resource-intensive, and time-consuming–offer an attractive variety of impactful use cases.

## How to build propensity to buy models

You can build models to predict propensity to buy and use those scores directly in your stack to gain the benefits that come with knowing when a lead is [likely to convert](/predict/category/lead_scores), a customer churn, or some other use case. Let’s dig into some key aspects of building propensity to buy into your stack.

### A strong foundation of data

Data is the most important feature of propensity to buy and customer predictions. Without a strong enough foundation, the predictive models that your data science team builds won’t have enough data to predict strongly off of, leaving you with questionable-at-best insights. You’ll want a strong mix of [first-and-third-party data](/blog/first-party-vs-third-party-data-marketing), and that data should include demographics, psychographics, lifestyle, property, financial, and more so that your data is well-rounded.

### Data science to build predictive models

With your data locked in, you need data scientists to build predictive models. Regardless of which machine learning algorithms your team uses, they’ll need time to build the models and for the models to iterate–machine learning needs time to learn the data that it’s working with, and will grow smarter the longer the algorithms run. This often takes anywhere from 5 months to over a year.

### Systems integration to fit in your stack

The last major pillar of building propensity to buy models is finding a place for it in your stack. You can have propensity to buy scores sent directly to your database or warehouse, to your CRM, or any other activation platform. Depending on where you decide to implement, you’ll need to build an integration of some kind that enables your workflows to continue uninterrupted–a process that can take several weeks to complete.

## How Faraday can help implement propensity to buy in your stack

Faraday is purpose-built to cut the build-it-yourself timeline of building propensity to buy down from monthslong down to mere days. With Faraday, your brand has access to [built-in, cookieless, permissioned consumer data](/features/data) on nearly every U.S. adult, providing rich insight into your customers, and a data science team that has been at it for over a decade. Propensity scores for any action your brand is interested in can be embedded anywhere in your stack through hundreds of [pre-built integrations](/integrations) to place your predictions exactly where you need them.

### Predict propensity to buy to stay proactive

Make use of Faraday’s propensity models to predict–in just a few clicks–who’s (or where’s) most [likely to buy](/predict/category/likely_buyers), who’s likely to churn, what leads are most likely to convert, & more. Propensity to buy models with machine learning make your brand proactive so that you’re always delivering the right message at the right time.

[![Faraday propensity to buy outcome](/images/blog/propensity_to_buy_machine_learning/propensity-to-buy-machine-learning-outcome.png)](/images/blog/propensity_to_buy_machine_learning/propensity-to-buy-machine-learning-outcome.png)

**Ready to try it out? [Create a free account](https://faraday.ai/signup).**

Using propensity to buy models to target your most likely-to-buy customers is an attractive prospect, but it can be a long, resource-draining process. Read about how customer prediction platforms can drastically shorten your time-to-value.

Predicting a customer’s propensity to buy using machine learning

**See our complete guide:** [How Faraday validates predictions — and how you can too](/blog/holdout-testing-with-faraday).

---

If you've ever used AI to make predictions, chances are you've asked yourself "How do I know these are . . . correct?" It's a fair question, and healthy skepticism is a critical component of any Responsible AI practice.

Luckily there are plenty of great options for confirming that the predictions you're getting are mostly right. First, a bit of background.

## Prediction basics: How do AI models make predictions?

To set the stage, let's review how prediction works. Throughout this post, I'll use the example of a company wanting to predict which of its leads are most likely to "convert" (become a customer). In Faraday you'd represent this prediction objective with an outcome like so:

![Lead conversion outcome configuration](/images/blog/testing_predictions/outcome.png)

### Finding patterns

Now we'll take you behind the scenes. To predict something, a system must comb through historical examples of that thing happening to find patterns.

Once you know the patterns, you can compare newly observed behavior to the crucial patterns that signal "true" - if they are similar, you predict "true," otherwise "false."

So, to follow our example, the first step is to pull together some historical lead data, including some leads that converted and some that didn't.

![Training data diagram](/images/blog/testing_predictions/training_data.png)

Here, the top row represents your leads over time at their first capture point. Some of them—the **<span style={{color: "green"}}>green</span>** dots—"convert" by becoming customers: we call these _examples_. Others—the **<span style={{color: "red"}}>red</span>** dots—go "stale" and represent failures: we call these _counterexamples_. It's too early to say what the few remaining gray dots will do, so we'll just ignore them.

One useful thing to note is that 6/13 of our leads converted, or about half the time. This will come in handy down the road.

Your prediction system will then use one of a variety of methods to look for interesting patterns. Is there something in common among the green leads? What about the red ones? How are they different? This identification of patterns (the model) is what lets the system make predictions about new leads.

### Using patterns to make predictions

So now, let's take those pesky remaining gray dots, as well as some new ones that might have popped up since we figured out our patterns. I've put a dark border around them:

![Data diagram including new leads](/images/blog/testing_predictions/new_leads.png)

Your prediction system will scrutinize each of these possibilities to see which patterns in your list it follows. If it follows a lot of the patterns associated with conversion, the system will predict that it will convert. If it follows a lot of the patterns associated with non-conversion, the system will predict non-conversion.

![Diagram showing predictions](/images/blog/testing_predictions/prediction.png)

### Beware silly patterns

Easy, right? Too easy you might say. The reason to be skeptical here is that the patterns your prediction system found may be coincidences rather than reflect some underlying truth.

For example, your prediction system may say:

> Here's a pattern I found! Leads with one of these exact email addresses convert, all others don't: `amy@example.com, leon@example.com, ..., pat@example.com`

Sounds silly, but it happens all the time when the wrong kind of data is examined for patterns. The methods your prediction system uses aren't always clever enough to distinguish between goofy patterns and serious ones. Based on the data available, this pattern is actually incredibly powerful! In fact, it's 100% accurate.

Problem is, it will never work with any new leads, because they'll all have different email addresses and the system will therefore predict non-conversion for all of them.

## Three holdout testing methods to see if your predictions are accurate

The way to avoid this kind of silliness is to "test" your patterns to see if they're right. If they pass the test, you can trust them going forward.

Think about your tests in school: your teacher asked you questions and you provided answers. Your teacher knew the true answers. If your answers matched the teacher's true answers, you got an A.

The hardest part of all this is that somebody has to "know" the "true" answers. If only we had some lead data where we knew the right answer . . .

But of course we do! We know which leads actually converted and which didn't. We just can't use the _same_ data to both find patterns and put those patterns to the test.

There are a few great ways to split up your data for these two simultaneous purposes. They all rely on a "holdout"—setting aside a portion of the data to use exclusively for testing. This allows you to see whether the system "understands" enough to assess new data versus only being able to regurgitate answers for the questions it has already seen. Let's dive into these methods now.

### Method 1: random holdout

#### Setup

This is the most common approach, and it involves choosing a random subset of your historical data to set aside as a holdout. Only then do you use your prediction system to find patterns among the remaining data.

![Cross validation diagram](/images/blog/testing_predictions/random_holdout.png)

Here, we've used 50% of our data as a holdout. The system will find patterns exclusively among the remaining leads.

#### Testing

Now we're ready to "take the test." The leads in the holdout are the questions, and their true disposition (converted or not) is the answer key. But for the time being we pretend we don't have the answer key and we treat the holdout leads like new leads. What can the patterns we have tell us about these leads?

I'll illustrate those predictions with colored borders on the holdout leads:

![Prediction step of cross validation](/images/blog/testing_predictions/random_holdout-prediction.png)

Once we've used the patterns to make predictions about the holdout leads, we can dramatically reveal the answer key and see how accurate the patterns really are. Let's see how we did:

![Cross validation results](/images/blog/testing_predictions/cross_validation.png)

We got 4 right and 2 wrong for about 67% accuracy. Is that good or bad?

#### Evaluation

In real life you'll rarely get an A on this test. In some cases you'll predict conversion when the lead didn't actually convert (false positive). In other cases, you'll predict non-conversion when the lead did actually convert (false negative).

Recall that our actual lead conversion rate is about 50%. That means that a really dumb prediction system could always predict "convert" and get it right about 50% of the time, or flip a virtual coin and get the same result. By this measure, 67% is quite a bit better!

So assuming you got more right than wrong, it's an improvement. Before, your patterns were always right about your historical leads, but were useless for assessing new leads. Now, maybe your patterns aren't as perfect, but at least they're useful!

#### Conclusion

Finding the right balance between false positives and false negatives is part of the art of prediction and ultimately comes down to risk versus reward. It's out of scope for this post but I'll be writing one on that soon—subscribe below to get notified.

The upshot here is that the patterns the system found on the non-holdout leads appear to be effective in making predictions about the holdout leads. This is a very good sign that the patterns themselves are "real" and not based on silly coincidences.

The name for this approach is _cross-validation_. In practice, you actually perform this random holdout method multiple times to _really_ make sure your patterns are real. The Faraday platform automatically uses 3 passes like this every time we look for patterns.

### Method 2: recent holdout

#### Setup

Another approach is to use your most recent data as your holdout, a method called a _backtest_. As a refresher, here's what we started with:

![Training data diagram](/images/blog/testing_predictions/training_data.png)

To use this method, we temporarily "rewind" time to a previous day, like so:

![Rewound training data diagram](/images/blog/testing_predictions/recent_holdout.png)

From here, the process is much like above. Our prediction system attempts to find patterns looking backwards from our "pretend now," ignoring the lead data that has transpired since then, which becomes the holdout.

#### Testing

Then it uses those "old" patterns to examine recent leads. Again, predictions here are represented by the colored borders on the holdout leads:

![Backtest prediction step](/images/blog/testing_predictions/recent_holdout-prediction.png)

And finally we use our answer key to see how we did:

![Backtest results](/images/blog/testing_predictions/backtest.png)

#### Evaluation

Here again, we went 4 for 6, or 67%. By the same logic as above, not bad.

The big advantage of the recent holdout versus the random holdout in my mind is that it's easier to understand and, ultimately, trust.

The main disadvantage of this type of holdout is the system can only look for patterns within data spanning a shorter period of time, so if the important patterns take time to appear, then using a recent holdout won't perform as well as using a random holdout.

Faraday has this feature built in as an additional option if you want it, although it's very compute-intensive so it's not available on all accounts by default. If you don't see it and want to use it, just ask support.

### Method 3: secret holdout

You may hear other folks call this a blind holdout, although I hope to see the end of that ableist terminology soon.

The idea here is that you go to extreme lengths to hide your holdout from the prediction system—or even the company running the prediction system!—in order to boost your confidence even further.

Here, the principal doesn't even trust the teacher with the answer key! The diagram looks something like this:

![Secret holdout](/images/blog/testing_predictions/secret_holdout.png)

The user has pre-emptively removed a random portion of the data to serve as the holdout and shared the remainder with the prediction system, which has no idea what's in the holdout.

The system finds patterns using all the data it has. Then the user performs the test by presenting the holdout leads to the system _without their true dispositions_. The system makes its predictions as usual:

![Secret test prediction step](/images/blog/testing_predictions/secret_holdout-prediction.png)

The user then secretly compares these predictions to the "true" answer key to determine accuracy.

In practice the secret holdout is exactly the same as the random holdout, just a heck of a lot more complicated and time-consuming. Nevertheless, you could consider it if you don't trust your prediction system for one reason or another.

Faraday supports secret holdout testing. When connecting your data, make sure to remove your holdout first, so Faraday never sees it. Configure your account as usual. Then upload your holdout—minus disposition—as a CSV dataset with a `holdout` trait, create a matching **Holdout** cohort, and use it as a the population in a new pipeline. Add the outcome you want to test to the payload, and deploy an identified CSV target. Now match Faraday's predictions back to your original holdout and analyze to evaluate accuracy.

## The real deal

Prediction systems finding silly patterns is a real problem—you could even say it's _the_ problem of machine learning. Luckily there are lots of ways to avoid these kinds of shenanigans, and good prediction systems employ them automatically.

Behind the scenes, Faraday runs random and/or recent holdout testing constantly to confirm it's finding legitimate patterns that are practically useful going forward.

I know all this (I helped build the system!), but I still treasure my ongoing skepticism, and you should too. Hopefully understanding how pattern validation works, and seeing how it operates behind the scenes, will help you develop a careful confidence in the predictions you build your workflows around.

Three holdout testing methods you can use to evaluate the accuracy of a model’s predictions.

How to set up holdout tests to evaluate prediction accuracy

In the world of machine learning, we often encounter two important concepts: the raw propensity score and probability. Though they may sound similar, they represent distinct aspects of a machine learning model's predictions.

## Example of propensity scores vs predicted probabilities

Imagine you have a model that predicts whether it will rain tomorrow. The raw propensity score is like the model's initial hunch or gut feeling about the outcome. It's a numeric value that the model calculates based on various factors like historical weather data, humidity, temperature, and more. The [propensity](https://faraday.ai/blog/propensity-models-predict-customer-behavior) score indicates how likely the model thinks it will rain or not. However, the propensity score itself doesn't give us a clear understanding of how confident the model is in its prediction.

This is where [probability](/blog/faraday-probability-scores) comes into play. The predicted probability aims at providing an answer to the question “how likely is the prediction?” It represents the likelihood of a particular outcome occurring. For example, a score of 0.75 for rain means the model is fairly confident it will rain tomorrow. However, a probability of 0.75 indicates that there is a 75% chance that it will rain tomorrow.

In summary, the raw propensity score is the model's initial guess, while probability quantifies the model's confidence in that guess. Understanding this difference can help us interpret machine learning [predictions](/features/predictions) and make better-informed decisions based on the model's level of certainty.

## Understanding model calibration

The model calibration process refers to the alignment between a model's predicted probabilities and the actual likelihood of events occurring. A good calibration ensures that the predicted probabilities are close to the true theoretical probabilities. Not all machine learning models are perfectly calibrated after training. Some models might be overconfident, providing probabilities that are too extreme, while others might be underconfident, consistently underestimating or overestimating probabilities.

### Importance of model calibration

Calibration is crucial because it impacts decision-making. If a model is well-calibrated, we can trust its probability estimates and use them to make informed choices, such as whether to take an umbrella based on the probability of rain.

Calibrating a model involves post-processing its probability scores to bring them in line with the true probabilities. Various calibration techniques can be employed, such as Platt scaling or isotonic regression, which adjust the raw propensity scores to produce well-calibrated probabilities. These techniques are typically applied during model validation and fine-tuning to ensure that the model's predictions are more accurate and reliable.

The calibration process ensures that when the model predicts a probability, it reflects the true likelihood of the event occurring. For example, if a model predicts a 70% probability of rain, it should be accurate in approximately 70% of the cases where it makes that prediction.

In summary, model calibration in machine learning is all about making sure a model's predicted probabilities accurately reflect the real-world likelihood of events happening. A well-calibrated model is more reliable and can lead to more effective decision-making.

## Benefits of using predicted probabilities

Using predicted probabilities instead of raw propensity scores offers several significant benefits that enhance the interpretability, decision-making, and overall performance of the models:

### Interpretability

Predicted probabilities provide a more intuitive and understandable representation of a model's output. Raw propensity scores are often abstract values that lack clear context, while probabilities are easily interpretable as the likelihood of an event occurring. This makes it easier for stakeholders and end-users to comprehend the model's predictions and build trust in its results.

### Uncertainty quantification

Predicted probabilities convey the model's level of certainty about its predictions. Models can provide probability scores ranging from 0 to 1, indicating their confidence in the predicted outcomes. This uncertainty quantification is valuable in decision-making, as it allows users to consider the risk associated with a prediction and adjust their actions accordingly.

### Threshold selection

Predicted probabilities offer flexibility in choosing decision thresholds. Depending on the specific application and requirements, decision-makers can adjust the probability threshold for classifying positive and negative outcomes. This adaptability allows for a fine-tuning of the model's behavior to suit the desired trade-offs between false positives and false negatives. For example, in the case of a lead conversion, a profitability threshold can be established by taking into account cost of acquisition for each lead, expected profit from actual customers, and probability of conversion.

### Calibration and reliability

When models are well-calibrated, predicted probabilities accurately reflect the real-world likelihood of events. Calibrated probabilities provide more reliable estimates, leading to improved decision-making and better performance in critical applications.

### Ensemble and stacking

Predicted probabilities facilitate model ensembling and stacking techniques. By combining the probabilities from multiple models, ensemble methods can often yield better overall performance compared to individual models or raw propensity score-based approaches.

## Benefits and drawbacks of raw propensity scores

Using raw predictions from models instead of calibrated scores has its own set of pros and cons. Let's explore them.

### Pros of using raw propensity scores:

**Simplicity and Speed**: Raw predictions are the direct output of the model without any additional post-processing, making them easier and faster to obtain. In real-time applications or scenarios where efficiency is critical, using raw propensity scores can be advantageous.

**Suitable for Ranking**: Raw propensity scores can still be useful for ranking predictions. Even though they may not directly represent probabilities, they can help rank instances based on their likelihood of being a positive or negative outcome, which is often sufficient in certain applications.

### Cons of using raw predictions:

**Inconsistent Ranking**: Raw propensity scores might not consistently reflect the true likelihood of events occurring. Two instances with similar raw propensity scores might have significantly different actual probabilities, leading to inaccuracies in ranking and decision-making.

**Suboptimal Decision Thresholds**: Using raw predictions often means relying on fixed decision thresholds (e.g., 0.5 for binary classification), which might not be suitable for all applications. This can lead to suboptimal trade-offs between precision and recall.

**Misinterpretation of Scores**: When raw propensity scores are not calibrated, users might incorrectly interpret them as reliable probabilities, leading to misguided decisions and a lack of trust in the model's predictions.

In summary, while using raw predictions can be straightforward and computationally efficient, it comes with the trade-off of lacking calibrated probabilities and potential inconsistencies in ranking. In many real-world applications, calibrated scores are preferred as they provide interpretable probabilities, better decision thresholds, and improved trust in the model's predictions.

## When should you use propensity score percentiles vs. predicted probabilities?

In the field of machine learning, the choice between using percentiles and probabilities depends on the specific context and the nature of the problem being addressed.

### Using propensity score percentiles

**Outliers are important**: If you are concerned about extreme values or outliers in your data, percentiles can be more robust in representing the distribution. For instance, the 95th percentile captures the value below which 95% of the data lies, making it less sensitive to extreme values.

**Data skewness**: When dealing with heavily skewed datasets, especially those with long tails, percentiles can be more informative than probabilities. Percentiles provide a clearer understanding of the data's spread and central tendencies, even in the presence of skewed distributions.

**Interpretability**: Percentiles can be easier to interpret and explain to non-technical stakeholders. For example, stating that a value is at the 75th percentile means that it is higher than 75% of the other values in the dataset.

### Using predicted probabilities

**Well-calibrated predictions**: Probabilities are essential when you need calibrated predictions. If you want reliable estimates of the likelihood of certain events, probabilities are more suitable. Well-calibrated models can provide accurate and informative probabilities for decision-making.

**Decision thresholds**: In many applications, using probabilities allows you to set appropriate decision thresholds based on your specific needs. For instance, in medical testing, you might want to set a higher threshold to prioritize precision over recall, depending on the consequences of false positives and false negatives.

**Uncertainty quantification**: Probabilities are crucial for quantifying uncertainty around predictions. Confidence intervals and prediction intervals can be constructed using probabilities, giving a range of likely outcomes.

**In summary, use percentiles when you need robust representation of data distribution and are concerned about outliers or skewed data. On the other hand, use probabilities when you require calibrated predictions, decision thresholds, and uncertainty quantification, especially in the context of machine learning models.**

**Ready to try Faraday? [Create a free account](https://faraday.ai/signup).**

Propensity scores and predicted probabilities represent distinct aspects of a machine learning model's predictions. Understand the difference and when you should use them.

Propensity vs probability: Understanding the difference between raw scores and probabilities

If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

Last month, a friendly brand exported about 10,000 payment records from Stripe as a CSV. Then they uploaded them to Faraday and, within 24 hours, got their results:

![Buyer signal showing importance of different features](/images/blog/buyer-signal.png)

Let's focus on one feature - "Science/new technology." If a person is interested in this, they are much more likely to buy this brand's products:

![Likelihood to buy based on science interest](/images/blog/science-directionality.png)

Faraday is batteries-included - we know things like whether people like science thanks to our [built-in consumer data](https://faraday.ai/features/data).

Now, this brand can predict the likelihood of more than **240 million Americans** to buy their product. Faraday can take any list of people and sort it by likelihood to buy... or Faraday can generate the list in the first place, using a ["Deployment"](https://faraday.ai/docs/abstractions/deployments).

So... how did they get this? First, by doing a [Stripe Payment Report](https://support.stripe.com/questions/exporting-payment-reports). In particular, doing a [**Successful Payments**](https://dashboard.stripe.com/payments?status%5B%5D=successful) report.

![You want Succeeded payments](/images/blog/stripe-payment-export-succeeded.png)

Faraday only needs the following fields:

![PII, date, and amount fields](/images/blog/stripe-custom-columns.png)

(Later, you can set up an automated integration. But for now, since this is a relatively stable model based on inherent consumer characteristics, it should be stable for at least a few weeks.)

In Faraday, you map the fields:

![Address mapping](/images/blog/stripe-dataset-identity-mapping.png)

Then, [follow the standard process for importing transactions](https://faraday.ai/docs/cohorts/customers_from_hosted_csv#option-a-each-row-represents-a-transaction-event).

And finally, you make a [likely buyers model](/docs/abstractions/outcomes).

In total, if this takes longer than 15 minutes, [email the CTO](mailto:seamus@faraday.ai) because that's a critical bug!

Happy Signal Hunting!

-Seamus

PS. When this brand loaded our model into Facebook, we immediately started beating Facebook's internal model, even though we were still in learning phase...

![Beating Facebook](/images/blog/stripe-adset.png)

[Sign up](https://faraday.ai/signup) free today.

If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

How to determine buyer signal from Stripe payments

**See our complete guide:** [Personalization tactics that actually drive conversions](/blog/personalization-tactics-that-drive-conversions).

---

Every marketer knows that personalization is critical. Once considered a nice touch to spruce up campaigns, today it’s a full-on necessity for digital-first brands. Plus, the bar for personalization is getting higher and higher, to the point that consumers expect real-time, highly-curated interactions on every marketing channel.

The groundwork for personalized marketing begins with knowing your customer. Before you can so much as sketch out a campaign idea, you need to know who you’re marketing to. However, even this seemingly simple question is getting a lot more complicated thanks to the phase-out of the third-party cookie.

Nowadays marketers need to take stock of their cookieless identity resolution options and ensure they have the rich customer profiles they need to achieve true personalization.

Looking for cookieless options for enhanced customer data? Eager to add more sophisticated personalization strategies to your arsenal? Here's what this article covers:

- Customer insights and segmentation
- Personalized advertising
- Personalized product recommendations
- Personalized promotions

## Personalization starts with segmentation

Third-party cookies have always gone hand in hand with personalized marketing, since they’ve unlocked access to swaths of data that marketers can use for surgically precise segmentation. But [the end of the cookie](https://faraday.ai/blog/death-of-third-party-cookies-strategies) doesn’t have to sound the death knell for highly personalized ads. If anything, it’s ushering in a new era of more meaningful personalization strategies, where people see content that speaks to them as people, not a click to be retargeted.

### Adding consumer data to the mix

Segmentation is still key to achieving human-centric interactions. Without third-party cookies, you can certainly segment your first-party data, but frankly, it won’t offer enough nuance to craft personalized campaigns. Now more than ever, marketers need to prioritize [data enrichment from responsible sources](https://faraday.ai/features/responsible-ai). This cookieless consumer data enables brands to build out rich customer profiles based on a wealth of demographic, psychographic, and firmographic data.

Once you’ve enhanced your first-party data, you’ll have the full picture you need to segment your audience in a meaningful way. This might mean segmenting by income level, household size, or even hobbies and interests. The savviest brands go even further and leverage machine learning algorithms to parse through billions of data points, surface the most significant patterns in their audience, and cluster their customers into highly accurate persona groups.

All of this paves the way for true [personalization at scale](https://faraday.ai/predict/category/persona_assignments). By segmenting your audience into clusters or personas, you can craft highly personalized campaigns that speak to each unique segment and orchestrate more meaningful interactions across channels.

If that sounds like a plan, read on for the strategies you can use to start bringing those campaigns to life.

## Personalizing ad campaigns

Personalized advertising is a fantastic tool for connecting with good fits across the buyer’s journey. Since consumers are bombarded with ads all day long — social media, TV, radio, podcast, flyers, billboards — meaningfully personalized ads help cut through the noise and create an immediate connection.

### Using personas to create custom audiences

It’s one thing to segment your existing audience based on [rich consumer data](https://faraday.ai/features/data), but quite another to personalize ads for potential customers who aren’t yet known to you. Lookalike audience options from platforms like Facebook used to reliably expand brands’ reach to new pools of good fits, but with the advent of iOS 14 and Apple’s privacy updates, these lookalikes have lost a lot of signal and are less cost-effective by the day. You’ll need additional strategies up your sleeve to reach new audiences.

One such strategy involves seeding Facebook lookalikes with AI-driven personas, which gives Facebook a more defined seed audience to work with and enhances its ability to surface consumers that fit your personas. Think of it like giving Facebook’s algorithm a bit of a boost. Then, you can craft highly personalized ad campaigns for individuals you would never have reached otherwise. Brands doing this typically see increased conversion rates and lower CPA.

Another option is to skip Facebook’s native lookalikes and leverage AI to build your own custom lookalike audiences. In this strategy, algorithms can uncover great fits from an identity graph and match them to your brand’s personas, providing you with fresh audiences to upload to Facebook or any other ad platform — greatly expanding your audience beyond what these platforms would normally surface.

With these in place, and your personas already established, you can personalize your ad campaigns with confidence in spite of having never interacted with the audience before.

## Personalized product recommendations

Once someone has made themselves known to your brand— for instance, subscribing to your email list, creating a free online account, or signing up for a rewards program — personalized product recommendations are another invaluable strategy. You can leverage insights in your CRM to create a truly personalized experience on your website, in-store, via email marketing, and more.

Crucially, consumers have high expectations for these experiences. In such a competitive market, where shoppers can find dozens of similar products and services online with just a quick Google search, it’s the real-time, personalized interactions that help brands stand out from the crowd and earn their trust. Orchestrating this type of personalization requires sophisticated automation and algorithms to make recommendations instantaneously and at scale.

### Personalized recommendations based on journey stage

Truly personalized recommendations tend to come back to great life cycle marketing. Once you have an accurate picture of where current and potential customers fall on the buyer’s journey, you can map recommendations to each stage — such as promoting a product quiz or survey to your social followers in order to collect more first-party data, launching email campaigns that are tailored to your unique personas, or nudging customers about new releases.

With most modern ESPs and email marketing platforms, such as Iterable, Klaviyo, and MailChimp, you can also build powerful workflows to trigger personalized product recommendations based on specific actions. For a financial organization, this could look like an email with more information on home loan options after a lead schedules a consultation. For an apparel brand, it could mean triggering an alert that an item has come back in stock for every rewards member who'd added it to their virtual wish list.

By aligning recommendations with an individual’s journey stage and automating them to be delivered at the perfect moment, you create the real-time, personalized interactions today’s consumers crave — and you’ll see the results in your revenue.

### Personalized recommendations with predicted purchases

For marketing teams that have already dialed up product recommendations and want to offer even more valuable suggestions to potential customers, there’s also the option to take it one step further with predictive journey mapping. Rather than make an assumption about what a person is going to purchase, this entails using AI to actually predict what they’ll get.

These [predictions](https://faraday.ai/features/predictions) are far more accurate than any other product recommendation you could make, allowing your marketing efforts to become less like a sales pitch and more like the advice of a trusted friend. Just like automated email campaigns, these predictions can be triggered from a platform like Iterable to deliver the perfect recommendation to a given individual at the perfect time.

## Personal discounts and pomotions

Sometimes consumers need a little extra encouragement to get them to commit. For these individuals, a well-timed promotion or discount can make an enormous difference. The trick is to tailor the promotion to the consumer in order to protect your bottom line — after all, you don’t want to offer 15% off or a first month free to someone who’s likely to pay full price, but you definitely want that to be an option for valuable lead that’s on the fence.

### Using personalized discounts to build brand affinity

To be certain that you’re only offering discounts to the customers that matter most, customer lifetime value (LTV) predictions can eliminate an enormous amount of guesswork. LTV is difficult to determine on your own, but a wide variety of AI tools available to marketers make it a piece of cake.

With LTV predictions uploaded to Salesforce, HubSpot, or wherever you track your customer information, it becomes much clearer how to personalize discounts for distinct audiences. You may want to try rewarding your most valuable customers with a discount, as a small thanks for their brand loyalty, or you may want to test out special offers for customers with middling LTV to see if that extra push inspires them to spend even more.

### Using personalized discounts to retain high-value customers

Another proven discount strategy relies on [churn prediction](https://faraday.ai/predict/category/churn_scores). As with all things predictive, AI is the best tool to utilize here, with churn predictions at least five times more accurate than those that traditional methods produce. Once you understand exactly which customers are at the highest risk of falling away, you can intervene proactively, with a personalized touch, rather than reactively.

For instance, you can tailor the discount to the level of churn risk. High-value customers at great risk of churn might deserve a generous discount offer, whereas customers with lower risk can get a smaller one. And just like with product recommendations and LTV predictions, churn predictions can be delivered seamlessly to wherever your team needs them most, whether that’s triggering an email drip or refreshing the customer profile in your CRM — empowering you to provide a seamless customer experience.

## Level up your personalization strategies today

Whether you’re just getting started with personalization or are looking to truly master 1:1 communication at scale, you’ve got numerous opportunities available at every stage of the buyer’s journey. Consumers overwhelmingly prefer personalized offers, and an increasingly crowded digital space is making it harder and harder to gain an edge — so it’s more important than ever to double down on personalization before it’s too late.

Explore practical personalization strategies that leading brands use to create more enjoyable experiences across the customer lifecycle.

5 ways to personalize experiences across the customer lifecycle

With brand trust at an [all-time low](https://www.havasgroup.com/press_release/havas-meaningful-brands-report-2021-finds-we-are-entering-the-age-of-cynicism/), it’s a good time to ask: is your organization doing the right thing with consumer data? 50% of consumers say they’d pay more for a brand that takes a stand on important issues. With privacy at the heart of a global reassessment of social media, big tech, and surveillance, organizations at the vanguard of this issue will win in 2022.

The good news is that playing fair isn’t all that complicated. “When in doubt, follow your heart,” says Andy Rossmeissl, CEO of Faraday, the leading customer prediction platform. “Our motto from the start has been ‘don’t be creepy,’ and it’s remarkable how far that can take you.”

Looking for specifics? Here are the top 3 rules for using customer data ethically:

## Rule 1: Use identifiers people can recite from memory.

This rule sounds simple, but it’s surprisingly powerful. “Ask your mom if she knows any of her tracking cookies by heart,” suggests Rossmeissl. “Ask yourself what your mobile device ID is.” The digital surveillance panopticon relies on these [long, unique strings](https://www.quora.com/What-does-the-code-for-a-tracking-cookie-look-like) of gibberish assigned to virtually every eyeball in the world.

Instead, ask yourself what identifiers you have that you can actually recite. “I have an address on my mailbox for all the world to see, and you probably do too,” says Rossmeissl. “I put my number in the phone book, and my email address is all over the Internet.”

Relying on real-world identifiers, or “known identities,” has the additional benefit of steering brands away from collecting and using anonymous behavior, such as data collected from unidentified web and mobile visitors. “Until somebody has identified themselves to you," explains Rossmeissl, "tracking their behavior is, frankly, creepy.”

## Rule 2: Use data that comes from commercial behavior

“I call it the ‘laying in bed rule,’” Rossmeissl shares. “When I’m laying in bed and I read a news article on my tablet or like something on Instagram, that’s private. Watching me do that is creepy.”
The surveillance of private behavior has rapidly normalized, much to the dismay of both privacy experts and, increasingly, consumers alike. A 2019 study found that nearly two thirds of Americans believe they can’t go through daily life without their data being collected.

And yet marketers have long known the power of data-driven growth practices. How can brands [use consumer data while still respecting privacy](https://faraday.ai/features/responsible-ai)? Rossmeissl shares:

> We thought long and hard about this when we started Faraday in 2012. We drew on anthropology, architecture, history, and other disciplines for our research, but the result was fairly simple.

> It turns out that commerce is one of the main reasons humans started to live in communities. Markets are some of the earliest forms of non-residential architecture we know of. No matter how you look at it, commerce and the marketplace have been with us longer than virtually any other social structure.

This provides a clue: humans are quite used to buying and selling, and they’re used to doing it in public. We have evolved an innate ability to operate in the marketplace, and an accompanying understanding that this behavior is public.

“What we put in our cart at the grocery store is public,” says Rossmeissl. “What we buy walking around a farmer's market is public. And when we buy online, there's an extension of this understanding. Engaging in commerce is a collaborative act — there's a buyer and a seller — and also a performative one.”

The upshot is that we instinctually treat data arising from commercial transactions as fair game. Luckily, this kind of data is not only the most ethical, but also some of the most powerful available for predicting subsequent consumer behavior. Faraday includes over 300 data points like this on virtually every US adult, but you can also procure your own from reputable licensing groups like Epsilon.

## Rule 3: Let people easily opt out

“Imagine being in a store and being told you can’t leave,” says Rossmeissl. “That goes from creepy to nightmarish very quickly.”

The right to opt out is at the heart of CCPA, the landmark privacy legislation. And while the law only protects California residents, brands should meet its requirements across the board. Luckily, that’s quite simple: provide an easy to use opt-out link, and pass along opt-outs both up- and down-stream to your partners.

Beyond the fact that [honoring opt-outs is actually good for business](https://sendgrid.com/blog/honoring-opt-outs-is-good-business/), it’s the right thing to do.

Is your organization doing doing the right things with consumer data? Here are 3 common-sense rules to follow.

3 common-sense rules for ethical data use 

**UPDATED FOR 2017** Now with easy subquery support and a more sensible argument order!

We adapted [this excellent piece by Dmitri Fontaine](http://tapoueh.org/blog/2014/02/21-PostgreSQL-histogram) and turned it into a function.

It can be invoked like this:

```
SELECT * FROM histogram($table_name_or_subquery, $column_name);
```

. . . to give sweet results like this, in a check of the distribution of 2016 political contributions in Vermont:

```
fec=# SELECT * FROM histogram('(SELECT * FROM small_donors_vt LIMIT 50000)', 'transaction_amt');

 bucket |   range   | freq |       bar
--------+-----------+------+-----------------
      1 | [0,9]     | 2744 | ******
      2 | [10,19]   | 5630 | *************
      3 | [20,29]   | 6383 | ***************
      4 | [30,39]   | 1290 | ***
      5 | [40,49]   |  369 | *
      6 | [50,59]   | 3541 | ********
      7 | [60,69]   |  174 |
      8 | [70,79]   |  313 | *
      9 | [80,89]   |  171 |
     10 | [90,99]   |   65 |
     11 | [100,109] | 2363 | ******
     12 | [110,119] |   51 |
     13 | [120,129] |  115 |
     14 | [130,139] |   32 |
     15 | [140,146] |   11 |
     16 | [150,159] |  187 |
     17 | [160,169] |   24 |
     18 | [170,177] |   33 |
     19 | [180,189] |   19 |
     20 | [191,199] |   24 |
     21 | [200,200] |  795 | **

```

Use it yourself by adding this to your postgres setup:

```
CREATE OR REPLACE FUNCTION histogram(table_name_or_subquery text, column_name text)
RETURNS TABLE(bucket int, "range" numrange, freq bigint, bar text)
AS $func$
BEGIN
RETURN QUERY EXECUTE format('
  WITH
  source AS (
    SELECT * FROM %s
  ),
  min_max AS (
    SELECT min(%s) AS min, max(%s) AS max FROM source
  ),
  histogram AS (
    SELECT
      width_bucket(%s, min_max.min, min_max.max, 20) AS bucket,
      numrange(min(%s)::numeric, max(%s)::numeric, ''[]'') AS "range",
      count(%s) AS freq
    FROM source, min_max
    WHERE %s IS NOT NULL
    GROUP BY bucket
    ORDER BY bucket
  )
  SELECT
    bucket,
    "range",
    freq::bigint,
    repeat(''*'', (freq::float / (max(freq) over() + 1) * 15)::int) AS bar
  FROM histogram',
  table_name_or_subquery,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name
  );
END
$func$ LANGUAGE plpgsql;
```

Note:

1. You don't need to use a subquery - you can also just provide a table name as the first argument.
1. Subqueries are useful for sampling. For example, `(SELECT * FROM bigtable LIMIT 500 ORDER BY RANDOM()`. Don't forget parentheses!
1. You can call this as `SELECT histogram()` or `SELECT * FROM histogram()`. The latter form is much more legible!

Happy querying!

Learn how to create in-table histograms with PostgreSQL.

How to do histograms in PostgreSQL

At Faraday, we've long used [csvkit](https://csvkit.readthedocs.io/en/latest/) to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to [xsv](https://github.com/BurntSushi/xsv).

xsv is a fast CSV-parsing toolkit written in [Rust](https://www.rust-lang.org/en-US/) that mostly matches the functionality of csvkit (including the clutch ability to pipe between modules), with a few extras tacked on (like smart sampling). Did I mention it's fast? In a standup comparison, I ran the "stats" module of XSV against "csvstat" from csvkit, on a 30k-line, 400-column CSV file:

- Python-based csvkit chews through it in a respectable-and-now-expected 4m16s.

- **xsv takes 1.8 seconds.** I don't even have time for a sip of my coffee.

The difference between csvkit and xsv is partly defined by scale; both tools are plenty fast on smaller datasets. But once you get into 10MB-and-upward range, xsv's processing speed pulls away exponentially.

If you've been using csvkit forever (like me), or if you want to be able to transform and analyze CSVs without loading them into a DB, give xsv a shot:

### Install Rust

```bash
curl https://sh.rustup.rs -sSf | sh
```

. . . which also gives you the rust package manager `cargo`, which lets you:

### Install xsv

```bash
cargo install xsv
```

Then be sure your PATH is configured correctly:

```bash
export PATH=~/.cargo/bin:$PATH
```

. . . and try it out on a demo CSV with 10k rows, some messy strings, and multiple data types:

```bash
curl https://gist.githubusercontent.com/wboykinm/044e2af62fc0c7f77e17f6ccd55b8fb0/raw/fca391e6c03a06a7be770fefca6c47a9acdd2305/mock_data.csv \
| xsv stats \
| xsv table
```

_(`xsv table` formats the data so it's readable in the console):_

```bash
field           type     sum                 min                  max                  min_length  max_length  mean                stddev
id              Integer  5005000             1                    1000                 1           4           500.49999999999994  288.6749902572106
first_name      Unicode                      Aaron                Willie               3           11
last_name       Unicode                      Adams                Young                3           10
email           Unicode                      aadamsp5@senate.gov  wwrightd8@upenn.edu  12          34
gender          Unicode                      Female               Male                 4           6
ip_address      Unicode                      0.111.40.87          99.50.37.244         9           15
value           Unicode                      $1007.98             $999.37              0           8
company         Unicode                      Abata                Zoovu                0           13
lat             Float    243963.82509999987  -47.75034            69.70287             0           9           24.42080331331331   24.98767816017553
lon             Float    443214.19009999954  -179.12198           170.29993            0           10          44.36578479479489   71.16647723898215
messed_up_data  Unicode                      !@#$%^&*()           𠜎𠜱𠝹𠱓𠱸𠲖𠳏       0           393
version         Unicode                      0.1.1                9.99                 3           14
```

Happy parsing!

A data science hack to process CSVs faster using XSV.

How we made our CSV processing 142x faster

The Rust compiler just saved me from a nasty threading bug. I was working on an open source development tool for Docker apps with lots of microservices, and I decided to parallelize the routine that transformed `docker-compose.yml` files. This was mostly an excuse to check out the awesome [rayon][] library, but it turned into a great example of what real-world Rust development is like.

[rayon]: https://github.com/nikomatsakis/rayon

The original routine looked something like this:

```rust
/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {
            continue;
        }

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        try!(file.make_standalone(&self.pods_dir()));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));
        try!(file.write_to_path(out_path));
    }
    Ok(())
}
```

To convert this to a parallel loop, I started by changing:

```rust
for pod in &self.pods {
```

To:

```rust
self.pods.par_iter().map(|pod| -> Result<()> {
```

Here, `Result<()>` means "this closure might return an error, or it might return an empty tuple, basically `void`." (For more information on Rust error handling, check out [the Rust book][errors].) But it couldn't be that easy, could it?

[errors]: https://doc.rust-lang.org/book/ch09-00-error-handling.html

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

```
error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied
   --> src/project.rs:397:30
    |
397 |         self.pods.par_iter().map(|pod| -> Result<()> {
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    |
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/project.rs:397:34: 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`
```

The key bit to take away here is that "`Sync` is not satisfied" for the trait `plugins::PluginTransform`. In Rust, `Sync` is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my `PluginTransform` objects live inside my `plugins::Manager` type, which in turn lives inside my `Project`. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that `PluginTransform` inherits from the `Plugin` trait:

```rust
pub trait PluginTransform: Plugin {
```

So we can fix this problem by changing the declaration of `Plugin` from:

```rust
pub trait Plugin {
```

To:

```rust
pub trait Plugin: Sync {
```

This says, "All types implementing `Plugin` must also allow me to access them from multiple threads." With that fixed, I re-run `cargo test` and get a similar message:

```
error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:230:6
    |
230 | impl plugins::Plugin for Plugin {
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied
    |
```

This time, I change:

```rust
trait GenerateToken: Debug {
```

To:

```rust
trait GenerateToken: Debug + Sync {
```

This says, "All types which implement `GenerateToken` must also implement `Debug` (so I can print them), as well as `Sync` (so I can share them). We're making progress!

Re-running `cargo test`, however, reveals the actual bug, and it would have been a nightmare to debug:

```
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:124:6
    |
124 | impl GenerateToken for MockVault {
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied
    |
```

Uh-oh. In our test harness, we have a type `MockVault`, which contains an `Rc<RefCell<_>>`:

```rust
type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
#[derive(Debug)]
#[cfg(test)]
struct MockVault {
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,
}

```

We use the `MockVault` to simulate a connection to Hashicorp's [Vault][], a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the `calls` member here to record all the requests that we would have made to Vault.

[Vault]: https://www.vaultproject.io/

The type `Rc<RefCell<_>>` is a hack. The `GenerateToken` API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see [the Rust book][mutability].) But `Rc<RefCell<_>>` is a lightweight mechanism designed for single-threaded code.

[mutability]: https://doc.rust-lang.org/beta/book/mutability.html

The fix is to replace `Rc<RefCell<_>>` with `Arc<RwLock<_>>`, which is fully thread-safe:

```rust
type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;
```

Once we make this change, Rust reminds us to change all the code that accesses `calls`, too:

```
error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:132:20
    |
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));
    |                    ^^^^^^^^^^
    |
...
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:368:23
    |
368 |     let calls = calls.borrow();
    |                       ^^^^^^
    |
```

Phew. If we hadn't caught that `Rc<RefCell<_>>`, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites _could_ trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

## Wrapping it up

Rust also complains about that fact that we're calling `continue` from inside a closure:

```
error[E0267]: `continue` inside of a closure
   --> src/project.rs:402:17
    |
402 |                 continue;
    |                 ^^^^^^^^ cannot break inside of a closure
```

We can fix this by moving this code:

```rust
// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {
    continue;
}
```

...into a `filter` call on our parallel iterator:

```rust
self.pods.par_iter()
    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {
```

The final problem is that we now have a parallel computation that returns a bunch of `Result<()>` objects, and we need to boil them down to a single `Result<()>` like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

```rust
    .map(|pod| -> Result<()> {
        // ...
    }
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))
    .unwrap_or(Ok(()))
```

This says, "Given any two results named `result1` and `result2`, take `result1` if it's an error, otherwise take `result2`." It's basically a short-circuit `&&` operator, but for `Result` values instead of booleans. I've filed an issue suggesting that `rayon` should provide those last two lines as a built-in function.

[issue113]: https://github.com/nikomatsakis/rayon/issues/113

And yes, that's a genuine parallel [map-reduce][] in Rust!

[map-reduce]: http://research.google.com/archive/mapreduce.html

So here's our final loop, using work-stealing parallelism:

```rust
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.
    self.pods.par_iter()

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            try!(file.make_standalone(&self.pods_dir()));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));
            try!(file.write_to_path(out_path));
            Ok(())
        })

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))
        .unwrap_or(Ok(()))
}
```

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

A detailed example of how to parallelize a loop with Rust and rayon.

Tech