In the world of machine learning, we often encounter two important concepts: the raw propensity score and probability. Though they may sound similar, they represent distinct aspects of a machine learning model's predictions.

## Example of propensity scores vs predicted probabilities

Imagine you have a model that predicts whether it will rain tomorrow. The raw propensity score is like the model's initial hunch or gut feeling about the outcome. It's a numeric value that the model calculates based on various factors like historical weather data, humidity, temperature, and more. The [propensity](https://faraday.ai/blog/propensity-models-predict-customer-behavior) score indicates how likely the model thinks it will rain or not. However, the propensity score itself doesn't give us a clear understanding of how confident the model is in its prediction.

This is where [probability](/blog/faraday-probability-scores) comes into play. The predicted probability aims at providing an answer to the question “how likely is the prediction?” It represents the likelihood of a particular outcome occurring. For example, a score of 0.75 for rain means the model is fairly confident it will rain tomorrow. However, a probability of 0.75 indicates that there is a 75% chance that it will rain tomorrow.

In summary, the raw propensity score is the model's initial guess, while probability quantifies the model's confidence in that guess. Understanding this difference can help us interpret machine learning [predictions](/features/predictions) and make better-informed decisions based on the model's level of certainty.

## Understanding model calibration

The model calibration process refers to the alignment between a model's predicted probabilities and the actual likelihood of events occurring. A good calibration ensures that the predicted probabilities are close to the true theoretical probabilities. Not all machine learning models are perfectly calibrated after training. Some models might be overconfident, providing probabilities that are too extreme, while others might be underconfident, consistently underestimating or overestimating probabilities.

### Importance of model calibration

Calibration is crucial because it impacts decision-making. If a model is well-calibrated, we can trust its probability estimates and use them to make informed choices, such as whether to take an umbrella based on the probability of rain.

Calibrating a model involves post-processing its probability scores to bring them in line with the true probabilities. Various calibration techniques can be employed, such as Platt scaling or isotonic regression, which adjust the raw propensity scores to produce well-calibrated probabilities. These techniques are typically applied during model validation and fine-tuning to ensure that the model's predictions are more accurate and reliable.

The calibration process ensures that when the model predicts a probability, it reflects the true likelihood of the event occurring. For example, if a model predicts a 70% probability of rain, it should be accurate in approximately 70% of the cases where it makes that prediction.

In summary, model calibration in machine learning is all about making sure a model's predicted probabilities accurately reflect the real-world likelihood of events happening. A well-calibrated model is more reliable and can lead to more effective decision-making.

## Benefits of using predicted probabilities

Using predicted probabilities instead of raw propensity scores offers several significant benefits that enhance the interpretability, decision-making, and overall performance of the models:

### Interpretability

Predicted probabilities provide a more intuitive and understandable representation of a model's output. Raw propensity scores are often abstract values that lack clear context, while probabilities are easily interpretable as the likelihood of an event occurring. This makes it easier for stakeholders and end-users to comprehend the model's predictions and build trust in its results.

### Uncertainty quantification

Predicted probabilities convey the model's level of certainty about its predictions. Models can provide probability scores ranging from 0 to 1, indicating their confidence in the predicted outcomes. This uncertainty quantification is valuable in decision-making, as it allows users to consider the risk associated with a prediction and adjust their actions accordingly.

### Threshold selection

Predicted probabilities offer flexibility in choosing decision thresholds. Depending on the specific application and requirements, decision-makers can adjust the probability threshold for classifying positive and negative outcomes. This adaptability allows for a fine-tuning of the model's behavior to suit the desired trade-offs between false positives and false negatives. For example, in the case of a lead conversion, a profitability threshold can be established by taking into account cost of acquisition for each lead, expected profit from actual customers, and probability of conversion.

### Calibration and reliability

When models are well-calibrated, predicted probabilities accurately reflect the real-world likelihood of events. Calibrated probabilities provide more reliable estimates, leading to improved decision-making and better performance in critical applications.

### Ensemble and stacking

Predicted probabilities facilitate model ensembling and stacking techniques. By combining the probabilities from multiple models, ensemble methods can often yield better overall performance compared to individual models or raw propensity score-based approaches.

## Benefits and drawbacks of raw propensity scores

Using raw predictions from models instead of calibrated scores has its own set of pros and cons. Let's explore them.

### Pros of using raw propensity scores:

**Simplicity and Speed**: Raw predictions are the direct output of the model without any additional post-processing, making them easier and faster to obtain. In real-time applications or scenarios where efficiency is critical, using raw propensity scores can be advantageous.

**Suitable for Ranking**: Raw propensity scores can still be useful for ranking predictions. Even though they may not directly represent probabilities, they can help rank instances based on their likelihood of being a positive or negative outcome, which is often sufficient in certain applications.

### Cons of using raw predictions:

**Inconsistent Ranking**: Raw propensity scores might not consistently reflect the true likelihood of events occurring. Two instances with similar raw propensity scores might have significantly different actual probabilities, leading to inaccuracies in ranking and decision-making.

**Suboptimal Decision Thresholds**: Using raw predictions often means relying on fixed decision thresholds (e.g., 0.5 for binary classification), which might not be suitable for all applications. This can lead to suboptimal trade-offs between precision and recall.

**Misinterpretation of Scores**: When raw propensity scores are not calibrated, users might incorrectly interpret them as reliable probabilities, leading to misguided decisions and a lack of trust in the model's predictions.

In summary, while using raw predictions can be straightforward and computationally efficient, it comes with the trade-off of lacking calibrated probabilities and potential inconsistencies in ranking. In many real-world applications, calibrated scores are preferred as they provide interpretable probabilities, better decision thresholds, and improved trust in the model's predictions.

## When should you use propensity score percentiles vs. predicted probabilities?

In the field of machine learning, the choice between using percentiles and probabilities depends on the specific context and the nature of the problem being addressed.

### Using propensity score percentiles

**Outliers are important**: If you are concerned about extreme values or outliers in your data, percentiles can be more robust in representing the distribution. For instance, the 95th percentile captures the value below which 95% of the data lies, making it less sensitive to extreme values.

**Data skewness**: When dealing with heavily skewed datasets, especially those with long tails, percentiles can be more informative than probabilities. Percentiles provide a clearer understanding of the data's spread and central tendencies, even in the presence of skewed distributions.

**Interpretability**: Percentiles can be easier to interpret and explain to non-technical stakeholders. For example, stating that a value is at the 75th percentile means that it is higher than 75% of the other values in the dataset.

### Using predicted probabilities

**Well-calibrated predictions**: Probabilities are essential when you need calibrated predictions. If you want reliable estimates of the likelihood of certain events, probabilities are more suitable. Well-calibrated models can provide accurate and informative probabilities for decision-making.

**Decision thresholds**: In many applications, using probabilities allows you to set appropriate decision thresholds based on your specific needs. For instance, in medical testing, you might want to set a higher threshold to prioritize precision over recall, depending on the consequences of false positives and false negatives.

**Uncertainty quantification**: Probabilities are crucial for quantifying uncertainty around predictions. Confidence intervals and prediction intervals can be constructed using probabilities, giving a range of likely outcomes.

**In summary, use percentiles when you need robust representation of data distribution and are concerned about outliers or skewed data. On the other hand, use probabilities when you require calibrated predictions, decision thresholds, and uncertainty quantification, especially in the context of machine learning models.**

**Ready to try Faraday? [Create a free account](https://faraday.ai/signup).**

Propensity scores and predicted probabilities represent distinct aspects of a machine learning model's predictions. Understand the difference and when you should use them.

Propensity vs probability: Understanding the difference between raw scores and probabilities

If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

Last month, a friendly brand exported about 10,000 payment records from Stripe as a CSV. Then they uploaded them to Faraday and, within 24 hours, got their results:

![Buyer signal showing importance of different features](/images/blog/buyer-signal.png)

Let's focus on one feature - "Science/new technology." If a person is interested in this, they are much more likely to buy this brand's products:

![Likelihood to buy based on science interest](/images/blog/science-directionality.png)

Faraday is batteries-included - we know things like whether people like science thanks to our [built-in consumer data](https://faraday.ai/features/data).

Now, this brand can predict the likelihood of more than **240 million Americans** to buy their product. Faraday can take any list of people and sort it by likelihood to buy... or Faraday can generate the list in the first place, using a ["Deployment"](https://faraday.ai/docs/abstractions/deployments).

So... how did they get this? First, by doing a [Stripe Payment Report](https://support.stripe.com/questions/exporting-payment-reports). In particular, doing a [**Successful Payments**](https://dashboard.stripe.com/payments?status%5B%5D=successful) report.

![You want Succeeded payments](/images/blog/stripe-payment-export-succeeded.png)

Faraday only needs the following fields:

![PII, date, and amount fields](/images/blog/stripe-custom-columns.png)

(Later, you can set up an automated integration. But for now, since this is a relatively stable model based on inherent consumer characteristics, it should be stable for at least a few weeks.)

In Faraday, you map the fields:

![Address mapping](/images/blog/stripe-dataset-identity-mapping.png)

Then, [follow the standard process for importing transactions](https://faraday.ai/docs/cohorts/customers_from_hosted_csv#option-a-each-row-represents-a-transaction-event).

And finally, you make a [likely buyers model](/docs/abstractions/outcomes).

In total, if this takes longer than 15 minutes, [email the CTO](mailto:seamus@faraday.ai) because that's a critical bug!

Happy Signal Hunting!

-Seamus

PS. When this brand loaded our model into Facebook, we immediately started beating Facebook's internal model, even though we were still in learning phase...

![Beating Facebook](/images/blog/stripe-adset.png)

[Sign up](https://faraday.ai/signup) free today.

If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

How to determine buyer signal from Stripe payments

Every marketer knows that personalization is critical. Once considered a nice touch to spruce up campaigns, today it’s a full-on necessity for digital-first brands. Plus, the bar for personalization is getting higher and higher, to the point that consumers expect real-time, highly-curated interactions on every marketing channel.

The groundwork for personalized marketing begins with knowing your customer. Before you can so much as sketch out a campaign idea, you need to know who you’re marketing to. However, even this seemingly simple question is getting a lot more complicated thanks to the phase-out of the third-party cookie.

Nowadays marketers need to take stock of their cookieless identity resolution options and ensure they have the rich customer profiles they need to achieve true personalization.

Looking for cookieless options for enhanced customer data? Eager to add more sophisticated personalization strategies to your arsenal? Here's what this article covers:

- Customer insights and segmentation
- Personalized advertising
- Personalized product recommendations
- Personalized promotions

## Personalization starts with segmentation

Third-party cookies have always gone hand in hand with personalized marketing, since they’ve unlocked access to swaths of data that marketers can use for surgically precise segmentation. But [the end of the cookie](https://faraday.ai/blog/death-of-third-party-cookies-strategies) doesn’t have to sound the death knell for highly personalized ads. If anything, it’s ushering in a new era of more meaningful personalization strategies, where people see content that speaks to them as people, not a click to be retargeted.

### Adding consumer data to the mix

Segmentation is still key to achieving human-centric interactions. Without third-party cookies, you can certainly segment your first-party data, but frankly, it won’t offer enough nuance to craft personalized campaigns. Now more than ever, marketers need to prioritize [data enrichment from responsible sources](https://faraday.ai/features/responsible-ai). This cookieless consumer data enables brands to build out rich customer profiles based on a wealth of demographic, psychographic, and firmographic data.

Once you’ve enhanced your first-party data, you’ll have the full picture you need to segment your audience in a meaningful way. This might mean segmenting by income level, household size, or even hobbies and interests. The savviest brands go even further and leverage machine learning algorithms to parse through billions of data points, surface the most significant patterns in their audience, and cluster their customers into highly accurate persona groups.

All of this paves the way for true [personalization at scale](https://faraday.ai/predict/category/persona_assignments). By segmenting your audience into clusters or personas, you can craft highly personalized campaigns that speak to each unique segment and orchestrate more meaningful interactions across channels.

If that sounds like a plan, read on for the strategies you can use to start bringing those campaigns to life.

## Personalizing ad campaigns

Personalized advertising is a fantastic tool for connecting with good fits across the buyer’s journey. Since consumers are bombarded with ads all day long — social media, TV, radio, podcast, flyers, billboards — meaningfully personalized ads help cut through the noise and create an immediate connection.

### Using personas to create custom audiences

It’s one thing to segment your existing audience based on [rich consumer data](https://faraday.ai/features/data), but quite another to personalize ads for potential customers who aren’t yet known to you. Lookalike audience options from platforms like Facebook used to reliably expand brands’ reach to new pools of good fits, but with the advent of iOS 14 and Apple’s privacy updates, these lookalikes have lost a lot of signal and are less cost-effective by the day. You’ll need additional strategies up your sleeve to reach new audiences.

One such strategy involves seeding Facebook lookalikes with AI-driven personas, which gives Facebook a more defined seed audience to work with and enhances its ability to surface consumers that fit your personas. Think of it like giving Facebook’s algorithm a bit of a boost. Then, you can craft highly personalized ad campaigns for individuals you would never have reached otherwise. Brands doing this typically see increased conversion rates and lower CPA.

Another option is to skip Facebook’s native lookalikes and leverage AI to build your own custom lookalike audiences. In this strategy, algorithms can uncover great fits from an identity graph and match them to your brand’s personas, providing you with fresh audiences to upload to Facebook or any other ad platform — greatly expanding your audience beyond what these platforms would normally surface.

With these in place, and your personas already established, you can personalize your ad campaigns with confidence in spite of having never interacted with the audience before.

## Personalized product recommendations

Once someone has made themselves known to your brand— for instance, subscribing to your email list, creating a free online account, or signing up for a rewards program — personalized product recommendations are another invaluable strategy. You can leverage insights in your CRM to create a truly personalized experience on your website, in-store, via email marketing, and more.

Crucially, consumers have high expectations for these experiences. In such a competitive market, where shoppers can find dozens of similar products and services online with just a quick Google search, it’s the real-time, personalized interactions that help brands stand out from the crowd and earn their trust. Orchestrating this type of personalization requires sophisticated automation and algorithms to make recommendations instantaneously and at scale.

### Personalized recommendations based on journey stage

Truly personalized recommendations tend to come back to great life cycle marketing. Once you have an accurate picture of where current and potential customers fall on the buyer’s journey, you can map recommendations to each stage — such as promoting a product quiz or survey to your social followers in order to collect more first-party data, launching email campaigns that are tailored to your unique personas, or nudging customers about new releases.

With most modern ESPs and email marketing platforms, such as Iterable, Klaviyo, and MailChimp, you can also build powerful workflows to trigger personalized product recommendations based on specific actions. For a financial organization, this could look like an email with more information on home loan options after a lead schedules a consultation. For an apparel brand, it could mean triggering an alert that an item has come back in stock for every rewards member who'd added it to their virtual wish list.

By aligning recommendations with an individual’s journey stage and automating them to be delivered at the perfect moment, you create the real-time, personalized interactions today’s consumers crave — and you’ll see the results in your revenue.

### Personalized recommendations with predicted purchases

For marketing teams that have already dialed up product recommendations and want to offer even more valuable suggestions to potential customers, there’s also the option to take it one step further with predictive journey mapping. Rather than make an assumption about what a person is going to purchase, this entails using AI to actually predict what they’ll get.

These [predictions](https://faraday.ai/features/predictions) are far more accurate than any other product recommendation you could make, allowing your marketing efforts to become less like a sales pitch and more like the advice of a trusted friend. Just like automated email campaigns, these predictions can be triggered from a platform like Iterable to deliver the perfect recommendation to a given individual at the perfect time.

## Personal discounts and pomotions

Sometimes consumers need a little extra encouragement to get them to commit. For these individuals, a well-timed promotion or discount can make an enormous difference. The trick is to tailor the promotion to the consumer in order to protect your bottom line — after all, you don’t want to offer 15% off or a first month free to someone who’s likely to pay full price, but you definitely want that to be an option for valuable lead that’s on the fence.

### Using personalized discounts to build brand affinity

To be certain that you’re only offering discounts to the customers that matter most, customer lifetime value (LTV) predictions can eliminate an enormous amount of guesswork. LTV is difficult to determine on your own, but a wide variety of AI tools available to marketers make it a piece of cake.

With LTV predictions uploaded to Salesforce, HubSpot, or wherever you track your customer information, it becomes much clearer how to personalize discounts for distinct audiences. You may want to try rewarding your most valuable customers with a discount, as a small thanks for their brand loyalty, or you may want to test out special offers for customers with middling LTV to see if that extra push inspires them to spend even more.

### Using personalized discounts to retain high-value customers

Another proven discount strategy relies on [churn prediction](https://faraday.ai/predict/category/churn_scores). As with all things predictive, AI is the best tool to utilize here, with churn predictions at least five times more accurate than those that traditional methods produce. Once you understand exactly which customers are at the highest risk of falling away, you can intervene proactively, with a personalized touch, rather than reactively.

For instance, you can tailor the discount to the level of churn risk. High-value customers at great risk of churn might deserve a generous discount offer, whereas customers with lower risk can get a smaller one. And just like with product recommendations and LTV predictions, churn predictions can be delivered seamlessly to wherever your team needs them most, whether that’s triggering an email drip or refreshing the customer profile in your CRM — empowering you to provide a seamless customer experience.

## Level up your personalization strategies today

Whether you’re just getting started with personalization or are looking to truly master 1:1 communication at scale, you’ve got numerous opportunities available at every stage of the buyer’s journey. Consumers overwhelmingly prefer personalized offers, and an increasingly crowded digital space is making it harder and harder to gain an edge — so it’s more important than ever to double down on personalization before it’s too late.

Explore practical personalization strategies that leading brands use to create more enjoyable experiences across the customer lifecycle.

5 ways to personalize experiences across the customer lifecycle

With brand trust at an [all-time low](https://www.havasgroup.com/press_release/havas-meaningful-brands-report-2021-finds-we-are-entering-the-age-of-cynicism/), it’s a good time to ask: is your organization doing the right thing with consumer data? 50% of consumers say they’d pay more for a brand that takes a stand on important issues. With privacy at the heart of a global reassessment of social media, big tech, and surveillance, organizations at the vanguard of this issue will win in 2022.

The good news is that playing fair isn’t all that complicated. “When in doubt, follow your heart,” says Andy Rossmeissl, CEO of Faraday, the leading customer prediction platform. “Our motto from the start has been ‘don’t be creepy,’ and it’s remarkable how far that can take you.”

Looking for specifics? Here are the top 3 rules for using customer data ethically:

## Rule 1: Use identifiers people can recite from memory.

This rule sounds simple, but it’s surprisingly powerful. “Ask your mom if she knows any of her tracking cookies by heart,” suggests Rossmeissl. “Ask yourself what your mobile device ID is.” The digital surveillance panopticon relies on these [long, unique strings](https://www.quora.com/What-does-the-code-for-a-tracking-cookie-look-like) of gibberish assigned to virtually every eyeball in the world.

Instead, ask yourself what identifiers you have that you can actually recite. “I have an address on my mailbox for all the world to see, and you probably do too,” says Rossmeissl. “I put my number in the phone book, and my email address is all over the Internet.”

Relying on real-world identifiers, or “known identities,” has the additional benefit of steering brands away from collecting and using anonymous behavior, such as data collected from unidentified web and mobile visitors. “Until somebody has identified themselves to you," explains Rossmeissl, "tracking their behavior is, frankly, creepy.”

## Rule 2: Use data that comes from commercial behavior

“I call it the ‘laying in bed rule,’” Rossmeissl shares. “When I’m laying in bed and I read a news article on my tablet or like something on Instagram, that’s private. Watching me do that is creepy.”
The surveillance of private behavior has rapidly normalized, much to the dismay of both privacy experts and, increasingly, consumers alike. A 2019 study found that nearly two thirds of Americans believe they can’t go through daily life without their data being collected.

And yet marketers have long known the power of data-driven growth practices. How can brands [use consumer data while still respecting privacy](https://faraday.ai/features/responsible-ai)? Rossmeissl shares:

> We thought long and hard about this when we started Faraday in 2012. We drew on anthropology, architecture, history, and other disciplines for our research, but the result was fairly simple.

> It turns out that commerce is one of the main reasons humans started to live in communities. Markets are some of the earliest forms of non-residential architecture we know of. No matter how you look at it, commerce and the marketplace have been with us longer than virtually any other social structure.

This provides a clue: humans are quite used to buying and selling, and they’re used to doing it in public. We have evolved an innate ability to operate in the marketplace, and an accompanying understanding that this behavior is public.

“What we put in our cart at the grocery store is public,” says Rossmeissl. “What we buy walking around a farmer's market is public. And when we buy online, there's an extension of this understanding. Engaging in commerce is a collaborative act — there's a buyer and a seller — and also a performative one.”

The upshot is that we instinctually treat data arising from commercial transactions as fair game. Luckily, this kind of data is not only the most ethical, but also some of the most powerful available for predicting subsequent consumer behavior. Faraday includes over 300 data points like this on virtually every US adult, but you can also procure your own from reputable licensing groups like Epsilon.

## Rule 3: Let people easily opt out

“Imagine being in a store and being told you can’t leave,” says Rossmeissl. “That goes from creepy to nightmarish very quickly.”

The right to opt out is at the heart of CCPA, the landmark privacy legislation. And while the law only protects California residents, brands should meet its requirements across the board. Luckily, that’s quite simple: provide an easy to use opt-out link, and pass along opt-outs both up- and down-stream to your partners.

Beyond the fact that [honoring opt-outs is actually good for business](https://sendgrid.com/blog/honoring-opt-outs-is-good-business/), it’s the right thing to do.

Is your organization doing doing the right things with consumer data? Here are 3 common-sense rules to follow.

3 common-sense rules for ethical data use 

During the pandemic, interest rates for home mortgages reached all-time low. As people were spending significantly more time at home, buying a house — especially for those who relocated out of denser cities — has become a more practical option than renting. In August 2020, [home sales shot up](https://www.businessinsider.com/how-2020-broke-the-housing-market-inventory-could-run-out-2020-9) to reach a 14-year high.

Millennials in particular have made their mark on the real estate business, as they now make up the largest share of home buyers in the country, according to the [National Association of Realtors](https://www.nar.realtor/research-and-statistics/research-reports/home-buyer-and-seller-generational-trends). However, many businesses that service the home, like HVAC contractors, report that they [do not focus on millennials](https://www.achrnews.com/articles/144348-ignore-millennials-if-you-dont-want-to-reach-homeowners) for their marketing campaigns. This is their loss, especially since millennial homeowners are at their fingertips on social media and digital platforms, where targeting younger demographics is a breeze.

So, if you’re in the business of home renovation, DIY tools, residential energy efficiency, or HVAC, consider the following when marketing to millennials — a generation eager to take you up on your services.

## Find millennial homeowners on the right channels

Millennials are a digitally native generation. A [global survey](https://www.gwi.com/reports/millennials) conducted by the Global Web Index in Q3 2020 looked at more than 66,000 internet users and found that millennials ages 24–37 spent an average of 2 hours and 30 minutes each day on social media alone.

If your brand is not capitalizing on social media, digital ads, and OTT options like streaming channels to reach millennial homeowners, you’re missing out on crucial brand awareness and conversion opportunities.

In addition to actively targeting specific audiences, be sure your SEM strategy is up to par, as millennials are big online researchers. Back in the day, newspaper and yellow pages ads may have cut it, but that’s all switched over to online search engines now. Ensure that your areas of interest and/or operation are prioritized in any search your brand may be relevant to.

And once you’ve got your lead on the hook, you can ease their journey to conversion by communicating on the platforms that are most convenient for them; this may be SMS text, video chat, or a website chatbox. From getting a quote for a solar installation to matching a desired paint color, millennials value seamless communication that meets them where they already are in the digital realm.\

## Cash in on millennials' renovation obsession

To accommodate growing families, remote work, and even houseplant obsessions, millennials are big into remodeling and DIY renovations — they’ve even been coined the “renovation generation.” Compared to both older and younger groups, millennials are [3X more likely](https://www.homeadvisor.com/r/wp-content/uploads/2020/12/DP6355-StateOfHomeSpending-2020-R4.pdf?ref=faraday-blog) to take on home office projects and additions. It’s imperative that contractors and home service providers take these trends into account when fleshing out marketing campaigns.

This could mean kitchen remodelers including messaging around spending more time cooking in the kitchen since dine-in options are still limited around the country; or perhaps construction companies advertising the ease of adding on a home office to accommodate remote work. Even video-based consultations can be a deciding factor for homeowners concerned about social distancing or eager for ease of access to an expert.

No matter the product or service, using digital marketing campaigns to introduce ideas for how millennials can improve their at-home lifestyles may spark interests homeowners didn’t know they had, bringing them closer to converting.

## Emphasize eco-friendly products and services

The millennial homeowner is no stranger to “going green” — this messaging has been prevalent throughout much of their life, so much so that [73% of millenials](https://www.inc.com/melanie-curtin/73-percent-of-millennials-are-willing-to-spend-more-money-on-this-1-type-of-product.html) are willing to pay more for sustainable products. Green features are especially popular when they’re sure to lower utility bills in the future.

[During the pandemic](https://porch.com/advice/home-improvement-trends-covid), 34% of homeowners have made their homes more environmentally friendly, with half of that percentage being millennials and Gen Z. On average, millennials are going to spend $10,000 this year alone on home improvement projects, and they recognize that their return is greater if those projects include eco-friendly materials and products.

So whether you’re offering the installation of more environmentally friendly heat pumps, solar panels, or even sustainably harvested hardwood flooring or low VOC paints, make a point to emphasize that your products are eco-friendly to further appeal to millennials. This angle may be even more effective if used to support social media campaigns, as those platforms serve as a hub for eco-conscious consumers proliferating sustainable home improvement solutions.

## Tap into trends to market to millennials

As a digitally native generation, millennials easily take to social media trends, especially those that are good for their wallets, the environment, and lifestyles at home. Tapping into the right channels with messaging relevant to millennial interests can improve the performance of your campaigns and attract the consumers who hold the most spending power in the economy.

Digital campaigns, relevant messaging, and eco-friendly products are key to wooing the largest share of homebuyers in the country.

Marketing to millenials: Tips for home services companies

**UPDATED FOR 2017** Now with easy subquery support and a more sensible argument order!

We adapted [this excellent piece by Dmitri Fontaine](http://tapoueh.org/blog/2014/02/21-PostgreSQL-histogram) and turned it into a function.

It can be invoked like this:

```
SELECT * FROM histogram($table_name_or_subquery, $column_name);
```

. . . to give sweet results like this, in a check of the distribution of 2016 political contributions in Vermont:

```
fec=# SELECT * FROM histogram('(SELECT * FROM small_donors_vt LIMIT 50000)', 'transaction_amt');

 bucket |   range   | freq |       bar
--------+-----------+------+-----------------
      1 | [0,9]     | 2744 | ******
      2 | [10,19]   | 5630 | *************
      3 | [20,29]   | 6383 | ***************
      4 | [30,39]   | 1290 | ***
      5 | [40,49]   |  369 | *
      6 | [50,59]   | 3541 | ********
      7 | [60,69]   |  174 |
      8 | [70,79]   |  313 | *
      9 | [80,89]   |  171 |
     10 | [90,99]   |   65 |
     11 | [100,109] | 2363 | ******
     12 | [110,119] |   51 |
     13 | [120,129] |  115 |
     14 | [130,139] |   32 |
     15 | [140,146] |   11 |
     16 | [150,159] |  187 |
     17 | [160,169] |   24 |
     18 | [170,177] |   33 |
     19 | [180,189] |   19 |
     20 | [191,199] |   24 |
     21 | [200,200] |  795 | **

```

Use it yourself by adding this to your postgres setup:

```
CREATE OR REPLACE FUNCTION histogram(table_name_or_subquery text, column_name text)
RETURNS TABLE(bucket int, "range" numrange, freq bigint, bar text)
AS $func$
BEGIN
RETURN QUERY EXECUTE format('
  WITH
  source AS (
    SELECT * FROM %s
  ),
  min_max AS (
    SELECT min(%s) AS min, max(%s) AS max FROM source
  ),
  histogram AS (
    SELECT
      width_bucket(%s, min_max.min, min_max.max, 20) AS bucket,
      numrange(min(%s)::numeric, max(%s)::numeric, ''[]'') AS "range",
      count(%s) AS freq
    FROM source, min_max
    WHERE %s IS NOT NULL
    GROUP BY bucket
    ORDER BY bucket
  )
  SELECT
    bucket,
    "range",
    freq::bigint,
    repeat(''*'', (freq::float / (max(freq) over() + 1) * 15)::int) AS bar
  FROM histogram',
  table_name_or_subquery,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name
  );
END
$func$ LANGUAGE plpgsql;
```

Note:

1. You don't need to use a subquery - you can also just provide a table name as the first argument.
1. Subqueries are useful for sampling. For example, `(SELECT * FROM bigtable LIMIT 500 ORDER BY RANDOM()`. Don't forget parentheses!
1. You can call this as `SELECT histogram()` or `SELECT * FROM histogram()`. The latter form is much more legible!

Happy querying!

Learn how to create in-table histograms with PostgreSQL.

How to do histograms in PostgreSQL

At Faraday, we've long used [csvkit](https://csvkit.readthedocs.io/en/latest/) to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to [xsv](https://github.com/BurntSushi/xsv).

xsv is a fast CSV-parsing toolkit written in [Rust](https://www.rust-lang.org/en-US/) that mostly matches the functionality of csvkit (including the clutch ability to pipe between modules), with a few extras tacked on (like smart sampling). Did I mention it's fast? In a standup comparison, I ran the "stats" module of XSV against "csvstat" from csvkit, on a 30k-line, 400-column CSV file:

- Python-based csvkit chews through it in a respectable-and-now-expected 4m16s.

- **xsv takes 1.8 seconds.** I don't even have time for a sip of my coffee.

The difference between csvkit and xsv is partly defined by scale; both tools are plenty fast on smaller datasets. But once you get into 10MB-and-upward range, xsv's processing speed pulls away exponentially.

If you've been using csvkit forever (like me), or if you want to be able to transform and analyze CSVs without loading them into a DB, give xsv a shot:

### Install Rust

```bash
curl https://sh.rustup.rs -sSf | sh
```

. . . which also gives you the rust package manager `cargo`, which lets you:

### Install xsv

```bash
cargo install xsv
```

Then be sure your PATH is configured correctly:

```bash
export PATH=~/.cargo/bin:$PATH
```

. . . and try it out on a demo CSV with 10k rows, some messy strings, and multiple data types:

```bash
curl https://gist.githubusercontent.com/wboykinm/044e2af62fc0c7f77e17f6ccd55b8fb0/raw/fca391e6c03a06a7be770fefca6c47a9acdd2305/mock_data.csv \
| xsv stats \
| xsv table
```

_(`xsv table` formats the data so it's readable in the console):_

```bash
field           type     sum                 min                  max                  min_length  max_length  mean                stddev
id              Integer  5005000             1                    1000                 1           4           500.49999999999994  288.6749902572106
first_name      Unicode                      Aaron                Willie               3           11
last_name       Unicode                      Adams                Young                3           10
email           Unicode                      aadamsp5@senate.gov  wwrightd8@upenn.edu  12          34
gender          Unicode                      Female               Male                 4           6
ip_address      Unicode                      0.111.40.87          99.50.37.244         9           15
value           Unicode                      $1007.98             $999.37              0           8
company         Unicode                      Abata                Zoovu                0           13
lat             Float    243963.82509999987  -47.75034            69.70287             0           9           24.42080331331331   24.98767816017553
lon             Float    443214.19009999954  -179.12198           170.29993            0           10          44.36578479479489   71.16647723898215
messed_up_data  Unicode                      !@#$%^&*()           𠜎𠜱𠝹𠱓𠱸𠲖𠳏       0           393
version         Unicode                      0.1.1                9.99                 3           14
```

Happy parsing!

A data science hack to process CSVs faster using XSV.

How we made our CSV processing 142x faster

The Rust compiler just saved me from a nasty threading bug. I was working on an open source development tool for Docker apps with lots of microservices, and I decided to parallelize the routine that transformed `docker-compose.yml` files. This was mostly an excuse to check out the awesome [rayon][] library, but it turned into a great example of what real-world Rust development is like.

[rayon]: https://github.com/nikomatsakis/rayon

The original routine looked something like this:

```rust
/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {
            continue;
        }

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        try!(file.make_standalone(&self.pods_dir()));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));
        try!(file.write_to_path(out_path));
    }
    Ok(())
}
```

To convert this to a parallel loop, I started by changing:

```rust
for pod in &self.pods {
```

To:

```rust
self.pods.par_iter().map(|pod| -> Result<()> {
```

Here, `Result<()>` means "this closure might return an error, or it might return an empty tuple, basically `void`." (For more information on Rust error handling, check out [the Rust book][errors].) But it couldn't be that easy, could it?

[errors]: https://doc.rust-lang.org/book/ch09-00-error-handling.html

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

```
error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied
   --> src/project.rs:397:30
    |
397 |         self.pods.par_iter().map(|pod| -> Result<()> {
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    |
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/project.rs:397:34: 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`
```

The key bit to take away here is that "`Sync` is not satisfied" for the trait `plugins::PluginTransform`. In Rust, `Sync` is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my `PluginTransform` objects live inside my `plugins::Manager` type, which in turn lives inside my `Project`. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that `PluginTransform` inherits from the `Plugin` trait:

```rust
pub trait PluginTransform: Plugin {
```

So we can fix this problem by changing the declaration of `Plugin` from:

```rust
pub trait Plugin {
```

To:

```rust
pub trait Plugin: Sync {
```

This says, "All types implementing `Plugin` must also allow me to access them from multiple threads." With that fixed, I re-run `cargo test` and get a similar message:

```
error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:230:6
    |
230 | impl plugins::Plugin for Plugin {
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied
    |
```

This time, I change:

```rust
trait GenerateToken: Debug {
```

To:

```rust
trait GenerateToken: Debug + Sync {
```

This says, "All types which implement `GenerateToken` must also implement `Debug` (so I can print them), as well as `Sync` (so I can share them). We're making progress!

Re-running `cargo test`, however, reveals the actual bug, and it would have been a nightmare to debug:

```
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:124:6
    |
124 | impl GenerateToken for MockVault {
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied
    |
```

Uh-oh. In our test harness, we have a type `MockVault`, which contains an `Rc<RefCell<_>>`:

```rust
type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
#[derive(Debug)]
#[cfg(test)]
struct MockVault {
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,
}

```

We use the `MockVault` to simulate a connection to Hashicorp's [Vault][], a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the `calls` member here to record all the requests that we would have made to Vault.

[Vault]: https://www.vaultproject.io/

The type `Rc<RefCell<_>>` is a hack. The `GenerateToken` API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see [the Rust book][mutability].) But `Rc<RefCell<_>>` is a lightweight mechanism designed for single-threaded code.

[mutability]: https://doc.rust-lang.org/beta/book/mutability.html

The fix is to replace `Rc<RefCell<_>>` with `Arc<RwLock<_>>`, which is fully thread-safe:

```rust
type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;
```

Once we make this change, Rust reminds us to change all the code that accesses `calls`, too:

```
error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:132:20
    |
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));
    |                    ^^^^^^^^^^
    |
...
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:368:23
    |
368 |     let calls = calls.borrow();
    |                       ^^^^^^
    |
```

Phew. If we hadn't caught that `Rc<RefCell<_>>`, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites _could_ trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

## Wrapping it up

Rust also complains about that fact that we're calling `continue` from inside a closure:

```
error[E0267]: `continue` inside of a closure
   --> src/project.rs:402:17
    |
402 |                 continue;
    |                 ^^^^^^^^ cannot break inside of a closure
```

We can fix this by moving this code:

```rust
// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {
    continue;
}
```

...into a `filter` call on our parallel iterator:

```rust
self.pods.par_iter()
    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {
```

The final problem is that we now have a parallel computation that returns a bunch of `Result<()>` objects, and we need to boil them down to a single `Result<()>` like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

```rust
    .map(|pod| -> Result<()> {
        // ...
    }
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))
    .unwrap_or(Ok(()))
```

This says, "Given any two results named `result1` and `result2`, take `result1` if it's an error, otherwise take `result2`." It's basically a short-circuit `&&` operator, but for `Result` values instead of booleans. I've filed an issue suggesting that `rayon` should provide those last two lines as a built-in function.

[issue113]: https://github.com/nikomatsakis/rayon/issues/113

And yes, that's a genuine parallel [map-reduce][] in Rust!

[map-reduce]: http://research.google.com/archive/mapreduce.html

So here's our final loop, using work-stealing parallelism:

```rust
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.
    self.pods.par_iter()

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            try!(file.make_standalone(&self.pods_dir()));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));
            try!(file.write_to_path(out_path));
            Ok(())
        })

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))
        .unwrap_or(Ok(()))
}
```

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

A detailed example of how to parallelize a loop with Rust and rayon.

Saved by the compiler: Parallelizing a loop with Rust and rayon

At Faraday, we use [redux-saga] in our client-side webapp because it offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps. I recently upgraded our version of redux-saga from an embarrassingly old version to the latest, in part to take advantage of a new (but largely undocumented) feature: context.

### The problem

I want to be able to write sagas without having to implicitly import any statically-declared singletons. The reason for that is simple: if I'm testing the saga that creates new users, I don't want to drag along the API layer or the router instance.

There are also a few modules in the codebase that _only_ work in a browser and will fail loudly when I try to run them in mocha. If one of these browser-only modules ends up in an import chain, suddenly my tests start failing for bad reasons. At worst, I have to rethink my testing strategy to include a mock browser environment. I'm lazy and hate testing, and I think mocks are a gross awful code smell that should be avoided at all costs.

### The solution

One of redux-saga's greatest features is that I can test a saga's behavior without actually executing that behavior, or even really knowing much about the details of how that behavior alters the world. It it should be possible to apply that to dependencies as well. And it is possible in redux-saga 0.15.0 and later using context.

Right now, there are very limited docs for context, but acts as shared value across all sagas that can be read with the `getContext` effect and written to by the `setContext` effect. It can also be set when the saga middleware is created by passing a context object as configuration.

### Example

Let's say we're writing an app that fetches game inventory data from the server using some kind of authentication. We don't want tokens and authentication to bleed into all our sagas, so we wrap it up in a nice singleton API service:

```js
class ApiService {
  getInventory = () => {
    return fetch("/api/inventory", {
      headers: {
        Authorization: `Bearer ${this.token}`,
      },
    }).then((res) => res.json());
  };
}

const api = new ApiService();
api.token = localStorage.token;

export default api;
```

Without context, our saga would probably statically import the API singleton:

```js
import { call } from "redux-saga/effects";
import api from "./api";

export function* fetchInventorySaga() {
  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

All is well until we try to test it in nodejs/mocha:

```js
import { fetchInventorySaga } from '../src/sagas/inventorySaga.js
// ReferenceError: localStorage is not defined
```

There is no localStorage in the nodejs global context. We can either pull in a testing harness to change how `import api from './api'` is resolved, attempt to run the tests in a browser, or roll our own late-binding mechanism so that you don't need to import API and can pass the API instance in at runtime.

We need to solve the same problem for `fetch`, because that's also absent in nodejs.

Or, we could use context. Our saga changes only a little bit:

```js
import { call, getContext } from "redux-saga/effects";
// No more API import:
// import api from './api'

export function* fetchInventorySaga() {
  // Get the api value out of the shared context:
  const api = yield getContext("api");

  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

We can now test the `getContext` effect just like we would any other redux-saga effect, and we can insert a mock value into `fetchInventorySaga` at test-time if we need to.

Setting up context in the main application is very straightforward. When you're creating your saga middleware:

```js
import createSagaMiddleware from "redux-saga";
import api from "./api";

const saga = createSagaMiddleware({
  context: {
    api, // our singleton API value
  },
});
```

Being able to late-bind singleton values like this has been enormously helpful writing robust tests in a complex codebase. I'll be steadily migrating the application code to use `getContext` more frequently, now that I have it as an option.

[redux-saga]: https://redux-saga.js.org
[a couple cryptic sentences]: https://github.com/redux-saga/redux-saga/releases/tag/v0.15.0

Redux-Saga offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps.

Using context in redux-saga

[Terraform's RDS support](https://www.terraform.io/docs/providers/aws/r/db_instance.html) makes it easy to create a database instance. Here's a cheatsheet:

```
resource "aws_db_instance" "mydb1" {
  allocated_storage        = 256 # gigabytes
  backup_retention_period  = 7   # in days
  db_subnet_group_name     = "${var.rds_public_subnet_group}"
  engine                   = "postgres"
  engine_version           = "9.5.4"
  identifier               = "mydb1"
  instance_class           = "db.r3.large"
  multi_az                 = false
  name                     = "mydb1"
  parameter_group_name     = "mydbparamgroup1" # if you have tuned it
  password                 = "${trimspace(file("${path.module}/secrets/mydb1-password.txt"))}"
  port                     = 5432
  publicly_accessible      = true
  storage_encrypted        = true # you should always do this
  storage_type             = "gp2"
  username                 = "mydb1"
  vpc_security_group_ids   = ["${aws_security_group.mydb1.id}"]
}
```

Here's the security group you need:

```
resource "aws_security_group" "mydb1" {
  name = "mydb1"

  description = "RDS postgres servers (terraform-managed)"
  vpc_id = "${var.rds_vpc_id}"

  # Only postgres in
  ingress {
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Allow all outbound traffic.
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
```

You can get these values from the EC2 console (don't forget them!):

```
variable "rds_vpc_id" {
  default = "vpc-XXXXXXXX"
  description = "Our default RDS virtual private cloud (rds_vpc)."
}

variable "rds_public_subnets" {
  default = "subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY"
  description = "The public subnets of our RDS VPC rds-vpc."
}

variable "rds_public_subnet_group" {
  default = "default-vpc-XXXXXXXX"
  description = "Apparently the group name, according to the RDS launch wizard."
}
```

Check out our Terraform RDS cheatsheet that makes it easy to create a database instance.

Tech