If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

Last month, a friendly brand exported about 10,000 payment records from Stripe as a CSV. Then they uploaded them to Faraday and, within 24 hours, got their results:

![Buyer signal showing importance of different features](/images/blog/buyer-signal.png)

Let's focus on one feature - "Science/new technology." If a person is interested in this, they are much more likely to buy this brand's products:

![Likelihood to buy based on science interest](/images/blog/science-directionality.png)

Faraday is batteries-included - we know things like whether people like science thanks to our [built-in consumer data](https://faraday.ai/features/data).

Now, this brand can predict the likelihood of more than **240 million Americans** to buy their product. Faraday can take any list of people and sort it by likelihood to buy... or Faraday can generate the list in the first place, using a ["Deployment"](https://faraday.ai/docs/abstractions/deployments).

So... how did they get this? First, by doing a [Stripe Payment Report](https://support.stripe.com/questions/exporting-payment-reports). In particular, doing a [**Successful Payments**](https://dashboard.stripe.com/payments?status%5B%5D=successful) report.

![You want Succeeded payments](/images/blog/stripe-payment-export-succeeded.png)

Faraday only needs the following fields:

![PII, date, and amount fields](/images/blog/stripe-custom-columns.png)

(Later, you can set up an automated integration. But for now, since this is a relatively stable model based on inherent consumer characteristics, it should be stable for at least a few weeks.)

In Faraday, you map the fields:

![Address mapping](/images/blog/stripe-dataset-identity-mapping.png)

Then, [follow the standard process for importing transactions](https://faraday.ai/docs/cohorts/customers_from_hosted_csv#option-a-each-row-represents-a-transaction-event).

And finally, you make a [likely buyers model](/docs/abstractions/outcomes).

In total, if this takes longer than 15 minutes, [email the CTO](mailto:seamus@faraday.ai) because that's a critical bug!

Happy Signal Hunting!

-Seamus

PS. When this brand loaded our model into Facebook, we immediately started beating Facebook's internal model, even though we were still in learning phase...

![Beating Facebook](/images/blog/stripe-adset.png)

[Sign up](https://faraday.ai/signup) free today.

If you have at least 2000 payments, Stripe data probably contains your buyer signal - the unique patterns that predict who will buy from you.

How to determine buyer signal from Stripe payments

Since the dawn of commerce, sellers have invested significant time and effort into figuring out what influences prospective buyers. Identifying your ideal customer—who they are, what they do, why they do it, and how they feel about various products and services—can help you craft powerful, people-first marketing campaigns that accelerate every sales cycle.

Building a customer insight strategy is the most efficient way to understand that ideal customer, evaluate your brand’s current situation, and adapt for the future. Here’s how you can get started.

## How to develop a customer insight strategy

To develop a customer insight strategy, you will need to focus on both qualitative and quantitative data. Gathering this information ensures you have a complete picture of who your ideal customer is, how they behave, and what their wants and needs are.

Identifying the key team players involved in the process and employing the right tools, techniques, and resources can take a lot of the guesswork out of building an insight strategy.

## Why customer insights are important for business growth

Competitive advantage begins with clear customer insights. Without a comprehensive customer insight strategy, brands are bound to miss out on crucial data that has the power to significantly impact their bottom line.

A customer insight strategy can unlock a wealth of information about your customers and what influences their purchases. Through a combination of consumer data, feedback, social listening, and website analytics, you can begin to draw conclusions about how your customers behave and why. This knowledge can help you improve the customer journey and create an experience that is truly memorable for those who purchase your product or service.

Having a strong foundational awareness of customer insights is the first step to developing a strategy of your own.

## Key components of a customer insight strategy

### Objectives

A clear understanding of why you want to gather customer insights can make developing a strategy easier. What is your reasoning behind gathering insights, and what do you hope to do with this data once you have it?

Maybe you’re hoping to increase revenue by a certain percentage next quarter or determine whether your current pricing strategy aligns with your competitors’. Maybe your ad campaigns aren’t converting like they used to and need to be refreshed. Or perhaps you’re exploring ways to optimize your current content strategy and attract a new audience. All these objectives can be achieved through a smart customer insight strategy.

### Research and data

A viable customer insight strategy calls for a combination of high-quality research and data. From customer interviews and social listening to surveys and focus groups, gathering data about your customers’ preferences can help influence your marketing strategy and content creation.

This emphasis on first-party data is increasingly important as the [end of third-party cookies](https://faraday.ai/blog/death-of-third-party-cookies-strategies) approaches, but you may also consider [responsibly sourced data](https://faraday.ai/features/data) from other vendors to add even more depth to your customer profiles.

Of course, high-quality data is of little use if there is no one to analyze it. A strong analytics team can help brands make sense of the noise and pull focus to the trends that matter most.

### Audience

There is no one-size-fits-all approach to customer insights. Before you can begin gathering valuable data, you’ll need to take a close look at which segment of customers you want to explore. For example, analyzing your current customer base will require a different approach than analyzing potential customers. Knowing whose insight is valuable in achieving certain objectives will ensure you use your time and resources as effectively as possible.

Keep in mind: customer feedback doesn’t always have to be about you to be useful. It doesn’t have to be gathered exclusively from your own customers, either. Finding out what people think about your competitors, for example, can provide a glimpse into what drives customers within your industry to buy.

### Customer journey map

Data-driven insights have the power to affect every level of decision-making when creating and marketing a product. As you gather data about your target audience, it should help you build a better understanding of your brand’s customer journey. Leverage those insights to create or refine a map of customer interactions with your brand—that way, you can skip the guesswork when it comes to what messaging and offers your audience should get at every step of their journey.

For instance, if most customers hear about your brand through digital channels but make their final purchase in-store, your campaigns should facilitate that leap from online to offline interactions. On the other hand, if most customers interact with multiple digital touchpoints before buying online, you’ll want to use those insights to guide them from touchpoint to touchpoint.

### Customer personas

Once you’ve thoroughly collected and analyzed your customer data, and determined what the typical customer journey looks like, you can take your customer insight strategy one step further to develop customer personas.

Personas are a means of clustering your customer base into distinct groups that share certain similarities. One persona group may contain customers who mostly have young families, live in the suburbs, and like to shop at brick-and-mortar stores; another may be digitally-native young professionals or wealthy urbanites. By honing in on the key customer insights that define each persona, you can tailor your campaigns to their unique interests and [better personalize every customer interaction](https://faraday.ai/predict/category/persona_assignments).

### AI enhancements

Sifting through such an enormous amount of data and unifying insights across platforms can be a huge endeavor. For some brands, off-the-shelf AI tools are the most straightforward way to enrich and make sense of all their customer data. Some of the ways they achieve this with AI include:

- Enhancing customer data with an identity graph to create detailed customer profiles
- Scoring leads based on how likely they are to purchase a product
- Predicting the lifetime value of potential or current customers
- Developing powerful quantitative persona groups
- Building lookalike audiences of good fits for use on any marketing channel
- Identifying which customers are at highest risk of churning

## Conclusion

Just about every successful brand relies on customer insights to inform their decision-making. Once you’ve pulled together all the components of your customer insight strategy, you can review them holistically to determine which insights offer the most potential for your team to work with.

An effective customer insight strategy can take the pressure off your sales and marketing teams, improve overall customer satisfaction, drive customer retention, and increase profitability over time. There’s no question that it’s worth your team’s time to get it right.

At Faraday, we offer a number of customer insight tools that allow brands to enhance their first-party data with responsibly sourced, opt-in consumer data and develop powerful, personalized marketing campaigns.

Learn what goes into a successful customer insight strategy and how you can start gathering the data you need to truly understand your customers.

How to create a customer insight strategy

Every marketer knows that personalization is critical. Once considered a nice touch to spruce up campaigns, today it’s a full-on necessity for digital-first brands. Plus, the bar for personalization is getting higher and higher, to the point that consumers expect real-time, highly-curated interactions on every marketing channel.

The groundwork for personalized marketing begins with knowing your customer. Before you can so much as sketch out a campaign idea, you need to know who you’re marketing to. However, even this seemingly simple question is getting a lot more complicated thanks to the phase-out of the third-party cookie.

Nowadays marketers need to take stock of their cookieless identity resolution options and ensure they have the rich customer profiles they need to achieve true personalization.

Looking for cookieless options for enhanced customer data? Eager to add more sophisticated personalization strategies to your arsenal? Here's what this article covers:

- Customer insights and segmentation
- Personalized advertising
- Personalized product recommendations
- Personalized promotions

## Personalization starts with segmentation

Third-party cookies have always gone hand in hand with personalized marketing, since they’ve unlocked access to swaths of data that marketers can use for surgically precise segmentation. But [the end of the cookie](https://faraday.ai/blog/death-of-third-party-cookies-strategies) doesn’t have to sound the death knell for highly personalized ads. If anything, it’s ushering in a new era of more meaningful personalization strategies, where people see content that speaks to them as people, not a click to be retargeted.

### Adding consumer data to the mix

Segmentation is still key to achieving human-centric interactions. Without third-party cookies, you can certainly segment your first-party data, but frankly, it won’t offer enough nuance to craft personalized campaigns. Now more than ever, marketers need to prioritize [data enrichment from responsible sources](https://faraday.ai/features/responsible-ai). This cookieless consumer data enables brands to build out rich customer profiles based on a wealth of demographic, psychographic, and firmographic data.

Once you’ve enhanced your first-party data, you’ll have the full picture you need to segment your audience in a meaningful way. This might mean segmenting by income level, household size, or even hobbies and interests. The savviest brands go even further and leverage machine learning algorithms to parse through billions of data points, surface the most significant patterns in their audience, and cluster their customers into highly accurate persona groups.

All of this paves the way for true [personalization at scale](https://faraday.ai/predict/category/persona_assignments). By segmenting your audience into clusters or personas, you can craft highly personalized campaigns that speak to each unique segment and orchestrate more meaningful interactions across channels.

If that sounds like a plan, read on for the strategies you can use to start bringing those campaigns to life.

## Personalizing ad campaigns

Personalized advertising is a fantastic tool for connecting with good fits across the buyer’s journey. Since consumers are bombarded with ads all day long — social media, TV, radio, podcast, flyers, billboards — meaningfully personalized ads help cut through the noise and create an immediate connection.

### Using personas to create custom audiences

It’s one thing to segment your existing audience based on [rich consumer data](https://faraday.ai/features/data), but quite another to personalize ads for potential customers who aren’t yet known to you. Lookalike audience options from platforms like Facebook used to reliably expand brands’ reach to new pools of good fits, but with the advent of iOS 14 and Apple’s privacy updates, these lookalikes have lost a lot of signal and are less cost-effective by the day. You’ll need additional strategies up your sleeve to reach new audiences.

One such strategy involves seeding Facebook lookalikes with AI-driven personas, which gives Facebook a more defined seed audience to work with and enhances its ability to surface consumers that fit your personas. Think of it like giving Facebook’s algorithm a bit of a boost. Then, you can craft highly personalized ad campaigns for individuals you would never have reached otherwise. Brands doing this typically see increased conversion rates and lower CPA.

Another option is to skip Facebook’s native lookalikes and leverage AI to build your own custom lookalike audiences. In this strategy, algorithms can uncover great fits from an identity graph and match them to your brand’s personas, providing you with fresh audiences to upload to Facebook or any other ad platform — greatly expanding your audience beyond what these platforms would normally surface.

With these in place, and your personas already established, you can personalize your ad campaigns with confidence in spite of having never interacted with the audience before.

## Personalized product recommendations

Once someone has made themselves known to your brand— for instance, subscribing to your email list, creating a free online account, or signing up for a rewards program — personalized product recommendations are another invaluable strategy. You can leverage insights in your CRM to create a truly personalized experience on your website, in-store, via email marketing, and more.

Crucially, consumers have high expectations for these experiences. In such a competitive market, where shoppers can find dozens of similar products and services online with just a quick Google search, it’s the real-time, personalized interactions that help brands stand out from the crowd and earn their trust. Orchestrating this type of personalization requires sophisticated automation and algorithms to make recommendations instantaneously and at scale.

### Personalized recommendations based on journey stage

Truly personalized recommendations tend to come back to great life cycle marketing. Once you have an accurate picture of where current and potential customers fall on the buyer’s journey, you can map recommendations to each stage — such as promoting a product quiz or survey to your social followers in order to collect more first-party data, launching email campaigns that are tailored to your unique personas, or nudging customers about new releases.

With most modern ESPs and email marketing platforms, such as Iterable, Klaviyo, and MailChimp, you can also build powerful workflows to trigger personalized product recommendations based on specific actions. For a financial organization, this could look like an email with more information on home loan options after a lead schedules a consultation. For an apparel brand, it could mean triggering an alert that an item has come back in stock for every rewards member who'd added it to their virtual wish list.

By aligning recommendations with an individual’s journey stage and automating them to be delivered at the perfect moment, you create the real-time, personalized interactions today’s consumers crave — and you’ll see the results in your revenue.

### Personalized recommendations with predicted purchases

For marketing teams that have already dialed up product recommendations and want to offer even more valuable suggestions to potential customers, there’s also the option to take it one step further with predictive journey mapping. Rather than make an assumption about what a person is going to purchase, this entails using AI to actually predict what they’ll get.

These [predictions](https://faraday.ai/features/predictions) are far more accurate than any other product recommendation you could make, allowing your marketing efforts to become less like a sales pitch and more like the advice of a trusted friend. Just like automated email campaigns, these predictions can be triggered from a platform like Iterable to deliver the perfect recommendation to a given individual at the perfect time.

## Personal discounts and pomotions

Sometimes consumers need a little extra encouragement to get them to commit. For these individuals, a well-timed promotion or discount can make an enormous difference. The trick is to tailor the promotion to the consumer in order to protect your bottom line — after all, you don’t want to offer 15% off or a first month free to someone who’s likely to pay full price, but you definitely want that to be an option for valuable lead that’s on the fence.

### Using personalized discounts to build brand affinity

To be certain that you’re only offering discounts to the customers that matter most, customer lifetime value (LTV) predictions can eliminate an enormous amount of guesswork. LTV is difficult to determine on your own, but a wide variety of AI tools available to marketers make it a piece of cake.

With LTV predictions uploaded to Salesforce, HubSpot, or wherever you track your customer information, it becomes much clearer how to personalize discounts for distinct audiences. You may want to try rewarding your most valuable customers with a discount, as a small thanks for their brand loyalty, or you may want to test out special offers for customers with middling LTV to see if that extra push inspires them to spend even more.

### Using personalized discounts to retain high-value customers

Another proven discount strategy relies on [churn prediction](https://faraday.ai/predict/category/churn_scores). As with all things predictive, AI is the best tool to utilize here, with churn predictions at least five times more accurate than those that traditional methods produce. Once you understand exactly which customers are at the highest risk of falling away, you can intervene proactively, with a personalized touch, rather than reactively.

For instance, you can tailor the discount to the level of churn risk. High-value customers at great risk of churn might deserve a generous discount offer, whereas customers with lower risk can get a smaller one. And just like with product recommendations and LTV predictions, churn predictions can be delivered seamlessly to wherever your team needs them most, whether that’s triggering an email drip or refreshing the customer profile in your CRM — empowering you to provide a seamless customer experience.

## Level up your personalization strategies today

Whether you’re just getting started with personalization or are looking to truly master 1:1 communication at scale, you’ve got numerous opportunities available at every stage of the buyer’s journey. Consumers overwhelmingly prefer personalized offers, and an increasingly crowded digital space is making it harder and harder to gain an edge — so it’s more important than ever to double down on personalization before it’s too late.

Explore practical personalization strategies that leading brands use to create more enjoyable experiences across the customer lifecycle.

5 ways to personalize experiences across the customer lifecycle

Iterable interviews Faraday's CEO to unlock the secrets to automated personalization using consumer data and AI responsibly.

AI and the future of personalized marketing with Iterable & Faraday

With brand trust at an [all-time low](https://www.havasgroup.com/press_release/havas-meaningful-brands-report-2021-finds-we-are-entering-the-age-of-cynicism/), it’s a good time to ask: is your organization doing the right thing with consumer data? 50% of consumers say they’d pay more for a brand that takes a stand on important issues. With privacy at the heart of a global reassessment of social media, big tech, and surveillance, organizations at the vanguard of this issue will win in 2022.

The good news is that playing fair isn’t all that complicated. “When in doubt, follow your heart,” says Andy Rossmeissl, CEO of Faraday, the leading customer prediction platform. “Our motto from the start has been ‘don’t be creepy,’ and it’s remarkable how far that can take you.”

Looking for specifics? Here are the top 3 rules for using customer data ethically:

## Rule 1: Use identifiers people can recite from memory.

This rule sounds simple, but it’s surprisingly powerful. “Ask your mom if she knows any of her tracking cookies by heart,” suggests Rossmeissl. “Ask yourself what your mobile device ID is.” The digital surveillance panopticon relies on these [long, unique strings](https://www.quora.com/What-does-the-code-for-a-tracking-cookie-look-like) of gibberish assigned to virtually every eyeball in the world.

Instead, ask yourself what identifiers you have that you can actually recite. “I have an address on my mailbox for all the world to see, and you probably do too,” says Rossmeissl. “I put my number in the phone book, and my email address is all over the Internet.”

Relying on real-world identifiers, or “known identities,” has the additional benefit of steering brands away from collecting and using anonymous behavior, such as data collected from unidentified web and mobile visitors. “Until somebody has identified themselves to you," explains Rossmeissl, "tracking their behavior is, frankly, creepy.”

## Rule 2: Use data that comes from commercial behavior

“I call it the ‘laying in bed rule,’” Rossmeissl shares. “When I’m laying in bed and I read a news article on my tablet or like something on Instagram, that’s private. Watching me do that is creepy.”
The surveillance of private behavior has rapidly normalized, much to the dismay of both privacy experts and, increasingly, consumers alike. A 2019 study found that nearly two thirds of Americans believe they can’t go through daily life without their data being collected.

And yet marketers have long known the power of data-driven growth practices. How can brands [use consumer data while still respecting privacy](https://faraday.ai/features/responsible-ai)? Rossmeissl shares:

> We thought long and hard about this when we started Faraday in 2012. We drew on anthropology, architecture, history, and other disciplines for our research, but the result was fairly simple.

> It turns out that commerce is one of the main reasons humans started to live in communities. Markets are some of the earliest forms of non-residential architecture we know of. No matter how you look at it, commerce and the marketplace have been with us longer than virtually any other social structure.

This provides a clue: humans are quite used to buying and selling, and they’re used to doing it in public. We have evolved an innate ability to operate in the marketplace, and an accompanying understanding that this behavior is public.

“What we put in our cart at the grocery store is public,” says Rossmeissl. “What we buy walking around a farmer's market is public. And when we buy online, there's an extension of this understanding. Engaging in commerce is a collaborative act — there's a buyer and a seller — and also a performative one.”

The upshot is that we instinctually treat data arising from commercial transactions as fair game. Luckily, this kind of data is not only the most ethical, but also some of the most powerful available for predicting subsequent consumer behavior. Faraday includes over 300 data points like this on virtually every US adult, but you can also procure your own from reputable licensing groups like Epsilon.

## Rule 3: Let people easily opt out

“Imagine being in a store and being told you can’t leave,” says Rossmeissl. “That goes from creepy to nightmarish very quickly.”

The right to opt out is at the heart of CCPA, the landmark privacy legislation. And while the law only protects California residents, brands should meet its requirements across the board. Luckily, that’s quite simple: provide an easy to use opt-out link, and pass along opt-outs both up- and down-stream to your partners.

Beyond the fact that [honoring opt-outs is actually good for business](https://sendgrid.com/blog/honoring-opt-outs-is-good-business/), it’s the right thing to do.

Is your organization doing doing the right things with consumer data? Here are 3 common-sense rules to follow.

3 common-sense rules for ethical data use 

During the pandemic, interest rates for home mortgages reached all-time low. As people were spending significantly more time at home, buying a house — especially for those who relocated out of denser cities — has become a more practical option than renting. In August 2020, [home sales shot up](https://www.businessinsider.com/how-2020-broke-the-housing-market-inventory-could-run-out-2020-9) to reach a 14-year high.

Millennials in particular have made their mark on the real estate business, as they now make up the largest share of home buyers in the country, according to the [National Association of Realtors](https://www.nar.realtor/research-and-statistics/research-reports/home-buyer-and-seller-generational-trends). However, many businesses that service the home, like HVAC contractors, report that they [do not focus on millennials](https://www.achrnews.com/articles/144348-ignore-millennials-if-you-dont-want-to-reach-homeowners) for their marketing campaigns. This is their loss, especially since millennial homeowners are at their fingertips on social media and digital platforms, where targeting younger demographics is a breeze.

So, if you’re in the business of home renovation, DIY tools, residential energy efficiency, or HVAC, consider the following when marketing to millennials — a generation eager to take you up on your services.

## Find millennial homeowners on the right channels

Millennials are a digitally native generation. A [global survey](https://www.gwi.com/reports/millennials) conducted by the Global Web Index in Q3 2020 looked at more than 66,000 internet users and found that millennials ages 24–37 spent an average of 2 hours and 30 minutes each day on social media alone.

If your brand is not capitalizing on social media, digital ads, and OTT options like streaming channels to reach millennial homeowners, you’re missing out on crucial brand awareness and conversion opportunities.

In addition to actively targeting specific audiences, be sure your SEM strategy is up to par, as millennials are big online researchers. Back in the day, newspaper and yellow pages ads may have cut it, but that’s all switched over to online search engines now. Ensure that your areas of interest and/or operation are prioritized in any search your brand may be relevant to.

And once you’ve got your lead on the hook, you can ease their journey to conversion by communicating on the platforms that are most convenient for them; this may be SMS text, video chat, or a website chatbox. From getting a quote for a solar installation to matching a desired paint color, millennials value seamless communication that meets them where they already are in the digital realm.\

## Cash in on millennials' renovation obsession

To accommodate growing families, remote work, and even houseplant obsessions, millennials are big into remodeling and DIY renovations — they’ve even been coined the “renovation generation.” Compared to both older and younger groups, millennials are [3X more likely](https://www.homeadvisor.com/r/wp-content/uploads/2020/12/DP6355-StateOfHomeSpending-2020-R4.pdf?ref=faraday-blog) to take on home office projects and additions. It’s imperative that contractors and home service providers take these trends into account when fleshing out marketing campaigns.

This could mean kitchen remodelers including messaging around spending more time cooking in the kitchen since dine-in options are still limited around the country; or perhaps construction companies advertising the ease of adding on a home office to accommodate remote work. Even video-based consultations can be a deciding factor for homeowners concerned about social distancing or eager for ease of access to an expert.

No matter the product or service, using digital marketing campaigns to introduce ideas for how millennials can improve their at-home lifestyles may spark interests homeowners didn’t know they had, bringing them closer to converting.

## Emphasize eco-friendly products and services

The millennial homeowner is no stranger to “going green” — this messaging has been prevalent throughout much of their life, so much so that [73% of millenials](https://www.inc.com/melanie-curtin/73-percent-of-millennials-are-willing-to-spend-more-money-on-this-1-type-of-product.html) are willing to pay more for sustainable products. Green features are especially popular when they’re sure to lower utility bills in the future.

[During the pandemic](https://porch.com/advice/home-improvement-trends-covid), 34% of homeowners have made their homes more environmentally friendly, with half of that percentage being millennials and Gen Z. On average, millennials are going to spend $10,000 this year alone on home improvement projects, and they recognize that their return is greater if those projects include eco-friendly materials and products.

So whether you’re offering the installation of more environmentally friendly heat pumps, solar panels, or even sustainably harvested hardwood flooring or low VOC paints, make a point to emphasize that your products are eco-friendly to further appeal to millennials. This angle may be even more effective if used to support social media campaigns, as those platforms serve as a hub for eco-conscious consumers proliferating sustainable home improvement solutions.

## Tap into trends to market to millennials

As a digitally native generation, millennials easily take to social media trends, especially those that are good for their wallets, the environment, and lifestyles at home. Tapping into the right channels with messaging relevant to millennial interests can improve the performance of your campaigns and attract the consumers who hold the most spending power in the economy.

Digital campaigns, relevant messaging, and eco-friendly products are key to wooing the largest share of homebuyers in the country.

Marketing to millenials: Tips for home services companies

**UPDATED FOR 2017** Now with easy subquery support and a more sensible argument order!

We adapted [this excellent piece by Dmitri Fontaine](http://tapoueh.org/blog/2014/02/21-PostgreSQL-histogram) and turned it into a function.

It can be invoked like this:

```
SELECT * FROM histogram($table_name_or_subquery, $column_name);
```

. . . to give sweet results like this, in a check of the distribution of 2016 political contributions in Vermont:

```
fec=# SELECT * FROM histogram('(SELECT * FROM small_donors_vt LIMIT 50000)', 'transaction_amt');

 bucket |   range   | freq |       bar
--------+-----------+------+-----------------
      1 | [0,9]     | 2744 | ******
      2 | [10,19]   | 5630 | *************
      3 | [20,29]   | 6383 | ***************
      4 | [30,39]   | 1290 | ***
      5 | [40,49]   |  369 | *
      6 | [50,59]   | 3541 | ********
      7 | [60,69]   |  174 |
      8 | [70,79]   |  313 | *
      9 | [80,89]   |  171 |
     10 | [90,99]   |   65 |
     11 | [100,109] | 2363 | ******
     12 | [110,119] |   51 |
     13 | [120,129] |  115 |
     14 | [130,139] |   32 |
     15 | [140,146] |   11 |
     16 | [150,159] |  187 |
     17 | [160,169] |   24 |
     18 | [170,177] |   33 |
     19 | [180,189] |   19 |
     20 | [191,199] |   24 |
     21 | [200,200] |  795 | **

```

Use it yourself by adding this to your postgres setup:

```
CREATE OR REPLACE FUNCTION histogram(table_name_or_subquery text, column_name text)
RETURNS TABLE(bucket int, "range" numrange, freq bigint, bar text)
AS $func$
BEGIN
RETURN QUERY EXECUTE format('
  WITH
  source AS (
    SELECT * FROM %s
  ),
  min_max AS (
    SELECT min(%s) AS min, max(%s) AS max FROM source
  ),
  histogram AS (
    SELECT
      width_bucket(%s, min_max.min, min_max.max, 20) AS bucket,
      numrange(min(%s)::numeric, max(%s)::numeric, ''[]'') AS "range",
      count(%s) AS freq
    FROM source, min_max
    WHERE %s IS NOT NULL
    GROUP BY bucket
    ORDER BY bucket
  )
  SELECT
    bucket,
    "range",
    freq::bigint,
    repeat(''*'', (freq::float / (max(freq) over() + 1) * 15)::int) AS bar
  FROM histogram',
  table_name_or_subquery,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name
  );
END
$func$ LANGUAGE plpgsql;
```

Note:

1. You don't need to use a subquery - you can also just provide a table name as the first argument.
1. Subqueries are useful for sampling. For example, `(SELECT * FROM bigtable LIMIT 500 ORDER BY RANDOM()`. Don't forget parentheses!
1. You can call this as `SELECT histogram()` or `SELECT * FROM histogram()`. The latter form is much more legible!

Happy querying!

Learn how to create in-table histograms with PostgreSQL.

How to do histograms in PostgreSQL

At Faraday, we've long used [csvkit](https://csvkit.readthedocs.io/en/latest/) to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to [xsv](https://github.com/BurntSushi/xsv).

xsv is a fast CSV-parsing toolkit written in [Rust](https://www.rust-lang.org/en-US/) that mostly matches the functionality of csvkit (including the clutch ability to pipe between modules), with a few extras tacked on (like smart sampling). Did I mention it's fast? In a standup comparison, I ran the "stats" module of XSV against "csvstat" from csvkit, on a 30k-line, 400-column CSV file:

- Python-based csvkit chews through it in a respectable-and-now-expected 4m16s.

- **xsv takes 1.8 seconds.** I don't even have time for a sip of my coffee.

The difference between csvkit and xsv is partly defined by scale; both tools are plenty fast on smaller datasets. But once you get into 10MB-and-upward range, xsv's processing speed pulls away exponentially.

If you've been using csvkit forever (like me), or if you want to be able to transform and analyze CSVs without loading them into a DB, give xsv a shot:

### Install Rust

```bash
curl https://sh.rustup.rs -sSf | sh
```

. . . which also gives you the rust package manager `cargo`, which lets you:

### Install xsv

```bash
cargo install xsv
```

Then be sure your PATH is configured correctly:

```bash
export PATH=~/.cargo/bin:$PATH
```

. . . and try it out on a demo CSV with 10k rows, some messy strings, and multiple data types:

```bash
curl https://gist.githubusercontent.com/wboykinm/044e2af62fc0c7f77e17f6ccd55b8fb0/raw/fca391e6c03a06a7be770fefca6c47a9acdd2305/mock_data.csv \
| xsv stats \
| xsv table
```

_(`xsv table` formats the data so it's readable in the console):_

```bash
field           type     sum                 min                  max                  min_length  max_length  mean                stddev
id              Integer  5005000             1                    1000                 1           4           500.49999999999994  288.6749902572106
first_name      Unicode                      Aaron                Willie               3           11
last_name       Unicode                      Adams                Young                3           10
email           Unicode                      aadamsp5@senate.gov  wwrightd8@upenn.edu  12          34
gender          Unicode                      Female               Male                 4           6
ip_address      Unicode                      0.111.40.87          99.50.37.244         9           15
value           Unicode                      $1007.98             $999.37              0           8
company         Unicode                      Abata                Zoovu                0           13
lat             Float    243963.82509999987  -47.75034            69.70287             0           9           24.42080331331331   24.98767816017553
lon             Float    443214.19009999954  -179.12198           170.29993            0           10          44.36578479479489   71.16647723898215
messed_up_data  Unicode                      !@#$%^&*()           𠜎𠜱𠝹𠱓𠱸𠲖𠳏       0           393
version         Unicode                      0.1.1                9.99                 3           14
```

Happy parsing!

A data science hack to process CSVs faster using XSV.

How we made our CSV processing 142x faster

The Rust compiler just saved me from a nasty threading bug. I was working on an open source development tool for Docker apps with lots of microservices, and I decided to parallelize the routine that transformed `docker-compose.yml` files. This was mostly an excuse to check out the awesome [rayon][] library, but it turned into a great example of what real-world Rust development is like.

[rayon]: https://github.com/nikomatsakis/rayon

The original routine looked something like this:

```rust
/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {
            continue;
        }

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        try!(file.make_standalone(&self.pods_dir()));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));
        try!(file.write_to_path(out_path));
    }
    Ok(())
}
```

To convert this to a parallel loop, I started by changing:

```rust
for pod in &self.pods {
```

To:

```rust
self.pods.par_iter().map(|pod| -> Result<()> {
```

Here, `Result<()>` means "this closure might return an error, or it might return an empty tuple, basically `void`." (For more information on Rust error handling, check out [the Rust book][errors].) But it couldn't be that easy, could it?

[errors]: https://doc.rust-lang.org/book/ch09-00-error-handling.html

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

```
error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied
   --> src/project.rs:397:30
    |
397 |         self.pods.par_iter().map(|pod| -> Result<()> {
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    |
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/project.rs:397:34: 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`
```

The key bit to take away here is that "`Sync` is not satisfied" for the trait `plugins::PluginTransform`. In Rust, `Sync` is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my `PluginTransform` objects live inside my `plugins::Manager` type, which in turn lives inside my `Project`. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that `PluginTransform` inherits from the `Plugin` trait:

```rust
pub trait PluginTransform: Plugin {
```

So we can fix this problem by changing the declaration of `Plugin` from:

```rust
pub trait Plugin {
```

To:

```rust
pub trait Plugin: Sync {
```

This says, "All types implementing `Plugin` must also allow me to access them from multiple threads." With that fixed, I re-run `cargo test` and get a similar message:

```
error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:230:6
    |
230 | impl plugins::Plugin for Plugin {
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied
    |
```

This time, I change:

```rust
trait GenerateToken: Debug {
```

To:

```rust
trait GenerateToken: Debug + Sync {
```

This says, "All types which implement `GenerateToken` must also implement `Debug` (so I can print them), as well as `Sync` (so I can share them). We're making progress!

Re-running `cargo test`, however, reveals the actual bug, and it would have been a nightmare to debug:

```
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:124:6
    |
124 | impl GenerateToken for MockVault {
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied
    |
```

Uh-oh. In our test harness, we have a type `MockVault`, which contains an `Rc<RefCell<_>>`:

```rust
type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
#[derive(Debug)]
#[cfg(test)]
struct MockVault {
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,
}

```

We use the `MockVault` to simulate a connection to Hashicorp's [Vault][], a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the `calls` member here to record all the requests that we would have made to Vault.

[Vault]: https://www.vaultproject.io/

The type `Rc<RefCell<_>>` is a hack. The `GenerateToken` API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see [the Rust book][mutability].) But `Rc<RefCell<_>>` is a lightweight mechanism designed for single-threaded code.

[mutability]: https://doc.rust-lang.org/beta/book/mutability.html

The fix is to replace `Rc<RefCell<_>>` with `Arc<RwLock<_>>`, which is fully thread-safe:

```rust
type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;
```

Once we make this change, Rust reminds us to change all the code that accesses `calls`, too:

```
error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:132:20
    |
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));
    |                    ^^^^^^^^^^
    |
...
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:368:23
    |
368 |     let calls = calls.borrow();
    |                       ^^^^^^
    |
```

Phew. If we hadn't caught that `Rc<RefCell<_>>`, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites _could_ trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

## Wrapping it up

Rust also complains about that fact that we're calling `continue` from inside a closure:

```
error[E0267]: `continue` inside of a closure
   --> src/project.rs:402:17
    |
402 |                 continue;
    |                 ^^^^^^^^ cannot break inside of a closure
```

We can fix this by moving this code:

```rust
// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {
    continue;
}
```

...into a `filter` call on our parallel iterator:

```rust
self.pods.par_iter()
    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {
```

The final problem is that we now have a parallel computation that returns a bunch of `Result<()>` objects, and we need to boil them down to a single `Result<()>` like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

```rust
    .map(|pod| -> Result<()> {
        // ...
    }
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))
    .unwrap_or(Ok(()))
```

This says, "Given any two results named `result1` and `result2`, take `result1` if it's an error, otherwise take `result2`." It's basically a short-circuit `&&` operator, but for `Result` values instead of booleans. I've filed an issue suggesting that `rayon` should provide those last two lines as a built-in function.

[issue113]: https://github.com/nikomatsakis/rayon/issues/113

And yes, that's a genuine parallel [map-reduce][] in Rust!

[map-reduce]: http://research.google.com/archive/mapreduce.html

So here's our final loop, using work-stealing parallelism:

```rust
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.
    self.pods.par_iter()

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            try!(file.make_standalone(&self.pods_dir()));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));
            try!(file.write_to_path(out_path));
            Ok(())
        })

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))
        .unwrap_or(Ok(()))
}
```

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

A detailed example of how to parallelize a loop with Rust and rayon.

Saved by the compiler: Parallelizing a loop with Rust and rayon

At Faraday, we use [redux-saga] in our client-side webapp because it offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps. I recently upgraded our version of redux-saga from an embarrassingly old version to the latest, in part to take advantage of a new (but largely undocumented) feature: context.

### The problem

I want to be able to write sagas without having to implicitly import any statically-declared singletons. The reason for that is simple: if I'm testing the saga that creates new users, I don't want to drag along the API layer or the router instance.

There are also a few modules in the codebase that _only_ work in a browser and will fail loudly when I try to run them in mocha. If one of these browser-only modules ends up in an import chain, suddenly my tests start failing for bad reasons. At worst, I have to rethink my testing strategy to include a mock browser environment. I'm lazy and hate testing, and I think mocks are a gross awful code smell that should be avoided at all costs.

### The solution

One of redux-saga's greatest features is that I can test a saga's behavior without actually executing that behavior, or even really knowing much about the details of how that behavior alters the world. It it should be possible to apply that to dependencies as well. And it is possible in redux-saga 0.15.0 and later using context.

Right now, there are very limited docs for context, but acts as shared value across all sagas that can be read with the `getContext` effect and written to by the `setContext` effect. It can also be set when the saga middleware is created by passing a context object as configuration.

### Example

Let's say we're writing an app that fetches game inventory data from the server using some kind of authentication. We don't want tokens and authentication to bleed into all our sagas, so we wrap it up in a nice singleton API service:

```js
class ApiService {
  getInventory = () => {
    return fetch("/api/inventory", {
      headers: {
        Authorization: `Bearer ${this.token}`,
      },
    }).then((res) => res.json());
  };
}

const api = new ApiService();
api.token = localStorage.token;

export default api;
```

Without context, our saga would probably statically import the API singleton:

```js
import { call } from "redux-saga/effects";
import api from "./api";

export function* fetchInventorySaga() {
  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

All is well until we try to test it in nodejs/mocha:

```js
import { fetchInventorySaga } from '../src/sagas/inventorySaga.js
// ReferenceError: localStorage is not defined
```

There is no localStorage in the nodejs global context. We can either pull in a testing harness to change how `import api from './api'` is resolved, attempt to run the tests in a browser, or roll our own late-binding mechanism so that you don't need to import API and can pass the API instance in at runtime.

We need to solve the same problem for `fetch`, because that's also absent in nodejs.

Or, we could use context. Our saga changes only a little bit:

```js
import { call, getContext } from "redux-saga/effects";
// No more API import:
// import api from './api'

export function* fetchInventorySaga() {
  // Get the api value out of the shared context:
  const api = yield getContext("api");

  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

We can now test the `getContext` effect just like we would any other redux-saga effect, and we can insert a mock value into `fetchInventorySaga` at test-time if we need to.

Setting up context in the main application is very straightforward. When you're creating your saga middleware:

```js
import createSagaMiddleware from "redux-saga";
import api from "./api";

const saga = createSagaMiddleware({
  context: {
    api, // our singleton API value
  },
});
```

Being able to late-bind singleton values like this has been enormously helpful writing robust tests in a complex codebase. I'll be steadily migrating the application code to use `getContext` more frequently, now that I have it as an option.

[redux-saga]: https://redux-saga.js.org
[a couple cryptic sentences]: https://github.com/redux-saga/redux-saga/releases/tag/v0.15.0

Redux-Saga offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps.

Tech