The use of third-party data isn't new, but it's becoming a necessity as businesses shift to a customer-centric approach. Marketers everywhere are realizing that first-party data is only the tip of the iceberg when it comes to understanding their customers.

The rise in digital technologies and interactions has increased the need for bringing outside data into their companies. While it may be as simple as understanding your existing customers' household and financial standing, with third-party data you can also gain deep insights into your leads on a known-identity basis.

Overall, leveraging third-party and first-party data adds breadth and depth to your customer insights, informing better branding and messaging decisions, personalization efforts across channels, and taking much of the guesswork out of targeted and location-based marketing campaigns.

## Customer identity resolution

The capability to bring together first-party data from various sources and third-party data to compile a unified picture of each customer is increasingly valuable. Spending dedicated to identity data assets [grew by more than 50% in 2018](https://www.iab.com/wp-content/uploads/2018/12/IAB_The-State-of-Data-2018_2018-12-05_Final.pdf?ref=faraday-blog), to a total of $846MM.

Personally identifiable information (PII) is crucial to connecting the dots across multiple touchpoints (online and offline) to form a holistic picture of your customers and prospects so you can reach them with content that will resonate with them through the channels that matter.

## Using third-party data for personalization

In a time when [almost all consumers](https://www.bondcap.com/report/it19/?ref=faraday-blog#view/148) prefer ads and products that are relevant to them, knowing how and when to serve your target market is crucial to effectively engaging your customers. According to [Forbes](https://www.forbes.com/sites/insights-treasuredata/2019/05/01/the-path-to-personalization/?ref=faraday-blog&sh=5bd84dc7a76b) 46% of marketing executives are not where they want to be in terms of delivering personalization, and 57% say that they hope to advance their personalization initiatives over the next year.

Enriching your first-party data with third-party information on your customers' financial, household, transactional, or lifestyle categories gives you a clearer idea of what your customers look like in real life.

To take it a step further, enriched customer data helps machine learning algorithms build better models. Cluster models, for example, sort individuals into distinct groups based on common attributes. These groups, or personas, can affirm your intuitions about what your customer base looks like, but may also surprise you by revealing significant groups you hadn't known about or previously considered important.

## Using third-party data for ad optimization

If you're using predictive lead scoring capabilities, the accuracy and precision of those scores improve with the quality and quantity of third-party data that you introduce. This not only benefits your leads' experiences, as they'll receive content that better aligns with their interests and where they are in their customer journeys, but it will save you time and money when you're deciding how to prioritize your outreach.

A popular way of leveraging third-party data with predictive modeling for digital marketing is Facebook lookalike audiences. But since the platform shut down advertisers' access to this data in April of 2018, gaining deep insights from Facebook has been a struggle for many businesses that are trying to better understand their audiences and target the right consumers.

If you enrich your first-party data with third-party attributes, you can gain insights on a granular level that you aren't getting with Facebook and other marketing channels. You can then segment known-identity audiences based on certain attributes — or develop in-depth personas for segmentation — and push the appropriate ad creative to those individual segments with the goal of increased, personalized engagement. This goes for not only new leads you're nurturing, but also for the customers you're actively engaging.

Another advantage of leveraging third-party data is the ability to match site's captured emails to real people. This allows you to bucket leads into certain personas for more personalized nurture campaigns that will pull them through their customer journeys with you. Your leads are more likely to receive messaging that aligns with where they are in their individual journeys, and it will save your team time and money when you decide how to prioritize your lead outreach.

## Using third-party data for location-based marketing efforts

Geographic data has obvious benefits, but incorporating additional third-party in geographic analyses improves the quality of the insights. Knowing where your ideal customers live is one piece of the geomarketing puzzle, but gaining an understanding of how your audiences will best interact with your ads offline is achieved by knowing more than just their geographic location.

Attributes like household information, transportation preferences, and lifestyle data helps you prioritize where you place ads, as well as where you might establish new brick-and-mortar presences. A subway ad or billboard placed in your customers' neighborhoods can be a great way to advertise your e-commerce business for those who don't live near a storefront, but for customers who are in your brick-and-mortar trade zones, perhaps a direct mailer advertising an in-store sale would better drive business.

From a creative perspective, audiences can vary widely depending on their physical locations, so the messaging and images chosen for your campaigns should be curated for specific audiences in particular geographies. Providing relevant content increases your chances of audience engagement, so capitalizing on the insights provided by third-party data is a necessity.

This is the case with SEM advertising, which segments audiences by geographic location. With the insights gained from third-party data, you can optimize your SEM spend by targeting areas where higher-value customers reside, spending less on areas that aren't as likely to bring in revenue.

## Using third-party data responsibly

With the rise of regulations like GDPR, privacy and validity of data are prominent concerns when using third-party data and PII. However, there are ways to ensure that your customers' and prospects' privacy is protected. You want to be sure your data comes from a verified source, isn't dangerously invasive, and is leveraged to benefit the customers you're targeting.

Social scraping is a common tactic, and it can provide large categories of data on behavioral tendencies, consumer habits, and some personal information, though this data may not always be entirely accurate. At Faraday we avoid social scraping in favor of canonical sources of information from established third-party data vendors. Vendors like Epsilon verify the accuracy of the high-quality data we license and leverage for our partners.

Faraday's Director of Data Science, William Morris, notes that we sidestep many consumer privacy issues by dealing with datasets that categorize data, such as consumer purchases, by type of product and frequency, rather than the exact stores where those purchases are made. This leads to a significant lift in machine learning models' predictive capabilities without being invasive from a privacy standpoint. Additionally, each of our clients' customer data is siloed using industry-standard practices to protect the privacy of each clients' customers.

Faraday also actively works to build models that are agnostic to a person's membership to protected classes. We exclude potentially harmful categories like ethnicity, religion, or primary language data, as these can disenfranchise certain groups of the consumer population. As an additional security measure, we employ a suite of tools for post-model checking to ensure that attributes that could indicate a person's race or class don't make the models biased or adversely impact a population.

Taking into consideration these types of ethical practices is important when exploring your own use of third-party data.

Using third-party data responsibly benefits brands and consumers alike. Learn how brand marketers are doing it right.

Using third-party data in marketing responsibly 

For nearly every B2C company, revenue is linked to the customer lifecycle. Customer acquisition, engagement, and retention initiatives directly or indirectly impact revenue growth and sustainability by either reducing customer acquisition costs, increasing customer lifetime value (LTV), or ideally, both.

Effective marketing strategies revolve around the customer lifecycle; understanding key lifecycle stages, identifying events that are likely to trigger transitions between stages, and finding ways to optimize interactions across the lifecycle.

Naturally, the more you know about your customers and how to influence them to transition into a desired stage, the more efficiently you can grow and sustain revenue with the resources at your disposal.

Simply put, that's what customer lifecycle optimization is all about: leveraging rich customer data and predictive analytics techniques to generate insights and make predictions that measurably improve outcomes at each stage of the customer lifecycle.

## How customer lifecycle optimization works

Customer lifecycle optimization (CLO) is a practice. As with other practices, CLO involves a series of prescribed steps to be done effectively. While specific lifecycle stage names, data sources, and analytics techniques will vary depending on your industry, company, and objectives, the canonical CLO process comprises four steps: lifecycle mapping; data discovery; predictive groundwork; and implementation and action.

### Customer lifecycle mapping

The first step in any CLO initiative is always identifying and defining key lifecycle stages and transitions between stages. Mapping these stages and transitions to a uniform customer lifecycle is crucial to uncovering rich, predictive data.

The uniform B2C customer lifecycle stages are:

1. Lead generation
2. Lead conversion
3. Retention and expansion
4. Reactivation

While terminology will change from business to business, we found that this formulation is rich enough to capture important boundaries, yet simple enough to avoid stages with ambiguous transitions.

Think about your customer lifecycle. How are stages defined? Which attributes qualify individuals to be placed in those stages? What events trigger transitions between stages?

### Customer data discovery

At each stage, prospects, leads, and customers will complete certain events that will individually or collectively trigger a transition in or out of that stage. Individuals will also have different attributes, which help determine whether or not they belong in any given stage.

Go through each cell of your customer lifecycle map and think about which events or attributes in your data could be used to trigger a transition or pass the litmus test. Then, think about where you can find that data. Is it in your ESP, CRM, or a custom data warehouse?

### Customer prediction groundwork

Once you've formalized your lifecycle and its representation in data, you can start recognizing patterns and eventually predict outcomes. Imagine loading up your converted leads (customers in "retention and expansion") alongside your stale leads that never converted. What differences can you find?

This comparison analysis is especially effective when you've added depth and breadth to your existing data.

As you experiment with this type of pattern recognition, you'll quickly realize that it's the kind of thing that computers do very well. That's where machine learning comes in handy.

### Predictive engagement

Whether you want to increase conversion rates from lead generation campaigns or reduce churn from existing customers, properly targeted outreach is essential to engaging and motivating the right leads and customers to take a desired action.

Outbound communication is the strongest and most versatile intervention at an organization's disposal to compel progress and therefore expand revenue. This includes individually targeted digital advertising, a form of direct outreach.

Having identified a stage transition you'd like to motivate with an outreach intervention, the question becomes, "Who do I reach?" Regardless of the desired transition, the general technique is called audience expansion, also known as "lookalike" audiences.

To apply the audience expansion technique, we must always identify the audience and a set of candidates with similar characteristics and attributes.

With these groups defined, the next step is to apply your predictive groundwork. This could involve using patterns you identified in your data to look for similar opportunities among your candidates, or in more advanced cases, using artificial intelligence to build a predictive model trained to discriminate between likely and unlikely transitioners.

Finally, you will be left with a well-defined group of candidates likely to transition into a desired lifecycle stage when reached with relevant content.

## Putting it all together

To summarize the CLO practice, it's useful to recall the original motivation: leveraging rich data and statistical predictions to optimize revenue-building initiatives throughout the customer lifecycle. This means motivating transitions from one stage to the next.

Customer lifecycle optimization is the practice of leveraging data to maximize revenue-generating customer behaviors throughout the lifecycle. Learn how it works.

What is customer lifecycle optimization and how does it work?

**UPDATED FOR 2017** Now with easy subquery support and a more sensible argument order!

We adapted [this excellent piece by Dmitri Fontaine](http://tapoueh.org/blog/2014/02/21-PostgreSQL-histogram) and turned it into a function.

It can be invoked like this:

```
SELECT * FROM histogram($table_name_or_subquery, $column_name);
```

. . . to give sweet results like this, in a check of the distribution of 2016 political contributions in Vermont:

```
fec=# SELECT * FROM histogram('(SELECT * FROM small_donors_vt LIMIT 50000)', 'transaction_amt');

 bucket |   range   | freq |       bar
--------+-----------+------+-----------------
      1 | [0,9]     | 2744 | ******
      2 | [10,19]   | 5630 | *************
      3 | [20,29]   | 6383 | ***************
      4 | [30,39]   | 1290 | ***
      5 | [40,49]   |  369 | *
      6 | [50,59]   | 3541 | ********
      7 | [60,69]   |  174 |
      8 | [70,79]   |  313 | *
      9 | [80,89]   |  171 |
     10 | [90,99]   |   65 |
     11 | [100,109] | 2363 | ******
     12 | [110,119] |   51 |
     13 | [120,129] |  115 |
     14 | [130,139] |   32 |
     15 | [140,146] |   11 |
     16 | [150,159] |  187 |
     17 | [160,169] |   24 |
     18 | [170,177] |   33 |
     19 | [180,189] |   19 |
     20 | [191,199] |   24 |
     21 | [200,200] |  795 | **

```

Use it yourself by adding this to your postgres setup:

```
CREATE OR REPLACE FUNCTION histogram(table_name_or_subquery text, column_name text)
RETURNS TABLE(bucket int, "range" numrange, freq bigint, bar text)
AS $func$
BEGIN
RETURN QUERY EXECUTE format('
  WITH
  source AS (
    SELECT * FROM %s
  ),
  min_max AS (
    SELECT min(%s) AS min, max(%s) AS max FROM source
  ),
  histogram AS (
    SELECT
      width_bucket(%s, min_max.min, min_max.max, 20) AS bucket,
      numrange(min(%s)::numeric, max(%s)::numeric, ''[]'') AS "range",
      count(%s) AS freq
    FROM source, min_max
    WHERE %s IS NOT NULL
    GROUP BY bucket
    ORDER BY bucket
  )
  SELECT
    bucket,
    "range",
    freq::bigint,
    repeat(''*'', (freq::float / (max(freq) over() + 1) * 15)::int) AS bar
  FROM histogram',
  table_name_or_subquery,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name,
  column_name
  );
END
$func$ LANGUAGE plpgsql;
```

Note:

1. You don't need to use a subquery - you can also just provide a table name as the first argument.
1. Subqueries are useful for sampling. For example, `(SELECT * FROM bigtable LIMIT 500 ORDER BY RANDOM()`. Don't forget parentheses!
1. You can call this as `SELECT histogram()` or `SELECT * FROM histogram()`. The latter form is much more legible!

Happy querying!

Learn how to create in-table histograms with PostgreSQL.

How to do histograms in PostgreSQL

Like many subscription services, meal delivery brand [Marley Spoon](https://marleyspoon.com/) faced the challenge of retaining customers. High churn rates made it difficult to cover acquisition costs and scale revenue, so they needed a way to identify risky customers. To improve their retention capabilities, Marley Spoon approached Faraday to assess the predictive accuracy between traditional segmentation and AI-driven churn modeling techniques.

## Traditional segmentation vs machine learning

With traditional methods, brands analyze trends among customer segments and take action to [prevent those with high churn rates](https://faraday.ai/predict/category/churn_scores) from unsubscribing. This is a relatively reactionary measure. In contrast, leveraging machine learning models to [score leads](https://faraday.ai/solutions/lead_management) and customers for their propensity to churn allows brands to segment customers based on churn risk and properly engage high-risk customers before they unsubscribe.

## Analysis and results

Because we’re most interested in the models’ ability to accurately predict churn, we evaluate performance based on how many churned customers were included in the top 10% and top 20% of predicted churners. Here are the results:

**Churn rate comparison:**

- 19% churn rate for likely churners identified with Faraday churn predictions
- 12% churn rate for likely churners identified through traditional segmentation

**Churn detection lift:**

- 70% lift over baseline with Faraday churn predictions
- 15% lift over baseline with traditional segmentation

To sum it up, Faraday's machine learning [predictions](https://faraday.ai/features/predictions) detected churn 5X better than the traditional technique.

Learn how the meal delivery subscription brand was able to identify likely churners 5X better with machine learning predictions.

How Marley Spoon predicted churn 5X better with machine learning

**Update** _We used to call this `jsonb_collect`, but then we realized it was very similar to `json_object_agg(name, value)`... except that there is no version of that function that just takes an expression. So, copying `jsonb_agg(expression)`, we give you..._

How do you aggregate (aka merge aka combine aka collect) JSONB fields in Postgres 9.5+? You define a new aggregate `jsonb_object_agg(expression)`:

```sql
CREATE AGGREGATE jsonb_object_agg(jsonb) (
  SFUNC = 'jsonb_concat',
  STYPE = jsonb,
  INITCOND = '{}'
);
```

Here's how you use it:

```console
# select * from greetings;
         data
-----------------------
 {"es":"Saludos"}
 {"en":"Hello"}
 {"ru":"Здравствуйте"}
(3 rows)

# select jsonb_object_agg(data) from greetings;
                        jsonb_object_agg
-------------------------------------------------------------
 { "es" : "Saludos", "en" : "Hello", "ru" : "Здравствуйте" }
(1 row)
```

If you're curious about the aggregate we just added, note that `jsonb_concat(jsonb, jsonb)` is the [function backing `||`](https://github.com/postgres/postgres/blob/REL9_5_STABLE/src/backend/utils/adt/jsonfuncs.c#L3349) that was [introduced in Postgres 9.5](https://www.postgresql.org/docs/9.5/static/release-9-5.html).

It's just that simple!

Learn how to aggregate (aka merge aka combine aka collect) JSONB fields in Postgres 9.5+.

How to aggregate JSONB in PostgreSQL 9.5+

At Faraday, we've long used [csvkit](https://csvkit.readthedocs.io/en/latest/) to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to [xsv](https://github.com/BurntSushi/xsv).

xsv is a fast CSV-parsing toolkit written in [Rust](https://www.rust-lang.org/en-US/) that mostly matches the functionality of csvkit (including the clutch ability to pipe between modules), with a few extras tacked on (like smart sampling). Did I mention it's fast? In a standup comparison, I ran the "stats" module of XSV against "csvstat" from csvkit, on a 30k-line, 400-column CSV file:

- Python-based csvkit chews through it in a respectable-and-now-expected 4m16s.

- **xsv takes 1.8 seconds.** I don't even have time for a sip of my coffee.

The difference between csvkit and xsv is partly defined by scale; both tools are plenty fast on smaller datasets. But once you get into 10MB-and-upward range, xsv's processing speed pulls away exponentially.

If you've been using csvkit forever (like me), or if you want to be able to transform and analyze CSVs without loading them into a DB, give xsv a shot:

### Install Rust

```bash
curl https://sh.rustup.rs -sSf | sh
```

. . . which also gives you the rust package manager `cargo`, which lets you:

### Install xsv

```bash
cargo install xsv
```

Then be sure your PATH is configured correctly:

```bash
export PATH=~/.cargo/bin:$PATH
```

. . . and try it out on a demo CSV with 10k rows, some messy strings, and multiple data types:

```bash
curl https://gist.githubusercontent.com/wboykinm/044e2af62fc0c7f77e17f6ccd55b8fb0/raw/fca391e6c03a06a7be770fefca6c47a9acdd2305/mock_data.csv \
| xsv stats \
| xsv table
```

_(`xsv table` formats the data so it's readable in the console):_

```bash
field           type     sum                 min                  max                  min_length  max_length  mean                stddev
id              Integer  5005000             1                    1000                 1           4           500.49999999999994  288.6749902572106
first_name      Unicode                      Aaron                Willie               3           11
last_name       Unicode                      Adams                Young                3           10
email           Unicode                      aadamsp5@senate.gov  wwrightd8@upenn.edu  12          34
gender          Unicode                      Female               Male                 4           6
ip_address      Unicode                      0.111.40.87          99.50.37.244         9           15
value           Unicode                      $1007.98             $999.37              0           8
company         Unicode                      Abata                Zoovu                0           13
lat             Float    243963.82509999987  -47.75034            69.70287             0           9           24.42080331331331   24.98767816017553
lon             Float    443214.19009999954  -179.12198           170.29993            0           10          44.36578479479489   71.16647723898215
messed_up_data  Unicode                      !@#$%^&*()           𠜎𠜱𠝹𠱓𠱸𠲖𠳏       0           393
version         Unicode                      0.1.1                9.99                 3           14
```

Happy parsing!

A data science hack to process CSVs faster using XSV.

How we made our CSV processing 142x faster

The Rust compiler just saved me from a nasty threading bug. I was working on an open source development tool for Docker apps with lots of microservices, and I decided to parallelize the routine that transformed `docker-compose.yml` files. This was mostly an excuse to check out the awesome [rayon][] library, but it turned into a great example of what real-world Rust development is like.

[rayon]: https://github.com/nikomatsakis/rayon

The original routine looked something like this:

```rust
/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {
            continue;
        }

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        try!(file.make_standalone(&self.pods_dir()));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));
        try!(file.write_to_path(out_path));
    }
    Ok(())
}
```

To convert this to a parallel loop, I started by changing:

```rust
for pod in &self.pods {
```

To:

```rust
self.pods.par_iter().map(|pod| -> Result<()> {
```

Here, `Result<()>` means "this closure might return an error, or it might return an empty tuple, basically `void`." (For more information on Rust error handling, check out [the Rust book][errors].) But it couldn't be that easy, could it?

[errors]: https://doc.rust-lang.org/book/ch09-00-error-handling.html

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

```
error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied
   --> src/project.rs:397:30
    |
397 |         self.pods.par_iter().map(|pod| -> Result<()> {
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    |
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/project.rs:397:34: 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`
```

The key bit to take away here is that "`Sync` is not satisfied" for the trait `plugins::PluginTransform`. In Rust, `Sync` is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my `PluginTransform` objects live inside my `plugins::Manager` type, which in turn lives inside my `Project`. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that `PluginTransform` inherits from the `Plugin` trait:

```rust
pub trait PluginTransform: Plugin {
```

So we can fix this problem by changing the declaration of `Plugin` from:

```rust
pub trait Plugin {
```

To:

```rust
pub trait Plugin: Sync {
```

This says, "All types implementing `Plugin` must also allow me to access them from multiple threads." With that fixed, I re-run `cargo test` and get a similar message:

```
error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:230:6
    |
230 | impl plugins::Plugin for Plugin {
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied
    |
```

This time, I change:

```rust
trait GenerateToken: Debug {
```

To:

```rust
trait GenerateToken: Debug + Sync {
```

This says, "All types which implement `GenerateToken` must also implement `Debug` (so I can print them), as well as `Sync` (so I can share them). We're making progress!

Re-running `cargo test`, however, reveals the actual bug, and it would have been a nightmare to debug:

```
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:124:6
    |
124 | impl GenerateToken for MockVault {
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied
    |
```

Uh-oh. In our test harness, we have a type `MockVault`, which contains an `Rc<RefCell<_>>`:

```rust
type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
#[derive(Debug)]
#[cfg(test)]
struct MockVault {
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,
}

```

We use the `MockVault` to simulate a connection to Hashicorp's [Vault][], a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the `calls` member here to record all the requests that we would have made to Vault.

[Vault]: https://www.vaultproject.io/

The type `Rc<RefCell<_>>` is a hack. The `GenerateToken` API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see [the Rust book][mutability].) But `Rc<RefCell<_>>` is a lightweight mechanism designed for single-threaded code.

[mutability]: https://doc.rust-lang.org/beta/book/mutability.html

The fix is to replace `Rc<RefCell<_>>` with `Arc<RwLock<_>>`, which is fully thread-safe:

```rust
type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;
```

Once we make this change, Rust reminds us to change all the code that accesses `calls`, too:

```
error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:132:20
    |
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));
    |                    ^^^^^^^^^^
    |
...
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:368:23
    |
368 |     let calls = calls.borrow();
    |                       ^^^^^^
    |
```

Phew. If we hadn't caught that `Rc<RefCell<_>>`, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites _could_ trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

## Wrapping it up

Rust also complains about that fact that we're calling `continue` from inside a closure:

```
error[E0267]: `continue` inside of a closure
   --> src/project.rs:402:17
    |
402 |                 continue;
    |                 ^^^^^^^^ cannot break inside of a closure
```

We can fix this by moving this code:

```rust
// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {
    continue;
}
```

...into a `filter` call on our parallel iterator:

```rust
self.pods.par_iter()
    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {
```

The final problem is that we now have a parallel computation that returns a bunch of `Result<()>` objects, and we need to boil them down to a single `Result<()>` like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

```rust
    .map(|pod| -> Result<()> {
        // ...
    }
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))
    .unwrap_or(Ok(()))
```

This says, "Given any two results named `result1` and `result2`, take `result1` if it's an error, otherwise take `result2`." It's basically a short-circuit `&&` operator, but for `Result` values instead of booleans. I've filed an issue suggesting that `rayon` should provide those last two lines as a built-in function.

[issue113]: https://github.com/nikomatsakis/rayon/issues/113

And yes, that's a genuine parallel [map-reduce][] in Rust!

[map-reduce]: http://research.google.com/archive/mapreduce.html

So here's our final loop, using work-stealing parallelism:

```rust
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.
    self.pods.par_iter()

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            try!(file.make_standalone(&self.pods_dir()));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));
            try!(file.write_to_path(out_path));
            Ok(())
        })

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))
        .unwrap_or(Ok(()))
}
```

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

A detailed example of how to parallelize a loop with Rust and rayon.

Saved by the compiler: Parallelizing a loop with Rust and rayon

At Faraday, we use [redux-saga] in our client-side webapp because it offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps. I recently upgraded our version of redux-saga from an embarrassingly old version to the latest, in part to take advantage of a new (but largely undocumented) feature: context.

### The problem

I want to be able to write sagas without having to implicitly import any statically-declared singletons. The reason for that is simple: if I'm testing the saga that creates new users, I don't want to drag along the API layer or the router instance.

There are also a few modules in the codebase that _only_ work in a browser and will fail loudly when I try to run them in mocha. If one of these browser-only modules ends up in an import chain, suddenly my tests start failing for bad reasons. At worst, I have to rethink my testing strategy to include a mock browser environment. I'm lazy and hate testing, and I think mocks are a gross awful code smell that should be avoided at all costs.

### The solution

One of redux-saga's greatest features is that I can test a saga's behavior without actually executing that behavior, or even really knowing much about the details of how that behavior alters the world. It it should be possible to apply that to dependencies as well. And it is possible in redux-saga 0.15.0 and later using context.

Right now, there are very limited docs for context, but acts as shared value across all sagas that can be read with the `getContext` effect and written to by the `setContext` effect. It can also be set when the saga middleware is created by passing a context object as configuration.

### Example

Let's say we're writing an app that fetches game inventory data from the server using some kind of authentication. We don't want tokens and authentication to bleed into all our sagas, so we wrap it up in a nice singleton API service:

```js
class ApiService {
  getInventory = () => {
    return fetch("/api/inventory", {
      headers: {
        Authorization: `Bearer ${this.token}`,
      },
    }).then((res) => res.json());
  };
}

const api = new ApiService();
api.token = localStorage.token;

export default api;
```

Without context, our saga would probably statically import the API singleton:

```js
import { call } from "redux-saga/effects";
import api from "./api";

export function* fetchInventorySaga() {
  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

All is well until we try to test it in nodejs/mocha:

```js
import { fetchInventorySaga } from '../src/sagas/inventorySaga.js
// ReferenceError: localStorage is not defined
```

There is no localStorage in the nodejs global context. We can either pull in a testing harness to change how `import api from './api'` is resolved, attempt to run the tests in a browser, or roll our own late-binding mechanism so that you don't need to import API and can pass the API instance in at runtime.

We need to solve the same problem for `fetch`, because that's also absent in nodejs.

Or, we could use context. Our saga changes only a little bit:

```js
import { call, getContext } from "redux-saga/effects";
// No more API import:
// import api from './api'

export function* fetchInventorySaga() {
  // Get the api value out of the shared context:
  const api = yield getContext("api");

  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

We can now test the `getContext` effect just like we would any other redux-saga effect, and we can insert a mock value into `fetchInventorySaga` at test-time if we need to.

Setting up context in the main application is very straightforward. When you're creating your saga middleware:

```js
import createSagaMiddleware from "redux-saga";
import api from "./api";

const saga = createSagaMiddleware({
  context: {
    api, // our singleton API value
  },
});
```

Being able to late-bind singleton values like this has been enormously helpful writing robust tests in a complex codebase. I'll be steadily migrating the application code to use `getContext` more frequently, now that I have it as an option.

[redux-saga]: https://redux-saga.js.org
[a couple cryptic sentences]: https://github.com/redux-saga/redux-saga/releases/tag/v0.15.0

Redux-Saga offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps.

Using context in redux-saga

[Terraform's RDS support](https://www.terraform.io/docs/providers/aws/r/db_instance.html) makes it easy to create a database instance. Here's a cheatsheet:

```
resource "aws_db_instance" "mydb1" {
  allocated_storage        = 256 # gigabytes
  backup_retention_period  = 7   # in days
  db_subnet_group_name     = "${var.rds_public_subnet_group}"
  engine                   = "postgres"
  engine_version           = "9.5.4"
  identifier               = "mydb1"
  instance_class           = "db.r3.large"
  multi_az                 = false
  name                     = "mydb1"
  parameter_group_name     = "mydbparamgroup1" # if you have tuned it
  password                 = "${trimspace(file("${path.module}/secrets/mydb1-password.txt"))}"
  port                     = 5432
  publicly_accessible      = true
  storage_encrypted        = true # you should always do this
  storage_type             = "gp2"
  username                 = "mydb1"
  vpc_security_group_ids   = ["${aws_security_group.mydb1.id}"]
}
```

Here's the security group you need:

```
resource "aws_security_group" "mydb1" {
  name = "mydb1"

  description = "RDS postgres servers (terraform-managed)"
  vpc_id = "${var.rds_vpc_id}"

  # Only postgres in
  ingress {
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Allow all outbound traffic.
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
```

You can get these values from the EC2 console (don't forget them!):

```
variable "rds_vpc_id" {
  default = "vpc-XXXXXXXX"
  description = "Our default RDS virtual private cloud (rds_vpc)."
}

variable "rds_public_subnets" {
  default = "subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY"
  description = "The public subnets of our RDS VPC rds-vpc."
}

variable "rds_public_subnet_group" {
  default = "default-vpc-XXXXXXXX"
  description = "Apparently the group name, according to the RDS launch wizard."
}
```

Check out our Terraform RDS cheatsheet that makes it easy to create a database instance.

How to create an RDS instance with Terraform

(The more valuable and massive a data set is, the less likely it's in a format you can just parse. Has anybody else noticed that?)

Here's how to convert a fixed-width file to CSV with the [standard GNU unix tool `gawk`](https://www.gnu.org/software/gawk/):

### Theoretical (i.e., see "real life" below)

Thanks to [stackoverflow](http://stackoverflow.com/questions/28561786/convert-a-fixed-width-file-from-text-to-csv): (reproducing verbatim)

```
gawk '$1=$1' OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv
```

Where `FIELDWIDTHS` is a list of field widths and `OFS` is the output file separator.

### Real life

In real life, fixed width files contain commas and double quotes.

```
# put this in a file called fixed2csv.awk
{
  for (i=1;i<=NF;i++) {
    sub(/\s+$/,"",$i)
    sub("\"","\"\"",$i)
    printf "\"%s\"%s", $i, (i<NF?OFS:ORS)
  }
}
```

Then run it on your data:

```
gawk -f fixed2csv.awk OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv
```

Thanks to [Ed Morton on Stackoverflow](http://stackoverflow.com/questions/31947383/fixed-width-to-csv) for inspiration!

Learn how to convert a fixed-width file to CSV with the standard GNU unix tool gawk

All posts