**Update** _We used to call this `jsonb_collect`, but then we realized it was very similar to `json_object_agg(name, value)`... except that there is no version of that function that just takes an expression. So, copying `jsonb_agg(expression)`, we give you..._

How do you aggregate (aka merge aka combine aka collect) JSONB fields in Postgres 9.5+? You define a new aggregate `jsonb_object_agg(expression)`:

```sql
CREATE AGGREGATE jsonb_object_agg(jsonb) (
  SFUNC = 'jsonb_concat',
  STYPE = jsonb,
  INITCOND = '{}'
);
```

Here's how you use it:

```console
# select * from greetings;
         data
-----------------------
 {"es":"Saludos"}
 {"en":"Hello"}
 {"ru":"Здравствуйте"}
(3 rows)

# select jsonb_object_agg(data) from greetings;
                        jsonb_object_agg
-------------------------------------------------------------
 { "es" : "Saludos", "en" : "Hello", "ru" : "Здравствуйте" }
(1 row)
```

If you're curious about the aggregate we just added, note that `jsonb_concat(jsonb, jsonb)` is the [function backing `||`](https://github.com/postgres/postgres/blob/REL9_5_STABLE/src/backend/utils/adt/jsonfuncs.c#L3349) that was [introduced in Postgres 9.5](https://www.postgresql.org/docs/9.5/static/release-9-5.html).

It's just that simple!

Learn how to aggregate (aka merge aka combine aka collect) JSONB fields in Postgres 9.5+.

How to aggregate JSONB in PostgreSQL 9.5+

At Faraday, we've long used [csvkit](https://csvkit.readthedocs.io/en/latest/) to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to [xsv](https://github.com/BurntSushi/xsv).

xsv is a fast CSV-parsing toolkit written in [Rust](https://www.rust-lang.org/en-US/) that mostly matches the functionality of csvkit (including the clutch ability to pipe between modules), with a few extras tacked on (like smart sampling). Did I mention it's fast? In a standup comparison, I ran the "stats" module of XSV against "csvstat" from csvkit, on a 30k-line, 400-column CSV file:

- Python-based csvkit chews through it in a respectable-and-now-expected 4m16s.

- **xsv takes 1.8 seconds.** I don't even have time for a sip of my coffee.

The difference between csvkit and xsv is partly defined by scale; both tools are plenty fast on smaller datasets. But once you get into 10MB-and-upward range, xsv's processing speed pulls away exponentially.

If you've been using csvkit forever (like me), or if you want to be able to transform and analyze CSVs without loading them into a DB, give xsv a shot:

### Install Rust

```bash
curl https://sh.rustup.rs -sSf | sh
```

. . . which also gives you the rust package manager `cargo`, which lets you:

### Install xsv

```bash
cargo install xsv
```

Then be sure your PATH is configured correctly:

```bash
export PATH=~/.cargo/bin:$PATH
```

. . . and try it out on a demo CSV with 10k rows, some messy strings, and multiple data types:

```bash
curl https://gist.githubusercontent.com/wboykinm/044e2af62fc0c7f77e17f6ccd55b8fb0/raw/fca391e6c03a06a7be770fefca6c47a9acdd2305/mock_data.csv \
| xsv stats \
| xsv table
```

_(`xsv table` formats the data so it's readable in the console):_

```bash
field           type     sum                 min                  max                  min_length  max_length  mean                stddev
id              Integer  5005000             1                    1000                 1           4           500.49999999999994  288.6749902572106
first_name      Unicode                      Aaron                Willie               3           11
last_name       Unicode                      Adams                Young                3           10
email           Unicode                      aadamsp5@senate.gov  wwrightd8@upenn.edu  12          34
gender          Unicode                      Female               Male                 4           6
ip_address      Unicode                      0.111.40.87          99.50.37.244         9           15
value           Unicode                      $1007.98             $999.37              0           8
company         Unicode                      Abata                Zoovu                0           13
lat             Float    243963.82509999987  -47.75034            69.70287             0           9           24.42080331331331   24.98767816017553
lon             Float    443214.19009999954  -179.12198           170.29993            0           10          44.36578479479489   71.16647723898215
messed_up_data  Unicode                      !@#$%^&*()           𠜎𠜱𠝹𠱓𠱸𠲖𠳏       0           393
version         Unicode                      0.1.1                9.99                 3           14
```

Happy parsing!

A data science hack to process CSVs faster using XSV.

How we made our CSV processing 142x faster

The Rust compiler just saved me from a nasty threading bug. I was working on an open source development tool for Docker apps with lots of microservices, and I decided to parallelize the routine that transformed `docker-compose.yml` files. This was mostly an excuse to check out the awesome [rayon][] library, but it turned into a great example of what real-world Rust development is like.

[rayon]: https://github.com/nikomatsakis/rayon

The original routine looked something like this:

```rust
/// Process our pods, flattening and transforming them using our
/// plugins, and output them to the specified directory.
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.
    for pod in &self.pods {
        // Don't export pods which aren't enabled.
        if !pod.enabled_in(&self.current_target) {
            continue;
        }

        // Figure out where to put our pod.
        // ...

        // Combine overrides, make it standalone, tweak as needed, and
        // output.
        let mut file = try!(pod.merged_file(&self.current_target));
        try!(file.make_standalone(&self.pods_dir()));
        let ctx = plugins::Context::new(self, pod);
        try!(self.plugins().transform(op, &ctx, &mut file));
        try!(file.write_to_path(out_path));
    }
    Ok(())
}
```

To convert this to a parallel loop, I started by changing:

```rust
for pod in &self.pods {
```

To:

```rust
self.pods.par_iter().map(|pod| -> Result<()> {
```

Here, `Result<()>` means "this closure might return an error, or it might return an empty tuple, basically `void`." (For more information on Rust error handling, check out [the Rust book][errors].) But it couldn't be that easy, could it?

[errors]: https://doc.rust-lang.org/book/ch09-00-error-handling.html

Nope. The Rust compiler showed me the following error message. It's long, but I'll translate:

```
error[E0277]: the trait bound `plugins::PluginTransform + 'static: std::marker::Sync` is not satisfied
   --> src/project.rs:397:30
    |
397 |         self.pods.par_iter().map(|pod| -> Result<()> {
    |                              ^^^ trait `plugins::PluginTransform + 'static: std::marker::Sync` not satisfied
    |
    = note: `plugins::PluginTransform + 'static` cannot be shared between threads safely
    = note: required because it appears within the type `Box<plugins::PluginTransform + 'static>`
    = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `alloc::raw_vec::RawVec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `std::vec::Vec<Box<plugins::PluginTransform + 'static>>`
    = note: required because it appears within the type `plugins::Manager`
    = note: required because it appears within the type `std::option::Option<plugins::Manager>`
    = note: required because it appears within the type `project::Project`
    = note: required because it appears within the type `&project::Project`
    = note: required because it appears within the type `&&project::Project`
    = note: required because it appears within the type `[closure@src/project.rs:397:34: 424:10 op:&plugins::Operation, self:&&project::Project, export_dir:&&std::path::Path]`
```

The key bit to take away here is that "`Sync` is not satisfied" for the trait `plugins::PluginTransform`. In Rust, `Sync` is a special trait that tells the compiler that it's safe to share an object between two threads. That whole long list of "notes" afterwards looks intimidating, but it's really just telling me that my `PluginTransform` objects live inside my `plugins::Manager` type, which in turn lives inside my `Project`. The Rust compiler is verbose, but it tries to be helpful.

A bit of investigation reveals that `PluginTransform` inherits from the `Plugin` trait:

```rust
pub trait PluginTransform: Plugin {
```

So we can fix this problem by changing the declaration of `Plugin` from:

```rust
pub trait Plugin {
```

To:

```rust
pub trait Plugin: Sync {
```

This says, "All types implementing `Plugin` must also allow me to access them from multiple threads." With that fixed, I re-run `cargo test` and get a similar message:

```
error[E0277]: the trait bound `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:230:6
    |
230 | impl plugins::Plugin for Plugin {
    |      ^^^^^^^^^^^^^^^ trait `plugins::transform::vault::GenerateToken + 'static: std::marker::Sync` not satisfied
    |
```

This time, I change:

```rust
trait GenerateToken: Debug {
```

To:

```rust
trait GenerateToken: Debug + Sync {
```

This says, "All types which implement `GenerateToken` must also implement `Debug` (so I can print them), as well as `Sync` (so I can share them). We're making progress!

Re-running `cargo test`, however, reveals the actual bug, and it would have been a nightmare to debug:

```
error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` is not satisfied
   --> src/plugins/transform/vault.rs:124:6
    |
124 | impl GenerateToken for MockVault {
    |      ^^^^^^^^^^^^^ trait `std::rc::Rc<std::cell::RefCell<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>: std::marker::Sync` not satisfied
    |
```

Uh-oh. In our test harness, we have a type `MockVault`, which contains an `Rc<RefCell<_>>`:

```rust
type MockVaultCalls = Rc<RefCell<Vec<(String, Vec<String>, VaultDuration)>>>;

/// A fake interface to vault for testing purposes.
#[derive(Debug)]
#[cfg(test)]
struct MockVault {
    /// The tokens we were asked to generate.  We store these in a RefCell
    /// so that we can have "interior" mutability, because we don't want
    /// `generate_token` to be `&mut self` in the general case.
    calls: MockVaultCalls,
}

```

We use the `MockVault` to simulate a connection to Hashicorp's [Vault][], a secure central storage for passwords and other secrets, which issues time-limited credentials. And when we test our Vault code, we use the `calls` member here to record all the requests that we would have made to Vault.

[Vault]: https://www.vaultproject.io/

The type `Rc<RefCell<_>>` is a hack. The `GenerateToken` API assumes that our token-generator is a read-only object. This is good, because we want to access it from multiple threads! But in the test code, we need to create some "interior" mutable state. Basically, we ask Rust to replace compile-time borrow checks with run-time borrow checks. (For more details, see [the Rust book][mutability].) But `Rc<RefCell<_>>` is a lightweight mechanism designed for single-threaded code.

[mutability]: https://doc.rust-lang.org/beta/book/mutability.html

The fix is to replace `Rc<RefCell<_>>` with `Arc<RwLock<_>>`, which is fully thread-safe:

```rust
type MockVaultCalls = Arc<RwLock<Vec<(String, Vec<String>, VaultDuration)>>>;
```

Once we make this change, Rust reminds us to change all the code that accesses `calls`, too:

```
error: no method named `borrow_mut` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:132:20
    |
132 |         self.calls.borrow_mut().push((display_name.to_owned(), policies, ttl));
    |                    ^^^^^^^^^^
    |
...
error: no method named `borrow` found for type `std::sync::Arc<std::sync::RwLock<std::vec::Vec<(std::string::String, std::vec::Vec<std::string::String>, vault::client::VaultDuration)>>>` in the current scope
   --> src/plugins/transform/vault.rs:368:23
    |
368 |     let calls = calls.borrow();
    |                       ^^^^^^
    |
```

Phew. If we hadn't caught that `Rc<RefCell<_>>`, our test suites might have randomly segfaulted a couple of times a week, and it would have taken us weeks to track it down. I don't even think our current test suites _could_ trigger this bug, but I bet a future version would have been able to, leaving a nasty surprise for us someday. The Rust compiler dug down through layers of data structures and found the one bit that wasn't thread safe.

## Wrapping it up

Rust also complains about that fact that we're calling `continue` from inside a closure:

```
error[E0267]: `continue` inside of a closure
   --> src/project.rs:402:17
    |
402 |                 continue;
    |                 ^^^^^^^^ cannot break inside of a closure
```

We can fix this by moving this code:

```rust
// Don't export pods which aren't enabled.
if !pod.enabled_in(&self.current_target) {
    continue;
}
```

...into a `filter` call on our parallel iterator:

```rust
self.pods.par_iter()
    // Don't export pods which aren't enabled.
    .filter(|pod| pod.enabled_in(&self.current_target))
    // Process each pod in parallel.
    .map(|pod| -> Result<()> {
```

The final problem is that we now have a parallel computation that returns a bunch of `Result<()>` objects, and we need to boil them down to a single `Result<()>` like we had before. If we have a bunch of error messages, I'm happy to just pick one. No need to overwhelm the user.

I spoke to Josh Stone, and he helped me come up with the following:

```rust
    .map(|pod| -> Result<()> {
        // ...
    }
    // If more than one parallel branch fails, just return one error.
    .reduce_with(|result1, result2| result1.and(result2))
    .unwrap_or(Ok(()))
```

This says, "Given any two results named `result1` and `result2`, take `result1` if it's an error, otherwise take `result2`." It's basically a short-circuit `&&` operator, but for `Result` values instead of booleans. I've filed an issue suggesting that `rayon` should provide those last two lines as a built-in function.

[issue113]: https://github.com/nikomatsakis/rayon/issues/113

And yes, that's a genuine parallel [map-reduce][] in Rust!

[map-reduce]: http://research.google.com/archive/mapreduce.html

So here's our final loop, using work-stealing parallelism:

```rust
fn output_helper(&self, op: Operation, export_dir: &Path) -> Result<()> {
    // Output each pod.  This isn't especially slow (except maybe the
    // Vault plugin), but parallelizing things is easy.
    self.pods.par_iter()

        // Don't export pods which aren't enabled.
        .filter(|pod| pod.enabled_in(&self.current_target))

        // Process each pod in parallel.
        .map(|pod| -> Result<()> {
            // Figure out where to put our pod.
            // ...

            // Combine overrides, make it standalone, tweak as needed, and
            // output.
            let mut file = try!(pod.merged_file(&self.current_target));
            try!(file.make_standalone(&self.pods_dir()));
            let ctx = plugins::Context::new(self, pod);
            try!(self.plugins().transform(op, &ctx, &mut file));
            try!(file.write_to_path(out_path));
            Ok(())
        })

        // If more than one parallel branch fails, just return one error.
        .reduce_with(|result1, result2| result1.and(result2))
        .unwrap_or(Ok(()))
}
```

This was pretty painless! I have a lot more things to parallelize, of course. But this is what day-to-day Rust development is like: I have to do a bit of extra work to satisfy the compiler (which mostly becomes a reflex). But in turn, the compiler ferrets out all kinds of subtle concurrency errors and generally watches my back. It's an interesting tradeoff, and overall I like it.

A detailed example of how to parallelize a loop with Rust and rayon.

Saved by the compiler: Parallelizing a loop with Rust and rayon

At Faraday, we use [redux-saga] in our client-side webapp because it offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps. I recently upgraded our version of redux-saga from an embarrassingly old version to the latest, in part to take advantage of a new (but largely undocumented) feature: context.

### The problem

I want to be able to write sagas without having to implicitly import any statically-declared singletons. The reason for that is simple: if I'm testing the saga that creates new users, I don't want to drag along the API layer or the router instance.

There are also a few modules in the codebase that _only_ work in a browser and will fail loudly when I try to run them in mocha. If one of these browser-only modules ends up in an import chain, suddenly my tests start failing for bad reasons. At worst, I have to rethink my testing strategy to include a mock browser environment. I'm lazy and hate testing, and I think mocks are a gross awful code smell that should be avoided at all costs.

### The solution

One of redux-saga's greatest features is that I can test a saga's behavior without actually executing that behavior, or even really knowing much about the details of how that behavior alters the world. It it should be possible to apply that to dependencies as well. And it is possible in redux-saga 0.15.0 and later using context.

Right now, there are very limited docs for context, but acts as shared value across all sagas that can be read with the `getContext` effect and written to by the `setContext` effect. It can also be set when the saga middleware is created by passing a context object as configuration.

### Example

Let's say we're writing an app that fetches game inventory data from the server using some kind of authentication. We don't want tokens and authentication to bleed into all our sagas, so we wrap it up in a nice singleton API service:

```js
class ApiService {
  getInventory = () => {
    return fetch("/api/inventory", {
      headers: {
        Authorization: `Bearer ${this.token}`,
      },
    }).then((res) => res.json());
  };
}

const api = new ApiService();
api.token = localStorage.token;

export default api;
```

Without context, our saga would probably statically import the API singleton:

```js
import { call } from "redux-saga/effects";
import api from "./api";

export function* fetchInventorySaga() {
  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

All is well until we try to test it in nodejs/mocha:

```js
import { fetchInventorySaga } from '../src/sagas/inventorySaga.js
// ReferenceError: localStorage is not defined
```

There is no localStorage in the nodejs global context. We can either pull in a testing harness to change how `import api from './api'` is resolved, attempt to run the tests in a browser, or roll our own late-binding mechanism so that you don't need to import API and can pass the API instance in at runtime.

We need to solve the same problem for `fetch`, because that's also absent in nodejs.

Or, we could use context. Our saga changes only a little bit:

```js
import { call, getContext } from "redux-saga/effects";
// No more API import:
// import api from './api'

export function* fetchInventorySaga() {
  // Get the api value out of the shared context:
  const api = yield getContext("api");

  const inventory = yield call(api.getInventory);
  // Do something with the inventory data...
}
```

We can now test the `getContext` effect just like we would any other redux-saga effect, and we can insert a mock value into `fetchInventorySaga` at test-time if we need to.

Setting up context in the main application is very straightforward. When you're creating your saga middleware:

```js
import createSagaMiddleware from "redux-saga";
import api from "./api";

const saga = createSagaMiddleware({
  context: {
    api, // our singleton API value
  },
});
```

Being able to late-bind singleton values like this has been enormously helpful writing robust tests in a complex codebase. I'll be steadily migrating the application code to use `getContext` more frequently, now that I have it as an option.

[redux-saga]: https://redux-saga.js.org
[a couple cryptic sentences]: https://github.com/redux-saga/redux-saga/releases/tag/v0.15.0

Redux-Saga offers a robust and testable set of tools for managing complex and asynchronous effects within Redux apps.

Using context in redux-saga

[Terraform's RDS support](https://www.terraform.io/docs/providers/aws/r/db_instance.html) makes it easy to create a database instance. Here's a cheatsheet:

```
resource "aws_db_instance" "mydb1" {
  allocated_storage        = 256 # gigabytes
  backup_retention_period  = 7   # in days
  db_subnet_group_name     = "${var.rds_public_subnet_group}"
  engine                   = "postgres"
  engine_version           = "9.5.4"
  identifier               = "mydb1"
  instance_class           = "db.r3.large"
  multi_az                 = false
  name                     = "mydb1"
  parameter_group_name     = "mydbparamgroup1" # if you have tuned it
  password                 = "${trimspace(file("${path.module}/secrets/mydb1-password.txt"))}"
  port                     = 5432
  publicly_accessible      = true
  storage_encrypted        = true # you should always do this
  storage_type             = "gp2"
  username                 = "mydb1"
  vpc_security_group_ids   = ["${aws_security_group.mydb1.id}"]
}
```

Here's the security group you need:

```
resource "aws_security_group" "mydb1" {
  name = "mydb1"

  description = "RDS postgres servers (terraform-managed)"
  vpc_id = "${var.rds_vpc_id}"

  # Only postgres in
  ingress {
    from_port = 5432
    to_port = 5432
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Allow all outbound traffic.
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
```

You can get these values from the EC2 console (don't forget them!):

```
variable "rds_vpc_id" {
  default = "vpc-XXXXXXXX"
  description = "Our default RDS virtual private cloud (rds_vpc)."
}

variable "rds_public_subnets" {
  default = "subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY,subnet-YYYYYYYY"
  description = "The public subnets of our RDS VPC rds-vpc."
}

variable "rds_public_subnet_group" {
  default = "default-vpc-XXXXXXXX"
  description = "Apparently the group name, according to the RDS launch wizard."
}
```

Check out our Terraform RDS cheatsheet that makes it easy to create a database instance.

How to create an RDS instance with Terraform

(The more valuable and massive a data set is, the less likely it's in a format you can just parse. Has anybody else noticed that?)

Here's how to convert a fixed-width file to CSV with the [standard GNU unix tool `gawk`](https://www.gnu.org/software/gawk/):

### Theoretical (i.e., see "real life" below)

Thanks to [stackoverflow](http://stackoverflow.com/questions/28561786/convert-a-fixed-width-file-from-text-to-csv): (reproducing verbatim)

```
gawk '$1=$1' OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv
```

Where `FIELDWIDTHS` is a list of field widths and `OFS` is the output file separator.

### Real life

In real life, fixed width files contain commas and double quotes.

```
# put this in a file called fixed2csv.awk
{
  for (i=1;i<=NF;i++) {
    sub(/\s+$/,"",$i)
    sub("\"","\"\"",$i)
    printf "\"%s\"%s", $i, (i<NF?OFS:ORS)
  }
}
```

Then run it on your data:

```
gawk -f fixed2csv.awk OFS=, FIELDWIDTHS='4 2 5 1 1' infile > outfile.csv
```

Thanks to [Ed Morton on Stackoverflow](http://stackoverflow.com/questions/31947383/fixed-width-to-csv) for inspiration!

Learn how to convert a fixed-width file to CSV with the standard GNU unix tool gawk

How to convert a fixed-width file into CSV

Amazon's [explanation of deleting a versioned object](http://docs.aws.amazon.com/AmazonS3/latest/dev/RemDelMarker.html) and the [SDK documentation](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#delete-instance_method) do not give an example of permanently deleting a versioned object. Here's how to do it.

```ruby
require 'aws-sdk'

s3 = Aws::S3::Resource.new(
  region: 'us-east-1',
  access_key_id: ACCESS_KEY_ID,
  secret_access_key: SECRET_ACCESS_KEY
)
bucket = s3.bucket('my-versioned-bucket')

bucket.objects.each do |object_summary|
  o = bucket.object object_summary.key
  # this is the secret: specify the version while deleting
  o.delete version_id: o.version_id
end
```

If you don't specify the version, you get a delete marker, which you can proceed to delete infinite times and it will not go away :)

Learn how to permanently deleted version objects from AWS S3.

How to permanently delete versioned objects from AWS S3

"How do you do `strftime` in postgres?"

[The answer](https://www.postgresql.org/docs/9.5/static/functions-formatting.html): `to_char(date, format)`.

If you want to group by month, this is what you're looking for:

```
psql=> select count(*), to_char(created_at, 'YYYY-MM') from employees group by to_char(created_at, 'YYYY-MM') order by to_char(created_at,'YYYY-MM') desc;
 count | to_char
-------+---------
    27 | 2016-08
    32 | 2016-07
    58 | 2016-06
    17 | 2016-05
    57 | 2016-04
    44 | 2016-03
    28 | 2016-02
    45 | 2016-01
    10 | 2015-12
    10 | 2015-11
    24 | 2015-10
    15 | 2015-09
    32 | 2015-08
    38 | 2015-07
    31 | 2015-06
    18 | 2015-05
    19 | 2015-04
     5 | 2015-03
     8 | 2015-02
    10 | 2015-01
     7 | 2014-12
    22 | 2014-11
(22 rows)
```

That's it.

How do you do strftime in postgres? Read this straightforward tutorial.

All posts