Using spatial aggregation and consumer predictions to analyze markets
A fundamental part of location intelligence, spatial aggregation combines individuals into groups and generalizes information about them.
Read more
Bill Morris
Hails from the mountains. Mapslinging and data plumbing with faraday.io.
The weighted scale: Mitigating implicit bias in data science
An algorithm contains the biases of its builder. At Faraday, we have a handful of approaches we use to minimize these effects at each level of our machine learning pipeline.
Read more
Customer insight fundamentals: Understand your customer data to make better predictions
Most data scientists practice some version of what could be called a data survey; the goal is to surface meaningful patterns, gaps, and anomalies with an eye toward prediction.
Read moreHow to get U.S. Census data as CSV — censusapi2csv
This post is part of our data science series. The U.S. Census and American Community Survey (ACS) are the crown jewels of open data (bother your Representative today to make sure they stay that way), but working with data from the Census API isn't always intuitive. Here's an example
Read more
Geochunk: fast, intelligent splitting for piles of address data
This post is part of our practical cartography and data science series. The problem: you want to split up a few million U.S. address records into equally-sized chunks that retain spatial hierarchy. You want to do this without anything other than a street address (geocoding is expensive!). Maybe you
Read more
Be not afraid of ZCTAs
This post is part of our practical cartography series. Most American geographers will note that - as much as we'd like it to be otherwise - ZIP Codes are not polygons. Rather, they're constantly-changing lines used by the USPS to coordinate delivery in an efficient network. Many of us polygon-happy
Read more
Plancha: how to flatten multi-sheet excel workbooks
This is part of our series on data science because it belongs in your toolchain. If you work with data long enough - actually scratch that; if you work with data for more than a week - you'll run into the dreaded multi sheet (or tab) excel workbook. Sometimes the
Read more
How we made our CSV processing 142x faster
This post is part of our data science hacks series At Faraday, we've long used csvkit to understand, transform, and beat senseless our many streams of data. However, even this inimitable swiss army knife can be improved on - we've switched to xsv. xsv is a fast CSV-parsing toolkit written
Read more
How to reverse geocode in bulk
This post is part of our practical cartography series. We just rebuilt our Argo reverse-geocoding module as a proper command-line tool. Got a pile of coordinates in a table like this? Pipe them through argo to get the context of an address assigned to each of them: npm install argo-geo
Read more
How to crunch lots of geodata in parallel
This post is part of our data science and practical cartography series. GNU parallel + ogr2ogr = happy data scientists These power tools in combination make it very easy to process lots of geodata at once, in as many parallel operations as your local machine or server can support. Reprojecting in bulk
Read more
How to do histograms in PostgreSQL
This post is part of our data science and PostgreSQL series. UPDATED FOR 2017 Now with easy subquery support and a more sensible argument order! We adapted this excellent piece by Dmitri Fontaine and turned it into a function. It can be invoked like this: SELECT * FROM histogram($table_name_
Read more
How to preview PostGIS maps on your command line
This is part of our practical cartography and PostgreSQL series. Put a map on it! Sometimes it's a pain to open up QGIS and load a PostGIS-enabled DB. Sometimes I don't feel like writing a custom tileserver and hooking it up to Leaflet or Mapbox GL just so I can
Read more
Mountains of Census geodata for all
U.S. Census data gives our modeling a good predictive boost, and it's a robust quality assurance tool for all the third-party data we've got flowing through our wires. The Census offers its geographic data in easy-to get, familiar formats via the TIGER portal, but distribution is split up by
Read moreThe big picture: exportable maps
Today we're introducing exportable audience maps . . . Do you use maps in your Faraday workflow? Prints? Presentations? Twitter? Let us know how these maps can help, and how we can improve them. Getting a map image from the platform is now simple: From any saved audience, just click the "Export&
Read more
Getting bite-sized chunks of OpenStreetmap
#post-getting-bite-sized-bits-of-openstreetmap { background-image: url(/blog/content/images/2016/03/roads.svg); background-repeat: no-repeat; background-size: 100%; background-position: 50% 0; } At Faraday, we dig OSM. OpenStreetmap (OSM) is the foundation of our basemap and a model of the power of open data. It guides customers on our platform to their ideal audiences . . . . . . and
Read more