Changelog

Incremental column support for datasets

Ben Rose
Ben Rose
on

In datasets console, you can now set an incremental column (e.g. updated_at, order_updated_at) on databases and data warehouses. This dramatically improves refresh efficiency by only processing rows that have changed since the last update.

When paired with a primary key, the refresh process works like this:

  1. Select rows from the source where incremental_column is newer than the last run.
  2. Merge (upsert) those rows into the dataset using primary_key.

This makes dataset refreshes faster and more efficient, especially for large tables.