List datasets

gethttps://api.faraday.ai/v1/datasets

Get a list of the datasets available in the developer’s account

Authentication

Bearer Authentication

Provide your API key in the Authorization header. You can find your API key in the Settings page of the dashboard.

Authorization: Bearer YOUR_TOKEN

Responses

200A list of datasets
Items of array[object]
archived_atstring<date-time>

If not null, this resource will no longer receive updates, but will still be visable.

connection_idstring<uuid>

If this is a "retrieve" dataset, the UUID of a connection - see /connections for more detail.

Only a subset of connection types...

Example: "5e0dfa56-2d52-4c06-a870-bc79c71e86a3"
created_atstring<date-time>required

When this resource was created.

detected_columnsarray[object]

An array of columns

Example: 
[
  {
    "data_type": "text",
    "is_nullable": false,
    "name": "id"
  },
  {
    "data_type": "long",
    "is_nullable": true,
    "name": "amount"
  }
]
Items of array[object]
data_typeone of 2 typesrequired

What type of data this field contains.

Properties of one of 2 types
One of:
stringenum:
"boolean""long""double""string""date"
fill_ratenumber

Count of non-null values divided by total count of values. 0.75 means 75% of the values are not null.

Example: 0.75
identifier_recommendationarray[string]

A list of recommended identity set keys to map this column to. Ordered by relevance.

is_nullablebooleanrequired
Example: true
namestringrequired

Name of the column found in the client data.

Example: "id"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
enrichmentdictionary[string, object]

A mapping of enrichment sources (like FIG) to their enrichment metadata

Values of dictionary[string, object]
Properties of each dictionary value:
anyintegerrequired

How many of the identities were enriched with either person-level or residence-level data

Example: 123
personintegerrequired

How many of the identities were enriched with person-level data

Example: 123
residenceintegerrequired

How many of the identities were enriched with residence-level data

Example: 123
idstring<uuid>required

A unique ID for this resource.

Example: "8cd2dcf6-f2b3-4318-b8b3-eb19ab18d29d"
identified_countinteger

The number of unique people identified in this dataset.
This can be different from the row_count, for example, in a table of orders.
The same person can order multiple things, so there are more rows t...

identity_providersarray[object]

Which identity providers to use for matching, in order of priority.
By default, all datasets will match on 'fig' data.
The dataset's match-rate can be boosted by adding other identity providers.
Please contact support to get access to this feature.

Example: [[{"provider":"fig"}]]
Items of array[object]
forceboolean

If true, then use this identity provider even if a match was found by another, higher priority provider.
Please contact support to gain access to this feature for any provider other than FIG.

providerstringrequired

Which data provider to use for matching.
By default, all Faraday accounts can match on the 'Faraday Identity Graph' (FIG) provider.
Please contact support to gain access to additional providers.

Allowed values: fig, match_boost
selectstring

If 'first_recognized', send to match-boost providers one-at-a-time until a match is found.
If 'all', send to all match-boost vendors. Please access support to gain access to this feature.

Allowed values: first_recognized, all
identity_setsdictionary[string, object]required

A mapping of {identity set name} (ex. shipping) -> {identity set object}.

Describes all the logical groupings of personally-identifiable information present in the dataset.

Identity set objects map...

Values of dictionary[string, object]
Properties of each dictionary value:
address_line_1stringdeprecated

Deprecated: use house_number_and_street instead

Example: "shipping_address_address1"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
address_line_2stringdeprecated

Deprecated: use house_number_and_street instead

Example: "shipping_address_address2"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
citystring
Example: "shipping_address_city"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
emailstring
Example: "email_address"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
email_hashstring

SHA-256 hash of the lowercase version of the email, with plus-addressing removed

Example: "email_hash"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
freeform_addressstring

House number and Street, City, State, and Zip all on one line if they are not available separately

Example: "shipping_address"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
house_number_and_streetarray[string]

Deprecated in favor of address_line_1 and address_line_2. If this field and either of those fields are populated, they will override this field's value when saved.

Example: ["shipping_address_address1","shipping_address_address2"]
person_first_namestring
Example: "shipping_address_first_name"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
person_full_namestring

Only provide this if you don't have first name and last name separately

Example: "shipping_address_full_name"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
person_last_namestring
Example: "shipping_address_last_name"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
phonestring
Example: "shipping_address_phone"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
postcodestring
Example: "shipping_address_zip"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
statestring
Example: "shipping_address_state"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
incremental_columnstring

A column specifying a date associated with a record.

Ideally incremental_column SHOULD be set to make data loading more efficient.

Ideally ALSO set upsert_columns to ensure that data is not dupl...

Example: "updated_at"
last_read_input_atstring<date-time>

The last time this resource's input was read.

last_updated_config_atstring<date-time>

The last time this resource's configuration was updated. If this is more recent than last_updated_output_at, the resource will be rebuilt.

last_updated_output_atstring<date-time>

The last time this resource successfully built.

managedboolean

A managed dataset requires special configuration from a Faraday admin, and is read-only.

Example: true
matched_countinteger

Deprecated: Use enrichment instead.

The number of identified people in this dataset that Faraday found a match for in its data.
This will only be displayed if the dataset built successfully.

merge_datasetsarray[object]

List of merge datasets using this dataset as a source.

Items of array[object]
dataset_idstring<uuid>required

The ID of the parent merge dataset.

Example: "5e0dfa56-2d52-4c06-a870-bc79c71e86a3"
join_columnstringrequired

The column from this dataset used to join with the parent merge dataset.

Example: "id"
namestringrequired

An identifying name for this dataset.

optionsdiscriminated type (type)required

Dataset connection options

Properties of discriminated type (type)
type
type#stringrequireddiscriminator
Value: "aws_aurora_mysql"
table_namestringrequired

Table name

Match pattern: ^[_a-zA-Z][a-zA-Z0-9_]+$
output_all_columns_as_traitsobject

If specified, all columns that are not excluded will be output as traits.

output_to_streams may not be not be specified when setting this parameter.

Properties of object
excludearray[string]required

A list of column names to exclude from bulk trait generation.

Example: ["id"]
includearray[string]

A list of column names that should only be used to generate traits.\

If unspecified, all columns will be considered and output_to_traits
must be unset. If specified, only these columns will be cons...

Example: ["id"]
output_to_streamsdictionary[string, object]

Describes how to transform the dataset into one or more streams.

Streams typically represent events. They can have multiple dataset sources and each dataset can be used to populate multiple streams....

Values of dictionary[string, object]
Properties of each dictionary value:
classicboolean

A classic dataset stream, created internally at Faraday.

conditionsarray[object]

List of conditions to use to filter events.

Items of array[object]
_eqstring

Equal to

_gtnumber

Greater than

_gtenumber

Greater than or equal to

_inarray[string]

Value is one of

_ltnumber

Less than

_ltenumber

Less than or equal to

_matchesstring

Value contains a match to the regex (re2) expression provided. For an exact regex match, use the ^ and $ characters as specified by the (re2 documentation)[https://github.com/google/re2/wiki/Syntax].

_neqstring

Not equal to

_ninarray[string]

Value is not one of

_nnullboolean

Value is not null

_nullboolean

Value is null (nulls are otherwise excluded)

column_namestringrequired

The column in the dataset that should be filtered on

optionalboolean

Optional conditions are unioned together, when combined they define events that meet either condition. At least one optional condition must be satisfied.

data_mapdictionary[string, object]required

A mapping from dataset columns to what they mean. It is recommended to include a "datetime" column if possible, as it will improve the accuracy of models.

Values of dictionary[string, object]
Properties of each dictionary value:
column_namestringrequired

The name of a column in the dataset. If format is "static_date_iso8601", then the value provided in this field is used as a static value.

Example: "skus"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
formatstring

Additional context for the column's data that isn't captured by its data type. For example, a 'revenue' column's data type would likely be 'int64', but format specifies if this number represents 'doll...

Allowed values: currency_cents, currency_dollars, mm_dd_yy_slash, mm_dd_yyyy_slash, mm_dd_yy_dash, mm_dd_yyyy_dash, yyyy_mm_dd_slash, yy_mm_dd_slash, yyyy_mm_dd_dash, yy_mm_dd_dash, yyyymmdd, yyyymm, yyyy_mm_dash, yyyy_mm_slash, dd_mm_yyyy_slash, dd_mm_yy_slash, dd_mm_yyyy_dash, dd_mm_yy_dash, date_iso8601, date_month_day_fullyear, date_month_day_shortyear, date_month_day_fullyear_hours_minutes, date_month_day_fullyear_hours_minutes_seconds, date_seconds_since_epoch_utc, date_milliseconds_since_epoch_utc, list_comma_separated, list_semicolon_separated, list_single_value, static_date_iso8601
stream_idstring<uuid>required
Example: "eefb0735-6ad6-4611-a832-40bab2968353"
output_to_streams_arrayarray[object]

An array-based approach to transforming datasets into streams. This structure allows multiple columns from the same dataset to map to the same stream, each with their own property configurations.

Unl...

Items of array[object]
propertiesdictionary[string, object]required

A flexible mapping of property names to their configurations. Property names can be any string (e.g., data, datetime, authority, precision, derivations, quality).

Each property can either reference a...

Values of dictionary[string, object]
Properties of each dictionary value:
column_namestring

The name of the column in the dataset to use for this property

Example: "age_in_years"
decodeobject

Configuration for decode transformations.

Properties of object
caststring

The data type to cast the transformed value to

Example: "string"
Allowed values: boolean, long, double, string, date
mapobject

A mapping of source values to target values

Example: {"a":1,"b":2}
sqlstring

SQL expression for transformation

Example: "SELECT * FROM table"
recodeobject

Configuration for recode transformations.

Properties of object
mapobject

A mapping of source values to target values

Example: {"a":1,"b":2}
sqlstring

SQL expression for transformation

Example: "SELECT * FROM table"
valueone of 3 types

A static value for this property (alternative to column_name)

Properties of one of 3 types
One of:
string
stream_idstring<uuid>
Example: "eefb0735-6ad6-4611-a832-40bab2968353"
stream_namestringrequired

The name of the stream to populate with this data

Example: "attribute_assertion_figv2_age"
output_to_traitsdictionary[string, object]

A mapping of trait name to trait definition, where the key is what the trait will be called in Faraday's system.

Traits are characteristics about people, that are unrelated to particular events.

Whe...

Values of dictionary[string, object]
Properties of each dictionary value:
column_namestring

The name of a column in the dataset. Required if no value is used.

Example: "skus"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
formatstring

Additional context for the column's data that isn't captured by its data type. For example, a 'revenue' column's data type would likely be 'int64', but format specifies if this number represents 'doll...

Allowed values: currency_cents, currency_dollars, mm_dd_yy_slash, mm_dd_yyyy_slash, mm_dd_yy_dash, mm_dd_yyyy_dash, yyyy_mm_dd_slash, yy_mm_dd_slash, yyyy_mm_dd_dash, yy_mm_dd_dash, yyyymmdd, yyyymm, yyyy_mm_dash, yyyy_mm_slash, dd_mm_yyyy_slash, dd_mm_yy_slash, dd_mm_yyyy_dash, dd_mm_yy_dash, date_iso8601, date_month_day_fullyear, date_month_day_shortyear, date_month_day_fullyear_hours_minutes, date_month_day_fullyear_hours_minutes_seconds, date_seconds_since_epoch_utc, date_milliseconds_since_epoch_utc, list_comma_separated, list_semicolon_separated, list_single_value, static_date_iso8601
null_valuesarray[string]

A list of values that should be treated as null.

transformation_tabledictionary[string, object]

A mapping of values as they appear in the dataset to how they should be used in models, where the key is what appears in the dataset and value is how it should be interpretted. This is used when multi...

valueone of 3 types

A static value for this trait. Required if no column_name is used.

Properties of one of 3 types
One of:
boolean
previewboolean

A dataset in preview mode will only detect columns and produce a data preview, but not ingest the data.

Defaults to undefined, which is equivalent to false.

Example: true
privacystring

Currently supported:

  • 'suppress' - data can be used for modeling but will be excluded from pipelines and deployments (do not contact)
  • 'delete' - data can not be used for modeling and will be ex...
Example: "suppress"
Allowed values: suppress, delete
reference_key_columnstring

Deprecated: use reference_key_columns instead
The name of the column that references an ID from an external system.

Setting this enables export of data via /targets that is keyed on this field.

Example: "customer_id"
Match pattern: ^[_a-zA-Z0-9][_a-zA-Z0-9 :/-]*$
reference_key_columnsarray[string]

The names of columns that reference IDs from an external system.

Setting this enables export of data via /targets that is keyed on this field.

Example: ["customer_id"]
resource_typestringrequired

The type of this resource.

Example: "datasets"
row_countinteger

The total number of rows in this dataset.
This will only be displayed if the dataset built successfully.

Example: 10000
sampleobject

If supported by the connection, a sample of the data.

statusstringrequired

The current state of this resource and any updates.

Example: "pending"
Allowed values: new, starting, running, ready, error
status_changed_atstring<date-time>

When the status of this resource was last updated.

status_errorstring

If this resource has status == "error", this will contain an error message.

updated_atstring<date-time>required

When this resource was last updated.

updatesarray[object]

A list of updates including how many rows were added.

If the dataset updates incrementally, these rows are added to the previous total. If the dataset is overwritten upon every ingestion, then these rows will be the new total row count.

Example: 
[
  {
    "datetime": "2021-10-05T14:48:00.000Z",
    "rows_added": 123
  },
  {
    "datetime": "2021-10-06T14:48:00.000Z",
    "rows_added": 32
  }
]
Items of array[object]
datetimestring<date-time>required

Time of the update

Example: "2021-10-05T14:48:00.000Z"
rows_addedintegerrequired

How many rows were added by the update

Example: 123
upsert_columnsarray[string]

Also known as the "primary key" of the dataset. A column or set of columns that uniquely identify an input row.

Ideally upsert_columns SHOULD be set so that data is not duplicated in the dataset.
...

Example: ["id"]
400The request was invalid.
errorstringrequired

A Faraday error code.

Some possible values include:

Generic HTTP Errors

  • BAD_REQUEST: The request could not be validated.
  • FORBIDDEN: You do not have permission to access the specified resour...
Example: "ERROR_TYPE"
Allowed values: BAD_REQUEST, FORBIDDEN, MAX_RESOURCES_REACHED, INTERNAL_SERVER_ERROR, INVALID_AUTHORIZATION, NOT_FOUND, MALFORMED_API_KEY, MISSING_API_KEY, EXPIRED_API_KEY, VALIDATION_FAILED, CONFLICT
idstring<uuid>required

A unique ID for this error. Please include this in bug reports.

Example: "082f9513-901c-4308-8081-902a8fe22d7e"
notestringrequired

A human-readable description of the error.

Example: "An error occurred"
validationErrorsarray[object]

JSON Schema validation errors, if any.

Items of array[object]
contextobjectrequired

More information about the error.

Properties of object
errorTypestringrequired

The type of validation error which occurred.

messagestringrequired

A human-readable error message.

pathstringrequired

The location in the document that failed validation.

suggestionstring

A suggestion for fixing this error.

401No API key was supplied.
errorstringrequired

A Faraday error code.

Some possible values include:

Generic HTTP Errors

  • BAD_REQUEST: The request could not be validated.
  • FORBIDDEN: You do not have permission to access the specified resour...
Example: "ERROR_TYPE"
Allowed values: BAD_REQUEST, FORBIDDEN, MAX_RESOURCES_REACHED, INTERNAL_SERVER_ERROR, INVALID_AUTHORIZATION, NOT_FOUND, MALFORMED_API_KEY, MISSING_API_KEY, EXPIRED_API_KEY, VALIDATION_FAILED, CONFLICT
idstring<uuid>required

A unique ID for this error. Please include this in bug reports.

Example: "082f9513-901c-4308-8081-902a8fe22d7e"
notestringrequired

A human-readable description of the error.

Example: "An error occurred"
validationErrorsarray[object]

JSON Schema validation errors, if any.

Items of array[object]
contextobjectrequired

More information about the error.

Properties of object
errorTypestringrequired

The type of validation error which occurred.

messagestringrequired

A human-readable error message.

pathstringrequired

The location in the document that failed validation.

suggestionstring

A suggestion for fixing this error.

403Access to this resource was forbidden.
errorstringrequired

A Faraday error code.

Some possible values include:

Generic HTTP Errors

  • BAD_REQUEST: The request could not be validated.
  • FORBIDDEN: You do not have permission to access the specified resour...
Example: "ERROR_TYPE"
Allowed values: BAD_REQUEST, FORBIDDEN, MAX_RESOURCES_REACHED, INTERNAL_SERVER_ERROR, INVALID_AUTHORIZATION, NOT_FOUND, MALFORMED_API_KEY, MISSING_API_KEY, EXPIRED_API_KEY, VALIDATION_FAILED, CONFLICT
idstring<uuid>required

A unique ID for this error. Please include this in bug reports.

Example: "082f9513-901c-4308-8081-902a8fe22d7e"
notestringrequired

A human-readable description of the error.

Example: "An error occurred"
validationErrorsarray[object]

JSON Schema validation errors, if any.

Items of array[object]
contextobjectrequired

More information about the error.

Properties of object
errorTypestringrequired

The type of validation error which occurred.

messagestringrequired

A human-readable error message.

pathstringrequired

The location in the document that failed validation.

suggestionstring

A suggestion for fixing this error.

404The requested resource ID was not found.
errorstringrequired

A Faraday error code.

Some possible values include:

Generic HTTP Errors

  • BAD_REQUEST: The request could not be validated.
  • FORBIDDEN: You do not have permission to access the specified resour...
Example: "ERROR_TYPE"
Allowed values: BAD_REQUEST, FORBIDDEN, MAX_RESOURCES_REACHED, INTERNAL_SERVER_ERROR, INVALID_AUTHORIZATION, NOT_FOUND, MALFORMED_API_KEY, MISSING_API_KEY, EXPIRED_API_KEY, VALIDATION_FAILED, CONFLICT
idstring<uuid>required

A unique ID for this error. Please include this in bug reports.

Example: "082f9513-901c-4308-8081-902a8fe22d7e"
notestringrequired

A human-readable description of the error.

Example: "An error occurred"
validationErrorsarray[object]

JSON Schema validation errors, if any.

Items of array[object]
contextobjectrequired

More information about the error.

Properties of object
errorTypestringrequired

The type of validation error which occurred.

messagestringrequired

A human-readable error message.

pathstringrequired

The location in the document that failed validation.

suggestionstring

A suggestion for fixing this error.

500An internal server error occurred.
errorstringrequired

A Faraday error code.

Some possible values include:

Generic HTTP Errors

  • BAD_REQUEST: The request could not be validated.
  • FORBIDDEN: You do not have permission to access the specified resour...
Example: "ERROR_TYPE"
Allowed values: BAD_REQUEST, FORBIDDEN, MAX_RESOURCES_REACHED, INTERNAL_SERVER_ERROR, INVALID_AUTHORIZATION, NOT_FOUND, MALFORMED_API_KEY, MISSING_API_KEY, EXPIRED_API_KEY, VALIDATION_FAILED, CONFLICT
idstring<uuid>required

A unique ID for this error. Please include this in bug reports.

Example: "082f9513-901c-4308-8081-902a8fe22d7e"
notestringrequired

A human-readable description of the error.

Example: "An error occurred"
validationErrorsarray[object]

JSON Schema validation errors, if any.

Items of array[object]
contextobjectrequired

More information about the error.

Properties of object
errorTypestringrequired

The type of validation error which occurred.

messagestringrequired

A human-readable error message.

pathstringrequired

The location in the document that failed validation.

suggestionstring

A suggestion for fixing this error.

Tags

datasets