Pipelines

Pipelines overview

Pipelines (referred to as scopes in the API) are how you choose which predictions you want to deploy, who you want to make those predictions on, and where they should go. When building one, you'll start with the population. Here, you'll choose which of your cohorts should be the population to include, or who you want to make predictions on. You can also optionally choose a population to exclude if you'd like to prevent a cohort from having predictions made.

Next, in the payload, you'll define the predictions you'd like to make:

• Predictions: Score the population with outcomes, discover their persona sets to enable rich insights & personalization, and use recommenders to serve next-best offers.

  • Available via the Dashboard or the API with the outcome_ids, persona_set_ids, and recommender_ids parameters in createscope or updatescope

• Cohort membership: Membership indicators can help you determine if members of the pipeline's population are also a member of a specific cohort you've created.

  • Available via the Dashboard or the API with the cohort_ids parameter in createscope or updatescope.

• Traits: Adding traits to the payload will display trait values for individuals in the population, which can be useful to add additional context when analyzing and segmenting individuals.

  • Available via the Dashboard or the API with the attributes parameter in createscope or updatescope.

• Prediction explanations: Prediction explanations can be enabled to provide explanations for why any individual prediction was given. For documentation on a deployment's explanation output, see deployments.

  • Available via the Dashboard or the API with explainability parameter in createscope or updatescope.

After your pipeline is created, you'll deploy your predictions by creating a deployment (referred to as a target in the API) so that your predictions can be delivered to any connection you've made to a database, data warehouse, cloud bucket, a Faraday-managed connection, or to a CSV file.

For pipeline creation instructions using both the Dashboard UI and API, see our how-to docs for use cases.

Additional pipeline information

Preview mode

All pipelines begin in preview mode, which includes a limited number of rows. This can be used to test if the output is in the desired format. To deploy all rows, simply turn off preview mode once you’ve created a deployment in the pipeline.

Pipeline diagnostics

Each pipeline includes a diagnostics section that runs checks to help ensure that your pipelines are properly configured. These checks will look at both the population and payload of a pipeline, then compare them with the eligible population used in the predictions (outcomes or recommenders) themselves.

For a pipeline diagnostic to look normal, we’d want to see the two curves have similar shapes, which indicates that the data used in both the outcome and in the pipeline are similar. You wouldn’t create an outcome to predict how likely everyone in the state of Vermont is to convert, and then use that model to score a population from Texas, right? Similarly, you wouldn’t want to use data from 2020 to create an outcome, then use that outcome in a pipeline to predict how likely to convert your 2024 new subscribers are.

If you’re finding that a pipeline shows two curves that are dissimilar, it’s likely there’s a discrepancy between the people used to build the prediction and the people you’re trying to make predictions on. Take a look at the populations for each resource, and consider refreshing the appropriate resource.

Membership indicators

Membership indicators, or including cohorts in your payload, are useful for segmentation. For example, say you want to know who in your customer base has an income of greater than $100k. Your population to include for your pipeline would be your Customers cohort, and as part of your payload, you could then select a cohort of customers with an income of greater than $100k. As a result, anyone who is indicated as being in that $100k or greater cohort in your pipeline is a current customer with more than $100k in income.

Eligibility restrictions

If the outcome you select in your payload specified an eligibility cohort, your pipeline is not restricted by that same eligibility cohort. For example, if your outcome's eligibility cohort is Leads (e.g. in a lead scoring outcome), and your pipeline's population to include is Everyone, then everyone will be scored–not just your leads.

Deleting or archiving a pipeline

Before deleting a pipeline, ensure that any deployments have first been deleted. Once there are no deployments using the persona set, you can safely delete it.

• Dashboard: click the ... menu on the far right of the pipeline you'd like to delete or archive (or upper right while viewing a specific one), then click the relevant option.
• API: use the delete scope or archive scope API request.