$customHeader
Skip to end of banner
Go to start of banner

Create Cache Pipelines (Private Beta)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 21 Current »

In this article

Overview

Cache Pipelines enable you to run pipelines that reference information from a main pipeline.

Prerequisites

  • Cache pipelines must have the following input and output views:

    • 0 unconnected input views (Binary or Document)

    • 1 unconnected Document output views

    • 0 unconnected Binary output views

  • The output document must have a flattened document structure. None of the values for primary keys can have objects or arrays as part of their structure.

If a document contains any value that is an array or object, it fails to process, which causes the pipeline to fail. If a cache pipeline fails to run, it will result in the preparation of the main pipeline failing (because it specifies the cache pipelines).

Design the Main Pipeline

Main pipelines reference the information in the cache pipeline. You can reference multiple main pipelines.

  1. In the Designer tab:

    1. Click your user name in the top banner.

    2. In the dropdown list, click User Settings.

    3. In the Settings dialog, under Preview Document Count, set the number of records (1, 2, 10. 50,100, 500, 1000, 11500, 2000). This value is the same for the number of records allowed for the expression language lookup during validation.

  2. Create or import a pipeline onto the Designer canvas.

  3. Click (blue star) to open the Edit Pipeline dialog.

  4. Fill out the form:

    1. Label. Enter a name for your main pipeline. We recommend that you choose a label that’s easily recognizable as the pipeline.

    2. Project. Confirm that the pipeline is the correct project. If you need to change the Project, refer to Moving a Pipeline.

    3. Resumable Mode. Leave deselected.

    4. Error Pipeline. Leave blank.

    5. Error Pipeline Parameters: Leave empty

    6. Parameters: Enter Pipeline Parameters

    7. Capture: Select to proliferate parameters downstream.

    8. Expression Libraries: Add libraries as needed.

  5. In Cache Pipelines properties, click (blue star) to enter the alias and the pipeline to run for the data stored in that alias. This is the first argument in the Cache Pipelines expression language functions as a reference.

  6. Click Save.

Build Cache Pipelines

  1. In Designer, create a new pipeline that will generate data to be referenced in the main pipeline.

  2. Use any Snaps you need to get the data to cache, ensuring the following:

    • The final output document needs to have a flattened structure.

    • The pipeline does not have an open input view.

  3. Optionally, end with a Mapper Snap to restrict the data in your cache or apply transformations for different needs as desired.

  4. Click (blue star) to run pipeline validation.

  5. Click (blue star) to open the Data Preview to verify the data that is stored in the cache.

  • No labels
$customFooter