Overview

This article describes the procedure for migrating your existing REST pipelines and accounts to the HTTP Client Snap and accounts. Refer to this example pipeline for migrating the REST pipeline and account.

Prerequisites

• Valid access to the Org and Project space.

• Optional. Identify the number of REST pipelines and accounts to be migrated. You can use the Scan pipeline to scan the project paths where the REST pipelines are located.

• Save a version of the pipelines containing the REST Snaps. You can use the automated Version creation pipeline to create a version of the given pipeline, if the Action type is “update".

• If you need to update the existing pipelines, set the Action type to update in your migration pipeline configuration.

• If you need to create new pipelines and accounts, set the Action type to create in your migration pipeline configuration.

  • Also, create a new project and copy the assets, such as files, to the same project folder.

Understanding the migration tool

The migration tool includes the following pipelines:

1. Scan pipeline: This pipeline scans the specified project path for all the available REST pipelines and accounts and provides both lists.

2. Version creation pipeline: This pipeline automatically creates versions for all the REST pipelines in a given project. Hence, we recommend that you use this pipeline if you want to update your existing pipelines because it assists you in the migration process by creating backup versions of the affected pipelines.

3. Child pipelines: Includes the following pipelines for migrating REST accounts and pipelines:

   1. Account migration: REST to HTTP Client Account migration

   2. Pipeline migration: REST to HTTP Client Pipeline migration

4. Parent pipeline: This pipeline runs the child pipelines to migrate REST pipelines and accounts in bulk to the HTTP Client.

Scan pipeline

This pipeline scans the metadata information and identifies the REST pipelines/accounts and writes their path to the output.

Step 1—Use the Scan pipeline to identify the REST pipeline and account paths by scanning the specified Project path

Open

Input

1. In the JSON Generator Snap, provide the Project path where the REST pipelines and accounts are located.

2. Pass the path value in the Pipeline parameters.

Output

• The pipeline scans the project path, gets the paths of the REST pipelines and accounts, and writes them to a file in SLDB.

Download the Scan pipeline.

Version Creation pipeline

This pipeline scans all pipelines in a project, identifies those containing REST snaps, and creates new versions for them.

Step 2—Use the Version creation pipeline to automatically create a version of all the REST pipelines in a given project

Open

Input

1. Configure the following Pipeline parameters:

• path: /<org>/<project-space>/<folder>

• instance_url: https://cdn.elastic.snaplogic.com or https://cdn.emea.snaplogic.com

Output

• List Pipelines: Lists all the pipelines in a project.

• Filter: This excludes the pipelines which are having * in the pipeline path, as the Metadata snap fails to read that pipeline path

• Fetch pipelines: Reads the pipeline metadata.

• Pipeline scan: Scans through pipeline configurations to identify those containing REST Snaps.

• Create version: Makes an HTTP POST request to create a new version of the pipeline, as shown below:

  Open

  

Download the REST to HTTP Client Version creation pipeline.

Child pipelines

Step 3—Use the REST to HTTP Client Account migration pipeline and the REST to HTTP Client Pipeline migration pipeline

Accounts migration pipeline: This pipeline identifies the REST accounts and maps them to the compatible HTTP Client accounts. Learn more about the mapping of the accounts.

Open

Input:

1. Configure the following pipeline parameters:

2. Set the Asset path in the SnapLogic Read Snap to read the asset (account).

Output:

Using the Script (Mapping Metadata) Snap the pipeline converts the REST accounts to the compatible HTTP Client accounts in the destination folder.

Download the REST accounts migration pipeline.

REST pipeline migration pipeline: This pipeline identifies the REST Snaps in a given pipeline and maps their properties to those of the HTTP Client Snap. Learn more about mapping the Snap settings.

Open

Input:

1. Configure the following pipeline parameters:

2. Set the Asset path in the SnapLogic Read Snap to read the asset (pipeline).

Output:

• The pipeline converts the REST metadata to HTTP client Snaps using the Script (Update Metadata) Snap.

• Based on the create action type, the pipeline creates a new pipeline.

Download the REST pipelines migration pipeline.

Parent pipeline

This pipeline routes to pipeline flow or account flow based on the migration type specified in the JSON Generator Snap. It scans the project path for REST pipelines and accounts and sends them to the Pipeline Execute Snap, which executes the child pipeline.

Step 4—Use the parent pipeline to migrate the pipelines/accounts to HTTP Client

Open

Input:

1. In the JSON Generator Snap, provide the following JSON input:

• Source project path: Project to migrate.

• Action: create/update

• Type: account/pipeline

• Destination folder (is required for create Action type).

• Build tag: <Snap Pack version number>

[     

{        

  "projectPath": "/<Org>/<Project>/<folder>",   

    "type": "pipeline",        

   "descFolder": "/<Org>/<Project>/<folder>",   

      "action": "create",      

    "apisuiteBuildTag": "440patches30202"     

}

]

2. In the Pipeline Execute Snap, the Pool size is set to 10 to migrate pipelines in parallell.

Output:

The migrated pipelines are saved in Manager, and the errors are routed to the error view, and written to a file.

Download the parent pipeline.

Recommendations

Limitations

• The following fields are not available in the HTTP Client Snap:

  • Batch Size: To work with input documents in a batch using the HTTP Client, refer to this example.

  • Process array

  • Normalize URL: The HTTP Client normalizes the URL by default at the backend.

• There is no property available to give the upload file Name for both single-file and multi-file uploads; the HTTP Client Snap uses the uploaded file name internally.

• There is no property available for the Upload transfer request type, so by default, the HTTP Client Snap calculates content length internally.

Accounts:

• As part of the migration, we are copying the active REST account token to the HTTP Client token for all the token-based accounts. As long as the token is valid and the target endpoint contains the valid redirect URI (https://elastic.snaplogic.com/api/1/rest/admin/oauth2callback/apisuite), the accounts will work as expected.

• REST In-memory OAuth2 accounts will be migrated to HTTP Client OAuth2 account by considering Header authenticated=true and Auth endpoint=token endpoint URL to avoid validation errors.

• The script migrates, Dynamic OAuth2 accounts to Secure Header account in HTTP Client.

Limitations that were fixed as part of the Migration Pipeline 

• The HTTP Client Snap writes an output document after evaluating Pagination Properties. Therefore, when snap.out.totalCount is used, it is not evaluated correctly. We have addressed this limitation as part of the script, by updating the snap.out.totalCount to snap.out.totalCount+1 for pagination properties.

• For multipart-related file uploads, in the REST Post snap, when an entity is not provided, the Snap constructs the JSON body based on the filename of the selected file. This is automated in the HTTP Client Snap, where the script automatically generates a new row in HTTP Client snap with entity fileName in case of multipart related.

• The REST Snap outputs the response as part of the response object, but the structure of the response is different for the HTTP Client. So, this conversion is addressed in the upstream Snap.

• View names between REST and HTTP Client have changed. As part of the script, we are making sure that output and error views are connected properly.

• In REST Snaps, we encode the query params using +, but in HTTP Client, by default, we use %20. As part of the script, we enabled the Use form encoding for spaces checkbox in the HTTP Client to ensure the same behavior as REST Snaps.

Things to note

• When the Pagination URL is provided, the Page size is set to 0 in the HTTP Client Get request.

• When single file upload and multi-file upload are selected and the Upload body type is multipart form-data, two upload requests occur as part of the REST POST snap, but after migration, only one request is sent that includes both single file upload and multi-file upload.

• The header names differ between REST and HTTP Client Snaps. The REST Snap converts all header names to lowercase, whereas the HTTP Client Snap preserves the original case. As a result, after migrating from REST to HTTP Client Snap, any downstream Snaps that reference header values from the HTTP Client Snap fail because of these differences in header property formatting before and after the migration.

Troubleshooting

Header properties

The header properties depend on the endpoint. However, if the upstream Snaps in your REST pipelines use the following headers, use the corresponding HTTP Client Snap header in title case, as shown in the following table:

Example of migrating a REST pipeline to an HTTP client