Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
minLevel1
maxLevel2
absoluteUrltrue

Overview

You can use this Snap to insert data in parallel parallelly across multi-threaded tasks to execute . Compared to the Oracle Bulk Load Snap, this Snap executes a high-performance data load.

  • The Oracle - Parallel Load Snap always auto-commits its database operations. The Auto commit setting either in the Oracle Thin Account or the Oracle Thin Dynamic Account configured for the Snap does not affect the Snap’s behavior.

  • This Snap supports Kerberos authentication for Oracle.

...

Snap Type

The <Snap name> Oracle - Parallel Load Snap is a Read/ Write/Parse/Transform/Flow/Format-type Snap that reads/fetches/writes/parses/executes/transforms/calls/creates…

Prerequisites

  • Valid client ID.

  • A valid account with the required permissions.

None.

Support for Ultra Pipelines

 

...

Works in Ultra Pipelines

...

Works in Ultra Pipelines if....

...

writes the specified data in the target table.

Prerequisites

Support for Ultra Pipelines  

Limitations

None.

Known Issues

NoneBoth the Schema name and the Table name settings accept schema from upstream Snap and offer suggestions for expressions. However, we recommend that you do not use those suggestions because they cause incorrect behavior.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

Binary

Binary or

Document

  • Min: 1

  • Max: 2

  • Mapper

  • Copy

  • ..

  • Requires the EDI data and internal ID as a JSON document

    Requires both the Schema name and the Table name to process a specific number of input documents into a prepared insert query and execute all of the prepared input document values on that query as a batch.

    Output

    Document

    Binary

    Binary or Document

    • Min: 0

    • Max: 1

    • ..

    • ..

    The EDI transaction ID and order confirmation
    • Mapper

    The Snap outputs the number of documents successfully inserted into the target table in the output.

    Error

    Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

    • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

    • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

    • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

    Learn more about the behavior of the Oracle - Parallel Load Snap with open error views.

    Snap Settings

    Info
    • Asterisk ( * ): Indicates a mandatory field.

    • Add icon ( (blue star) ): Indicates that you can add fields in the field set.

    • Remove icon ( (blue star)): Indicates that you can remove fields from the field set.

    • Expression icon ((blue star)): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

    • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

    • Upload icon ((blue star)): Indicates that you can upload files. Learn more about managing Files.

    Field Name

    Field Type

    Field Dependency

    Description

    Label*

    Default Value

    ELT Database Account

    Oracle - Parallel Load
    Example

    ELT RS Account

    Oracle - Parallel Load

    String

    None.

    The

    Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same

    Snap

    Snaps in your pipeline.

    Number of records

    Schema name

    Default Value:

     

    fieldset Name

    Specify advanced parameters that you want to include in the request.

    Field 1*

    Default Value<value> or None.
    Example<value>

    String

    Debug mode check box is not selected.

    Field 2

    Default Value<value> or None.
    Example<value>

    String

    None

     N/A
    Example: Sys

    String/Expression

    Sampling Type is Number of records.

    Enter the number of records to output.

    Specify the database schema name.

    You can pass the values as expressions using the pipeline parameters but cannot use values from the upstream Snap.

    Table name*

    Default Value: N/A
    Example: people

    String/Expression

    Specify the table on which to execute the load operation.

    You can pass the values as expressions using the pipeline parameters but cannot use values from the upstream Snap.

    Create table if not present

    Default Value: Deselected

    Checkbox

    Select this checkbox to automatically create the target table if it does not exist.

    • If a second input view is configured for the Snap and it contains a document with schema (metadata) from the source table, the Snap creates the new (target) table using the same schema (metadata). However, if the schema comes from a different database, the Snap might fail with the Unable to create table: "<table_name>" error because of data type incompatibility.

    • Without a second input view, the Snap creates a table based on the data types of the columns generated from the first row of the input document (first input view).

    The newly created table is not visible to subsequent database Snaps during runtime validation because of implementation details. If you want to immediately use the newly updated data, you must use a child pipeline invoked through the Pipeline Execute Snap.

    Maximum thread count*


    Default Value: 0
    Example: 20

    Integer/Expression

    Specify the maximum number of threads allowed to perform data insertions simultaneously. If the value is 0, the Snap automatically sets the number of threads equal to the number of available processors on the system.

    Note

    There is no upper limit for the Maximum thread count, therefore you should exercise caution when setting this value to avoid unexpected behavior or performance degrade.

    Batch capacity*


    Default Value: 1000
    Example: 500

    Integer/Expression

    Specify the maximum number of records allowed in a single batch for an insertion task.

    Note

    There is no upper limit for Batch capacity, therefore you should exercise caution when setting this value to avoid unexpected behavior or performance degrade.

    Snap memory usage: To gracefully handle failed record insertions, the Snap must retain some input document data in memory during the Snap execution. To calculate the memory usage follow this formula:
    Max thread count * Batch capacity * 2 * typical record size
    For example, with a Maximum thread count of 4, a Batch capacity of 10,000, and an average record size of 50 bytes, the input document data in Snap memory might peak at 4 million bytes.

    Maximum error count*


    Default Value50 
    Example:10

    Integer/Expression

    Specify the maximum number of record insertion errors allowed before execution stops on the next error. For example, if you specify a value of 10, the pipeline execution continues until the number of failed records exceeds 10.

    A value of -1 means no limit. At least one more error document than the specified maximum value is expected, but task concurrency may result in additional error documents. This applies only when the error behavior is set to Route Error Data to Error View.

    Truncate table before load


    Default Value: Deselected

    Checkbox/Expression

    Select this checkbox to truncate the target table before loading data.

    Snap Execution

    Default Value
    Example: Validate & Execute

    Dropdown list

    N/A

    Select one of the following three modes in which the Snap executes:

    • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

    • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

    • Disabled: Disables the Snap and all Snaps that are downstream from it.

    Troubleshooting

    ...

    Error

    Reason

    Resolution

    Interrupted while waiting for dataWriter

    dataWriter is still null

    "

    Pre-load truncation of the target table failed

    ";

    The

    Either the target table

    may

    might not exist

    , " + "

    or a database access error

    may

    might have occurred when trying to execute the

    " + "

    TRUNCATE statement

    ";

    .

    Verify your database credentials and ensure the target table exists, then

    try again

    retry.

    Unable to determine target table

    " + "

    column names

    ";

    Connection

    The connection used to retrieve table metadata

    threw

    displayed an exception

    while closing";

    Examples

    Excluding Fields from the Input Data Stream

    e can exclude the unrequired fields from the input data stream by omitting them in the Input schema fieldset. This example demonstrates how we can use the <Snap Name> to achieve this result:

    <screenshot of Pipeline/Snap and description>

    before closing.

    Verify the target table column names and retry.

    Behavior of Parallel Load Execution with Open Error Views

    The following table describes the expected behavior for the general outcomes of Parallel Load execution when the Snap has an open error view (that is, when you select Route Error Data to Error View for When error occurs).

    Committed Database Insertions

    Error View

    Output Preview

    Snap Outcome

    Final Snap State

    All insertions committed.

    N/A

    One document containing the
    <insert_count> field with an integer value indicating the number of successfully inserted records.

    All input records are successfully inserted into the target database.

    Success

    One error document for each record that failed to insert.

    One document containing the
    <insert_count> field with an integer value indicating the number of successfully inserted records.

    Some input records individually fail to insert into the target database, but do not exceed maximum error count.

    Success

    Insertions made successfully
    prior to the Maximum
    error count being
    exceeded may be
    committed.

    Multiple error documents, one for each record that failed to insert.
    The total number of error documents should be
    equal to at least the Maximum
    error count plus one, but might be more because of concurrency of attempted insertions.

    One document containing the
    <insert_count> field with an integer value indicating the number of inserted records successfully
    before exceeding maximum
    error count.

    Enough input records individually fail to insert to exceed the Maximum error count and cause the Snap to end execution early.

    Success

    Insertions made
    successfully prior to the
    Snap-failing error may be
    committed.

    One error document for
    each record that individually
    failed insertion (if any).

    One document containing
    <insert_count> field with an integer value indicating the number of inserted records successfully before the (Snap-failing) error.

    An error occurs beyond the scope of any individual record.

    Failure

    Individual record insertion failures occur when the database encounters an error during the insertion of a record. These failures are recoverable and specific to one input document. For example, an error might occur if the value of a document field is incompatible with the data type of the target column.

    Examples

    Load 1000 Document Insertions

    The following example pipeline demonstrates the use of Parallel Load to insert 1000 documents. This example showcases Parallel Load's efficiency and capability in managing large-scale data insertion tasks.

    ex-oracle-pl-pipeline.pngImage Added

    Configure the Mapper Snap with subscriber details (Subscriber ID, Last Login, and Subscriber’s full name) to pass them to the Oracle - Parallel Load Snap.

    ...

    Configure the Oracle - Parallel Load Snap with the following settings:

    • Maximum thread count: 4

    • Batch capacity: 100

    ...

    On validation, the pipeline inserts a maximum of 400 records simultaneously, across 4 threads handling 100 records each. The image below displays only 50 records because the Preview document count is set to 50.

    ...

    Download this pipeline

    Handle Database Insertion Failures with Open Error View

    This example demonstrates a Mapper Snap configuration designed to intentionally cause a small number of records to fail during database insertion. For every failed insertion, an error document is generated. Specifically, out of 1000 records, 7 are expected to fail, resulting in 7 error documents, and the remaining 993 records are successfully inserted. The count of successful insertions is also recorded in the output document. This pipeline demonstrates how to efficiently manage and log insertion errors.

    ...

    Configure the Mapper Snap with settings so that a small number of records fail during database insertion, and route the error documents to the error view.

    ...

    Configure the Oracle - Parallel Load Snap as follows to insert a total of 1000 documents.

    ...

    On validation, 993 records are successfully inserted and written to the output document, and 7 error documents are written on the error view. The image below displays only 47 records because the Preview document count is set to 50.

    Inserted records

    Insertion failed records

    ex-pl-output-inserted-records.pngImage Added

    ex-pl-error-output.pngImage Added

    Download this pipeline

    Downloads

    Info
    1. Download and import the pipeline into the SnapLogic application.

    2. Configure the Snap accounts, as applicable.

    3. Provide the pipeline parameters, as applicable.

    Attachments
    patterns*.slp, *.zip
    sortByname

    Snap Pack History

    ...

    Insert excerpt

    ...

    Oracle Snap Pack

    ...

    Oracle Snap Pack
    name

    Related Content

    ...

    Endpoint Doc Link 1

    ...

    Endpoint Doc Link 2

    Oracle SPH
    nopaneltrue

    ...

    Related Content