...
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document |
|
| Requires the Schema name and Table name to process a certain number of input documents into a prepared insert query and execute all of the prepared input document values on that query as a batch. |
Output | Document |
|
| Snap outputs the number of documents successfully inserted into the target table in the output view document. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about the behavior of Parallel Load Snap with open error views. Learn more about Error handling in Pipelines. |
Snap Settings
Info |
---|
|
Field Name | Field Type | Description | ||
---|---|---|---|---|
Label* Default Value: Oracle - Parallel Load | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | ||
Schema name Default Value: N/A | String/Expression | Specify the database schema name. You can pass the values as expressions using the pipeline parameters but cannot use values from the upstream Snap. | ||
Table name* Default Value: N/A | String/Expression | Specify the table on which to execute the load operation. You can pass the values as expressions using the pipeline parameters but cannot use values from the upstream Snap. | ||
Create table if not present Default Value: Deselected | Checkbox | Select this checkbox to automatically create the target table if it does not exist.
The newly created table is not visible to subsequent database Snaps during runtime validation because of implementation details. If you want to immediately use the newly updated data, you must use a child pipeline invoked through the Pipeline Execute Snap. | ||
Maximum thread count*
| Integer/Expression | Specify the maximum number of threads allowed to perform data insertions simultaneously. If the value is 0, the snap automatically sets the number of threads equal to the number of available processors on the system.
| ||
Batch capacity*
| Integer/Expression | Specify the maximum number of records allowed in a single batch for an insertion task.
| ||
Snap memory usage: To gracefully handle failed record insertions, the Snap must retain some input document data in memory during the Snap execution. To calculate the memory usage follow this formula: | ||||
Maximum error count*
| Integer/Expression | Specify the maximum number of record insertion errors allowed before execution stops on the next error. For example, if you specify a value of 10, the pipeline execution continues until the number of failed records exceeds 10. A value of -1 means no limit. At least one more error document than the specified maximum value is expected, but task concurrency may result in additional error documents. This applies only when the error behavior is set to Route Error Data to Error View. | ||
Truncate table before load
| Checkbox/Expression | Select this checkbox to truncate the target table before loading data. | ||
Snap Execution Default Value: | Dropdown list | Select one of the following three modes in which the Snap executes:
|
...
The following example pipeline demonstrates the use of Parallel Load to insert 1000 documents. It showcases Parallel Load's efficiency and capability in managing large-scale data insertion tasks.
Configure the Mapper Snap as followsSnap with subscriber details (Subscriber ID, Last Login, and Subscriber’s full name) to pass them to the Oracle - Parallel Load Snap.
...
Configure the Oracle - Parallel Load Snap with the following settings:
...
On validation, the pipeline inserts a maximum of 400 records simultaneously, across 4 threads handling 100 records each. The image below displays only 50 records because the Preview document count is set to 50.
...
Download this Pipelinepipeline.
Handle Database Insertion Failures with Open Error View
...
Inserted records | Insertion failed records |
---|---|
Download this Pipelinepipeline.
Downloads
Info |
---|
|
...
Expand | ||||||
---|---|---|---|---|---|---|
| ||||||
|
...
Related Content
...