...
You can use this Snap to insert data parallelly across multi-threaded tasks. Compared to the Oracle Bulk Load Snap, this Snap executes a high-performance data load.
The Oracle - Parallel Load Snap currently always auto-commits its database operations. The Auto commit setting either in the Oracle Thin Account or the Oracle Thin Dynamic Account configured for the Snap does not affect the Snap’s behavior.
This Snap supports Kerberos authentication for Oracle.
...
Limitations
None.
Known Issues
The Both the Schema name and the Table name settings both accept schema from upstream snaps Snap and offer suggestions for expressions; however. However, we recommend that you do not use those suggestions as because they cause incorrect behavior.
...
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document |
|
| Requires both the Schema name and the Table name to process a certain specific number of input documents into a prepared insert query and execute all of the prepared input document values on that query as a batch. |
Output | Document |
|
| The Snap outputs the number of documents successfully inserted into the target table in the output view document. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about the behavior of the Oracle - Parallel Load Snap with open error views. |
...
Field Name | Field Type | Description | ||
---|---|---|---|---|
Label* Default Value: Oracle - Parallel Load | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap Snaps in your pipeline. | ||
Schema name Default Value: N/A | String/Expression | Specify the database schema name. The suggestions in the Schema field are populated only when at least a single table exists in the schema. If no tables exist to use that schema, only SYS, SYSTEM, and XDB are populated.
| ||
Table name* Default Value: N/A | String/Expression | Specify the table on which to execute the load operation.that the rows will be inserted into. This list is populated based on the tables associated with the selected schema.
| ||
Create table if not present Default Value: Deselected | Checkbox | Select this checkbox to automatically create the target table if it does not exist.
The newly created table is not visible to subsequent database Snaps during runtime validation because of implementation details. If you want to immediately use the newly updated data, you must use a child pipeline invoked through the Pipeline Execute Snap. | ||
Maximum thread count*
| Integer/Expression | Specify the maximum number of threads allowed to perform data insertions simultaneously. If the value is 0, the snap Snap automatically sets the number of threads equal to the number of available processors on the system.
| ||
Batch capacity*
| Integer/Expression | Specify the maximum number of records allowed in a single batch for an insertion task.
| ||
Snap memory usage: To gracefully handle failed record insertions, the Snap must retain some input document data in memory during the Snap execution. To calculate the memory usage follow this formula: | ||||
Maximum error count*
| Integer/Expression | Specify the maximum number of record insertion errors allowed before execution stops on the next error. For example, if you specify a value of 10, the pipeline execution continues until the number of failed records exceeds 10. A value of -1 means no limit. At least one more error document than the specified maximum value is expected, but task concurrency may result in additional error documents. This applies only when the error behavior is set to Route Error Data to Error View. | ||
Truncate table before load
| Checkbox/Expression | Select this checkbox to truncate the target table before loading data. | ||
Snap Execution Default Value: | Dropdown list | Select one of the following three modes in which the Snap executes:
|
...
Error | Reason | Resolution |
---|---|---|
| The Either the target table may might not exist or a database access error may might have occurred when trying to execute the TRUNCATE statement. | Verify your database credentials and ensure the target table exists, then try againretry. |
| The connection used to retrieve table metadata displayed an exception while before closing. | Verify the target table column names and try againretry. |
Behavior of Parallel Load Execution with Open Error Views
The following table describes the expected behavior for the general outcomes of Parallel Load execution when the Snap has an open error view (that is, when you select Route Error Data to Error View for When error occurs).
Committed database insertionsDatabase Insertions | Error viewView | Output previewPreview | Snap outcomeOutcome | Final Snap stateState |
---|---|---|---|---|
All insertions committed. | N/A | One document containing the | All input records are successfully inserted to into the target database. | Success |
One error document for each record that failed to insert. | One document containing the | Some input records individually fail to insert into the target database, but do not exceed maximum error count. | Success | |
Insertions made successfully | Multiple error documents, one for each record that failed to insert. | One document containing the | Enough input records individually fail to insert to exceed the Maximum error count and cause the Snap to end execution early. | Success |
Insertions made | One error document for | One document containing | An error occurs beyond the scope of any individual record. | Failure |
Individual record insertion failures occur when the database encounters an error during the insertion of a record. These failures are recoverable and specific to one input document. For example, an error might occur if the value of a document field is incompatible with the data type of the target column. |
Examples
...
Load 1000 Document Insertions
The following example pipeline demonstrates the use of Parallel Load to insert 1000 documents. It This example showcases Parallel Load's efficiency and capability in managing large-scale data insertion tasks.
...
This example demonstrates a Mapper Snap configuration designed to intentionally cause a small number of records to fail during database insertion. For every failed insertion, an error document is generated. Specifically, out of 1000 records, 7 are expected to fail, resulting in 7 error documents, while and the remaining 993 records are successfully inserted. The count of successful insertions is also recorded in the output document. This pipeline demonstrates how to efficiently manage and log insertion errors efficiently.
...
Configure the Mapper Snap with settings so that a small number of records fail during database insertion, and route the error documents to the error view.
...
On validation, 993 records are successfully inserted and written to the output document, while and 7 error documents are written on the error view. The image below displays only 47 records because the Preview document count is set to 50.
...
Info |
---|
|
...