...
Table of Contents | ||||
---|---|---|---|---|
|
Overview
The You can use the Script Snap executes a to execute Javascript, Python, or Ruby script scripts using the JVM ScriptEngine mechanism. Accounts are not required to work with this Snap.
Once you select the language of your choice, click the Edit Script button to open the script editor.
By default, the editor is populated with a basic script you can modify.
The basic script reads an input document, wraps it in a map, and writes the wrapper to the output view. If a script file is present in the SLDB, you can upload it to the Snap using the Script file property; this property also accepts pipeline parameters and upstream parameters.
Note |
---|
You cannot create external process (like the |
...
Snap Type
The Script Snap is a Write write-type Snap.
Support for Ultra Pipelines
...
Note the following two arguments in the output.write()
method:
The first argument is the input document -- doc.
The second argument is the data for the output document -- wrapper.
Both the arguments are required so that the lineage of the output document can be tied to the input document. This is important for an Ultra pipeline responding to web requests. The initial request becomes an input document to the first Snap in the SnapLogic pipeline, eventually resulting in an output document from the last Snap in the pipeline. The pipeline must maintain the lineage of each document so that each response can be correlated to the request that generated it.
...
For JavaScript, objects written to the output view should be composed of serializable Java types. This is required by some Some downstream Snaps, such as the Copy Snap, require this. To write out a map to the output view in a JavaScript script, use a Java Map implementation, such as HashMap
or LinkedHashMap,
per the following:
Paste code macro |
---|
importClass(java.util.LinkedHashMap); ... var inDoc = this.input.next(); var outDoc = new LinkedHashMap(); outDoc.put("original", inDoc); this.output.write(inDoc, outDoc); |
...
To implement Python in the Script Snap, we use Jython. We recently have upgraded the Jython engine from version 2.7-b3 (a beta version from 2014) to the current version , 2.7.2(March , 2020). The following are the resultant issues and workarounds that we are suggesting:
There's an An open bug in 2.7 that introduced a backward-incompatible change in the SnapLogic platform wherein the Jython engine automatically converts BigInteger values to primitive long values. This impacts all your scripts that perform numeric manipulation of integer values from documents (SnapLogic uses the BigInteger type to represent integers in documents). Your Pipelines pipelines and Snaps with the Script Snap (or the deprecated Execute Script Snap) that use numeric manipulation scripts with integer or BigInteger data type may fail during execution. We recommend you to prospectively replace integer or BigInteger values with long values.
Example:
sum = a.intValue() + b.intValue()
Here Here,a
andb
are of BigInteger type that now fail as Jython 2.7.2 automatically and transparently callslongValue()
on any BigInteger value it encounters. Soa
andb
would need to use the long and not BigInteger type.The known fix is to rewrite the above calculation as
sum = a + b
by removing occurrences of.intValue()
or.longValue()
from your Python scripts.Before the 4.22 release (August 2020), when using the Script Snap with the Scripting language option selected as Python, requesting a key that did not exist in a dictionary (for example,
my_dict['missing_key']
) would returnNone
. Starting from the 4.22 release, the same request now returns aKeyError
exception. If you need to continue returningNone
, use the .get(key) method instead (for example,my_dict.get['missing_key']
).- zlib.compress()
: The zlib library compresses the JSON files retrieved from the SnapLogic APIs and backs-up Pipelines and accounts to a database. The following Python code, when trying to compress displays anascii … ordinal not in range(128)
error.
Original code:in_doc["json"] = zlib.compress(in_doc["json"])
Fix: in_doc["json"]= zlib.compress(in_doc["json"].encode("utf-8"))
{dictionary}.values().toArray()[i]:
Prior to Before the 4.22 release (August 2020), to subscript a{dictionary}.values()
method, you had to append thetoArray()
method tovalues()
;e
lse, you would see theFailure: ‘java.util.LinkedHashMap$LinkedValues’ object is unsubscriptable
error. After the 4.22 release,toArray()
returns Failure: ‘list’ object has no attribute ‘toArray’. However, the requirement fortoArray()
is no longer necessary for the subscript.
Original code:sLine = data.values().toArray()[0]
Fix: sLine =data.values()[0]
Additional Information
The document data can convert to and from the JSON data interchange language. By convention, the root of every document is conceptually a JSON object -- a collection of name/value pairs, where each name is a string and each value is an object, an array, a string, a number, a boolean, or a null. Every modern programming language has a corresponding type for this concept:
Java | Map |
---|---|
Python | Dictionary |
Ruby | Hash |
JavaScript | Object |
When you are writing a script for the Script Snap, each input document is an object that implements the Java Map interface, and can be accessed as an instance of the scripting language’s native object class; for example, as a Python dictionary.
To write an output document, your script must create a new object. In Python or Ruby, you can create an instance of the required language’s native object type, a Python dictionary or a Ruby hash. The values you add to these objects must be one of the JSON-compatible types including objects, arrays, strings, numbers, booleans. For an array, you can use the corresponding array or list type of the language. Objects written to the output view should be of Java types. This is required by some downstream Snaps. For example, the Join Snap. To write a Python map to the output view in a Python script, convert the map to a Java HashMap
General Instructions for All Scripting Languages
The script author should declare a global variable named 'hook' (note that this variable name is case sensitive). The Script engine makes the following four global variables available to the script as defined in the Script#ScriptHook Interface section:
Variable input is of type
ScriptHook.Input
Variable output is of type
ScriptHook.Output
Variable error is of type
ScriptHook.Error
Variable log is of type
org.slf4j.Logger
Type defined in the schema maps to the Java class per the following:
...
Snap Views
...
Type
...
Format
...
Number of Views
...
Examples of Upstream and Downstream Snaps
...
Description
...
Input
...
Document
...
Min: 1
Max: 1
...
Mapper
...
This Snap has at most one document input view.
...
Output
...
Document
...
Min: 1
Max: 1
...
Mapper
Copy
...
This Snap has at most one document output view.
...
Error
...
Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Stop pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.
Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.
Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.
Learn more about Error handling in Pipelines.
Snap Settings
Info |
---|
|
...
Field
...
Description
Label*
Default Value: Script
Example: Script
...
Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.
...
Scripting Language:
...
Required. Language in which the script is provided. The options available include:
Javascript
Python
Ruby
Example: Javascript
Default value: Javascript
...
Script file:
A script file that implements the ScriptHook interface. This field can be used if the script file is present in the SLDB. Click on the 'browse' icon to select the required script file from the SLDB.
Example: transform.py
Default value: [None]
Note |
---|
This field accepts Pipeline parameters as well as upstream parameters provided that the script file is present in the SLDB. |
...
Edit Script:
...
Required. This property enables you to edit a script within the Snap instead of through an external file. From this page, you can export the script to a file in a project, import a script, or generate a template for the selected Scripting Language.
Default value: A skeleton for the chosen scripting language. You can click the Generate Template button to regenerate the skeleton.
...
Multiexcerpt include macro | ||||
---|---|---|---|---|
|
...
Multiexcerpt include macro | ||||
---|---|---|---|---|
|
Note |
---|
You cannot create external process (like the |
ScriptHook Interface
Note |
---|
This example requires that an input view be defined for it to work. |
...
language | json |
---|
...
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document
|
|
| This Snap has at most one document input view. |
Output | Document
|
|
| This Snap has at most one document output view. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
Snap Settings
Info |
---|
|
Field Name | Field Type | Description | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Label* Default Value: Script | String | Specify a unique name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | ||||||||||||
Scripting Language* Default value: Javascript | Dropdown list | Choose a language for the script. The available options are:
| ||||||||||||
Script file Default value: None | String/Expression | Speciy or select a script file that implements the ScriptHook interface. This field can be used if the script file is present in the SLDB. Click on the Upload icon to upload the required script file from the SLDB. This field accepts pipeline parameters as well as upstream parameters provided that the script file is present in the SLDB. | ||||||||||||
Edit Script Default Value: A skeleton for the chosen scripting language. You can click the Generate Template link to regenerate the skeleton. | Button | Click the Edit Script button to edit a script within the Snap instead of through an external file. From this page, you can export the script to a file in a project, import a script, or generate a template for the selected Scripting Language. | ||||||||||||
Default Value: Execute only | Dropdown list |
|
ScriptHook Interface
This example requires that an input view be defined for it to work.
Paste code macro | ||
---|---|---|
| ||
package com.snaplogic.scripting.language;
import java.util.Iterator;
/**
* ScriptHook is the interface that should be implemented as a callback mechanism for
* ExecuteScript snap to call into the script.
*/
public interface ScriptHook {
/**
* Scripts should implement this method to provide application logic.
*/
void execute();
/**
* Scripts should implement this method to cleanup any resources allocated in execute().
*/
void cleanup();
/**
* Input is interface that is used by the script to read input from the snap's input view.
*/
interface Input extends Iterator<Object> {
}
/**
* Output is interface that is used by the script to send output data to the snap output view.
*/
interface Output {
/**
* Write the data to the snap output.
*
* @param data
*/
void write(Object data);
/**
* Write the data that was generated for the given incoming data to the snap output.
* This method carries the lineage data forward.
*
* @param incomingData
* @param data
*/
void write(Object incomingData, Object data);
}
/**
* Error is interface that is used by the script to send error data to snap error view.
*/
interface Error extends Output {
}
} |
Importing Third-Party Libraries
While SnapLogic does not support importing third-party libraries directly using the Script Snap, you can add their package/JAR files in a directory in your Groundplex nodes and then import them using this Snap. For example, consider that you have added the JAR file, mongo-java-driver-3.12.7.jar
, in the directory /opt/snaplogic/ext_jar/
. For your Python scripts to be able to use this library, create a file named .jython
in the home directory of the user running the JCC process. The .jython
file should specify a value for the python.path
, as follows:
python.path=/opt/snaplogic/ext_jar/mongo-java-driver-3.12.7.jar
You can find the user’s home directory (user running the jcc) in the jcc filename “jcc_output.log
" when you search with user.home
. If you have multiple jar files, you can add all the paths in the same .jython file separated by colon, as shown below:
python.path=jar1_path:jar2_path:jar3_path
Here’s an example of Python script that imports and uses code from this library. Use the cleanup
method to ensure that the mongoClient
object is appropriately closed.
Code Block |
---|
from com.snaplogic.scripting.language import ScriptHook from com.mongodb.client import MongoClients class TransformScript(ScriptHook): def __init__(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log def execute(self): try: self.mongoClient = MongoClients.create("mongodb://localhost:27017/?readPreference=primary&ssl=false") |
...
for d |
...
in self.mongoClient.listDatabases(): |
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
self.output.write(d) except |
...
Exception as e: errDoc = { |
...
|
...
|
...
|
...
'error' : str(e.args) |
...
|
...
|
...
} |
...
|
...
self.error.write( |
...
errDoc) |
...
|
...
|
...
def cleanup(self): |
...
|
...
self.mongoClient.close() hook = |
...
Importing Third Party Libraries
While SnapLogic does not support importing third party libraries directly using the Script Snap, you can add their package/JAR files in a directory in your Groundplex nodes and then import them using this Snap.
For example, consider that you have added the JAR file, mongo-java-driver-3.12.7.jar
, in the directory /opt/snaplogic/ext_jar/
. For your Python scripts to be able to use this library, create a file named .jython
in the home directory of the user running the JCC process. The .jython
file should specify a value for python.path
, as follows:
python.path=/opt/snaplogic/ext_jar/mongo-java-driver-3.12.7.jar
Here’s an example Python script that imports and uses code from this library. Note the use of the cleanup
method to ensure that the mongoClient
object is closed properly.
Code Block |
---|
from com.snaplogic.scripting.language import ScriptHook
from com.mongodb.client import MongoClients
class TransformScript(ScriptHook):
def __init__(self, input, output, error, log):
self.input = input
self.output = output
self.error = error
self.log = log
def execute(self):
try:
self.mongoClient = MongoClients.create("mongodb://localhost:27017/?readPreference=primary&ssl=false")
for d in self.mongoClient.listDatabases():
self.output.write(d)
except Exception as e:
errDoc = {
'error' : str(e.args)
}
self.error.write(errDoc)
def cleanup(self):
self.mongoClient.close()
hook = TransformScript(input, output, error, log) |
Note |
---|
The paths listed in The If you are using multiple Groundplex nodes then you must add the package/JAR files in each of those nodes. TransformScript(input, output, error, log) |
Note |
---|
|
Additional Information
The document data can be converted to and from the JSON data interchange language. By convention, the root of every document is conceptually a JSON object—a collection of name-value pairs, where each name is a string, and each value is an object, an array, a string, a number, a boolean, or a null. Every modern programming language has a corresponding type for this concept:
Script | Type |
---|---|
Java | Map |
Python | Dictionary |
Ruby | Hash |
JavaScript | Object |
When writing a script for the Script Snap, each input document is an object that implements the Java Map interface and can be accessed as an instance of the scripting language’s native object class, such as a Python dictionary.
To write an output document, your script must create a new object. In Python or Ruby, you can create an instance of the required language’s native object type, a Python dictionary, or a Ruby hash. The values you add to these objects must be one of the JSON-compatible types, including objects, arrays, strings, numbers, and booleans. You can use the corresponding array or list type of the language for an array. Objects written to the output view should be of Java types. Some downstream Snaps require this, for example, the Join Snap. To write a Python map to the output view in a Python script, convert the map to a Java HashMap.
General Instructions for all Scripting Languages
The script author should declare a global variable named 'hook' (note that this variable name is case-sensitive). The Script engine makes the following four global variables available to the script as defined in the Script#ScriptHook Interface section:
The variable input is of type
ScriptHook.Input
The variable output is of type
ScriptHook.Output
The variable error is of type
ScriptHook.Error
The variable log is of type
org.slf4j.Logger
Type defined in the schema maps to the Java class per the following:
Data Type | Java class |
---|---|
NUMBER | java.math.BigDecimal |
INTEGER | java.math.BigInteger |
STRING | java.lang.String |
DATETIME | org.joda.time. DateTime |
LOCALDATETIME | org.joda.time. LocalDateTime |
BOOLEAN | java.lang.Boolean |
DATE | org.joda.time.LocalDate |
TIME | org.joda.time.LocalTime |
BINARY | java.lang.Byte |
TABLE | java.util.List |
ANY | java.lang.Object |
COMPOSITE | java.util.Map |
...
Example Scripts
JavaScript
Paste code macro |
---|
script = { execute : function() { var fruits = ["apple", "banana", "cherry"] var map = {"fruits" : fruits} output.write(map) } } var hook = new com.snaplogic.scripting.language.ScriptHook(script) |
Python
...
This example requires an input view be defined in order for it to work.
Paste code macro | ||
---|---|---|
| ||
from com.snaplogic.scripting.language import ScriptHook from com.snaplogic.scripting.language.ScriptHook import * class TransformScript(ScriptHook): def __init__(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log def execute(self): self.log.info("Executing Transform script") while self.input.hasNext(): data = self.input.next() data["firstLast"] = "%s-%s" %(data["first"],data["last"]) data["firstLast2"] = data["first"] + data["last"] data["numberMath"] = data["counter"] + 22 data["numberMath2"] = data["counter"] + 23 data["dateMonthPlusOne"] = data["birthday"].plusMonths(1) data["numberMathType"] = type(data["counter"]) data["dateType"] = type(data["birthday"]) try: data["mathTryCatch"] = data["counter2"] + 33 self.output.write(data) except Exception as e: data["errorMessage"] = e.message self.error.write(data) self.log.info("Finished executing the Transform script") hook = TransformScript(input, output, error, log) |
...
...
Script Snap Configuration Using Python, JS Script, and Ruby Script
The following pipeline is a demonstration of demonstrates the Script Snap using all three supported languages. In this This pipeline uses a simple JSON file with information like First Name, Last Name, and Birthday are passed to the Script Snaps.
The following is a snapshot of the input for the Script Snap:
...
Python Script
Script Snap uses the Jython engine to execute the scripts written in Python.
Paste code macro | ||
---|---|---|
| ||
from com.snaplogic.scripting.language import ScriptHook from random import randint from time import sleep class TransformScript(ScriptHook): def __init__(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log def execute(self): self.log.info("Executing Transform script") i = 1 while self.input.hasNext(): data = self.input.next() sleep(randint(1,10)) map = {"out": data} self.output.write(map) self.log.info("Finished executing the Transform script") hook = TransformScript(input, output, error, log) |
JavaScriptJava Script
The Script Snap uses the Nashorn engine to execute the scripts written in JavaScript.
...
Expand | ||
---|---|---|
| ||
|
The exported Pipeline pipeline is available in the Script#Downloads section below.
Script Snap Execution Using Pipeline
...
Parameters
The following example demonstrates the execution of the Script Snap also accepts using the pipeline parameters, ; the script file in the SLDB can be is passed as a Pipeline parameter. The pipeline demonstrated above is modified to accept pipeline parameters in this example by configuring the Script file property. For the scripts to be passed as a pipeline parameter, the script file should be present in the project folder in SnapLogic. Confirm in the Files section inside the Manager that that the script files are present, ; if they are not then , upload them by clicking on the '+' icon.
In this example, there are three files, one for each type of scripting language supported:
...
Below is a snapshot of the Pipelinepipeline's properties with the and configured pipeline parameters also configured.
The individual Script Snaps are configured as shown below:
...
...
As in the example above, this pipeline also produces the same output upon execution. The exported Pipeline pipeline and sample script files used are available as a zip file in the Downloads section below.
...