Skip to end of banner
Go to start of banner

Script

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 50 Next »

On this Page

Snap type:Write
Description:

This Snap executes a Javascript, Python, or Ruby script using the JVM ScriptEngine mechanism. After selecting your language of choice, click the Edit Script button to open the script editor. By default, the editor will be populated with a skeleton script that you can modify. The skeleton simply reads an input document, wraps it in a map and writes the wrapper to the output view. If a script file is present in the SLDB, it can be uploaded to the Snap by using the Script file property; this property also accepts Pipeline parameters and upstream parameters. 

The document data can convert to and from the JSON data interchange language. By convention, the root of every document is conceptually a JSON object -- a collection of name/value pairs, where each name is a string and each value is an object, an array, a string, a number, a boolean, or a null. Every modern programming language has a corresponding type for this concept:

JavaMap
Python

Dictionary 

RubyHash
JavaScriptObject 

When you are writing a script for the Script Snap, each input document is an object that implements the Java Map interface, and can be accessed as an instance of the scripting language’s native object class; for example, as a Python dictionary.

To write an output document, your script must create a new object. In Python or Ruby, you can create an instance of the required language’s native object type, a Python dictionary or a Ruby hash. The values you add to these objects must be one of the JSON-compatible types including objects, arrays, strings, numbers, booleans. For an array, you can use the corresponding array or list type of the language. Objects written to the output view should be of Java types. This is required by some downstream Snaps. For example, the Join Snap. To write a Python map to the output view in a Python script, convert the map to a Java HashMap

General Instructions for All Scripting Languages

The script author should declare a global variable named 'hook' (note that this variable name is case sensitive). The Script engine makes the following four global variables available to the script as defined in the Script#ScriptHook Interface section:

  • Variable input is of type ScriptHook.Input
  • Variable output is of type ScriptHook.Output
  • Variable error is of type ScriptHook.Error
  • Variable log is of type org.slf4j.Logger


Type defined in the schema maps to the Java class per the following:

 Data TypeJava class
 NUMBER   java.math.BigDecimal
 INTEGER java.math.BigInteger 
 STRING java.lang.String
 DATETIME org.joda.time. DateTime  
 LOCALDATETIME org.joda.time. LocalDateTime 
 BOOLEAN java.lang.Boolean 
 DATE org.joda.time.LocalDate 
 TIME org.joda.time.LocalTime
 BINARY java.lang.Byte 
 TABLE java.util.List 
 ANY java.lang.Object 
 COMPOSITE java.util.Map 

Modes

  • Ultra Pipelines: Works in Ultra Pipelines if you pass the original document to the write() method for the output view. For example, in the JavaScript template, the body of the loop is per the following: 

    Note the two arguments to the output.write() method. The first argument is the input document -- doc. The second argument is the data for the output document -- wrapper. Both the arguments are needed so that the lineage of the output document can be tied to the input document. This is especially important for an Ultra pipeline responding to web requests. The initial request becomes an input document to the first Snap in the SnapLogic Pipeline, eventually resulting in an output document from the last Snap in the Pipeline. The Pipeline must maintain the lineage of each document so that each response can be correlated to the request that generated it.

Prerequisites:None.
Limitations and Known Issues

For JavaScript, objects written to the output view should be composed of serializable Java types. This is required by some downstream Snaps such as the Copy Snap. To write out a map to the output view in a JavaScript script, use a Java Map implementation, such as HashMap or LinkedHashMap per the following:

Breaking changes for Pipelines using the Script Snap (or the deprecated Execute Script Snap) with the Python engine

To implement Python in the Script Snap we use Jython.

We recently upgraded the Jython engine from version 2.7-b3 (a beta version from 2014) to the current version, 2.7.2 (March, 2020). The following are the resultant issues and workarounds that we are suggesting:

  • There's an open bug in 2.7 that introduced a backwards-incompatible change in the SnapLogic platform wherein the Jython engine automatically converts BigInteger values to primitive long values. This impacts all your scripts that perform numeric manipulation of integer values from documents (SnapLogic uses the BigInteger type to represent integers in documents). Your Pipelines and Snaps with the Script Snap (or the deprecated Execute Script Snap) that use numeric manipulation scripts with integer or BigInteger data type may fail during execution. We recommend you to prospectively replace integer or BigInteger values with long values.

    Example:
    sum = a.intValue() + b.intValue()
    Here a and b are of BigInteger type that now fail as Jython 2.7.2 automatically and transparently calls longValue() on any BigInteger value it encounters. So a and b would need to use the long and not BigInteger type.

    The known fix is to rewrite the above calculation as sum = a + b by removing occurrences of .intValue() or .longValue() from your Python scripts. 

  • Before the 4.22 release (August 2020), when using the Script Snap with the Scripting language option selected as Python, requesting a key that did not exist in a dictionary (for example, my_dict['missing_key']) would return None. Starting from the 4.22 release, the same request now returns a KeyError exception. If you need to continue returning None, use the .get(key) method instead (for example, my_dict.get['missing_key']).

  • - zlib.compress():
 The zlib library compresses the JSON files retrieved from the SnapLogic APIs and backs-up Pipelines and accounts to a database. The following Python code, when trying to compress displays an ascii … ordinal not in range(128) error.
    Original code: in_doc["json"] = zlib.compress(in_doc["json"])
    Fix: in_doc["json"] = zlib.compress(in_doc["json"].encode("utf-8"))

  • {dictionary}.values().toArray()[i]:
 Prior to the 4.22 release (August 2020), to subscript a {dictionary}.values() method, you had to append the toArray() method to values(); else, you would see the Failure: ‘java.util.LinkedHashMap$LinkedValues’ object is unsubscriptable error. After the 4.22 release, toArray() returns Failure: ‘list’ object has no attribute ‘toArray’. However, the requirement for toArray() is no longer necessary for the subscript.
    Original code: sLine = data.values().toArray()[0]
    Fix: sLine = data.values()[0]

Account:

Accounts are not used with this Snap.

Views:
InputThis Snap has at most one document input view.
Output

This Snap has at most one document output view. 

ErrorThis Snap has at most one document error view and produces zero or more documents in the view.

Settings

Label:

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Scripting Language:Required. Language in which the script is provided. The options available include:
  • Javascript
  • Python
  • Ruby

Example: Javascript
Default value: Javascript

Script file:

A script file that implements the ScriptHook interface. This field can be used if the script file is present in the SLDB. Click on the 'browse' icon to select the required script file from the SLDB. 

Example: transform.py

Default value: [None]

This field accepts Pipeline parameters as well as upstream parameters provided that the script file is present in the SLDB. 

Edit Script:

Required. This property enables you to edit a script within the Snap instead of through an external file. From this page, you can export the script to a file in a project, import a script, or generate a template for the selected Scripting Language.

Default value: A skeleton for the chosen scripting language.  You can click the Generate Template button to regenerate the skeleton.

Snap execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

You cannot create external process (like the popen function) on Cloudplex via the Script Snap or custom Snap. While external process creation on Groundplex is possible, this can be disabled upon your request to support@snaplogic.com.

ScriptHook Interface

This example requires that an input view be defined for it to work.

Importing Third Party Libraries

While SnapLogic does not support importing third party libraries directly using the Script Snap, you can add their package/JAR files in a directory in your Groundplex nodes and then import them using this Snap.

For example, consider that you have added a JAR file, sample.jar, in the directory /opt/snaplogic/ext_jar/. 

We recommend you to configure an external library that works with both the old and new Jython engines by using a .jython file in the home directory of the user running the JCC node. The .jython file should specify a value for the python.path per the following:
python.path=/opt/snaplogic/ext_jar/sample.jar

  • The paths listed in python.path can be .jar files (Java libraries), directories containing Python libraries (compatible with Python 2.7), or .zip files packaging those Python libraries. For more on using the .jython file, see Jython Registry.

  • The python.path variable is Jython's version of CPython’s PYTHONPATH variable. To learn more about the PYTHONPATH, see the official Python documentation.

  • If you are using multiple Groundplex nodes then you must add the package/JAR files in each of those nodes.

  • You can import third party libraries only on Groundplex nodes.

Example Scripts


JavaScript

Python

This example requires an input view be defined in order for it to work.

Example Use Case 


The following pipeline is a demonstration of the Script Snap using all three supported languages. In this pipeline a simple JSON file with information like First Name, Last Name, and Birthday are passed to the Script Snaps.

 

The following is a snapshot of the input for the Script Snap:


The following scripts are executed:

Python Script

Script Snap uses Jython engine to execute the scripts written in Python. 

JavaScript

The Script Snap uses the Nashorn engine to execute the scripts written in JavaScript.

Ruby Script

Script Snap uses JRuby engine to execute the scripts written in Ruby.

A sample preview output of the successful execution for all the three Snaps is shown below:

 Output from the Python script

 

 Output from the JS script

 Output from the Ruby script

  

The exported Pipeline is available in the Script#Downloads section below.

Script Execution Using Pipeline Parameter

The Script Snap also accepts pipeline parameters, the script file in the SLDB can be passed as a Pipeline parameter. The pipeline demonstrated above is modified to accept pipeline parameters in this example by configuring the Script file property. For the scripts to be passed as a pipeline parameter, the script file should be present in the project folder in SnapLogic. Confirm in the Files section inside the Manager that the script files are present, if they are not then upload them by clicking on the '+' icon.

In this example there are three files, one for each type of scripting language supported: 

Below is a snapshot of the Pipeline's properties with the pipeline parameters also configured.

 

The individual Script Snaps are configured as shown below:

 

 

 

As in the example above, this pipeline also produces the same output upon execution. The exported Pipeline and sample script files used are available as a zip file in the Downloads section below.

Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide Pipeline parameters as applicable.

  File Modified
You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account.
No files shared here yet.
  • Drag and drop to upload or browse for files
  •  

    See Also

    Snap Pack History

     Click to view/expand
    ReleaseSnap Pack VersionDateType Updates
    February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
    November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.

    August 2023

    main22460

     

    Stable

    Updated and certified against the current SnapLogic Platform release.

    May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
    February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
    November 2022main18944 StableUpgraded with the latest SnapLogic Platform release.
    August 2022main17386 StableUpgraded with the latest SnapLogic Platform release.

    4.29

    main15993

      

    Stable

    Upgraded with the latest SnapLogic Platform release.

    4.28main14627
     
    StableUpgraded with the latest SnapLogic Platform release.

    4.27

    main12833

     

    Stable

    Upgraded with the latest SnapLogic Platform release.
    4.26main11181
     
    StableUpgraded with the latest SnapLogic Platform release.
    4.25main9554
     
    StableUpgraded with the latest SnapLogic Platform release.
    4.24main8556
     
    StableUpgraded with the latest SnapLogic Platform release.
    4.23 Patch423patches7671
     
    Latest

    Fixes an issue with the PySpark Snap by removing the dependency on the json-path library, thus avoiding a conflict between the external library version and the SnapLogic json-path.jar.

    4.23main7430
     
    Stable

    Upgraded with the latest SnapLogic Platform release.

    4.22main6403
     
    Latest

    Upgraded the Jython engine from version 2.7-b3 (a beta version from 2014) to the current version, 2.7.2 (March, 2020). See the breaking changes note for Script Snap and the deprecated Execute Script Snap in the Limitations and Known Issues section for potential impacts of this upgrade.   

    4.21snapsmrc542-Stable

    Adds a new Cleanup method in the ScriptHook interface associated with the Script Snap. This method enables the Script Snap to automatically execute a clean-up once the configured script completes executing.

    4.20snapsmrc535-StableUpgraded with the latest SnapLogic Platform release.
    4.19snaprsmrc528-StableUpgraded with the latest SnapLogic Platform release.
    4.18snapsmrc523-Latest

    Enhanced the PySpark Snap to work in Kerberized Hadoop clusters with the addition of a Kerberos account.

    4.17 Patch ALL7402-Latest

    Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

    4.17snapsmrc515-Stable

    Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

    4.16 snapsmrc508-StableUpgraded with the latest SnapLogic Platform release.
    4.15snapsmrc500-StableUpgraded with the latest SnapLogic Platform release.

    4.14

    snapsmrc490

    -StableUpgraded with the latest SnapLogic Platform release.
    4.13snapsmrc486-StableUpgraded with the latest SnapLogic Platform release.

    4.12

    snapsmrc480

    -Stable

    Updated the PySpark Snap to enable users to stop a PySpark job in progress by pressing the STOP button.

    4.11

    snapsmrc465

    -Stable

    Added a new Snap, the PySpark Snap which is supported on the Groundplex on edge node and Hadooplex node.

    4.10

    snapsmrc414

    -StableUpgraded with the latest SnapLogic Platform release.

    4.9

    snapsmrc405

    -StableUpgraded with the latest SnapLogic Platform release.

    4.8

    snapsmrc398

    -StableUpgraded with the latest SnapLogic Platform release.

    4.7

    snapsmrc382

    -StableUpgraded with the latest SnapLogic Platform release.

    4.6.0

    snapsmrc344

    -StableUpgraded with the latest SnapLogic Platform release.

    4.4.1

    --Latest

    Updated default generated template to ensure compatibility with both JDK 7 and 8 JSR-223 Script Engines.

    March 2015

    --Stable

    Script Snap: Generate Template a link has been added within the Edit Script page to provide a template for each of the supported languages.

    January 2015

    --Stable

    Script Snap: Optional Execute during preview property added.

    December 20, 2014

    --Stable

    Documentation enhancement: updated script samples with a sample pipeline

    May 2014

    --Stable

    Script Snap was introduced in this release.

    Initial Release (June 2013)

    --Stable

    Execute Script introduced in this release.

     

    • No labels