Versions Compared

Old Version 121

changes.mady.by.user Ankur Parekh

Saved on

compared with

New Version 122

changes.mady.by.user Vidya Patil

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
maxLevel2

Note
  • On Saturday, December 15, 2018, at 9 PM PT, all customer Snaplex nodes across Orgs are upgraded to 4.15 per the new SnapLogic Release Process. In case of any questions, contact SnapLogic Support.
  • This release also includes the patches made to SnapLogic after the 4.14 GA release, as listed in the 4.14 Dot Releases page. 

...

  • Introducing Data Catalog as a Service. The catalog service adds a Table asset type which can store data location and schema information for files in an external file system. In conjunction with the Catalog Insert and Catalog Query Snaps, this feature provides data integrators with the tools required to interconnect and analyze enterprise metadata. For eXtremeplex users, this feature includes using the Catalog Writer and Catalog Reader Snaps (see Big Data Snaps). You can gain visibility of data (Table and Column Structure) stored in the Data Lake (usually in an AWS S3 account) for building Capture/Ingest/Conform/Refine Pipelines in a single UI.

  • Enhanced support in Runtime Archive to enable access to logs of all pipeline executions for auditing purposes. Prior to this release, logs for only 100 child pipelines per parent pipeline were archived; this limitation has been removed.

  • Added High Availability configurations for Triggered Tasks. Users relying on running their pipelines via the Task method can now deploy Triggered Tasks to multiple Snaplexes using one of the Override URLs  to achieve fault tolerance in the event of a failure. 

  • Added the API Usage Detail feature to API Meters in Manager, enabling administrators to monitor API workflows in their Org to prevent exceeding usage. You can now track API usage both daily and concurrently both summarized in pie graphs, along with a table of the top ten API uses.

  • Enhanced the recent SnapLogic release process to enable admin users to automatically update their respective Orgs immediately when a new release is available. Starting with the SnapLogic 4.16 release (around mid-February, 2019), you will be able to configure their SnapLogic org to update automatically to 4.16 and later in the Manager tab. The current process of manually updating all Orgs 35 days after GA will continue. 

  • Enhanced the API download functionality by giving an option to download tasks (APIs) in .yaml or .json formats for viewing in Swagger.

  • Added two API parameters. For the GET /api/1/rest/public/snaplex/config/<path:snaplexpath> API call, you can set an expiration time in hours for how long the URL is valid. For the POST /api/1/rest/public/users API, the returned document now has the basic_user field, which displays the last user login. 

UI Enhancements

Manager

  • You can now extract enterprise metadata from AWS S3 and store them as Table Assets in Manager. Metadata is stored in Tables accessible via projects; alternatively, you can search under Assets in Manager. Once opened, you can view partition keys and schema of the table data.
  • API Meters in Manager enables you to monitor API usage. Clicking Usage Detail displays a table of the top ten most run API calls, as well as pie graphs summarizing the usage. Both daily and concurrent views are available. 
  • Set your version of SnapLogic to upgrade automatically to the next available release of SnapLogic via the Auto Update Check box. With 4.15, this feature enables an automatic update to the 4.16 release.
  • Enhanced the Activity Log UI, enabling you to view the number of log entries available, and view log entries in sets of 100.

...

  • Updated the Snaplex health icons in the Dashboard Health Wall page to the following:
    •  Up and running
    •  Alert
    •  Not running
    •  Error status for nodes

Snap Enhancements

Info
  • Pipelines that are already started before a Snap update will continue to run with the older version of the Snaps. Only pipelines started after the new Snaps are deployed will run with the new Snaps.
  • For Ultra pipelines, the currently running instances will continue to run with the older Snap packs. Editing and saving the task instance in Manager will cause a rolling restart of the Ultra pipeline instances and the new Snap Packs will get picked up.
  • Customers using SQL Server with Windows authentication and SAP will need to restart their Groundplexes and Cloudplexes.
  • New Snap Packs are displayed in the Designer only after all the JCCs in an org are upgraded to the latest 4.15 version.

New Snap Packs

SnapLogic Data Science (Machine Learning): Introducing SnapLogic Data Science Snaps that enable you to conduct machine learning operations on your datasets. Using these Snap Packs, you can conduct data cleaning, data modeling, and data analytics. The Data Science solution includes the following: 

  • ML Data Preparation Snap Pack: Perform preparatory operations on datasets such as data type transformation, data cleanup, sampling, shuffling, and scaling. Snaps in this Snap Pack are: 
    • Categorical to Numeric: Convert categorical columns into numeric columns by using integer coding or one hot encoding.
    • Clean Missing Values: Replace missing values in datasets by dropping or imputing values.
    • Date Time Extractor: Extract components from datetime objects.
    • Numeric to Categorical: Convert numeric columns into categorical columns by using custom ranging or binning.
    • Sample: Generate sample datasets from an input dataset using sampling algorithms.
    • Scale: Scale values in columns to specify ranges or apply statistical transformations.
    • Shuffle: Randomize the order of the row data in the dataset.
    • Type Converter: Determine types of values in columns. There are four supported types: integer, floating point, text, and datetime.
  • ML Core Snap Pack: Perform data modeling operations such as model training, cross-validation, and model-based predictions. Additionally, you can also execute Python scripts remotely. Snaps in this Snap Pack are: 
  • ML Analytics Snap Pack: Perform analytic operations such as data profiling. Snaps in this Snap Pack are:
    • Profile: Generate statistical information for the input dataset.
    • Type Inspector: Analyze the data types present in the dataset.

Read more about Data Science Snaps in Getting Started with ML.

EDI Snap Pack: Integrate with SnapLogic B2B gateway and create/read EDI transaction data. Snaps in this Snap Pack are:

...

Users can automate reporting activities and creation of EDI transactions using these Snaps in Ultra tasks. Users can also generate analytics information to aid in their decision-making efforts by using the Analytic Snap Packs in conjunction with these Snaps. 

Workday Prism: Provides the ability to integrate data in Workday Prism to generate self-service Analytical reports that empower HR and finance teams to make business decisions. This Snap Pack includes the following Snap:

  • Bulk Load Snap: Use this Snap to upload multiple records/files in the SnapLogic pipeline from multiple sources (such as ERP, CRM, databases, and Workday HCM) into Workday Prism, and derive insightful visual analytics on the data.

...

  • Catalog Insert: Insert metadata information into metadata catalogs tables. You can also use this Snap to further enrich your metadata catalog with additional details, ensuring that you have all the meta-information you need to get actionable insights from your data.
  • Catalog Query: Query metadata information from the metadata catalog. You can use this Snap to retrieve metadata from tables in your catalog and select the elements that you want to reuse. For example, you could use a combination of the Catalog Query and Catalog Insert Snaps to retrieve metadata from one set of tables and create or enrich other tables in your catalog.

New Snaps in existing Snap Packs

Snowflake Snap Pack: Added the following Snaps:

  • Snowflake Multi Execute: Execute multiple Data Modifying Language (DML) and Data Definition Language (DDL) queries in the Snowflake database. 
  • Snowflake SCD2: Create SCD2 type field-historization in Snowflake tables.

Users can leverage these metadata Snaps by just referring to the table names without needing to know the actual file locations.

...

  • HDFS ZipFile Writer: Create and write ZIP files directly into the Hadoop file system. Use this Snap to collect files from various locations, zip them all together, and insert them directly into the Hadoop file system at one go.
  • HDFS ZipFile Reader: Read the contents of ZIP files that are located in the Hadoop file system. This Snap enables you to read all the ZIP files in a Hadoop directory that match specific criteria and make the unzipped files available to downstream Snaps.

Updated Snap Packs

Analytics

  • Anaplan Account: Anaplan is extending the deadline to migrate to Certificate Authority (CA) certificates to December, 2019. To support this, on December 8, 2018, Anaplan will update their production certificate. This will replace validation for Anaplan certificates and existing Anaplan certificates will cease to work once this certificate has been issued. Anaplan still recommends that customers update their integration clients to a version that supports Certificate Authentication 2.0 which uses a CA certificate. To support Anaplan v2.0 API, we have enhanced the Anaplan Account configuration.

    Note

    Depending on the authentication method you use, perform the steps mentioned in the table below:


    Authentication method
    Action required with the updated Anaplan Snap Pack
    Basic username and passwordNo action required
    CA certificateUpdate the Anaplan Account settings and enter relevant information in External certificate file, External certificate contents, and External private key properties. Click here for details.
    Self-signed Anaplan CertificateOnce you upgrade your Snaplexes to 4.15, you must roll back to the previous version of the Anaplan Snap Pack. Anaplan certificate expires on December 8, 2018. You must download the new certificate and update the Anaplan Snap account with the new certificate.



  • Anaplan Snap Pack: Added retry capability to Anaplan Action, Read, Write, and Upload Snaps that allow you to handle connection timeouts during Pipeline execution.

Binary

  • Binary Snap Pack: Added a new property, Binary Header Properties, to the CSV Formatter Snap. This property can be configured with the binary headers and their values in the incoming document. 

...

  • Hadoop Snap Pack: Enhanced the Parquet Reader and Parquet Writer Snaps to support the Data Catalog Snap Pack.
  • Hive Snap Pack: Added Zookeeper support to the Hive Account. You can now set up your Hive account using Zookeeper.

Database

  • Azure SQL Snap Pack: Added two new properties, Schema source and Use type default, to the PolyBase Bulk Load Snap that lets you specify the schema source and handle any missing values in the input documents.
  • Snowflake Snap Pack:

...

Limitations

  • Hive Snap Pack: The Hive Snap Pack does not validate with Apache Hive JDBC v1.2.1 jars or earlier because of a defect in Hive. HDP 2.6.3 and HDP 2.6.1 run on Apache Hive JDBC v1.2.1 jars. To validate Snaps that must work with HDP 2.6.3 and HDP 2.6.1, use JDBC v2.0.0 jars. 

Big Data Enhancements

General

...

Panel

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The expected schedule is:

  • UAT #1, October 19, 2018 (Release updates are published above)
  • UAT #2, October 26, 2018
  • UAT #3, November 02, 2018
  • GA, November 09, 2018 (9 pm PDT)

UAT #2

Platform Updates

  • Fixed an issue with the task timeout for local task execution.
  • Fixed an issue that causes excessive audit trail logging for user update.
  • Fixed an issue with the API meter cleanup.
  • Fixed an issue with the OAuth token refresh.
  • Fixed an issue regarding details displaying in Dashboard for Ultra tasks.

UI Updates

  • Fixed the following in the Node Properties tab (Create/Update Snaplex dialog):
    • Restart Max Wait property, which when set to 0, automatically resets to select Forever.
    • HTTPS Port property, wherein an error displays asking for an integer input, even when the user has entered an integer value. 
  • Added an option to overwrite an existing eXtremeplex when copying the same to create a new eXtremeplex.
  • Fixed a script error encountered by Snaps having the File property when specifying the file path. 

Snap Updates

  • Fixed an issue with Snowflake and Redshift SCD2 Snaps where the asterisk that indicates a mandatory field is missing.

  • Fixed an issue with the REST Snap Pack where API call response times are slower than typical or even failing.

  • Fixed an issue where the Directory Browser Snap is not providing the directory name in the Name column when using the SMB protocol.

Big Data Updates

  • Fixed an issue wherein if Pipelines on Hadooplex fail, the entire Pipeline turns red, instead of just the Snap that has an issue turning red.
  • Fixed an issue in Catalog Writer wherein the Table Schema is not displayed in Manager (Table > Show schema) for an eXtreme pipeline.
  • Added the yarn_suggest_queue property to the Execution Settings tab in the Pipeline properties dialog to enable the queue to validate a Pipeline. If no value is specified, then the default setting is derived from the myresource_queue value in the plex.properties file.
  • Fixed an issue with the Hadooplex, wherein no Spark jobs are submitted to Hadoop clusters.
  • Fixed an with the Catalog Writer that created an output file after validating a Pipeline, even when the Execute during preview check box is not selected. 
  • Fixed an issue with the Catalog Reader that returns an empty result for a partitioned table created by an eXtreme Pipeline.
  • Fixed an issue with the Catalog Reader that reads the records of only the last table partition in an eXtreme Pipeline, instead of reading all the records across partitions. 
  • Fixed an issue with the Copy Snap that previews only a single output, even when multiple preview outputs are expected. 
  • Fixed the doc links in the Avro Formatter Snap that did not open the related documentation.
  • Fixed an issue where an existing pipelines with the HDFS Writer Snap failed for HDP Kerberos and for CDH Kerberos as well.

UAT #3

Platform Updates

  • Fixed an issue that causes the ORC Writer Snap to fail while validating a Pipeline, displaying a "Could not initialize class" error. 
  • Moved the Catalog Query and Catalog Insert Snaps from the SnapLogic Metadata Catalog to a new Data Catalog Snap Pack.
  • Fixed an issue that causes private Snap Packs loading from the global shared project to fail during Pipeline validation.
  • Fixed an OAuth token refresh error with Google BigQuery account, while using the REST Snap Pack.

UI Updates

  • Fixed an issue in the Data Catalog Snaps (both Standard- and Spark-mode) wherein when the Table Name field expression (=) is enabled and disabled, the drop down arrow to select values disappears.
  • Fixed an issue across multiple Snaps wherein clicking on a Snap to zoom in/out when it is still loading throws various script errors.
  • Fixed an issue with the Project field in Add New Pipeline dialog wherein, when selecting a project, the special characters display as HTML entity numbers.  

Snap Updates

  • Fixed an issue with the File Operation Snap wherein the error message does not route to the Error View for an invalid file path.
  • Fixed the following Hadoop Snap Pack issues:
    • HDFS ZipFile Writer/Reader Snaps fail when using unicode characters for file names.
    • User impersonation fails for CDH HA-enabled cluster having no account when the User Impersonation is enabled.
    • Unable to define multiple input views; only a single input view is currently possible.
    • HDFS Writer Snap fails when using the WASB account protocol.
  • Fixed an issue with the Hive Execute Snap that does not authenticate with Kerberos for Hortonworks Data Platform (HDP) cluster.
  • Fixed the following Data Science Snap Pack issues:
    • Enhanced support in the Numeric to Categorical Snap wherein the bin value throws an error if the entered value is not a positive integer.
    • Enhanced support with the Numeric to Categorical Snap where the Snap throws an error if the user provides unsorted list or duplicated value.
    • Enhanced support with the Categorical to Numeric Snap where the Snap throws an error if the profile input view is not connected.
    • Fixed an initialization error with the Remote Python Script Snap Account.
  • Fixed an issue in the Snowflake Bulk Load Snap where the Snap would fail when the property Load empty strings is set to true and non-string field values are null.

Big Data Updates

  • Fixed the following Data Catalog Snap Pack issues:
    • Catalog Reader Snap cannot read an integer partition from the table created by Standard-mode Pipelines.
    • Catalog Reader cannot process expression parameters for Partition Key and Value fields for eXtreme Pipelines.
  • Fixed an issue with eXtreme Pipelines wherein the timestamp data written by Spark is not being read by the Standard-mode Pipelines. 
  • Fixed an issue with eXtreme Pipelines that fail to fetch the pricing data for Spot AWS instances.

GA Updates

Platform 

  • Added a configuration option to enable support for Unicode characters in SLFS (SnapLogic File System) file names.

UI

  • None

Snap

  • Fixed an issue with the Adobe Cloud Platform wherein the pipelines were failing because of an API update from Adobe. To handle the API update, the pagination limit for fetching datasets is changed from 0 to 20.

Big Data

  • Fixed an issue with the JSON Parser/Formatter Spark SQL 2.x Snaps wherein Pipelines ingested with a JSON file with multilines fails to validate.  

...

Panel

SnapLogic dot releases help optimize and continuously improve the platform. This dot release section documents all customer-impacting updates across the SnapLogic platform. 

Platform

Release #DateUpdatesBuildCustomer Action
4.15.2

 

Fixed an issue with pushing of pipeline runtime information to control plane that causes pipeline failures. master-4451None
4.15.1 Fixed an issue with the ForEach Snap and ground trigger passing in maps and arrays as input.master-4436None

For information on updating Snaplexes, refer to How to Update a Snaplex War File.

UI

Release #DateUpdatesActions
4.15.4 Enhanced user login behavior so that users are directly logged into the UI after a password reset.
4.15.3 
  • Fixed an issue with the Remote Python Script Snap in existing pipelines, wherein the "Copy Info" action throws an error.
  • Synchronized the expressions list for eXtreme-mode Snaps in the suggestion dropdown with that of Spark-mode Snaps.
  • Fixed an issue with some user accounts that are able to access the UI, despite the "Allow this user UI access" check box being disabled.
  • Users can no longer rearrange pipeline parameters in the Edit Pipeline dialog, since rearranging the parameters result in adding new fields.
NA

Snap Packs


Snap PackUpdate DateBuildUpdates
Google Spreadsheet google/spreadsheet6835

Added a new property, Parse data, to the Worksheet Writer Snap which helps preserve the date data type.

MongoDB db/mongo6783

Fixed connection timeout issue with MongoDB.

Binary binary6744

Fixed a timeout error while writing to S3 in the S3 File Writer Snap.

Transform transform6736

Fixed an issue wherein the XML Generator Snap was unable to escape some of the special characters.

SOAP soap6699

Fixed an issue with the SOAP Execute Snap wherein errors where not getting routed to the error view and pipeline execution was failing.

Google Spreadsheet google/spreadsheet6639

Added new properties, Preserve data types, Connection properties, Read timeout in seconds, and Connection timeout in seconds, to the Worksheet Reader Snap. These properties let you convert input data types to strings and handle connection timeouts. Also, fixed issues with header count, column count mismatch and rendering of columns with no headers.

Transform transform6680

Fixed an issue with the Excel Parser Snap wherein headers were not parsed properly and header columns were also maintained in an incorrect order.

DynamoDB dynamodb6670Added new functionality to the property Expression attribute values in the Delete Table Item, Scan, Update, and Bulk Get Snaps. The property now handles columns that are named after the DynamoDB reserve words.
MongoDB db/mongo6465

Fixed an issue wherein REST calls to Ultra tasks returned an error message.

Oracle db/oracle6534

Improved performance in Pipelines that contain child Pipelines.

Transform transform6402

Fixed an issue with an object type in the Fixed Width Formatter Snap.

Salesforce salesforce6405Fixed an issue with resource leak in PK Chunking in Salesforce Read and SOQL Snaps.
Binary binary6414

Fixed an issue wherein the hostname specified in the Directory Browser path failed to appear as a path in the output document.

MySQL MULTIPLE6413

Added 2 new properties, JDBC Driver Class and JDBC JARs. The new properties enable uploading JDBC JARs that help connect to MYSQL8.

Confluent Kafka confluentkafka6446

Fixed an issue with the Confluent Kafka Consumer Snap wherein some connections were not closed after a Pipeline is aborted.

Transform transform6386Fixed an issue with the Fixed Width Parser Snap wherein Turkish characters caused incorrect parsing of data on Windows plex.
Oracle db/oracle6417

Fixed an issue with data types conversion. All fields with byte, short, int, or long data types will be converted to BigInteger after fetching.

Email email6267Fixed an issue with handling null attachments in the Email Sender and Email Reader Snaps.
Workday workday6292Fixed an issue with the Workday Read Snap giving inconsistent results. Reverting the multithreading update resolved the issue.
SOAP soap6269

Removed synchronization from a WSDL service of SOAP Execute Snap to avoid hangs.

Transform transform6321Fixed an issue with the XML Parser Snap wherein the Snap failed to get data types from XSD.
Salesforce salesforce6349Fixed an issue with the Salesforce SOQL Snap timing out while waiting for a connection from the pool.
Redshift db/redshift6286

Fixed an issue with the Bulk Upsert Snap wherein there was no output for any input schema.

MongoDB db/mongo6312

Fixed an issue wherein connections were not released after pipeline execution.

RESTrest6310

Fixed an issue with the NLTM Rest account connection hanging infinitely.

Azure SQL db/azuresql6327Replaced Max idle time and Idle connection test period properties with Max life time and Idle Timeout properties, respectively, in the Account configuration dialog. The new properties fix the connection release issues that are occurring due to default/restricted DB Account settings.










Cassandra db/cassandra6329 
Hive db/hive6330 
MySQL db/mysql6332
PostgreSQL db/postgres6333
Redshift db/redshift6334
SAP HANA db/saphana6335
Snowflake db/snowflake6336
SQL Server db/sqlserver6337
Teradata db/teradata6338
Vertica db/vertica6339
Oracle db/oracle6284
Transform transform6265

Fixed an issue with the XML Parser wherein XML validation was failing if the XSD contained xsd:include.

Flow flow6263Improved the error messages to include the source path of the constraint for data validation failures.

For information on updating the Snap Packs, refer to How to upgrade to a new Snap Pack version.

...