On this Page
Types of Ultra Pipelines
When building an Ultra Pipeline, the Pipeline must either have:
- One unconnected input view and one or more unconnected output views: A FeedMaster must also be in the Snaplex for this to work. This is different from triggered tasks (one input and one output).
- No unconnected views: A FeedMaster is not required to be in the Snaplex in this case. This scenario is expected to be used if the data source is a Snap like a JMS Consumer.
Low-latency Feed (Request and Response)
Pipeline: One unconnected input view and one or more unconnected output views
The most popular Ultra Pipeline design is a straightforward request-response construct that is used as a data access layer for real-time web services. This design transforms a Pipeline into a continuously running job. Documents are supplied to the Pipeline through a FeedMaster that maintains a queue of documents, which are then parsed by the Pipeline while responses are returned through the FeedMaster to the caller.
At the simplest, an Ultra Pipeline Task can comprise a Mapper Snap that maps an incoming document to a target field. You can download this sample Pipeline from which you can create an Ultra Task.
Always-On (Continuous Data Flow)
Pipeline: No unconnected input view and no unconnected output view
This design is often used in a listener-consumer construct. With zero unconnected input and output views, an Ultra Pipeline can be used to poll and consume the documents from an endpoint without requiring a document feed from the FeedMaster.
A typical JMS-based polling Ultra Pipeline can comprise of a JMS Consumer Snap and a Snap that writes data to a destination, like a File Write Snap. You can download a sample Pipeline from which you can create an Ultra Task.
Guidelines for Ultra Pipeline Designs
Ultra Pipelines tools are similar to web service requests and conform to an HTTP response architecture. Consider these guidelines when designing Ultra Pipelines.
Input View Type
An unconnected input view type must be either binary or document:
- If binary, the HTTP headers are in the binary document's header.
- If document, the headers from the original HTTP request are in the root of the document and the body of the request is in the content field.
For example, you can reference the the User-Agent HTTP header in the input document as$['user-agent'].
In addition to the HTTP headers in the request, the following fields are added to the input document:
- uri: The original URI of the request.
- method: The HTTP request method.
- query: The parsed version of the query string. The value of this field will be an object whose fields correspond to query string parameters and a list of all the values for that parameter.
For example, the following query string:
foo=bar&foo=baz&one=1
- Results in the following query object:
{"foo" : ["bar", "baz"],"one": ["1"]}
- task_name: The name of the Task.
- path_info: The part of the path after the Task URL.
- server_ip: The IP address of the FeedMaster that received the request.
- server_port: The TCP port of the FeedMaster that received the request.
- client_ip: The IP address of the client that sent the request.
- client_port: The TCP port of the client that sent the request.
Output View Type
The unlinked output view type can be binary or document as well:
- If the view type is a document, the output should have a content field that is JSON-encoded and sent back to the client. The other fields in the output document are treated as HTTP response headers to send back to the client. If a content field is not present, the entire document is JSON-encoded and used as the response body to be sent back to the client.
- If the output document contains a status field that is an integer, it is treated as the HTTP response status code.
- If the view type is binary data, the binary file's header is sent back to the client as HTTP response headers. The body of the binary file is directly streamed back to the client.
- SnapLogic expects exactly one output document for every input.
- If output is not sent, then the original HTTP request sent to the FeedMaster hangs, eventually reaching a timeout.
- If more than one output document is generated, then SnapLogic only sends the first one back as the response to the original HTTP request. This behavior is different from a Triggered Task, where all of the documents sent to the unlinked output are sent back in the response.
- Error views are implicitly added to all Snaps when the Pipeline is executed in Ultra mode, regardless of how the Snaps are configured.
Naming
Avoid spaces in Pipeline names that are invoked as Tasks.
Downloads
