Pipeline
Pipelines move data between sources and targets while applying logic and transformations. For example, a pipeline can monitor an OPC UA tag for data changes. When a change occurs, it can read an Instance composed of data from the OPC UA server and publish it to MQTT on a unique topic.
Pipelines consist of Triggers and Stages, enabling them to process and manipulate data effectively.
Error Handler
The Error Handler allows users to define additional behavior when a pipeline encounters an error. If the Error Handler is set to None, the pipeline follows the default error handling behavior. Errors are tracked locally, and the pipeline status reflects the status of the most recent run. If the Error Handler is set to External, the pipeline will forward all stage-level errors to another pipeline while maintaining default error handling behavior locally. The template of the reported value and metadata of the error can be seen below.
{
value: any, // Event value
metadata: { // Metadata
pipelineMetadata: { // Metadata related to the incoming event value
},
pipelineError: { // Metadata regarding stage failure
name: string, // Name of pipeline that errored out
stage: string, // Name of stage that errored out
type: string, // Type of stage that errored out
timestamp: string, // Time the error occurred
error: string // Error message
}
}
}
Triggers
Triggers generate events that start the pipeline. Triggers can be callable, event, or time based. When data enters a pipeline, the data is referred to as an event. All events enter the pipeline at the pipeline start stage.
Stages
Stages perform transformation and I/O using the events. Stages have an input and one or more outputs depending on the stage type. Example stages include writes, reads, modeling, buffering, etc.
Creating a pipeline
- Click Pipeline in the configuration’s Main Menu and then click New Pipeline to get started.
- Enter a Name to represent the pipeline. Optionally enter a Description, Tags, and Grouping.
- Choose to create an Empty Pipeline or choose Build Flow.
- Empty Pipeline creates a pipeline with no triggers or stages that must be manually configured.
- Build Flow continues the wizard and displays flow triggers settings to create a simple pipeline that reads and writes.
- Once the Pipeline is created, navigate to Stages in the top left of the Pipeline to view and drag and drop Trigger or other Stages onto the Pipeline.
Pipeline Status
The list view shows the name, enabled status, processing status, active features, and tags for each Pipeline.
The Name, Enabled, and Tags columns can be sorted by clicking on the column header.
All triggers and the Callable flag can be enabled/disabled on multiple pipelines at the same time by selecting them (using the checkboxes on the left) and then using an action from the Actions dropdown
| Column | Description |
|---|---|
| Name | The name of the pipeline or group |
| Enabled | This will show a play button if the pipeline has any enabled triggers, or if it is Callable. |
| Status | Status is based off of the last execution of the pipeline. Not Started - The pipeline has never run Good - The pipeline is executing successfully Error - An error occurred while processing Warning - The pipeline queue is growing faster than it is being processed. |
| Features | A blue downward pointing array will be shown if this pipeline is Callable. A warning icon will be shown if Track Activity is enabled. |
Pipeline Auto Layout
For large pipelines use the auto layout feature in the lower left hand corner of the pipeline canvas to automatically layout the stages.
Pipeline Stage Cloning
To clone a stage, click on the desired stage and press the clone button.
Pipelines are a powerful tool to move and transform data. See the links below to learn more about pipeline components, debugging, monitoring, and more.