Error Handling

Pipeline Errors

Errors can occur either in the overall pipeline or in pipeline stages. Example errors could include exceeding the maximum inbound event queue size or individual stage errors like bad reads/writes or incorrect stage logic.

The pipeline is in error if any of the stages experienced an error in the most recent pipeline execution. If a pipeline experiences and error in one run, and then executes successfully in the next run, the pipeline is no longer in error.

Stage Errors

Some stages, like read, write, etc have success failure paths as part of the stage configuration. This allows you to handle errors inside the Pipeline without putting the pipeline in error. For example, here is what happens when the failure path is connected to the read stage and the read fails. You can see the event takes the failure path give the execution count.

If the failure path is not connected, the stage and pipeline go into error.

Catching Pipeline Errors

There are cases when you want to standardize the way pipeline errors are caught to send them to an internal ticket management system for example.

To do this you can set the Error Handler setting in the pipeline to External (defaults to None) and select a pipeline to send errors to. In this mode the pipeline continues to behave and reflect errors like before, but error events are sent to another pipeline.

The event sent to the external pipeline for errors looks as follows

{
   value: any, // Event value
   metadata: { // Metadata
      pipelineMetadata: { // Metadata related to the incoming event value

      },
      pipelineError: { // Metadata regarding stage failure
         name: string, // Name of pipeline that errored out
         stage: string, // Name of stage that errored out
         type: string, // Type of stage that errored out
         timestamp: string, // Time the error occurred
         error: string // Error message
      }
   }
}