Error Handling

Jump to Section

Pipeline Errors

Errors can occur either in the overall pipeline or in pipeline stages. Example errors could include exceeding the maximum inbound event queue size or individual stage errors like bad reads/writes or incorrect stage logic.

The pipeline is in error if any of the stages have experienced an error. Errors are also sticky, meaning that if an error only occurs on the 10th event processed by the pipeline, but the 11th event succeeds, the pipeline is still in an error state until you resave the pipeline or click the stage that had the error and clear it. This way intermittent errors are not lost. To clear an individual error, click the stage that’s in error and click “Dismiss”, shown below.

Stage Errors

Some stages, like read, write, etc have success failure paths as part of the stage configuration. This allows you to handle errors inside the Pipeline without putting the pipeline in error. For example, here is what happens when the failure path is connected to the read stage and the read fails. You can see the event takes the failure path give the execution count.

If the failure path is not connected, the stage and pipeline go into error.

To clear a stage error, click the stage and select “Dismiss” underneath the error message.

Catching Pipeline Errors

There are cases when you want to standardize the way pipeline errors are caught to send them to an internal ticket management system for example.

To do this you can set the Error Handler setting in the pipeline to External (defaults to None) and select a pipeline to send errors to. In this mode the pipeline continues to behave and reflect errors like before, but error events are sent to another pipeline.

The event sent to the external pipeline for errors looks as follows

{json}

{
   value: any, // Event value
   metadata: { // Metadata
      pipelineMetadata: { // Metadata related to the incoming event value

      },
      pipelineError: { // Metadata regarding stage failure
         name: string, // Name of pipeline that errored out
         stage: string, // Name of stage that errored out
         type: string, // Type of stage that errored out
         timestamp: string, // Time the error occurred
         error: string // Error message
      }
   }
}

Replay