On Change

The On Change stage is used to only send changed data.

The first event always passes through the stage. Subsequent events are compared against the last event’s value. The cache is cleared any time the pipeline is saved or if the runtime is restarted.

Output Mode

Set to All to emit the entire event if anything in the event has changed. Set to Changed to only emit the changed values from the event. Values that remain the same are filtered out.

Changed filters only changed elements from complex payloads and arrays of complex payloads. If the payload is a simple array or not a complex payload, the entire payload is emitted.

Inclusion List

When using the Changed output mode, this setting allows you to include parts of the event that may not have changed. For example, if the event contains an attribute SiteId and this never changes, add SiteId to the inclusion list to always include it in the emitted event.

Key Expression

The Key Expression allows you to create separate cache given a unique key in the value or metadata. For example, you could create a cache per Instance name or Model type.

Setting the key is controlled using JavaScript and the stage.setBufferKey function. stage.setBufferKey(null) is the default, and creates a single cache. stage.setBufferKey(event.name) creates a unique buffer per source name (i.e. input or instance name). New events are compared to the events unique cache.

Metadata Key

Specifies the name of the key used to retain the value passed to stage.setBufferKey(value) in the metadata object.

Examples

Below is an example payload and how it works with Changed. Mode

First Payload: All of this data passes through the stage on the first event.

{
  "attr1": 123,
  "attr2": 123.123,
  "attr3": true,
  "attr4": "123",
  "attr5": [{"attr5_1": 1, "attr5_2": 2}, {"attr5_1": 1, "attr5_2": 2}],
  "attr6": {"attr6": 1, "attr7": 2}
}

Second Payload:

{
  "attr1": 1234,     // change here
  "attr2": 123.123,
  "attr3": true,
  "attr4": "123",
  "attr5": [{"attr5_1": 1, "attr5_2": 2}, {"attr5_1": 3, "attr5_2": 4}, {"attr5_1": 5, "attr5_2": 6}],  // Added column
  "attr6": {"attr6": 1, "attr7": 3}   // change here
}

If the Output Mode is set to All, the full payload will pass through as is above because there are changes in it. If there were no changes, the event ends. If the Output Mode is Changed, the following event would pass through. Note only the changes pass through.

{
  "attr1": 1234,     // change here
  "attr5": [{"attr5_1": 1, "attr5_2": 2}, {"attr5_1": 3, "attr5_2": 4}, {"attr5_1": 5, "attr5_2": 6}],  // Added column
  "attr6": {"attr7": 3}   // change here
}

For Complex Payload arrays, the comparison is done row by row. If a row changes, the entire row passes the filter. If the row doesn’t change it is omitted. If there are new rows, these rows pass the filter. If rows are removed, but the remaining rows match the previous rows, nothing passes the filter.

First Payload: All of this data passes through the stage on the first event.

[
  {
    "attr_1": 1, 
    "attr_2": 2
  }, 
  {
    "attr1_1": 3, 
    "attr2_2": 4
  }, 
  {
    "attr1_1": 5, 
    "attr2_2": 6
  }
]

Second Payload

[
  {
    "attr_1": 1, 
    "attr_2": 2
  }, 
  {
    "attr1_1": 3, 
    "attr2_2": 6
  }, 
  {
    "attr1_1": 7, 
    "attr2_2": 8
  }
]

If the Output Mode is set to All, the full payload will pass through as is above because there are changes in it. If there were no changes, the event ends. If the Output Mode is Changed, the following event would pass through. Note only the changes pass through.

[
  {
    "attr1_1": 6
  }, 
  {
    "attr1_1": 7, 
    "attr2_2": 8
  }
]