Handling errors

Edit online

Processing Applications offers three different types of error handling to manage issues that may arise during the deserialization, processing, or aggregation of events. The error handling mode is set during the processing application configuration using the deserializationErrorMode parameter.

The three supported modes are:

LOG_ERROR_AND_CANCEL_JOB- (default) Cancels the processing job and logs the error.
LOG_ERROR_AND_SKIP_EVENT- Logs the error and continues processing the remaining events.
LOG_ERROR_AND_USE_DLQ- Logs the error and sends the event to a dedicated Kafka topic, known as the 'dead letter queue' (DLQ), for further analysis.

Log error and skip event

Test the two non-default deserialization error handling modes by sending an invalid event to the Kafka Ingress topic and analyze how the different modes handle the error.

Create a folder named test-app containing the following two files:

processing-conf.json - a file defining Kafka ingress and egress. In this file, a deserializationErrorMode of type LOG_ERROR_AND_SKIP_EVENT is specified.

  {
      "ingresses": [
        {
          "uid": "I0001",
          "type": "kafka",
          "topic": "icp4ba-bai-order",
          "selector": {
            "type": "jslt-inline",
            "expression": "test(.kind, \"order\")"
          }
        }
      ],
      "egresses": [
        {
          "uid": "E0001",
          "type": "kafka",
          "topic": "icp4ba-bai-order-completed",
          "enabled": true,
          "selector": {
            "type": "jslt-file",
            "filename": "filter-egress.jslt"
          }
        }
      ],
      "settings": {
        "deserializationErrorMode": "LOG_ERROR_AND_SKIP_EVENT"
      }
    }

filter-egress.jslt - a file defining the filter to apply at the egress level.
```
test(.status, "ORDER_DONE")
```

Create a folder named test-data containing order.events.txt file with following content. Note that the file is an invalid JSON file (raw event).
```
{ "id": "order_1", "kind": "order", "seq": 1, "timestamp": "2021-05-07T00:00:01.000-04:00", "status": "ORDER_CREATED"
```

Create a Processing Application.

management-cli processing-app create --name="test-app"

Import the configuration.

management-cli processing-conf import --name="test-app" --directory="./test-app"

Deploy the application.

management-cli processing-app deploy --name="test-app"

Run the following command in a dedicated terminal:

management-cli kafka consumer-json --topic=icp4ba-bai-order-completed

Send the test events

management-cli kafka producer-json --topic=icp4ba-bai-order --file=./test-data/order.events.txt

Result

To see that the job is still running despite an invalid event being sent to the ingress topic, run the following command:

./management-cli processing-jobs list

Log error and use dead letter queue

Push any errors that occur during deserialization or processing in a dedicated Kafka topic known as the dead letter queue by using the LOG_ERROR_AND_USE_DLQ error handling mode. This allows you to easily monitor and manage any issues that arise during the processing process. By piping relevant software to the dead letter queue Kafka topic, you can gain valuable insights into the types of errors that are occurring and take appropriate action to resolve them.

Create a folder named test-app containing the following two files:

processing-conf.json - a file defining Kafka ingress and egress. In this file, a deserializationErrorMode of type LOG_ERROR_AND_USE_DLQ is specified. A Kafka egress of kind: deadLetterQueue was also created.

{
      "ingresses": [
        {
          "uid": "I0001",
          "type": "kafka",
          "topic": "icp4ba-bai-order",
          "selector": {
            "type": "jslt-inline",
            "expression": "test(.kind, \"order\")"
          }
        }
      ],
      "egresses": [
        {
          "uid": "E0001",
          "type": "kafka",
          "topic": "icp4ba-bai-order-completed",
          "enabled": true,
          "selector": {
            "type": "jslt-file",
            "filename": "filter-egress.jslt"
          }
        },
        {
          "uid": "E0002",
          "type": "kafka",
          "topic": "icp4ba-bai-order-dlq",
          "enabled": true,
          "kind": "deadLetterQueue"
        }
      ],
      "settings": {
        "deserializationErrorMode": "LOG_ERROR_AND_USE_DLQ"
      }
    }

filter-egress.jslt - a file defining the filter to apply at the egress level.
```
test(.status, "ORDER_DONE")
```

Create a folder named test-data containing `order.events.txt` file with following content. Note that the file is an invalid JSON file (raw event).
```
{ "id": "order_1", "kind": "order", "seq": 1, "timestamp": "2021-05-07T00:00:01.000-04:00", "status": "ORDER_CREATED"
```