Outbox Event Router

Table of Contents

Example outbox message
Basic outbox table
Basic configuration
Options for applying the transformation selectively
Using Avro as the payload format
Emitting messages with additional fields
相依iguration options
Distributed tracing

The outbox pattern is a way to safely and reliably exchange data between multiple (micro) services. An outbox pattern implementation avoids inconsistencies between a service’s internal state (as typically persisted in its database) and state in events consumed by services that need the same data.

To implement the outbox pattern in a Debezium application, configure a Debezium connector to:

Capture changes in an outbox table
Apply the Debezium outbox event router single message transformation (SMT)

A Debezium connector that is configured to apply the outbox SMT should capture changes that occur in an outbox table only. For more information, seeOptions for applying the transformation selectively.

A connector can capture changes in more than one outbox table only if each outbox table has the same structure.

SeeReliable Microservices Data Exchange With the Outbox Patternto learn about why the outbox pattern is useful and how it works.

For an example that you can run, see theoutbox pattern demo, which is in the Debezium examples repository. It includes an example of how to configure a Debezium connector to run the outbox event router SMT.

The outbox event router SMT doesnotsupport the MongoDB connector.

Example outbox message

To learn about how to configure the Debezium outbox event router SMT, consider the following example of a Debezium outbox message:

# Kafka Topic: outbox.event.order # Kafka Message key: "1" # Kafka Message Headers: "id=4d47e190-0402-4048-bc2c-89dd54343cdc" # Kafka Message Timestamp: 1556890294484 { "{\"id\": 1, \"lineItems\": [{\"id\": 1, \"item\": \"Debezium in Action\", \"status\": \"ENTERED\", \"quantity\": 2, \"totalPrice\": 39.98}, {\"id\": 2, \"item\": \"Debezium for Dummies\", \"status\": \"ENTERED\", \"quantity\": 1, \"totalPrice\": 29.99}], \"orderDate\": \"2019-01-31T12:13:01\", \"customerId\": 123}" }

A Debezium connector that is configured to apply the outbox event router SMT generates the above message by transforming a Debezium raw message like this:

# Kafka Message key: "406c07f3-26f0-4eea-a50c-109940064b8f" # Kafka Message Headers: "" # Kafka Message Timestamp: 1556890294484 { "before": null, "after": { "id": "406c07f3-26f0-4eea-a50c-109940064b8f", "aggregateid": "1", "aggregatetype": "Order", "payload": "{\"id\": 1, \"lineItems\": [{\"id\": 1, \"item\": \"Debezium in Action\", \"status\": \"ENTERED\", \"quantity\": 2, \"totalPrice\": 39.98}, {\"id\": 2, \"item\": \"Debezium for Dummies\", \"status\": \"ENTERED\", \"quantity\": 1, \"totalPrice\": 29.99}], \"orderDate\": \"2019-01-31T12:13:01\", \"customerId\": 123}", "timestamp": 1556890294344, "type": "OrderCreated" }, "source": { "version": "1.5.4.Final", "connector": "postgresql", "name": "dbserver1-bare", "db": "orderdb", "ts_usec": 1556890294448870, "txId": 584, "lsn": 24064704, "schema": "inventory", "table": "outboxevent", "snapshot": false, "last_snapshot_record": null, "xmin": null }, "op": "c", "ts_ms": 1556890294484 }

This example of a Debezium outbox message is based on thedefault outbox event router configuration, which assumes an outbox table structure and event routing based on aggregates. To customize behavior, the outbox event router SMT provides numerousconfiguration options.

Basic outbox table

To apply the default outbox event router SMT configuration, your outbox table is assumed to have the following columns:

Column | Type | Modifiers --------------+------------------------+----------- id | uuid | not null aggregatetype | character varying(255) | not null aggregateid | character varying(255) | not null type | character varying(255) | not null payload | jsonb |

Table 1. Descriptions of expected outbox table columns
Column	Effect
`id`	Contains the unique ID of the event. In an outbox message, this value is a header. You can use this ID, for example, to remove duplicate messages. To obtain the unique ID of the event from a different outbox table column, set the`table.field.event.id`SMT optionin the connector configuration.
`aggregatetype`	Contains a value that the SMT appends to the name of the topic to which the connector emits an outbox message. The default behavior is that this value replaces the default`${routedByValue}`variable in the`route.topic.replacement`SMT option. For example, in a default configuration, the`route.by.field`SMT option is set to`aggregatetype`and the`route.topic.replacement`SMT option is set to`outbox.event.${routedByValue}`. Suppose that your application adds two records to the outbox table. In the first record, the value in the`aggregatetype`column is`customers`. In the second record, the value in the`aggregatetype`column is`orders`. The connector emits the first record to the`outbox.event.customers`topic. The connector emits the second record to the`outbox.event.orders`topic. To obtain this value from a different outbox table column, set the`route.by.field`SMT optionin the connector configuration.
`aggregateid`	包含事件的关键,它提供了一个ID为the payload. The SMT uses this value as the key in the emitted outbox message. This is important for maintaining correct order in Kafka partitions. To obtain the event key from a different outbox table column, set the`table.field.event.key`SMT optionin the connector configuration.
`type`	A user-defined value that helps categorize or organize events.
`payload`	The representation of the event itself. The default structure is JSON. The content in this field becomes one of these: Part of the outbox message`payload`. If other metadata, including`eventType`is delivered as headers, the payload becomes the message itself without encapsulation in an envelope. To obtain the event payload from a different outbox table column, set the`table.field.event.payload`SMT optionin the connector configuration.

Basic configuration

To configure a Debezium connector to support the outbox pattern, configure theoutbox.EventRouterSMT. For example, the basic configuration in a.propertiesfile looks like this:

transforms=outbox,... transforms.outbox.type=io.debezium.transforms.outbox.EventRouter

Options for applying the transformation selectively

In addition to the change event messages that a Debezium connector emits when a database change occurs, the connector also emits other types of messages, including heartbeat messages, and metadata messages about schema changes and transactions. Because the structure of these other messages different from the structure of the change event messages that the SMT is designed to process, it’s best to configure the connector to selectively apply the SMT, so that it processes only the intended data change messages. You can use one of the following methods to configure the connector to apply the SMT selectively:

相依igure an SMT predicate for the transformation.
Use thetopic.regexconfiguration option for the SMT.

Using Avro as the payload format

The outbox event router SMT supports arbitrary payload formats. Thepayloadcolumn value in an outbox table is passed on transparently. An alternative to working with JSON is to use Avro. This can be beneficial for message format governance and for ensuring that outbox event schemas evolve in a backwards-compatible way.

How a source application produces Avro formatted content for outbox message payloads is out of the scope of this documentation. One possibility is to leverage theKafkaAvroSerializerclass to serializeGenericRecordinstances. To ensure that the Kafka message value is the exact Avro binary data, apply the following configuration to the connector:

transforms=outbox,... transforms.outbox.type=io.debezium.transforms.outbox.EventRouter value.converter=io.debezium.converters.ByteBufferConverter

By default, thepayloadcolumn value (the Avro data) is the only message value. Configuration ofByteBufferConverteras the value converter propagates thepayloadcolumn value as-is into the Kafka message value.

The Debezium connectors may be configured to emit heartbeat, transaction metadata, or schema change events (support varies by connector). These events cannot be serialized by theByteBufferConverterso additional configuration must be provided so the converter knows how to serialize these events. As an example, the following configuration illustrates using the Apache KafkaJsonConverterwith no schemas:

transforms=outbox,... transforms.outbox.type=io.debezium.transforms.outbox.EventRouter value.converter=io.debezium.converters.ByteBufferConverter value.converter.delegate.converter.type=org.apache.kafka.connect.json.JsonConverter value.converter.delegate.converter.type.schemas.enable=false

The delegateConverterimplementation is specified by thedelegate.converter.typeoption. If any extra configuration options are needed by the converter, they can also be specified, such as the disablement of schemas shown above usingschemas.enable=false.

Emitting messages with additional fields

Your outbox table might contain columns whose values you want to add to the emitted outbox messages. For example, consider an outbox table that has a value ofpurchase-orderin theaggregatetypecolumn and another column,eventType, whose possible values areorder-createdandorder-shipped. To emit theeventTypecolumn value in the outbox message header, configure the SMT like this:

transforms=outbox,... transforms.outbox.type=io.debezium.transforms.outbox.EventRouter transforms.outbox.table.fields.additional.placement=type:header:eventType

To emit theeventType列值在发件箱消息信封,进行gure the SMT like this:

transforms=outbox,... transforms.outbox.type=io.debezium.transforms.outbox.EventRouter transforms.outbox.table.fields.additional.placement=type:envelope:eventType

相依iguration options

The following table describes the options that you can specify for the outbox event router SMT. In the table, theGroupcolumn indicates a configuration option classification for Kafka.

Table 2. Descriptions of outbox event router SMT configuration options
Option	Default	Group	Description
`table.field.event.id`	`id`	Table	Specifies the outbox table column that contains the unique event ID.
`table.field.event.key`	`aggregateid`	Table	Specifies the outbox table column that contains the event key. When this column contains a value, the SMT uses that value as the key in the emitted outbox message. This is important for maintaining correct order in Kafka partitions.
`table.field.event.timestamp`		Table	By default, the timestamp in the emitted outbox message is the Debezium event timestamp. To use a different timestamp in outbox messages, set this option to an outbox table column that contains the timestamp that you want to be in emitted outbox messages.
`table.field.event.payload`	`payload`	Table	Specifies the outbox table column that contains the event payload.
`table.field.event.payload.id`	`aggregateid`	Table	Specifies the outbox table column that contains the payload ID.
`table.fields.additional.placement`		Table, Envelope	Specifies one or more outbox table columns that you want to add to outbox message headers or envelopes. Specify a comma-separated list of pairs. In each pair, specify the name of a column and whether you want the value to be in the header or the envelope. Separate the values in the pair with a colon, for example: `id:header,my-field:envelope` To specify an alias for the column, specify a trio with the alias as the third value, for example: `id:header,my-field:envelope:my-alias` The second value is the placement and it must always be`header`or`envelope`. 相依iguration examples are inemitting additional fields in Debezium outbox messages.
`table.field.event.schema.version`		Table, Schema	When set, this value is used as the schema version as described in theKafka Connect SchemaJavadoc.
`route.by.field`	`aggregatetype`	Router	Specifies the name of a column in the outbox table. The default behavior is that the value in this column becomes a part of the name of the topic to which the connector emits the outbox messages. An example is in thedescription of the expected outbox table.
`route.topic.regex`	`(?.*)`	Router	Specifies a regular expression that the outbox SMT applies in the RegexRouter to outbox table records. This regular expression is part of the setting of the`route.topic.replacement`SMT option. The default behavior is that the SMT replaces the default`${routedByValue}`variable in the setting of the`route.topic.replacement`SMT option with the setting of the`route.by.field`outbox SMT option.
`route.topic.replacement`	`outbox.event.${routedByValue}`	Router	Specifies the name of the topic to which the connector emits outbox messages. The default topic name is`outbox.event.`followed by the`aggregatetype`column value in the outbox table record. For example, if the`aggregatetype`value is`customers`, the topic name is`outbox.event.customers`. To change the topic name, you can: Set the`route.by.field`option to a different column. Set the`route.topic.regex`option to a different regular expression.
`route.tombstone.on.empty.payload`	`false`	Router	指示是否一个空`null`payload causes the connector to emit a tombstone event.
`debezium.op.invalid.behavior`	`warn`	开云体育官方注册网址	Determines the behavior of the SMT when there is an`UPDATE`operation on the outbox table. Possible settings are: `warn`- The SMT logs a warning and continues to the next outbox table record. `error`- The SMT logs an error and continues to the next outbox table record. `fatal`- The SMT logs an error and the connector stops processing. All changes in an outbox table are expected to be`INSERT`operations. That is, an outbox table functions as a queue; updates to records in an outbox table are not allowed. The SMT automatically filters out`DELETE`operations on an outbox table.
`tracing.span.context.field`	`tracingspancontext`	Tracing	The name of the field containing tracing span context.
`tracing.operation.name`	`debezium-read`	Tracing	The operation name representing the Debezium processing span.
`tracing.with.context.field.only`	`false`	Tracing	When`true`only events that have serialized context field should be traced.

Distributed tracing

The extension has support for the distributed tracing. Seetracing documentationfor more details.