New Record State Extraction
Debezium connectors emits data change messages to represent each operation that they capture from a source database. The messages that a connector sends to Apache Kafka have a complex structure that faithfully represent the details of the original database event.
Although this complex message format accurately details information about changes that happen in the system, the format might not be suitable for some downstream consumers. Sink connectors, or other parts of the Kafka ecosystem might require messages that are formatted so that field names and values are presented in a simplified, flattened structure.
To simplify the format of the event records that the Debezium connectors produce, you can use the Debezium event flattening single message transformation (SMT). Configure the transformation to support consumers that require Kafka records to be in a format that is simpler than the default format that that the connector produces. Depending on your particular use case, you can apply the SMT to a Debezium connector, or to a sink connector that consumes messages that the Debezium connector produces. To enable Apache Kafka to retain the Debezium change event messages in their original format, configure the SMT for a sink connector.
The event flattening transformation is aKafka Connect SMT.
The information in this chapter describes the event flattening single message transformation (SMT) for Debezium SQL-based database connectors. For information about an equivalent SMT for the Debezium MongoDB connector, seeMongoDB New Document State Extraction. |
Change event structure
Debezium generates data change events that have a complex structure. Each event consists of three parts:
Metadata, which includes but is not limited to:
The type of operation that changed the data.
Source information, such as the names of the database and the table in which the change occurred.
Timestamp that identifies when the change was made.
Optional transaction information.
Row data before the change
Row data after the change
The following example shows part of the message structure for anUPDATE
change event:
{ "op": "u", "source": { ... }, "ts_ms" : "...", "before" : { "field1" : "oldvalue1", "field2" : "oldvalue2" }, "after" : { "field1" : "newvalue1", "field2" : "newvalue2" } }
For more information about the change event structure for a connector, seethe documentation for the connector.
After the event flattening SMT processes the message in the previous example, it simplifies the message format, resulting in the message in the following example:
{ "field1" : "newvalue1", "field2" : "newvalue2" }
Behavior
The event flattening SMT extracts theafter
field from a Debezium change event in a Kafka record. The SMT replaces the original change event with only itsafter
field to create a simple Kafka record.
You can configure the event flattening SMT for a Debezium connector or for a sink connector that consumes messages emitted by a Debezium connector. The advantage of configuring event flattening for a sink connector is that records stored in Apache Kafka contain whole Debezium change events. The decision to apply the SMT to a source or sink connector depends on your particular use case.
You can configure the transformation to do any of the following:
Add metadata from the change event to the simplified Kafka record. The default behavior is that the SMT does not add metadata.
Keep Kafka records that contain change events for
DELETE
人事处erations in the stream. The default behavior is that the SMT drops Kafka records forDELETE
人事处eration change events because most consumers cannot yet handle them.
A databaseDELETE
人事处eration causes Debezium to generate two Kafka records:
A record that contains
"op": "d",
thebefore
row data, and some other fields.A tombstone record that has the same key as the deleted row and a value of
null
. This record is a marker for Apache Kafka. It indicates thatlog compactioncan remove all records that have this key.
Instead of dropping the record that contains thebefore
row data, you can configure the event flattening SMT to do one of the following:
Keep the record in the stream and edit it to have only the
"value": "null"
field.Keep the record in the stream and edit it to have a
value
field that contains the key/value pairs that were in thebefore
field with an added"__deleted": "true"
entry.
Similarly, instead of dropping the tombstone record, you can configure the event flattening SMT to keep the tombstone record in the stream.
配置
Configure the Debezium event flattening SMT in a Kafka Connect source or sink connector by adding the SMT configuration details to your connector’s configuration. For example, to obtain the default behavior of the transformation, add it to the connector configuration without specifying any options, as in the following example:
transforms=unwrap,... transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
As with any Kafka Connect connector configuration, you can settransforms=
to multiple, comma-separated, SMT aliases in the order in which you want Kafka Connect to apply the SMTs.
The following.properties
example sets several event flattening SMT options:
transforms=unwrap,... transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState transforms.unwrap.drop.tombstones=false transforms.unwrap.delete.handling.mode=rewrite transforms.unwrap.add.fields=table,lsn
-
drop.tombstones=false
-
Keeps tombstone records for
DELETE
人事处erations in the event stream. -
delete.handling.mode=rewrite
-
For
DELETE
人事处erations, edits the Kafka record by flattening thevalue
field that was in the change event. Thevalue
field directly contains the key/value pairs that were in thebefore
field. The SMT adds__deleted
and sets it totrue
, for example:"value": { "pk": 2, "cola": null, "__deleted": "true" }
-
add.fields=table,lsn
-
Adds change event metadata for the
table
andlsn
fields to the simplified Kafka record.
The connector might emit many types of event messages (heartbeat messages, tombstone messages, or metadata messages about transactions or schema changes). To apply the transformation to a subset of events, you can definean SMT predicate statement that selectively applies the transformationto specific events only.
Adding metadata
You can configure the event flattening SMT to add original change event metadata to the simplified Kafka record. For example, you might want the simplified record’s header or value to contain any of the following:
The type of operation that made the change
The name of the database or table that was changed
Connector-specific fields such as the Postgres LSN field
For more information on what is available seethe documentation for each connector.
To add metadata to the simplified Kafka record’s header, specify theadd.headers
人事处tion. To add metadata to the simplified Kafka record’s value, specify theadd.fields
人事处tion. Each of these options takes a comma separated list of change event field names. Do not specify spaces. When there are duplicate field names, to add metadata for one of those fields, specify the struct as well as the field. For example:
transforms=unwrap,... transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState transforms.unwrap.add.fields=op,table,lsn,source.ts_ms transforms.unwrap.add.headers=db transforms.unwrap.delete.handling.mode=rewrite
With that configuration, a simplified Kafka record would contain something like the following:
{ ... "__op" : "c", "__table": "MY_TABLE", "__lsn": "123456789", "__source_ts_ms" : "123456789", ... }
Also, simplified Kafka records would have a__db
header.
In the simplified Kafka record, the SMT prefixes the metadata field names with a double underscore. When you specify a struct, the SMT also inserts an underscore between the struct name and the field name.
To add metadata to a simplified Kafka record that is for aDELETE
人事处eration, you must also configuredelete.handling.mode=rewrite
.
Options for applying the event-flattening transformation selectively
In addition to the change event messages that a Debezium connector emits when a database change occurs, the connector also emits other types of messages, including heartbeat messages, and metadata messages about schema changes and transactions. Because the structure of these other messages differs from the structure of the change event messages that the SMT is designed to process, it’s best to configure the connector to selectively apply the SMT, so that it processes only the intended data change messages.
For more information about how to apply the SMT selectively, seeConfigure an SMT predicate for the transformation.
配置人事处tions
The following table describes the options that you can specify to configure the event flattening SMT.
Option | Default | Description |
---|---|---|
|
Debezium generates a tombstone record for each |
|
|
Debezium generates a change event record for each |
|
To use row data to determine the topic to route the record to, set this option to an |
||
__ (double-underscore) |
Set this optional string to prefix a field. |
|
Set this option to a comma-separated list, with no spaces, of metadata fields to add to the simplified Kafka record’s value. When there are duplicate field names, to add metadata for one of those fields, specify the struct as well as the field, for example |
||
__ (double-underscore) |
Set this optional string to prefix a header. |
|
Set this option to a comma-separated list, with no spaces, of metadata fields to add to the header of the simplified Kafka record. When there are duplicate field names, to add metadata for one of those fields, specify the struct as well as the field, for example |
||
The Kafka message header name to use for listing field names in the source message that you want to drop from the output message. |
||
|
Specifies whether you want the SMT to remove fields that are listed in |
|
|
Specifies whether you want the SMT to remove non-optional fields that are included in the |