Exporting CloudEvents
CloudEventsis a specification for describing event data in a common way. Its aim is to provide interoperability across services, platforms and systems. Debezium enables you to configure a MongoDB, MySQL, PostgreSQL, or SQL Server connector to emit change event records that conform to the CloudEvents specification.
Support for CloudEvents is in an incubating state. This means that exact semantics, configuration options, and other details may change in future revisions based on feedback. Please let us know your specific requirements or if you encounter any problems while using this feature. |
The CloudEvents specification defines:
A set of standardized event attributes
Rules for defining custom attributes
Encoding rules for mapping event formats to serialized representations such as JSON or Avro
Protocol bindings for transport layers such as Apache Kafka, HTTP or AMQP
To configure a Debezium connector to emit change event records that conform to the CloudEvents specification, Debezium provides theio.debezium.converters.CloudEventsConverter
, which is a Kafka Connect message converter.
Currently, only structured mapping mode is supported. The CloudEvents change event envelope can be JSON or Avro and each envelope type supports JSON or Avro as thedata
format. It is expected that a future Debezium release will support binary mapping mode.
For information about using Avro, see:
Example event format
The following example shows what a CloudEvents change event record emitted by a PostgreSQL connector looks like. In this example, the PostgreSQL connector is configured to use JSON as the CloudEvents format envelope and also as thedata
format.
{ "id" : "name:test_server;lsn:29274832;txId:565",(1)"source" : "/debezium/postgresql/test_server",(2)"specversion" : "1.0",(3)"type" : "io.debezium.postgresql.datachangeevent",(4)"time" : "2020-01-13T13:55:39.738Z",(5)"datacontenttype" : "application/json",(6)"iodebeziumop" : "r",(7)"iodebeziumversion" : "1.4.2.Final",(8)"iodebeziumconnector" : "postgresql", "iodebeziumname" : "test_server", "iodebeziumtsms" : "1578923739738", "iodebeziumsnapshot" : "true", "iodebeziumdb" : "postgres", "iodebeziumschema" : "s1", "iodebeziumtable" : "a", "iodebeziumtxId" : "565", "iodebeziumlsn" : "29274832", "iodebeziumxmin" : null, "iodebeziumtxid": "565",(9)"iodebeziumtxtotalorder": "1", "iodebeziumtxdatacollectionorder": "1", "data" : {(10)"before" : null, "after" : { "pk" : 1, "name" : "Bob" } } }
1 | Unique ID that the connector generates for the change event based on the change event’s content. |
2 | The source of the event, which is the logical name of the database as specified by thedatabase.server.name property in the connector’s configuration. |
3 | The CloudEvents specification version. |
4 | Connector type that generated the change event. The format of this field isio.debezium.CONNECTOR_TYPE.datachangeevent . The value ofCONNECTOR_TYPE ismongodb ,mysql ,postgresql , orsqlserver . |
5 | Time of the change in the source database. |
6 | Describes the content type of thedata attribute, which is JSON in this example. The only alternative is Avro. |
7 | An operation identifier. Possible values arer for read,c for create,u for update, ord for delete. |
8 | Allsource attributes that are known from Debezium change events are mapped to CloudEvents extension attributes by using theiodebezium prefix for the attribute name. |
9 | When enabled in the connector, eachtransaction attribute that is known from Debezium change events is mapped to a CloudEvents extension attribute by using theiodebeziumtx prefix for the attribute name. |
10 | The actual data change itself. Depending on the operation and the connector, the data might containbefore ,after and/orpatch fields. |
The following example also shows what a CloudEvents change event record emitted by a PostgreSQL connector looks like. In this example, the PostgreSQL connector is again configured to use JSON as the CloudEvents format envelope, but this time the connector is configured to use Avro for thedata
format.
{ "id" : "name:test_server;lsn:33227720;txId:578", "source" : "/debezium/postgresql/test_server", "specversion" : "1.0", "type" : "io.debezium.postgresql.datachangeevent", "time" : "2020-01-13T14:04:18.597Z", "datacontenttype" : "application/avro",(1)"dataschema" : "http://my-registry/schemas/ids/1",(2)“io开云体育官方注册网址debeziumop”:“r”、“iodebeziumversion”:“1.4.2.Final", "iodebeziumconnector" : "postgresql", "iodebeziumname" : "test_server", "iodebeziumtsms" : "1578924258597", "iodebeziumsnapshot" : "true", "iodebeziumdb" : "postgres", "iodebeziumschema" : "s1", "iodebeziumtable" : "a", "iodebeziumtxId" : "578", "iodebeziumlsn" : "33227720", "iodebeziumxmin" : null, "iodebeziumtxid": "578", "iodebeziumtxtotalorder": "1", "iodebeziumtxdatacollectionorder": "1", "data" : "AAAAAAEAAgICAg=="(3)}
1 | Indicates that thedata attribute contains Avro binary data. |
2 | URI of the schema to which the Avro data adheres. |
3 | Thedata attribute contains base64-encoded Avro binary data. |
It is also possible to use Avro for the envelope as well as thedata
attribute.
Example configuration
Configureio.debezium.converters.CloudEventsConverter
in your Debezium connector configuration. The following example shows how to configure the CloudEvents converter to emit change event records that have the following characteristics:
Use JSON as the envelope.
Use the schema registry at
http://my-registry/schemas/ids/1
to serialize thedata
attribute as binary Avro data.
... "value.converter": "io.debezium.converters.CloudEventsConverter", "value.converter.serializer.type" : "json",(1)"value.converter.data.serializer.type" : "avro", "value.converter.avro.schema.registry.url": "http://my-registry/schemas/ids/1" ...
1 | Specifying theserializer.type is optional, becausejson is the default. |
The CloudEvents converter converts Kafka record values. In the same connector configuration, you can specifykey.converter
if you want to operate on record keys. For example, you might specifyStringConverter
,LongConverter
,JsonConverter
, orAvroConverter
.
配置options
When you configure a Debezium connector to use the CloudEvent converter you can specify the following options.
Option |
Default |
Description |
|
The encoding type to use for the CloudEvents envelope structure. The value can be |
|
|
The encoding type to use for the |
|
N/A |
Any configuration options to be passed through to the underlying converter when using JSON. The |
|
N/A |
Any configuration options to be passed through to the underlying converter when using Avro. The |