apache kafka - Debezium JDBC sink Connector and MSSQL varbinary - Stack Overflow

admin管理员组

文章数量:1391998

I have Debezium 3.0.8. I setup source SQL connector, which produce data from CDC SQL Server (MSSQL). Trying to setup io.debezium.connector.jdbc.JdbcSinkConnector for a table with varbinary field

Source connector successfully produces messages to topic and schema-registry. Message like

{
    "after": {
        "Value": {
            "Field0": "ÞÊ0T",
            "Field1": 4
        }
    },
.....

Schema for value is:

{
    "type": "record",
    "name": "Envelope",
    "namespace": "test.Domino.dbo._del_test_kafka25_2",
    "fields": [
        {
            "name": "before",
            "type": [
                "null",
                {
                    "type": "record",
                    "name": "Value",
                    "fields": [
                        {
                            "name": "Field0",
                            "type": "bytes"
                        },
                        {
                            "name": "Field1",
                            "type": "int"
                        }
                    ],
                    "connect.name": "test.Domino.dbo._del_test_kafka25_2.Value"
                }
            ],

So this part looks ok for me. When I try run io.debezium.connector.jdbc.JdbcSinkConnector it fails with :

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The conversion from UNKNOWN to VARBINARY is unsupported

I turned on TRACE logging and got this:

Bind field 'Field0' at position 2 with type io.debezium.connector.jdbc.type.connect.ConnectBytesType: java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]

I looked over the source of debezium and also jdbc source and I think that the root cause is that somewhy value of debezium type BYTES was created as java.nio.HeapByteBuffer instead of byte[] which is expected by microsoft jdbc driver.

Can anybody help with the solution for this?

After digging into source code i think

I have Debezium 3.0.8. I setup source SQL connector, which produce data from CDC SQL Server (MSSQL). Trying to setup io.debezium.connector.jdbc.JdbcSinkConnector for a table with varbinary field

Source connector successfully produces messages to topic and schema-registry. Message like

{
    "after": {
        "Value": {
            "Field0": "ÞÊ0T",
            "Field1": 4
        }
    },
.....

Schema for value is:

{
    "type": "record",
    "name": "Envelope",
    "namespace": "test.Domino.dbo._del_test_kafka25_2",
    "fields": [
        {
            "name": "before",
            "type": [
                "null",
                {
                    "type": "record",
                    "name": "Value",
                    "fields": [
                        {
                            "name": "Field0",
                            "type": "bytes"
                        },
                        {
                            "name": "Field1",
                            "type": "int"
                        }
                    ],
                    "connect.name": "test.Domino.dbo._del_test_kafka25_2.Value"
                }
            ],

So this part looks ok for me. When I try run io.debezium.connector.jdbc.JdbcSinkConnector it fails with :

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The conversion from UNKNOWN to VARBINARY is unsupported

I turned on TRACE logging and got this:

Bind field 'Field0' at position 2 with type io.debezium.connector.jdbc.type.connect.ConnectBytesType: java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]

I looked over the source of debezium and also jdbc source and I think that the root cause is that somewhy value of debezium type BYTES was created as java.nio.HeapByteBuffer instead of byte[] which is expected by microsoft jdbc driver.

Can anybody help with the solution for this?

After digging into source code i think

• apache-kafka
• jdbc
• debezium
• mssql-jdbc

Share Improve this question Follow asked Mar 14 at 9:50 Yuriy VikulovYuriy Vikulov 2,50955 gold badges2626 silver badges3232 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

So the problem was that there were no correct handling from BYTES type to VARBINARY SQL Server type in JDBC Sink connector. Made a PR to fix this issue: https://github/debezium/debezium/pull/6235

Some time ago same issue was for Postgres: https://github/debezium/debezium-connector-jdbc/pull/36

本文标签： apache kafkaDebezium JDBC sink Connector and MSSQL varbinaryStack Overflow