sql - Problem while fetch the data from snowflake to kafka topic - Stack Overflow

admin管理员组

文章数量:1122832

I am trying to fetch the data from snowflake to Kafka using the confluent jdbc connector, everything seems to be setup correctly but having a problem while running the query. I am following this post

The problem which I can see in the logs is related with the invalid identifier '"id"':

 [2024-11-22 10:08:26,121] ERROR [snowflake.to.kafka.connector|task-0] SQL exception while running query for table: TimestampIncrementingTableQuerier{table=null, query='select * from oslo_cycle_trip', topicPrefix='snowflake-kafka-etl', incrementingColumn='id', timestampColumns=[]}, net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error: error line 1 at position 36
invalid identifier '"id"'. Attempting retry 2 of -1 attempts. (io.confluent.connect.jdbc.source.JdbcSourceTask:477)

Here is my snowflake table formate:

create or replace table cycle_trip (
    id integer,
    started_at datetime,
    ended_at datetime,
    duration integer
);
CREATE OR REPLACE SEQUENCE my_sequence
START WITH 1
INCREMENT BY 1;

I use to insert the my_sequence.NEXTVAL in insert statement when fill the data. Here it is how look like:

Connector configuration:

connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
timestamp.column.name=
incrementing.column.name=id
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
connection.password=PASSWORD
query=select * from cycle_trip
connection.attempts=100
transforms=AddNamespace,createKey,AddKeyNamespace
connection.backoff.ms=300000
transforms.AddNamespace.type=org.apache.kafka.connect.transforms.SetSchemaMetadata$Value
timestamp.delay.interval.ms=3000
table.types=table
mode=incrementing
topic.prefix=snowflake-kafka-etl
transforms.AddKeyNamespace.type=org.apache.kafka.connect.transforms.SetSchemaMetadata$Key
connection.user=LEARNER
transforms.AddNamespace.schema.name=inc.evil.coursecatalog.InstructorAggregate
transforms.createKey.fields=started_at
poll.interval.ms=1500
transforms.AddKeyNamespace.schema.name=inc.evil.coursecatalog.Key
numeric.mapping=best_fit
connection.url=jdbc:snowflake://MY-ACCOUNT.snowflakecomputing/?warehouse=compute_wh&db=cycle_db&schema=public

本文标签： sqlProblem while fetch the data from snowflake to kafka topicStack Overflow