admin管理员组

文章数量:1356597

I initially create an iceberg table with two columns of type int and string. Later I try to overwrite the table with one extra column of type date being added in the middle. The operation errors out. I have attached the code and error log below:

import .apache.spark.sql.functions._
spark.range(10).withColumn("tmp", lit("hi")).writeTo("test.sample").using("iceberg").tableProperty("write.spark.accept-any-schema", "true").createOrReplace() // creating a table for the first time
spark.range(10).withColumn("cur_date", current_date).withColumn("tmp", lit("hi")).writeTo("test.sample").using("iceberg").tableProperty("write.spark.accept-any-schema", "true").option("mergeSchema", "true").createOrReplace()

Caused by: .apache.hadoop.hive.metastore.api.InvalidOperationException: The following columns have types incompatible with the existing columns in their respective positions : cur_date

Version Details

spark : version 3.4.1.3.3.6.4-7

scala version: 2.12.17

java: 17.0.14

iceberg-spark-runtime: iceberg-spark-3.4_2.12-1.4.3.3.3.6.4-7.jar

As per documentation, the existing table's configuration and data must be replaced with that of the dataframe.

本文标签：

版权声明：本文标题：scala - createOrReplace() of DataFrameWriterV2 not replacing the data and configuration of Iceberg table - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1744046505a2581620.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。