pyspark - Providing table options while writing to iceberg table using spark - Stack Overflow

admin管理员组

文章数量:1418922

I am trying to write to a iceberg table(which does not exist before write, hence creating during write) and would like to provide few table properties. Is it possible to do so using the dataframeWriter? I do not want to fire a sql query using spark.sql()

the following are some of the configs that I am using.

"spark.sql.catalog.spark_catalog": ".apache.iceberg.spark.SparkSessionCatalog"

"spark.sql.extensions": ".apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"

"spark.sql.catalogImplementation": "hive"

I am trying to write to a iceberg table(which does not exist before write, hence creating during write) and would like to provide few table properties. Is it possible to do so using the dataframeWriter? I do not want to fire a sql query using spark.sql()

the following are some of the configs that I am using.

"spark.sql.catalog.spark_catalog": ".apache.iceberg.spark.SparkSessionCatalog"

"spark.sql.extensions": ".apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"

"spark.sql.catalogImplementation": "hive"

• apache-spark
• pyspark
• apache-spark-sql
• apache-iceberg

Share Improve this question Follow asked Jan 29 at 12:06 trutru 16322 silver badges1111 bronze badges 3

• 1 can you share sample of what you came up with till now – NNM Commented Jan 29 at 21:46
• I believe neither of the config options you mentioned are table properties, i.e. what you'd define while writing a table. TBLPROPERTIES that can be set for dataframe writer using .option() are listed in Iceberg docs: iceberg.apache./docs/latest/spark-configuration/… – mazaneicha Commented Mar 21 at 21:49
• 1 @mazaneicha I shared the spark session configs that I using and I know those are not table properties. I wanted to provide table properties using dataframe api and I have found DataFrameWriterV2 to help me do so. – tru Commented Mar 28 at 9:21

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Using DataFrameWriterV2, this is possible:

spark.range(10).withColumn("tmp", lit("hi")).writeTo("test.sample").using("iceberg").tableProperty("write.spark.accept-any-schema", "true").createOrReplace()

本文标签： pysparkProviding table options while writing to iceberg table using sparkStack Overflow