pyspark - GCS Error getting access token from metadata server at: http:169.254.169.254computeMetadatav1instanceservice-accountsd

admin管理员组

文章数量:1291595

I’m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, I’m encountering the following error:

Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: .html

I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: .html

spark.conf.set("spark.hadoop.google.cloud.auth.service.account.enable", "true")
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.email", client_email)
spark.conf.set("spark.hadoop.fs.gs.project.id", project_id)
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key", private_key)
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key.id", private_key_id)
spark.conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.conf.set("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")

Attempting to read the test csv file from my GCS bucket:

gcs_path = "gs://ddfsdfts/events/31dfsdfs4_2025_02_01_000000000000.csv"

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(gcs_path)

df.show()

The error happens when trying df.show()

I've seen a few other questions like this but no straight forward answers. Why is it trying to get to the metadata server token?

I’m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, I’m encountering the following error:

Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks/en/connect/storage/gcs.html

I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks/en/connect/storage/gcs.html

spark.conf.set("spark.hadoop.google.cloud.auth.service.account.enable", "true")
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.email", client_email)
spark.conf.set("spark.hadoop.fs.gs.project.id", project_id)
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key", private_key)
spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key.id", private_key_id)
spark.conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.conf.set("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")

Attempting to read the test csv file from my GCS bucket:

gcs_path = "gs://ddfsdfts/events/31dfsdfs4_2025_02_01_000000000000.csv"

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(gcs_path)

df.show()

The error happens when trying df.show()

I've seen a few other questions like this but no straight forward answers. Why is it trying to get to the metadata server token?

• pyspark
• google-cloud-storage
• databricks

Share Improve this question Follow asked Feb 13 at 13:22 KristiLogosKristiLogos 1311 silver badge55 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Instead of setting the configs using spark.conf.set which affects only the driver node, please try setting them at the cluster level like below as mentioned here. Setting these at the cluster level propogates these to the worker nodes as well.

spark.hadoop.google.cloud.auth.service.account.enable true
spark.hadoop.fs.gs.auth.service.account.email <client-email>
spark.hadoop.fs.gs.project.id <project-id>
spark.hadoop.fs.gs.auth.service.account.private.key {{secrets/scope/gsa_private_key}}
spark.hadoop.fs.gs.auth.service.account.private.key.id {{secrets/scope/gsa_private_key_id}}

本文标签：