Airflow environment variable is empty - Stack Overflow

admin管理员组

文章数量:1122826

I am trying to pass the ID of a playlist using an environment variable. It is initially set through the execution of this Python script from the user.

def get_input(prompt):
    return input(prompt)

playlist_id = get_input("Bitte geben Sie die PLAYLIST_ID ein: ")

with open('.env', 'w') as f:
    f.write(f"AIRFLOW_VAR_PLAYLIST_ID={playlist_id}\n")

Afterward, the .env file is created, which contains the ID.

In my docker-compose setup, I have configured it as follows:

services:
  airflow:
    build:
      context: .
      dockerfile: Dockerfile.airflow
    container_name: airflow
    environment:
      - AIRFLOW_VAR_PLAYLIST_ID=${PLAYLIST_ID}
    volumes:
...

Now, my Airflow DAG should access the content of the environment variable so that it can use the ID for a request. I am doing it like this and I've shortened the code a bit for better clarity. The access token is also retrieved within the same task.

playlist_id = os.getenv("AIRFLOW_VAR_PLAYLIST_ID")

get_playlist_tracks = BashOperator(
    task_id='get_playlist_tracks',
    bash_command="""

    PLAYLIST_ID=$PLAYLIST_ID

    echo $AIRFLOW_VAR_PLAYLIST_ID
    echo 'PLAYLIST_ID = '$PLAYLIST_ID
    
    curl -X "GET" "/$PLAYLIST_ID/tracks" \
        -H "Accept: application/json" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        > /user/hadoop/spotify/track_data/raw/playlist_tracks.json
    echo "Playlist tracks fetched and saved to HDFS"
    """,  
    dag=dag 
)

The problem is that it doesn't work properly. I also had this form:

PLAYLIST_ID={{ playlist_id }}

But this also not worked. When I try to display the content of the environment variable, it shows as empty. I've read a lot and tried various solutions, but there's an error somewhere, and I can't find it.

I am trying to pass the ID of a playlist using an environment variable. It is initially set through the execution of this Python script from the user.

def get_input(prompt):
    return input(prompt)

playlist_id = get_input("Bitte geben Sie die PLAYLIST_ID ein: ")

with open('.env', 'w') as f:
    f.write(f"AIRFLOW_VAR_PLAYLIST_ID={playlist_id}\n")

Afterward, the .env file is created, which contains the ID.

In my docker-compose setup, I have configured it as follows:

services:
  airflow:
    build:
      context: .
      dockerfile: Dockerfile.airflow
    container_name: airflow
    environment:
      - AIRFLOW_VAR_PLAYLIST_ID=${PLAYLIST_ID}
    volumes:
...

Now, my Airflow DAG should access the content of the environment variable so that it can use the ID for a request. I am doing it like this and I've shortened the code a bit for better clarity. The access token is also retrieved within the same task.

playlist_id = os.getenv("AIRFLOW_VAR_PLAYLIST_ID")

get_playlist_tracks = BashOperator(
    task_id='get_playlist_tracks',
    bash_command="""

    PLAYLIST_ID=$PLAYLIST_ID

    echo $AIRFLOW_VAR_PLAYLIST_ID
    echo 'PLAYLIST_ID = '$PLAYLIST_ID
    
    curl -X "GET" "https://api.spotify.com/v1/playlists/$PLAYLIST_ID/tracks" \
        -H "Accept: application/json" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        > /user/hadoop/spotify/track_data/raw/playlist_tracks.json
    echo "Playlist tracks fetched and saved to HDFS"
    """,  
    dag=dag 
)

The problem is that it doesn't work properly. I also had this form:

PLAYLIST_ID={{ playlist_id }}

But this also not worked. When I try to display the content of the environment variable, it shows as empty. I've read a lot and tried various solutions, but there's an error somewhere, and I can't find it.

• airflow
• airflow-taskflow

Share Improve this question Follow asked Nov 22, 2024 at 0:36 Holly2207Holly2207 1133 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

AIRFLOW_VAR_PLAYLIST_ID is following the convention for creating Airflow variables via environment variables.

You can retrieve the value for the playlist_id from the var template variable.

get_playlist_tracks = BashOperator(
    task_id='get_playlist_tracks',
    bash_command="""
    PLAYLIST_ID="{{ var.value.playlist_id }}"

    echo "PLAYLIST_ID = $PLAYLIST_ID"
    
    curl -X "GET" "https://api.spotify.com/v1/playlists/$PLAYLIST_ID/tracks" \
        -H "Accept: application/json" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $ACCESS_TOKEN" \
        > /user/hadoop/spotify/track_data/raw/playlist_tracks.json
    echo "Playlist tracks fetched and saved to HDFS"
    """,  
    dag=dag 
)

本文标签： Airflow environment variable is emptyStack Overflow