google cloud platform - How to pass {{job.trigger.time.iso_date}} - 1 day as a Databricks pipeline parameter? - Stack Overflow

admin管理员组

文章数量:1402964

I'm running a Databricks Workflow and using {{job.trigger.time.iso_date}} to pass the job execution date as a parameter. However, I need to pass the previous day's date instead (i.e., {{job.trigger.time.iso_date}} - 1 day). I tried adjusting it inside a notebook, but I want to set it at the pipeline level instead. How can I modify the workflow to subtract one day before passing the parameter?

What I Tried:

Using {{job.trigger.time.iso_date}} directly in the workflow parameters.

    Expected: It would allow me to subtract one day.

    Result: The syntax does not support direct date arithmetic.

Attempted to adjust it inside the notebook using Python:

from datetime import datetime, timedelta job_date = dbutils.widgets.get("job_date") # Passed from workflow yesterday = (datetime.strptime(job_date, "%Y-%m-%d") - timedelta(days=1)).strftime('%Y-%m-%d') print(yesterday)

Expected: I could compute yesterday in the notebook.

Result: This works, but I want to adjust it before execution, not inside the notebook.

Tried to use {{job.trigger.time.iso_date}} in combination with an expression in workflow parameters.

Expected: Some way to compute -1 day at the workflow level.

Result: Couldn’t find syntax that supports it.

Looked for Task Output Variables as a workaround.

Thought about creating a Python task to compute the date and pass it as an output variable to the next task.

But is there a better way to do this at the pipeline level?

Checked Databricks documentation but found no direct solution.

Does Databricks have built-in support for adjusting {{job.trigger.time.iso_date}}?

I'm running a Databricks Workflow and using {{job.trigger.time.iso_date}} to pass the job execution date as a parameter. However, I need to pass the previous day's date instead (i.e., {{job.trigger.time.iso_date}} - 1 day). I tried adjusting it inside a notebook, but I want to set it at the pipeline level instead. How can I modify the workflow to subtract one day before passing the parameter?

What I Tried:

Using {{job.trigger.time.iso_date}} directly in the workflow parameters.

    Expected: It would allow me to subtract one day.

    Result: The syntax does not support direct date arithmetic.

Attempted to adjust it inside the notebook using Python:

from datetime import datetime, timedelta job_date = dbutils.widgets.get("job_date") # Passed from workflow yesterday = (datetime.strptime(job_date, "%Y-%m-%d") - timedelta(days=1)).strftime('%Y-%m-%d') print(yesterday)

Expected: I could compute yesterday in the notebook.

Result: This works, but I want to adjust it before execution, not inside the notebook.

Tried to use {{job.trigger.time.iso_date}} in combination with an expression in workflow parameters.

Expected: Some way to compute -1 day at the workflow level.

Result: Couldn’t find syntax that supports it.

Looked for Task Output Variables as a workaround.

Thought about creating a Python task to compute the date and pass it as an output variable to the next task.

But is there a better way to do this at the pipeline level?

Checked Databricks documentation but found no direct solution.

Does Databricks have built-in support for adjusting {{job.trigger.time.iso_date}}?

• google-cloud-platform
• azure-data-factory
• pipeline
• azure-databricks

Share Improve this question Follow asked Mar 27 at 13:27 naveennaveen 1 1

• Databricks Workflows does not priviledge to do date_add or date_sub in pipeline level. But we can use other external orchestration tools to fetch Yesterdays Date. – Rakesh Govindula Commented Mar 28 at 12:18

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

As you want to adjust the value of job_date before execution and not inside notebook.

Databricks workflow does not provide that much priviledge to use arithmetic functions like date_add or date_sub inside task parameter. For that you can use other external orchestration tools which has greater flexibility.

You can use Azure Data Factory to calculate the previous date using expression and send it as a parameter in your Databricks notebook. Please follow the below steps:

Step 1: You need to create a notebook with below code:

dbutils.widgets.text("job_date", "")
Yesterday = dbutils.widgets.get("job_date")

print(f"Received job_date: {Yesterday}")
print('Pipeline Triggered')

Step 2: Now you need to create pipeline and use Databricks Notebook Activity in ADF.

Step 3: Navigate to settings inside Notebook Activity and provide the Notebook path of Databricks and create a base parameter with name job_date use the following expression: @formatDateTime(addDays(utcNow(), -1), 'yyyy-MM-dd')

Step 4: Debug the pipeline and wait for the pipeline to get succeeded and get the output by clicking on details link below:

Output : Now you can use the yesterday variable according your requirement.

Please Refer the Microsoft Document :

Azure Databricks - Dynamic Value Reference

本文标签：