How to trigger an Azure Machine Learning Studio notebook from a Databricks notebook? - Stack Overflow

admin管理员组

文章数量:1202394

I have a notebook in Azure Machine Learning Studio Workspace, and I have also created a compute cluster in the workspace. Now, I want to trigger this notebook from a Databricks notebook.

I have already created a minimal configuration cluster in Databricks, and I have set up a Databricks notebook as well. However, I am unsure how to trigger the main notebook located in Azure ML Studio workspace from the Databricks notebook.

What I have tried so far:

• I created a compute cluster in Azure ML workspace.
• I have set up a Databricks cluster and notebook.
• I am looking for a way to invoke or trigger the Azure ML notebook from the Databricks notebook programmatically.

My environment:

• Azure Machine Learning Studio
• Databricks
• Azure ML compute cluster

Question: How can I trigger an Azure ML Studio notebook from a Databricks notebook? What would be the best way to do this using the available APIs or services in Azure?

I have a notebook in Azure Machine Learning Studio Workspace, and I have also created a compute cluster in the workspace. Now, I want to trigger this notebook from a Databricks notebook.

I have already created a minimal configuration cluster in Databricks, and I have set up a Databricks notebook as well. However, I am unsure how to trigger the main notebook located in Azure ML Studio workspace from the Databricks notebook.

What I have tried so far:

• I created a compute cluster in Azure ML workspace.
• I have set up a Databricks cluster and notebook.
• I am looking for a way to invoke or trigger the Azure ML notebook from the Databricks notebook programmatically.

My environment:

• Azure Machine Learning Studio
• Databricks
• Azure ML compute cluster

Question: How can I trigger an Azure ML Studio notebook from a Databricks notebook? What would be the best way to do this using the available APIs or services in Azure?

• azure
• azure-databricks
• azure-machine-learning-service
• azureml-python-sdk
• rnotebook

Share Improve this question Follow asked Jan 21 at 6:35 Sadman SaadatSadman Saadat 2133 bronze badges 7

• Did you tried any api to trigger? – JayashankarGS Commented Jan 21 at 6:36
• Actually the notebook in ml studio is for purely development purpose and not for job. if you want to run any kind job you create job/pipeline for it and trigger. – JayashankarGS Commented Jan 21 at 6:39
• What exactly you want to run in azure ml notebook? whatever it is, you can directly run it in dbx notebook. – JayashankarGS Commented Jan 21 at 6:42
• I can access my Azure ML workspace by using the subscription_id, resource_group, and workspace_name credentials and also can initiate the compute cluster in my Azure ML workspace( compute_target.start(wait_for_completion=True) ). But after that I cannot trigger the main notebook in Azure ML studio. I tried the ScriptRunConfig to trigger the notebook and also access the notebook by using the papermill library. But no luck :) I am not sure if this is the standard way to do this. I am totally new in this field – Sadman Saadat Commented Jan 21 at 6:46
• I am running a ML training job in Azure ML – Sadman Saadat Commented Jan 21 at 6:47

 |  Show 2 more comments

1 Answer 1

Sorted by: Reset to default 0

You create a pipeline with python function components for your current training script.

You check how to build a pipeline with python function components here.

After you are creating the pipeline, you publish it.

Below is the sample one i created with 3 components.

After successfully run you publish it as shown in above image.

After publishing you will get the pipeline endpoints.

and

Copy that rest endpoint and use below code.

from azure.identity import DefaultAzureCredential
import requests

credential = DefaultAzureCredential()
token  = credential.get_token("https://management.azure.com/.default").token

response = requests.post("<Endpoint_you_copied_earlier>",
                         headers={"Authorization":f"Bearer {token}","Content-Type": "application/json"},
                         json={"ExperimentName": "My_Pipeline"})

Here, for authentication i used default credential you can use any of the way mentioned here.

Refer this for more about running published pipeline.

本文标签： How to trigger an Azure Machine Learning Studio notebook from a Databricks notebookStack Overflow