Facing problems in Azure Cloud evaluation using Python SDK - Stack Overflow

admin管理员组

文章数量:1123159

I am using Azure AI Projects and Evaluation SDK to evaluate my dataset for evaluation purpose. I am able to successfully run the code but facing issues with portal, the evaluation status is just showing "Queued" for almost 1 hour. I have followed the code provided in the document and Azure Python SDK github.

For code reproducibility:

import os, time
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.ai.projects.models import Evaluation, Dataset, EvaluatorConfiguration, ConnectionType
from azure.ai.evaluation import F1ScoreEvaluator, RelevanceEvaluator, ViolenceEvaluator

# Load your Azure OpenAI config
deployment_name = "gpt-4o"
api_version = "2024-08-01-preview"


project_client = AIProjectClient.from_connection_string(
    credential=DefaultAzureCredential(),
    conn_str=""
)


data_id = project_client.upload_file('val.jsonl')

default_connection = project_client.connections.get_default(connection_type=ConnectionType.AZURE_OPEN_AI)


model_config = default_connection.to_evaluator_model_config(deployment_name=deployment_name, api_version=api_version)

evaluators = {
    "relevance": EvaluatorConfiguration(
        id=RelevanceEvaluator.id,
        init_params={"model_config": model_config},
        data_mapping={"query": "${data.Input}", "response": "${data.Output}"},
    ),
}


evaluation = Evaluation(
    display_name="Cloud Evaluation",
    description="Cloud Evaluation of dataset",
    data=Dataset(id=data_id),
    evaluators=evaluators,
)


evaluation_response = project_client.evaluations.create(
    evaluation=evaluation,
)



get_evaluation_response = project_client.evaluations.get(evaluation_response.id)


print("----------------------------------------------------------------")
print("Created evaluation, evaluation ID: ", get_evaluation_response.id)
print("Evaluation status: ", get_evaluation_response.status)
print("AI Foundry Portal URI: ", get_evaluation_response.properties["AiStudioEvaluationUri"])
print("----------------------------------------------------------------")

My val.jsonl looks like:

enter image description here

Issue with Azure Portal:

enter image description here

The cloud is running for more than 1+ hour.

I have tried the azure-ai-samples files as same but didn't get the desired output, also found something similar like this something called Batch evaluation, but currently I am more focused on this approach as this is straight forward.

Need some assistance for this.

Thank you.

本文标签： Facing problems in Azure Cloud evaluation using Python SDKStack Overflow