Azure AI Studio: How to evaluate and upgrade your models, using the Prompt Flow SDK

On this page

Before you upgrade a model deployment in the Azure OpenAI Service, run the new model version and your current one against the same evaluation dataset and compare the scores. If the new version doesn't do at least as well on your data, you've found a regression before your users did.

This tutorial shows how to do that with the Microsoft Prompt Flow SDK and Azure AI Studio Evaluations. We deploy GPT-4o as a "judge" model, evaluate GPT-4o and GPT-35-turbo on a small question-answer dataset, and compare the two runs in the Azure AI Studio evaluation dashboard. In our example, GPT-4o came out ahead on almost every metric.

Why bother? New model versions keep arriving, often faster, cheaper or safer than the last one, and old versions get retired. You will have to upgrade at some point. The real question is whether the new model improves your application or introduces regressions.

We'll cover:

• Using Azure AI Studio Evaluations to assess new model versions
• Comparing models in code and in the Azure AI Studio UI
• Best practices for upgrading your deployments

What is Azure AI Studio?

Azure AI Studio is a platform in Microsoft Azure for building and deploying AI-powered applications. It has tools for building generative AI applications for developers with different levels of experience, and you can either use existing AI models or customize them with your own data.

That flexibility lets organizations start with pre-built models and tailor them with proprietary data later. Azure AI Studio also supports team collaboration and integrates with development tools like GitHub and Visual Studio.

The main selling point at the moment is the integration of OpenAI's models. Most OpenAI models can be managed and provisioned through Azure AI Studio. You can add content filters, deploy models, log metrics and, as this post shows, evaluate and upgrade models with the Prompt Flow SDK.

What is the Microsoft Prompt Flow SDK?

The Microsoft Prompt Flow SDK is a toolkit for developing applications based on large language models (LLMs). It covers the lifecycle of an LLM app, from the first prototype to production deployment and ongoing monitoring.

Prompt Flow SDK features

For prototyping, Prompt Flow lets you experiment with different prompt configurations to optimize model outputs. You can also put Python code directly into your workflows for custom logic and data manipulation next to your prompts.

For testing, it provides unit tests for LLM prompts, which catch regressions when prompts or the underlying models change. Its evaluation features measure how well different prompts and configurations perform. This testing and evaluation part is what we use in this tutorial.

For deployment, the SDK integrates with CI/CD pipelines and comes with monitoring and logging for deployed applications. It supports version control and shared environments for teams, and you can extend it with custom plugins and third-party integrations.

The benefit of all this is one toolkit for every stage of LLM app development. That shortens the path from prototype to production, and the built-in tests and evaluations help keep your application reliable, whether it's a small prototype or a large production system.

Why upgrade your Azure OpenAI models?

Before the hands-on part, let's step back. Why upgrade models at all, and why do you need an automated process to validate upgrades? There are two reasons.

First, models get better very quickly. The OpenAI model changelog shows the pace:

• GPT-3.5 was released in March 2022
• GPT-3.5-turbo was released in November 2022
• GPT-4 was released in March 2023
• GPT-4-turbo was released in November 2023
• GPT-4o was released in May 2024

That's a new model roughly every 6 months, and the list doesn't include minor versions like GPT-4o-mini, GPT-3.5-turbo-16k or GPT-3.5-turbo-0125. Each of these newer models offered much better performance or 10x lower cost and latency than the previous one, so not upgrading was almost negligent.

Second, Microsoft and other model providers retire older models aggressively as new ones arrive. Microsoft has fairly transparent model deprecation policies, and one of the main statements is: models can be retired 1 year after their initial release. So whatever you think about upgrading, a year after a model's release you have to.

What to evaluate when upgrading a model

The authors of the Prompt Flow SDK suggest three groups of evaluation criteria. Performance and quality metrics are groundedness, relevance, coherence, fluency, similarity and F1 score. Risk and safety metrics assess violence, sexual content, self-harm and hate/unfairness. Composite metrics combine evaluations for question-answer pairs or chat messages, and for content safety.

Here is what each metric measures:

• Groundedness: how well the model's output aligns with factual information.
• Relevance: how relevant the response is to the input prompt.
• Coherence: the logical flow of the response ("does the output make sense?").
• Fluency: the linguistic quality and naturalness of the response.
• Similarity: how similar the model output is to a reference text.
• F1 score: used for evaluations based on precision (the accuracy of the positive predictions) and recall (the ability to find all positive instances).
• Violence: content with violent or aggressive language.
• Sexual content: content with sexual references or inappropriate language.
• Self-harm: whether the content encourages or discusses self-harm.
• Hate/unfairness: the presence of hateful, biased or discriminatory language.
• Question-answer pairs: accuracy and completeness of the model's responses in a Q&A context.
• Content safety: a combination of the risk and safety metrics that rates overall content safety.

That's a lot to validate, and not every metric is relevant for every use case. Pick the ones that match what your application does.

Hands-on: evaluating models with Azure AI Studio and Prompt Flow

The rest of this post walks through a complete evaluation of two model versions with Azure AI Studio and the Prompt Flow SDK.

Prerequisites for the Prompt Flow evaluation

Prepare the following before you start:

1. Sign up for Microsoft Azure AI Studio and create a project, as outlined here.

   Azure AI Studio "Create a project" dialog, review and finish step

   Note: Pay attention to step 9 of that guide: "Select an existing Azure AI services resource (including Azure OpenAI) from the dropdown or create a new one." This tutorial needs an Azure OpenAI service connected to your project. Either select an existing one or create a new one.

   You'll also need the Azure CLI installed on your computer. Run az login to authenticate your Azure account and select the subscription you want to use.

2. Install the Prompt Flow evaluation and Azure packages:

   1pip install promptflow-evals promptflow-azure
   

3. Open the "Settings" page of your Azure AI Studio project and note down the following information:

   • project name
   • resource group name
   • subscription id

   Azure AI Studio project settings page

4. Next, deploy the AI models to evaluate, plus one model that acts as "judge" for the evaluation.

   This is critical: Prompt Flow uses an LLM to judge the other LLMs. Use the model with the best reasoning quality you have as the judge. Otherwise, a weak judge model can distort the evaluation results.

   Click on "Model catalog" in the Azure AI Studio left-hand menu, then select "GPT-4o".

   Azure AI Studio model catalog with GPT-4o

   On the next screen, click "Deploy" and a modal opens. Set the options as you see fit. For GPT-4o, the settings below worked well at the time of writing. In the "Connected Azure OpenAI resource" dropdown, select the service you connected or created in step 1 (during project creation).

   Deploy model dialog for gpt-4o with Global Standard deployment

   Hit "Deploy" to provision the model.

   Note: This model acts as the judge for the evaluation. If you just want to follow along and don't have a "real" model to validate yet, repeat the steps above and deploy a second model, for example GPT-35-TURBO.

5. Click "Deployment" in the sidebar menu and select the model you just created. The next screen gives an overview of the deployed model. The important part for us is the "Endpoints" section: note down both the endpoint URL and the endpoint key.

6. Now create the model configurations in Python. Create an object for the Azure AI project as follows, using the values from step 3.

   1azure_ai_project = {
   2  "subscription_id": "<your subscription id>",
   3  "resource_group_name": "<name of your resource group>",
   4  "project_name": "<name of azure ai studio project>"
   5}
   

   Then create a model configuration object for the judge model. It needs the azure_endpoint, api_key, api_version and azure_deployment, all of which come from the deployment screen in step 5.

   The api_key is simply the endpoint key from the deployment screen.

   For the other values, break down the endpoint URL, which is structured like this: https://<deployment>.openai.azure.com/openai/deployments/<model_name>/<model_type>?api-version=<api_version> Example: https://openai-docusearch.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-02-01

   For the example above, the values are:

   • azure_endpoint: https://openai-docusearch.openai.azure.com
   • api_version: 2024-02-01
   • azure_deployment: gpt-4o

   1from promptflow.core import AzureOpenAIModelConfiguration
   2judge_config = AzureOpenAIModelConfiguration(
   3    azure_endpoint="https://openai-docusearch.openai.azure.com",
   4    api_key="<api-key>",
   5    api_version="2024-02-01",
   6    azure_deployment="gpt-4o",
   7)
   

   Finally, define the endpoints of the models you want to evaluate. For each model, you need two pieces of information:

   1. The model API endpoint
   2. The API key

   If the models are deployed in Azure AI Studio, use the endpoint and API key from the deployment screen. For other API providers, check their documentation.

   1gpt_4o = {
   2    "api_endpoint": "https://openai-docusearch.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-02-01",
   3    "api_key": "<api-key>"
   4}
   5
   6gpt_35_turbo = {
   7    "api_endpoint": "https://openai-docusearch.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2024-02-01",
   8    "api_key": "<api-key>"
   9}
   

Azure permissions for Prompt Flow

This isn't in the official documentation: Prompt Flow uploads your code, the sample data and the evaluation results to an Azure blob storage account that is created automatically along with the project.

Provisioning the Azure AI Studio project also creates a service principal with the same name as the project (blog in our example).

Prompt Flow uses this service principal to upload files to the blob storage. By default, it seems to lack a required permission/role, which results in the following error:

1This request is not authorized to perform this operation using this permission.

If you see this error, follow these steps:

1. Navigate to "Settings" in the Azure AI Studio project and select the "workspaceartifactstore" resource.

   Azure AI Studio settings with the workspaceartifactstore resource

2. On the next screen, click "View in Azure Portal". There, click "Access control (IAM)", "Role Assignments" and then "Add".

   Access control (IAM) role assignments in the Azure Portal

3. In the "Role" tab, select "Storage Blob Data Contributor", then click the "Members" tab.

   Selecting the Storage Blob Data Contributor role

4. Select "Managed Identity" and click "Select members". In the sidebar that opens, select "Azure AI Project" in the managed identity dropdown. Then select the service principal with the same name as the project. Selecting the project's managed identity as a member

5. Click "Select" and then "Review + assign".

Your permissions are now set.

Creating the evaluation dataset for Prompt Flow

Next comes the evaluation dataset. As always when evaluating AI systems, the dataset is the most important part of the whole process.

For Prompt Flow, create a JSONL file with the following fields:

• question: The reference question for the model
• context: Context for the model to answer the question upon
• ground_truth: The correct answer to the question

For this example, we use a simple evaluation dataset kindly provided by the Prompt Flow SDK

1{"question":"What is the capital of France?","context":"France is the country in Europe.","ground_truth":"Paris"}
2{"question": "Which tent is the most waterproof?", "context": "#TrailMaster X4 Tent, price $250,## BrandOutdoorLiving## CategoryTents## Features- Polyester material for durability- Spacious interior to accommodate multiple people- Easy setup with included instructions- Water-resistant construction to withstand light rain- Mesh panels for ventilation and insect protection- Rainfly included for added weather protection- Multiple doors for convenient entry and exit- Interior pockets for organizing small ite- Reflective guy lines for improved visibility at night- Freestanding design for easy setup and relocation- Carry bag included for convenient storage and transportatio## Technical Specs**Best Use**: Camping  **Capacity**: 4-person  **Season Rating**: 3-season  **Setup**: Freestanding  **Material**: Polyester  **Waterproof**: Yes  **Rainfly**: Included  **Rainfly Waterproof Rating**: 2000mm", "ground_truth": "The TrailMaster X4 tent has a rainfly waterproof rating of 2000mm"}
3{"question": "Which camping table is the lightest?", "context": "#BaseCamp Folding Table, price $60,## BrandCampBuddy## CategoryCamping Tables## FeaturesLightweight and durable aluminum constructionFoldable design with a compact size for easy storage and transport## Technical Specifications- **Weight**: 15 lbs- **Maximum Weight Capacity**: Up to a certain weight limit (specific weight limit not provided)", "ground_truth": "The BaseCamp Folding Table has a weight of 15 lbs"}
4{"question": "How much does TrailWalker Hiking Shoes cost? ", "context": "#TrailWalker Hiking Shoes, price $110## BrandTrekReady## CategoryHiking Footwear", "ground_truth": "The TrailWalker Hiking Shoes are priced at $110"}

Create a file named data.jsonl and paste the content above into it.

Creating the model router (Prompt Flow target)

Prompt Flow lets you define a target: a callable Python class that takes the evaluation questions and routes them to the respective model endpoints. This is useful because it means you can evaluate not only different models but also full RAG pipelines.

The target class __call__ method needs to take a question and context parameter and return a dictionary with question and answer keys.

Add this class to a separate file named target.py:

1import requests
2from typing_extensions import Self
3from typing import TypedDict
4from promptflow.tracing import trace
5
6
7class ModelRouter:
8    def __init__(self: Self, model_type: str, model_config: dict):
9        self.model_config = model_config
10        self.model_type = model_type
11
12    class Response(TypedDict):
13        question: str
14        answer: str
15
16    @trace
17    def __call__(self: Self, question: str, context: str) -> Response:
18        if self.model_type == "gpt-4o":
19            output = self.call_gpt4o_endpoint(question, context)
20        elif self.model_type == "gpt-35-turbo":
21            output = self.call_gpt35_turbo_endpoint(question, context)
22        else:
23          raise ValueError("Model type not supported")
24        return output
25
26    def query(self: Self, endpoint: str, headers: str, payload: str) -> str:
27        response = requests.post(url=endpoint, headers=headers, json=payload)
28        return response.json()
29
30    def call_gpt4o_endpoint(self: Self, question: str, context: str) -> Response:
31        endpoint = self.model_config["api_endpoint"]
32        key = self.model_config["api_key"]
33
34        headers = {"Content-Type": "application/json", "api-key": key}
35
36        question = question + \
37        "\n Context: " + context if context else question
38
39        payload = {"messages": [{"role": "user", "content": question}], "max_tokens": 1000}
40
41        output = self.query(endpoint=endpoint, headers=headers, payload=payload)
42        answer = output["choices"][0]["message"]["content"]
43        return {"question": question, "answer": answer}
44
45    def call_gpt35_turbo_endpoint(self: Self, question: str, context: str) -> Response:
46        endpoint = self.model_config["api_endpoint"]
47        key = self.model_config["api_key"]
48
49        headers = {"Content-Type": "application/json", "api-key": key}
50
51        question = question + \
52        "\n Context: " + context if context else question
53
54        payload = {"messages": [{"role": "user", "content": question}], "max_tokens": 1000}
55
56        output = self.query(endpoint=endpoint, headers=headers, payload=payload)
57        answer = output["choices"][0]["message"]["content"]
58        return {"question": question, "answer": answer}

A few things to note about the model router:

1. Note the @trace decorator. Traces in Prompt Flow record specific events or the state of an application during execution, such as function calls, variable values and system events. See here for more information about tracing.

2. The call_gpt4o_endpoint and call_gpt35_turbo_endpoint methods are basically the same and could be a single function. I kept them separate to show how you could implement a different API provider.

3. You can extend the call-endpoint functions for your specific application. You could even add a full RAG retrieval pipeline here, for example one built on Azure AI Search.

Running the Prompt Flow evaluation

All that's left is to define the evaluations and run them.

1. Define the evaluators. For a list of available evaluators see here. You can also create your own evaluators.

   1from promptflow.evals.evaluators \
   2import ContentSafetyEvaluator, RelevanceEvaluator, CoherenceEvaluator, GroundednessEvaluator, FluencyEvaluator, SimilarityEvaluator
   3content_safety_evaluator = ContentSafetyEvaluator(project_scope=azure_ai_project)
   4relevance_evaluator = RelevanceEvaluator(model_config=judge_config)
   5coherence_evaluator = CoherenceEvaluator(model_config=judge_config)
   6groundedness_evaluator = GroundednessEvaluator(model_config=judge_config)
   7fluency_evaluator = FluencyEvaluator(model_config=judge_config)
   8similarity_evaluator = SimilarityEvaluator(model_config=judge_config)
   

2. Next, create the evaluation configuration and run it for each model.

   1import pathlib
   2import random
   3from promptflow.evals.evaluate import evaluate
   4from target import ModelRouter
   5
   6path = str(pathlib.Path(pathlib.Path.cwd())) + "/data.jsonl"
   7models = ["gpt-4o", "gpt-35-turbo"]
   8for model in models:
   9    randomNum = random.randint(111, 999)
   10    results = evaluate(
   11        azure_ai_project=azure_ai_project,
   12        evaluation_name="Eval-Run-"+str(randomNum)+"-"+model.title(),
   13        data=path,
   14        # The model_config refers to the configuration objects we set up some steps before
   15        target=ModelRouter(model, model_config = gpt_4o if model == "gpt-4o" else gpt_35_turbo),
   16        evaluators={
   17            "content_safety": content_safety_evaluator,
   18            "coherence": coherence_evaluator,
   19            "relevance": relevance_evaluator,
   20            "groundedness": groundedness_evaluator,
   21            "fluency": fluency_evaluator,
   22            "similarity": similarity_evaluator,
   23        },
   24        evaluator_config={
   25            "content_safety": {
   26                "question": "${data.question}",
   27                "answer": "${target.answer}"
   28            },
   29            "coherence": {
   30                "answer": "${target.answer}",
   31                "question": "${data.question}"
   32            },
   33            "relevance": {
   34                "answer": "${target.answer}",
   35                "context": "${data.context}",
   36                "question": "${data.question}"
   37            },
   38            "groundedness": {
   39                "answer": "${target.answer}",
   40                "context": "${data.context}",
   41                "question": "${data.question}"
   42            },
   43            "fluency": {
   44                "answer": "${target.answer}",
   45                "context": "${data.context}",
   46                "question": "${data.question}"
   47            },
   48            "similarity": {
   49                "answer": "${target.answer}",
   50                "context": "${data.context}",
   51                "question": "${data.question}"
   52            }
   53        }
   54    )
   

To view the results, print them, or turn them into a pandas DataFrame for better readability.

1import pandas as pd
2
3pd.DataFrame(results["rows"])

Analyzing Prompt Flow results in Azure AI Studio

Because we passed the (optional) Azure AI Studio project configuration, we can view the results of the Prompt Flow evaluations, including charts, in Azure AI Studio.

In Azure AI Studio, select "Tools -> Evaluations" from the left-hand menu. All model evaluation runs that you logged to your project from the SDK show up here.

1. In the Evaluations menu, click "Switch to Dashboard" in the top-right corner. The dashboard gives a better overview for comparing runs.

2. Select one or more runs to compare. Each model we defined in the code above shows up as a separate run. Let's select both the GPT-4o and the GPT-35-TURBO runs. As the screenshot below shows, this gives a clear comparison of all the metrics we defined. In our example, GPT-4o seems to be the better model in almost every regard (which is not too surprising).

   Evaluation dashboard comparing the GPT-4o and GPT-35-TURBO runs

   Scroll down for detailed comparisons of the individual answers of each run.

   Side-by-side answers of each evaluation run

3. Besides comparing runs, you can open each run for its detailed evaluation results. You get charts for each metric, plus the questions, answers and grades for every individual evaluation.

   Detailed evaluation results of a single run

That's the whole setup. It takes quite a few steps, but you end up with a repeatable way to evaluate models before upgrading: add the new model to the model endpoints and evaluate it against your existing baseline model.

An evaluation run checks the model once, before the upgrade. To see how the new version behaves with real user traffic afterwards, trace your production calls with an observability tool such as Langfuse or Opik. If you would rather not set up this evaluation and upgrade process on your own, our AI consulting team in Vorarlberg builds and runs Azure-based RAG and agent applications for companies.

Using Prompt Flow without Azure AI Studio

You can also use Prompt Flow without Azure AI Studio. The process is the same as above, minus the code that refers to the Azure AI Studio project (and the content safety evaluator, which needs it). Here is the complete code for evaluating models without Azure AI Studio.

1from promptflow.core import AzureOpenAIModelConfiguration
2import requests
3import pathlib
4import random
5from typing_extensions import Self
6from typing import TypedDict
7from promptflow.tracing import trace
8from promptflow.evals.evaluate import evaluate
9from promptflow.evals.evaluators \
10import ContentSafetyEvaluator, RelevanceEvaluator, CoherenceEvaluator, GroundednessEvaluator, FluencyEvaluator, SimilarityEvaluator
11
12from target import ModelRouter
13
14import pandas as pd
15
16judge_config = AzureOpenAIModelConfiguration(
17    azure_endpoint="https://openai-docusearch.openai.azure.com",
18    api_key="<api-key>",
19    api_version="2024-02-01",
20    azure_deployment="gpt-4o",
21)
22
23
24gpt_4o = {
25    "api_endpoint": "https://openai-docusearch.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-02-01",
26    "api_key": "<api-key>"
27}
28
29gpt_35_turbo = {
30    "api_endpoint": "https://openai-docusearch.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2024-02-01",
31    "api_key": "<api-key>"
32}
33
34
35relevance_evaluator = RelevanceEvaluator(model_config=judge_config)
36coherence_evaluator = CoherenceEvaluator(model_config=judge_config)
37groundedness_evaluator = GroundednessEvaluator(model_config=judge_config)
38fluency_evaluator = FluencyEvaluator(model_config=judge_config)
39similarity_evaluator = SimilarityEvaluator(model_config=judge_config)
40
41
42path = str(pathlib.Path(pathlib.Path.cwd())) + "/data.jsonl"
43models = ["gpt-4o", "gpt-35-turbo"]
44try:
45  for model in models:
46      randomNum = random.randint(111, 999)
47      results = evaluate(
48          evaluation_name="Eval-Run-"+str(randomNum)+"-"+model.title(),
49          data=path,
50          # The model_config refers to the configuration objects we set up some steps before
51          target=ModelRouter(model, model_config = gpt_4o if model == "gpt-4o" else gpt_35_turbo),
52          evaluators={
53              # "content_safety": content_safety_evaluator,
54              "coherence": coherence_evaluator,
55              "relevance": relevance_evaluator,
56              "groundedness": groundedness_evaluator,
57              "fluency": fluency_evaluator,
58              "similarity": similarity_evaluator,
59          },
60          evaluator_config={
61              "content_safety": {
62                  "question": "${data.question}",
63                  "answer": "${target.answer}"
64              },
65              "coherence": {
66                  "answer": "${target.answer}",
67                  "question": "${data.question}"
68              },
69              "relevance": {
70                  "answer": "${target.answer}",
71                  "context": "${data.context}",
72                  "question": "${data.question}"
73              },
74              "groundedness": {
75                  "answer": "${target.answer}",
76                  "context": "${data.context}",
77                  "question": "${data.question}"
78              },
79              "fluency": {
80                  "answer": "${target.answer}",
81                  "context": "${data.context}",
82                  "question": "${data.question}"
83              },
84              "similarity": {
85                  "answer": "${target.answer}",
86                  "context": "${data.context}",
87                  "question": "${data.question}"
88              }
89          }
90      )
91except Exception as e:
92  print(e)
93
94print(results)
95
96pd.DataFrame(results["rows"])

Further reading

• How to fine-tune GPT-4o mini (for free until September 2024)
• Chat with your confluence using LLMs
• Need a privacy-friendly ChatGPT alternative?
• Building enterprise AI agents with the Microsoft Agent Framework

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

• Understanding the 14 systematic failure modes in multi-agent systems
• Evidence-based best practices for agent design
• Structured communication protocols and verification mechanisms

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.

Talk to Andreas about your projectFollow on LinkedIn