Creative Writing Assistant: Working with Agents using Promptflow (Python Implementation)

This sample demonstrates how to create and work with AI agents driven by Azure OpenAI. It includes a Flask app that takes a topic and instruction from a user and then calls a research agent that uses the Bing Search API to research the topic, a product agent that uses Azure AI Search to do a semantic similarity search for related products from a vector store, a writer agent to combine the research and product information into a helpful article, and an editor agent to refine the article that's finally presented to the user.

Table of Contents

• Features
• Azure account requirements
• Opening the project
  • GitHub Codespaces
  • VS Code Dev Containers
  • Local environment
    • Prerequisites
    • Initializing the project
• Deployment
• Testing the sample
  • Evaluating prompt flow results
• Costs
• Security Guidelines
• Resources
• Code of Conduct

Features

This project template provides the following features:

• Azure OpenAI to drive the various agents
• Prompty and Prompt Flow to create, manage and evaluate the prompt into our code.
• Bing Search API to research the topic provided
• Azure AI Search for performing semantic similarity search

Azure account requirements

IMPORTANT: In order to deploy and run this example, you'll need:

• Azure account. If you're new to Azure, get an Azure account for free and you'll get some free Azure credits to get started. See guide to deploying with the free trial.
• Azure subscription with access enabled for the Azure OpenAI service. You can request access with this form. If your access request to Azure OpenAI service doesn't match the acceptance criteria, you can use OpenAI public API instead.
  • Ability to deploy gpt-35-turbo-0613, gpt-4-1106-Preview, and gpt-4-0613.
  • We recommend using France Central, as this region has access to all models and services required.
• Azure subscription with access enabled for Bing Search API
• Azure subscription with access enabled for Azure AI Search

Opening the project

You have a few options for setting up this project. The easiest way to get started is GitHub Codespaces, since it will setup all the tools for you, but you can also set it up locally.

GitHub Codespaces

1. You can run this template virtually by using GitHub Codespaces. The button will open a web-based VS Code instance in your browser:

   

2. Open a terminal window.

3. Sign in to your Azure account:

   azd auth login

4. Provision the resources and deploy the code:

   azd up

   This project uses gpt-35-turbo-0613, gpt-4-1106-Preview and gpt-4-0613which may not be available in all Azure regions. Check for up-to-date region availability and select a region during deployment accordingly. For this project we recommend France Central.

5. Install the necessary Python packages:

   src/api
   pip install -r requirements.txt
   

Once the above steps are completed you can jump straight to testing the sample.

VS Code Dev Containers

A related option is VS Code Dev Containers, which will open the project in your local VS Code using the Dev Containers extension:

1. Start Docker Desktop (install it if not already installed)

2. Open the project:

   

3. In the VS Code window that opens, once the project files show up (this may take several minutes), open a terminal window.

4. Install required packages:

   cd src/api
   pip install -r requirements.txt

   Once you've completed these steps jump to deployment.

Local environment

Prerequisites

• Azure Developer CLI (azd)
• Python 3.10+
• Docker Desktop
• Git

Note for Windows users: If you are not using a container to run this sample, our hooks are currently all shell scripts. To provision this sample correctly while we work on updates we recommend using git bash.

Initializing the project

1. Create a new folder and switch to it in the terminal, then run this command to download the project code:

   azd init -t agent-openai-python-prompty

   Note that this command will initialize a git repository, so you do not need to clone this repository.

2. Install required packages:

   cd src/api
   pip install -r requirements.txt

Deployment

Once you've opened the project in Codespaces, Dev Containers, or locally, you can deploy it to Azure.

1. Sign in to your Azure account:

   azd auth login

   If you have any issues with that command, you may also want to try azd auth login --use-device-code.

   This will create a folder under .azure/ in your project to store the configuration for this deployment. You may have multiple azd environments if desired.

2. Provision the resources and deploy the code:

   azd up

   This project uses gpt-35-turbo-0613 and gpt-4-1106-Preview which may not be available in all Azure regions. Check for up-to-date region availability and select a region during deployment accordingly. We recommend using East US 2 for this project.

   After running azd up, you may be asked the following question during Github Setup:

   Do you want to configure a GitHub action to automatically deploy this repo to Azure when you push code changes?
   (Y/n) Y

   You should respond with N, as this is not a necessary step, and takes some time to set up.

Testing the sample

This sample repository contains an agents folder that includes subfolders for each agent. Each agent folder contains a prompty file where the agent's prompty is defined and a python file with the code used to run it. Exploring these files will help you understand what each agent is doing. The agent's folder also contains an orchestrator.py file that can be used to run the entire flow and to create an article. When you ran azd up a catalogue of products was uploaded to the Azure AI Search vector store and index name contoso-products was created.

To test the sample:

1. Run the example web app locally using a Flask server.

   First navigate to the src/api folder

   cd ./src/api
   

   Run the Flask webserver

   flask --debug --app api.app:app run --port 8080
   

   If you open the server link in a browser, you will see a URL not found error, this is because we haven't created a home url route in flask. We have instead created a /get_article route which is used to pass context and instructions directly to the get_article.py file which runs the agent workflow.

   We have created a web interface which we will run next, but you can test the API is working as expected by running this in the browser:

   http://127.0.0.1:8080/get_article?context=Write an article about camping in alaska&instructions=find specifics about what type of gear they would need and explain in detail
   

2. Once the flask server is running you can now run the web app. To do this open a new terminal window and navigate to the web folder using this command:

   cd src/web
   

   First install node packages:

   npm install
   

   Then run the web app with a local dev web server:

   npm run dev
   

   This will launch the app, where you can use example context and instructions to get started. On the 'Creative Team' page you can examine the output of each agent by clicking on it. The app should look like this:

   The getting started tab to send your instructions and context to the prompt:

   

   The creative team tab that lets you follow and understand the agent's workflow:

   

   The document tab that displays the article that was created:

   

   Change the instructions and context to create an article of your choice.

3. For debugging purposes you may want to test in Python using the orchestrator Logic

   To run the sample using just the orchestrator logic use the following command:

   cd ./src/api
   python -m api.agents.orchestrator
   
   

Evaluating prompt flow results

To understand how well our prompt flow performs using defined metrics like groundedness, coherence etc we can evaluate the results. To evaluate the prompt flow, we need to be able to compare it to what we see as "good results" in order to understand how well it aligns with our expectations.

We may be able to evaluate the flow manually (e.g., using Azure AI Studio) but for now, we'll evaluate this by running the prompt flow using gpt-4 and comparing our performance to the results obtained there. To do this, follow the instructions and steps in the notebook evaluate-chat-prompt-flow.ipynb under the eval folder.

You can also view the evaluation metrics by running the following command from the src/api folder.

Run evaluation:

python -m api.evaluate.evaluate

Setting up CI/CD with GitHub actions

This template is set up to run CI/CD when you push changes to your repo. When CI/CD is configured, evaluations will in GitHub actions and then automatically deploy your app on push to main.

To set up CI/CD with GitHub actions on your repository, run the following command:

azd pipeline config

Workaround to enable CI/CD: in the output, look for the "Creating service principal" line and Copy the name of the service principal (looks like az-dev-), an example is shown below:

...
(✓) Done: Creating service principal az-dev-05-19-2024-02-02-02 (62c51d5e-17f7-4b06-bfab-fza2a4e1e6d8)
...

This will create a service principal in your Azure subscription for your GitHub actions to use when running azd and evaluations.

You will need to add a role assignment of this service principal on your tfstate storage account. Open the Azure portal, and take the following steps:

1. In the search box at the top of the screen, search for ENVIRONMENT_NAME-tfstate
2. Select the resource group that appears
3. Click to open the storage account in that resource group
4. Select the Access control (IAM) on the left nav
5. Then click Add > Role Assignment
6. Search for the Storage Blob Data Contributor role, and select Next
7. Click + Select Members, and add the az-dev- role that you copied earlier
8. Click Review + Assign twice

Currently, when your GitHub action runs, it will remove your access to call the OpenAI and AI Search resources from your development environment with your user identity. It essentially takes over this azd environment for CI/CD purposes, and if you want to continue working on this repo you should create a new development environment.

If you do want to re-enable access from your development environment, re-run the provision command:

azd provision

Costs

Pricing may vary per region and usage. Exact costs cannot be estimated. You may try the Azure pricing calculator for the resources below:

• Azure Container Apps: Pay-as-you-go tier. Costs are based on vCPU and memory used. Pricing
• Azure OpenAI: Standard tier, GPT, and Ada models. Pricing per 1K tokens used, and at least 1K tokens are used per question. Pricing
• Azure Monitor: Pay-as-you-go tier. Costs based on data ingested. Pricing

Security Guidelines

This template uses Managed Identity built in to eliminate the need for developers to manage these credentials. Applications can use managed identities to obtain Microsoft Entra tokens without having to manage any credentials. We also use Key Vault, specifically for Bing Search, since Managed Identity is currently not implemented for it. Additionally, we have added a GitHub Action tool that scans the infrastructure-as-code files and generates a report containing any detected issues. To ensure best practices in your repo we recommend anyone creating solutions based on our templates ensure that the Github secret scanning setting is enabled in your repos.

Resources

• Promptflow/Prompty Documentation
• Develop Python apps that use Azure AI services

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct.

Resources:

• Microsoft Open Source Code of Conduct
• Microsoft Code of Conduct FAQ
• Contact opencode@microsoft.com with questions or concerns

For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Responsible AI Guidelines

This project follows below responsible AI guidelines and best practices, please review them before using this project:

• Microsoft Responsible AI Guidelines
• Responsible AI practices for Azure OpenAI models
• Safety evaluations transparency notes