pamelafox

Evaluation:

• ai-rag-chat-evaluator: Repo with scripts to generate sample data based on Azure AI search index and evaluate responses from a chat app using GPT-4 model and custom metrics. Designed for use with azure-search-openai-demo but can be modified for use with other chat apps.
  • 📺 Quick video overview
  • 📺 Talk: Automated evaluations of LLM apps: Talk for Data Science Day. Slides also available.
  • 📝 Blog post: Evaluating a RAG Chat App - Approach, SDKs, Tools
• [📝 Blog post: Can your app say "I don't know?"](https://blog.pamelafox.org/2024/03/evaluating-rag-chat-apps-can-y

pamelafox

import os
import json

from dotenv import load_dotenv
from openai import AzureOpenAI
import requests

load_dotenv(".env", override=True)

pamelafox

import json
import logging
import re
from typing import Any, AsyncGenerator, Optional, Union

import aiohttp
import openai
from azure.search.documents.aio import SearchClient
from azure.search.documents.models import QueryType

pamelafox

import os
from collections import namedtuple

import aiohttp
import pytest
from azure.core.exceptions import ResourceNotFoundError
from azure.core.pipeline.transport import (
    AioHttpTransportResponse,
    AsyncHttpTransport,
    HttpRequest,

pamelafox

{
    "html.autoCreateQuotes": false,
    "editor.autoClosingBrackets": "never",
    "html.autoClosingTags": false,
    "cmake.configureOnOpen": true,
    "editor.quickSuggestions": {
        "other": "off"
    },
    "azureFunctions.showPysteinModel": true,
    "workbench.colorTheme": "Solarized Light",

pamelafox

• Codespaces Jupyter template
• GitHub Codespaces documentation
• Alfredo's Introduction to Codespaces
• Codespaces teaching template
• Blog post: How to optimize Codespaces quotas

pamelafox

FROM python:3.11.2-slim

# Install Git
RUN apt-get update && apt-get install -q -y --fix-missing \
	git && \
	rm -rf /var/lib/apt/lists/*

#[Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
COPY .devcontainer/requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \

pamelafox

https://tinyurl.com/aifair-links

https://aka.ms/oaiapply

ChatGPT with Cognitive Search (Manual):

• https://github.com/Azure-Samples/azure-search-openai-demo

ChatGPT with Cognitive Search Integrated: - PREVIEW

• https://github.com/Microsoft/sample-app-aoai-chatGPT/

pamelafox

name: Axe analysis

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

pamelafox

response=$(gh api \
  --method POST \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  /repos/pamelafox/pamelafox-site/code-scanning/sarifs \
  -f commit_sha='d9b327639e4e84337b9edd297a8cfb098bbfeefa' \
 -f ref='refs/heads/a11y' \
 -f sarif=$(gzip -c src/tests/axe_results.sarif | base64 -b 0))

url=$(echo $response | grep -o '"url": *"[^"]*"' | cut -d '"' -f 4)