izikeros

Synthetic Data Generation & RAG Evaluation: RAGAS + LangSmith

Introduction

Retrieval-Augmented Generation (RAG) has emerged as a powerful approach for enhancing Large Language Models (LLMs) with external knowledge. However, evaluating RAG pipelines presents significant challenges due to the complexity of retrieval quality, generation accuracy, and the overall coherence of responses. This document provides a comprehensive analysis of using RAGAS (Retrieval Augmented Generation Assessment) for synthetic test data generation and LangSmith for RAG pipeline evaluation, based on the Jupyter notebook example provided.

What is RAG?

Retrieval-Augmented Generation is a technique that enhances LLMs by providing them with relevant external knowledge. A typical RAG system consists of two main components[1]:

izikeros

"""Demo for displaying colored text in Jupyter Notebook."""

import matplotlib
import matplotlib.pyplot as plt
from IPython.display import HTML, display


def importance_to_color(importance):
    cmap = plt.get_cmap("Greys")  # YlOrRd, Greys
    rgba = cmap(importance)

izikeros

# Sets the default shell for executing commands as /bin/bash and specifies command should be executed in a Bash shell.
SHELL := /bin/bash

# Color codes for terminal output
COLOR_RESET=\033[0m
COLOR_CYAN=\033[1;36m
COLOR_GREEN=\033[1;32m

SRC_FILES = src
SRC_AND_TEST_FILES = src tests

izikeros

"""
Usage example:

@timer_decorator
def sum_list(numbers):
    return sum(numbers)

numbers = list(range(1, 1000000))  # Create a list of numbers from 1 to 999999
result = sum_list(numbers)
print(f"Sum: {result}")

izikeros

; this is not a complete tox.ini file but just one env!!! 

[testenv:coverage]
; it requires 'setup.py' or 'pyproject.toml' to have package installable.
deps = -rrequirements.txt
       -rtox-reqs/coverage.txt
; usedevelop - fix problems with 0% coverage https://stackoverflow.com/a/69462954/3247880
usedevelop=True
commands =
    pytest --cov trend_classifier/ --cov-report html --cov-fail-under=70 tests/

izikeros

name: pytest

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

izikeros

# This workflow will upload a Python Package using Twine when a release is created
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Upload Python Package

izikeros

name: Run Tests with Tox

on:
  push:
    branches:
      - '*'  # This will trigger the action on push to any branch

jobs:
  test:
    runs-on: ubuntu-latest

izikeros

---
name: Check links in README
# from: unixorn/git-extra-commands

on:
  push:
    branches: [ '*' ]
  pull_request:
    branches: [ '*' ]

izikeros

import io

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.animation import FuncAnimation
from PIL import Image


def create_plot(i):
    plt.clf()