Julien Bouquillon revolunet

GitOps for machine learning aka MLOps

For open-source projects, GitHub could provide the workflow and runners for total automation.

wandb looks used widely for reporting.

https://github.blog/enterprise-software/ci-cd/streamlining-your-mlops-pipeline-with-github-actions-and-arm64-runners/

repos:

Full GitHub CI/CD : https://github.com/BenedictusAryo/Simple-MLOps-Configuration/

File upload workflow

graph TD

browser-->|POST file|API
API-->Parser[Parser : html,md,pdf...]-->|content,metadata|Chunker
Chunker(LangchainRecursiveCharacterTextSplitter)-->Search

test mermaid charts

see all on https://mermaid.live

Pie

pie title Pets adopted by volunteers
    "Dogs" : 386
    "Cats" : 85

	#
	# for some reason this give "neither the id_token nor the profileURL set an email"
	#
	#
	#
	# https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview
	#
	OAUTH2_PROXY_HTTP_ADDRESS: "0.0.0.0:4180"
	OAUTH2_PROXY_WHITELIST_DOMAINS: "*.kube-dev.incubateur.net"
	OAUTH2_PROXY_DISPLAY_NAME: "test-pc provider"

	const query =
	"Fais-moi un texte de RAP à la gloire de betagouv et formattes le en markdown";

	const res = fetch(
	"https://api.scaleway.ai/xxx",
	{
	method: "POST",
	headers: {
	Authorization: "Bearer xxx",
	"Content-Type": "application/json",

	// @ts-check

	// run in your browser or with `node internet.mjs` with the ALBERT_API_KEY set below
	// see also: https://albert.api.etalab.gouv.fr/documentation

	const API_KEY = process.env.ALBERT_API_KEY;

	const API_URL = "https://albert.api.etalab.gouv.fr";
	const LANGUAGE_MODEL = "AgentPublic/llama3-instruct-8b"; // see https://albert.api.etalab.gouv.fr/v1/models

	// @ts-check

	// see also: https://albert.api.etalab.gouv.fr/documentation

	import { readFile, readdir } from "fs/promises";
	import pAll from "p-all";

	const API_KEY = process.env.ALBERT_API_KEY;

	const API_URL = "https://albert.api.etalab.gouv.fr";

	copy(Array.from($0.querySelectorAll("li.down .url")).map(li =>
	`${li.querySelector(".link a.internal").textContent}\t${li.querySelector("div.status:first-child").textContent}`
	).join("\n"))

	# run query from a plain prompt text file

	jq -Rs '{ stream: false, model: "llama3.2", prompt: . }' ./some.prompt \| curl -X POST http://127.0.0.1:11434/api/generate --data-binary @- \| jq -r '.response'