TLDR
Worked with Ragas as an AI & DX Consultant while the team launched its experiments-first SDK.
I authored end-to-end eval cookbooks and built a docs AI agent that checks SDK changes and proposes documentation updates.
The Problem
Ragas is an open-source evals framework with 14K+ GitHub stars and more than 1M monthly PyPI downloads.
The new SDK shifted from ready-made metrics toward an experiments-first workflow. Teams needed concrete examples showing how to design, run, and improve evaluations in real applications.
Documentation also had to keep up as the SDK changed.
What I Built
End-to-end eval cookbooks
- Text-to-SQL evaluation
- Improving RAG systems with MLflow
- Aligning an LLM-as-judge with human feedback
- Model benchmarking
Docs AI agent
The agent reviews SDK changes, identifies documentation that may be affected, and proposes updates. A human still reviews the changes before they are merged.
I also shipped:
llms.txtgeneration for LLM-friendly documentation- A copy-to-LLM workflow for moving documentation into AI assistants
Related
- AI Consulting: More about my consulting work