LLMs in Production: Observability, Monitoring, and Evaluation

Simone Civetta, expert at Hymaïa, shares his experience on the challenges of LLMs in production and the importance of observability. Through the story of Philip and his chatbot Zootbot, he explores the common pitfalls encountered when deploying LLM-based applications -- until the day the chatbot started recommending Swedish meatballs instead of the company's own products.

This talk answers three key questions:
- How do you ensure traceability and monitoring of your LLM applications in production?
- Why is prompt versioning crucial for collaboration and iteration?
- What are the benefits of evaluation (offline and online) to guarantee the expected behavior of your LLMs?

Practical solutions to ensure the reliability and performance of your LLM-based systems.