Evidently AI is an open-source evaluation and observability framework for testing and monitoring large language models, RAG applications, AI agents, and predictive machine-learning models. Teams can place quality, safety, and reliability checks in development pipelines and inspect results through shareable reports or a continuously updated dashboard. It spans both generative-AI output evaluation and conventional ML concerns such as data drift, data quality, and predictive performance.
The framework is designed for data scientists, machine-learning engineers, and AI product teams building or operating production systems. They can start with built-in metrics or combine rules, classifiers, LLM-based judges, and custom evaluations for their own quality system; the open-source framework is distributed under the Apache 2.0 license. Meaningful monitoring requires evaluation data, logs, reference expectations, or team-defined rules, and detected changes still need to be interpreted and acted on within the relevant business and risk context.