Arize Phoenix Assess
Overview
Arize Phoenix is an open observability and evaluation toolkit for LLM traces, embeddings, and RAG quality metrics (Phoenix).
Assess when you need a self-hostable eval UX without full LangSmith commitment. Integrate traces with OpenTelemetry export for central dashboards.
Adoption Signals
- Active, well-used repository: 11.7k stars and 1.2k forks on GitHub, with server release v20.16.0 published on 23 September 2026 (Phoenix GitHub, Phoenix releases).
- The
arize-phoenixserver package recorded 645,558 PyPI downloads in the month to 30 September 2026 (PyPI Stats); Arize itself reported more than two million monthly Phoenix downloads in February 2025 (Arize Series C announcement). - The vendor is well funded: Arize raised a $70M Series C in February 2025 led by Adams Street Partners, with M12 (Microsoft), Datadog, and PagerDuty participating (Arize Series C announcement).
- Framework vendors ship it as a default target: LlamaIndex lists Phoenix under one-click observability via
set_global_handler("arize_phoenix")(LlamaIndex observability). - The instrumentation layer is a separate Apache-2.0 project, OpenInference, with 40+ Python packages plus JavaScript, Java (LangChain4j, Spring AI, Google ADK), and Go instrumentations usable with any OTel collector (OpenInference).
- Enterprise access controls exist in the free server: OIDC (including Microsoft Entra ID and Keycloak), LDAP/Active Directory with group-to-role mapping, and system and user API keys (Phoenix authentication).
Risks
- Open by default. Phoenix deploys with authentication disabled; enabling it later stops trace collection and blocks API access until API keys are issued, so switching it on in a running pilot interrupts ingestion (Phoenix authentication).
- Traces are personal data. Spans hold prompts, retrieved documents, and model outputs, and the default retention (
PHOENIX_DEFAULT_RETENTION_POLICY_DAYS=0) keeps them forever unless a policy is set globally or per project (Phoenix configuration). - Upgrade churn. Eight major releases between February and August 2026; v14 removed the legacy client and the
/v1/evaluationsendpoint, and v20 changed the assistant'ssession.idformat, so saved filters and dashboards on the assistant's trace project must be rebuilt (Phoenix migration guide). - Growing built-in agent surface. Recent majors added sandboxed code evaluators with external providers such as E2B, Daytona, Vercel, and Modal (v16), the PXI in-app assistant (v17), and an OAuth2 authorization server with dynamic client registration that is on whenever auth is enabled (v19); each needs an explicit decision to allow or disable (Phoenix migration guide).
- Licence limits. Under ELv2, offering Phoenix to third parties as a hosted service is not permitted, so consultancies and group IT serving separate legal entities need a legal review before building a shared service on it (Phoenix licence).
Pros & Cons
Advantages
- Instrumentation is built on OpenTelemetry through the Apache-2.0 OpenInference conventions, which cover Python, TypeScript, Java, and Go and can export to any OTel-compatible backend, so the tracing code outlives a decision against Phoenix.
- Self-hosting is free with no feature gates and keeps trace data inside your own infrastructure, with PostgreSQL, Helm, OIDC or LDAP login, and Admin, Member, and Viewer roles available for a governed internal deployment.
- Tracing, LLM-as-judge and code evaluations, datasets, experiments, and prompt versioning sit in one tool, and uptake is broad: about 646,000 PyPI downloads of the server package in the 30 days to 30 September 2026 and 11.7k GitHub stars.
Disadvantages
- Phoenix is licensed under the Elastic License 2.0, not an OSI open-source licence, and may not be provided to third parties as a hosted or managed service, which matters for service providers and shared group platforms.
- The server moved through eight major versions (v13 to v20) between February and August 2026, with removed clients and endpoints along the way, so upgrades need pinned versions and regression tests.
- Defaults favour a quick start over control: authentication is off, trace retention is unlimited, and code evaluators may use external sandbox providers unless an allowlist is set.
Recommendation
Assess Phoenix as a self-hosted tracing and evaluation workbench for one RAG or agent application, instrumented with OpenInference so the same spans can later go to Langfuse, LangSmith, MLflow 3, or a central OpenTelemetry backend (all also on this radar). Phoenix fits teams that want evals, datasets, and experiments next to their traces without sending data to a SaaS vendor; it is a weaker fit where OpenTelemetry GenAI telemetry already lands in a central platform and only offline eval runs are missing, which a library such as DeepEval can cover.
Before the first real traces arrive, set up PostgreSQL rather than SQLite, enable authentication with your OIDC or LDAP directory, set a retention period that matches your data protection assessment, restrict PHOENIX_ALLOWED_SANDBOX_PROVIDERS to local runtimes, and disable the PXI assistant if its model provider is not approved. Pin the server version and test each upgrade against the migration guide. Promotion to Trial should depend on a quarter of stable upgrades and on evaluation results that the team actually uses to gate releases.
Sources
Consulting
From radar to implementation
You have assessed the technologies — we help you bring them into production systems safely and measurably.
