Experimentation (releases & versions)

You can track the effect of changes to your LLM app on metrics in Langfuse. This allows you to:

• Run experiments (A/B tests) in production and measure the impact on costs, latencies and quality.
  • Example: "What is the impact of switching to a new model?"
• Explain changes to metrics over time.
  • Example: "Why did latency in this chain increase?"

Releases

A release tracks the overall version of your application. Commonly it is set to the semantic version or git commit hash of your application.

SDK initialization

from langfuse import Langfuse
langfuse = Langfuse(
  ENV_PUBLIC_KEY,
  ENV_SECRET_KEY,
  ENV_HOST,
  release="<release_tag>"
)

Via environment variable

The SDKs will look for a LANGFUSE_RELEASE environment variable. Use it to configure the release e.g. in your CI/CD pipeline.

LANGFUSE_RELEASE = "<release_tag>" # <- github sha or other identifier

Automatically on popular platforms

If no other release is set, the Langfuse SDKs default to a set of known release environment variables.

Supported platforms include: Vercel, Heroku, Netlify. See the full list of support environment variables for JS/TS (opens in a new tab) and Python (opens in a new tab).

Versions

The version parameter can be added to traces and all observation types (span, generation, event). Thereby, you can track the effect of a new version on the metrics of an object with a specific name using Langfuse analytics.

from langfuse.model import InitialGeneration
langfuse.generation(InitialGeneration(
  name="guess-countries",
  version="1.0",
))

Version parameter in Langfuse interface