AI Monitoring (LLM Observability)

A first-class observability pillar for AI. Track spend, token usage, latency and errors across every model and provider — with full multi-step traces (agents, RAG, tool calls), per-user and per-session cost attribution, and opt-in prompt/completion capture. Major providers are auto-instrumented; anything else (custom, self-hosted, in-browser WebLLM) is one wrap call away. Cost is computed server-side from a maintained pricing table.

Prerequisites

  • A Node.js / edge runtime using @senzops/apm-node (v1.4+), or any runtime that can POST to the AI ingest endpoint.
  • Prompt/completion content is opt-in and OFF by default; enable it per-source in Settings if you need it for debugging.

1. Dashboard Configuration

Before injecting code into your application, you must provision an API key from the Senzor dashboard.

1

Create an AI Source

Click '+' next to AI Monitoring in the sidebar, name it, and choose Server or Browser (WebLLM).

2

Copy the Source Key

The dashboard issues a one-time sz_ai_ key — copy it now; it is shown only once.

3

Initialize the SDK

Call Senzor.init({ ai: { apiKey } }) as early as possible — AI Monitoring sources carry their own key. If you already run Senzor APM/Task, keep your existing top-level apiKey and just add the ai key. Supported providers (OpenAI, Anthropic, Gemini/Vertex, Azure OpenAI, Cohere, Mistral, Groq, Ollama, the Vercel AI SDK and LangChain) are then auto-instrumented.

2. SDK Installation

Select your environment below to view the initialization code.

npm install @senzops/apm-node

import Senzor from '@senzops/apm-node';

// Already using Senzor APM/Task? Keep your existing apiKey and just add `ai`.
Senzor.init({
  ai: { apiKey: "<YOUR_AI_KEY>", captureContent: false } // captureContent: store masked prompts/outputs
});

// OpenAI, Anthropic, Gemini, Cohere, Mistral, Groq, Ollama,
// the Vercel AI SDK and LangChain are now captured automatically.

AI sources use a dedicated key under ai.apiKey, separate from APM/Task. Cost is recomputed server-side — the SDK never sends a cost field. Set ai.sampleRate (0–1) for high-volume head sampling.

Troubleshooting & Edge Cases

Cost shows as $0 for some models.

The model isn't in the built-in pricing table (e.g. a new snapshot or a self-hosted model). Add a price under the source's Settings → pricing overrides (USD per 1M input/output tokens).

Streaming calls show 0 output tokens (OpenAI).

OpenAI only returns usage on streamed responses when you pass stream_options: { include_usage: true }. Anthropic and Gemini report streaming usage natively. Time-to-first-token is always captured.

Prompts/outputs aren't appearing in traces.

Content capture is opt-in. Enable it on the source (Settings → Capture prompt & output) AND set ai.captureContent: true in the SDK. It is masked on ingest and stored under your plan's retention.