Opik Integration
Opik (by Comet ML) provides LLM observability, tracing, and prompt optimization for OCapistaine.
Overview
┌─────────────────────────────────────────────────────────────────┐
│ Opik Integration │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Tracing Evaluation │
│ ├─ LLM calls ├─ Hallucination detection │
│ ├─ Token usage ├─ Output format scoring │
│ ├─ Latency └─ Confidence calibration │
│ └─ Error tracking │
│ │
│ Datasets Optimization │
│ ├─ Training data ├─ Prompt refinement │
│ ├─ Test cases └─ A/B testing │
│ └─ Evaluation sets │
│ │
└─────────────────────────────────────────────────────────────────┘
Documentation
| Document | Description |
|---|---|
| Continuous Improvement | Daily practices for keeping AI grounded |
| Experiment Workflow | Running evaluations and building datasets |
Quick Start
View Traces
from app.agents.tracing import get_tracer
tracer = get_tracer()
# Traces automatically logged to Opik dashboard
Run Evaluation
# Via scheduler
python -c "from app.services.scheduler import run_task_now; print(run_task_now('task_opik_evaluate'))"
Sync Prompts
# Sync all forseti prompts to Opik
python -m app.prompts.opik_sync --prefix forseti.
Related Tasks
- task_opik_evaluate - Automated evaluation
- task_prompt_sync - Prompt synchronization
- task_audierne_docs - Dataset generation
Configuration
Environment variables:
OPIK_API_KEY=your-api-key
OPIK_PROJECT_NAME=ocapistaine-test
OPIK_WORKSPACE=your-workspace
Config file: ~/.opik.config