Every app, site, job and AI agent we run reports its usage, performance and errors to one self-hosted dashboard. OpenTelemetry carries the data, so no vendor owns it and nothing gets forgotten.
The live dashboard is private: it asks for an approved Google sign-in, then a Grafana login.
When each service has its own logs and its own way of failing, problems hide in the gaps. With one place, "is it up, is it slow, who is using it, what broke" has one answer for the whole project, and a new service is visible from the day it ships instead of after its first outage.
OpenTelemetry (OTel) is an open standard, run by the CNCF, for three kinds of signal: traces (what one request did, step by step), metrics (counts and timings over time) and logs. Code sends them over one protocol, OTLP, to a collector, and the collector forwards them to whichever backend we like. That's the adapter pattern from Own your stack applied to monitoring: switching from one backend to another is a collector config change, not a rewrite.
apps, sites, AI agents ──OTLP──▶ OpenTelemetry Collector ──▶ backend (store + dashboards)
journald logs ─────────────────▶ (127.0.0.1) on our own disk
host, GPU, health checks ──────▶The collector also gathers what code can't send itself: CPU, memory and disk for the machine, the logs every systemd service already writes, and a health check against each registered service.
One file, observability/services.yml, lists every machine and every service we run. Each service has its systemd unit, a
health URL and where its code lives. Adding a solution means adding its entry in the same change, and the health
checks are built from that list. If it isn't in the registry, it doesn't exist.
- name: live
unit: live.service
health: http://127.0.0.1:8090/live
otel: true
source: apps/live/The official gems do most of the work: Rack, Rails, Net::HTTP, PostgreSQL and more get traced automatically.
# Gemfile
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"
# config.ru or an initializer
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
OpenTelemetry::SDK.configure do |c|
c.service_name = ENV.fetch("OTEL_SERVICE_NAME", "live")
c.use_all
endErrors become visible by recording them on the current span; slow parts show up as long spans in the trace:
span = OpenTelemetry::Trace.current_span
span.record_exception(error)
span.status = OpenTelemetry::Trace::Status.error(error.message)Usage: requests, active sessions, and for AI calls the model, tokens and cost.
Performance: latency percentiles (p50, p95, p99), time spent queueing, memory per process.
Machines: CPU and load average, memory and swap, disk space and I/O, network, GPU use, VRAM and temperature, for every host, including the one running the dashboard.
Errors: exceptions with their trace, error rate per service, and services failing their health check.
Every ruby.im request is sorted by its User-Agent and Referer: an AI crawler (GPTBot, ClaudeBot, PerplexityBot), a search engine spider (Googlebot, Bingbot), a link preview (Slack, X), a script, or a person arriving from a search engine, an AI assistant (ChatGPT, Claude, Perplexity, Gemini), a social site, another site, or directly. The answer is tagged on the request's trace, so the dashboard counts visits by source and lists the latest ones with the page they asked for. The lists of bots and sites are plain Ruby arrays, easy to extend.
TrafficSource.classify(user_agent: "Mozilla/5.0 (compatible; GPTBot/1.2)") # => ["ai-bot", "GPTBot"]
TrafficSource.classify(user_agent: browser, referer: "https://www.google.com/") # => ["search", "Google"]Every site also writes one log line per request and per page view, read by a second dashboard that counts
unique visitors and sessions, shows top pages, countries and devices, and flags vulnerability scans: requests for
/wp-admin, /.env or /.git, scanner User-Agents, 404 bursts and busy IPs. It
replaces an outside analytics service; the logs never leave our machine. How it works:
visitor and scan logging.
Claude Code, Codex and other agents can export OpenTelemetry themselves. Pointing them at the same collector puts their sessions, token use and cost next to everything else, which is exactly the data needed to play providers against each other on price and quality.
The stack runs on the same machine as the apps, so it has to stay light: single binaries under user systemd, listening on 127.0.0.1 only, keeping data on local disk with a retention limit. Good open-source backends for a box like ours include OpenObserve (one binary for logs, metrics and traces), the Grafana stack (Prometheus, Loki, Tempo and Grafana) and SigNoz. Because everything speaks OTLP, we can start with one and move later.
We started with the Grafana stack: the OpenTelemetry Collector feeds Prometheus (metrics, 30 days), Loki (logs, 30 days) and Tempo (traces, 7 days), and one Grafana dashboard shows every service's health, request rate, latency and errors next to the machine's CPU, load, memory, disk, network and GPU. Tempo turns traces into request metrics, so an instrumented app gets its graphs without writing any metrics code. The whole stack uses about 1 GB of the 30 GB of memory.
Provided as is, for information only, with no warranty and no liability. Not legal or professional advice. Security and radio material is for authorized, lawful use only. Full disclaimer.
Visits are logged on our own server: IP address, browser and device details, and pages viewed. We use them to understand traffic and spot attacks, and never share or sell them.