Elpe observability → Grafana

The Elpe daemon emits standard observability instead of shipping a bespoke UI: a Prometheus exporter and an optional Loki log feed. Point Grafana at both and import elpe-observability.json for a ready-made “Jobs” screen — the historical jobs table, a live “running now” table, and a single-job console — replacing the Nest’s hand-rolled CI panel.

Design rationale and scope: ~/Projets/elpe.roadmap/done/project_job_dashboard.md.

What the daemon exposes

The exporter binds a second port next to the gRPC service (:50051):

Env var	Default	Meaning
ELPE_METRICS_ADDR	0.0.0.0:50052	Prometheus exporter + control surface
ELPE_LOKI_URL	(unset → disabled)	Loki base URL, e.g. http://localhost:3100
ELPE_TRACES_URL	(unset → disabled)	Zipkin-v2 span endpoint, e.g. http://localhost:9411/api/v2/spans (Tempo’s Zipkin receiver)

Endpoints on ELPE_METRICS_ADDR:

• GET /metrics — Prometheus exposition (scrape target).
• GET /api/jobs — JSON snapshot of running + recent jobs (for a Grafana Infinity datasource, or ad-hoc tooling). Each entry carries input_hash.
• POST /api/jobs/{id}/kill — cancel an in-flight job.
• GET / — a signpost page (not a UI).

Metrics

• elpe_jobs_created_total (counter)
• elpe_jobs_finished_total{result="ok|error|hash_mismatch"} (counter)
• elpe_jobs_running (gauge)
• elpe_build_duration_seconds (histogram)
• elpe_pins_active (gauge) — active output-lock pins
• elpe_stdout_bytes_total, elpe_stderr_bytes_total (counters)
• elpe_job_info{id,kind,name,input_hash,phase} (gauge, value = age s) — only in-flight jobs, so cardinality is the concurrent job count, never the lifetime total. This is the “which jobs are running right now” view Prometheus otherwise can’t give you.

Logs (Loki)

Enabled only when ELPE_LOKI_URL is set. Two low-cardinality streams under the single label set {service="elpe", stream=…}:

• stream="lifecycle" — one logfmt line per job start/finish (event, job_id, kind, name, input_hash, result, exit, duration_ms). This is the historical Jobs table.
• stream="stdout" / stream="stderr" — the job’s console output.

job_id, name and kind ride as Loki structured metadata (the third element of each push value), so filtering by a specific job costs no label cardinality: {service="elpe"} | job_id="1234". Requires Loki ≥ 3.0.

Prometheus scrape config

scrape_configs:
  - job_name: elpe
    static_configs:
      - targets: ["ELPE_HOST:50052"]

Import the dashboard

1. In Grafana, add a Prometheus and a Loki datasource.
2. Dashboards → New → Import, upload elpe-observability.json, and map the two datasource inputs. (Or drop the file in a provisioning path — see below.)

Provisioning (file-based, low-maintenance):

# /etc/grafana/provisioning/dashboards/elpe.yaml
apiVersion: 1
providers:
  - name: elpe
    type: file
    options:
      path: /var/lib/grafana/dashboards/elpe

Copy elpe-observability.json into that path. Datasource UIDs are resolved from the ${DS_PROMETHEUS} / ${DS_LOKI} inputs at import; for pure provisioning, pin your datasource UIDs to prometheus / loki or edit the file’s uids.

Kill & retry — the control plane

Observability is read-only; actions are not Grafana’s job, so they live on the daemon’s control surface and are surfaced deliberately.

• Kill — POST /api/jobs/{id}/kill (pushes an internal Job::Cancel). To get a button inside Grafana, add a Business Actions panel (Volkov Labs, volkovlabs-form-panel / actions) — a popular community plugin — configured to POST {{ELPE_METRICS_ADDR}}/api/jobs/${id}/kill. Grafana core data links only navigate (GET), so they can’t trigger the kill on their own; the table’s id column instead drills into that job’s console.

• Retry — intentionally not in the daemon. The daemon does not retain a submitted derivation’s request (name/paths/script), so it cannot replay one. Retry belongs to whoever submitted the job (the frontend / the Nest). The snapshot and metrics expose each job’s input_hash precisely so the Nest can correlate and resubmit. This is the “kill here, retry at the Nest” split.

Traces (Tempo)

Set ELPE_TRACES_URL to a Zipkin v2 JSON ingest endpoint and the daemon emits one span per finished job. Zipkin (not OTLP) is used deliberately: the daemon hand-encodes the JSON with reqwest, adding no OpenTelemetry crate to the pinned graph — the same best-effort pattern as the Loki sink.

Enable Tempo’s Zipkin receiver:

# tempo.yaml
distributor:
  receivers:
    zipkin:            # listens on :9411 -> ELPE_TRACES_URL=http://tempo:9411/api/v2/spans

Span shape: serviceName=elpe, name = job name, SERVER kind, tags job.id / job.kind / job.input_hash / job.result / job.exit / otel.status_code. Trace/span ids are derived (blake3) from the job identity.

Trace → logs correlation. The span’s job.id tag equals the Loki job_id structured-metadata field, so wire the Tempo datasource’s Trace to logs to Loki with a custom query {service="elpe"} | job_id="$${__span.tags["job.id"]}" — clicking a span jumps to that job’s console.

Caveat — this is flat tracing (one span per trace), not a DAG flamegraph. The daemon sees each derivation/action independently; a “1 build → N derivations” flamegraph needs the submitting frontend/Nest to propagate a trace context so the daemon can parent spans under a shared trace id. That is roadmap item #2 (build↔derivation correlation) and is intentionally not wired here.

JSON datasource (Infinity)

Import elpe-jobs-infinity.json. It uses only the Infinity datasource (yesoreyeram-infinity-datasource) reading GET ${elpe}/api/jobs — a running-jobs and a recent-jobs table with no Prometheus or Loki required. Set the elpe dashboard variable to the daemon’s base URL (default http://localhost:50052). Useful for a minimal deployment or as a truthful cross-check of the info-metrics.