Sentrinel — Production Architecture

Designed for SaaS from day one: multi-tenant, two-store, sampling-aware. Both storage engines are Apache-2.0 licensed — nothing here restricts offering Sentrinel as a paid hosted product.

                     ┌─────────────────────────┐
   customer app      │  @sentrinel/plugin       │  in-process aggregation
   (Elysia.js)  ───▶ │  · exact 1-min counters │  + head sampling of raw logs
                     │  · sampled request logs │  (errors & slow always kept)
                     └───────────┬─────────────┘
                                 │ HTTPS batches (flushInterval)
                                 ▼
                     ┌─────────────────────────┐
                     │  Sentrinel API (Bun)     │  /api/ingest/*
                     │  LogStore abstraction   │  /api/requests|traffic|…
                     └─────┬──────────────┬────┘
              writes/reads │              │ writes/reads
                           ▼              ▼
              ┌────────────────┐   ┌──────────────────────┐
              │ PostgreSQL     │   │ ClickHouse           │
              │ CONTROL PLANE  │   │ TELEMETRY STORE      │
              │ orgs, apps,    │   │ request_logs, spans, │
              │ endpoints,     │   │ app_logs             │
              │ consumers,     │   │ · day partitions     │
              │ alerts, uptime,│   │ · ZSTD codecs        │
              │ app_sessions,  │   │ · TTL retention      │
              │ 1-min metrics  │   │                      │
              │ (+ optional    │   │ · 1-min MV rollup    │
              │  read replica) │   │   kept 2 years       │
              └────────────────┘   └──────────────────────┘
                           ▲
                           │
                 ┌──────────────────┐
                 │ Next.js dashboard│
                 └──────────────────┘

Why two stores

The measured data shape (see docs/storage sizing): metrics tables grow at ~130 MB/day even with 500 endpoints; request_logs grows at ~13 GB/day at a mere 100 req/s. One table is 99% of the volume and it's append-only, never-updated event data — the exact shape column stores crush (10–30× compression, sub-second scans over billions of rows). Meanwhile apps, alerts and orgs need foreign keys, updates and transactions — the exact shape Postgres is for. SigNoz and Uptrace ship this same split.

Where app_sessions sits, and why

One row per mobile app launch, in Postgres, not the telemetry store — even though it is high-volume, append-shaped event data that looks like it belongs next to request_logs.

Two reasons. It is upserted rather than appended: a session is reported when it starts and again when it ends or is found abandoned, and the status may only ever worsen (ok → errored → abnormal → crashed). Column stores are poor at read-modify-write, and ClickHouse's ReplacingMergeTree deduplicates eventually, at merge time — which would make crash-free rate wrong for an unpredictable window after every write.

And the volume is small. One row per launch is orders of magnitude below one row per request, so the thing that justifies ClickHouse for request_logs does not apply here.

Deliberately not named sessions — that table is dashboard logins. Two very different things under one obvious name is how you join the wrong one at 2am.

The LogStore abstraction (apps/api/src/lib/logstore/)

All firehose reads/writes go through one interface with two implementations:

Routes never touch a database directly for logs, so switching stores is an env var, not a migration.

Multi-tenancy

organizations (Postgres) is the tenant root; every app carries org_id, and every ClickHouse row carries org_id as the FIRST column of the sort key (ORDER BY (org_id, app_id, timestamp, …)) so tenant isolation is a sort-key prefix scan, not a filter. This was baked in now because changing a ClickHouse sort key later means rewriting the table. monthlyRequestQuota on organizations is the hook for plan enforcement at the ingest API.

Entitlement (apps/api/src/lib/entitlement.ts)

One function answers "may this org use Sentrinel right now?", because the dashboard and the ingest path must never disagree about it. Three conditions block, checked in this order:

  1. payment_required — billing_status set to past_due/expired by an operator
  2. trial_expired — trial_ends_at is in the past
  3. over_quota — this month's ingest passed the plan's quota

Order matters. An org that is both past due and over quota is told about the payment, because fixing the quota would not let them back in.

Blocking the dashboard does not immediately stop ingest. Data keeps flowing for SENTRINEL_INGEST_GRACE_DAYS past the block, so paying restores an unbroken history rather than one with a hole where the outage was. After that, ingest answers 402 — not 403: the credential is valid, the account owes money, and a 403 sends SDK authors hunting for a bad key.

Over-quota is exempt from that grace on purpose: quota.ts already rejects with 429 at the boundary, and two systems dropping the same payload for two different stated reasons is worse than one.

The ingest path resolves entitlement per payload, so it reads a 15-second in-process cache — the same trade-off quota.ts makes, for a boundary measured in days.

The dashboard gate fails open: if /api/entitlement is unreachable the app renders normally. This is the opposite of the auth gate, deliberately. Failing closed on auth protects data; failing closed on billing would lock every paying customer out during an outage, turning our problem into theirs at the moment they most need their telemetry. The gate is UI, not enforcement — the enforcement that matters is the 402 on ingest, which devtools cannot reach.

The gate also lets the org owner through regardless of the block, so the one person who can fix a past-due account or raise a quota is never locked out of their own Settings. This is a UI bypass only — ingest still refuses once the grace period or quota boundary is hit — and it costs no security, because the owner already holds the highest role in their tenant.

Platform admin (apps/api/src/lib/admin.ts)

A different axis from org membership: an org owner owns their tenant, a platform admin operates the instance. Membership comes from SENTRINEL_ADMIN_EMAILS rather than a database column, because a column is one compromised signup or one injection away from being set, while an env var needs redeploy access — and there is no "grant admin" button to find and abuse. The variable adds operators to the product owner rather than replacing them, so naming a colleague cannot be the act that removes your own access.

Non-admins receive 404, not 403, so the surface reads as absent rather than merely locked.

Source relations (every signal names its owner)

Requests, errors, app logs, traces and replays each store a consumer_identifier of their own; sessions store the same value as distinct_id. None of them resolve it by joining back through the request that produced them.

That denormalisation is the point. The tables expire on different clocks — raw logs at RETENTION_LOGS_DAYS, replays at RETENTION_REPLAY_DAYS — so a trail assembled by join shortens as the requests behind it age out, and "show me everything this user did" starts silently returning a subset. A log line that has lost its owner cannot be found by anyone looking for that user.

Two identity sources converge on it:

Source Field it sends
Backend plugin consumerIdentifier from the app's own resolver
Browser SDK userId, set by identify()
Flutter / native SDKs distinctId

The sessions ingest accepts userId and distinctId. It previously read only the latter, so every web session arrived anonymous and the browser and backend key-spaces could never meet — recordings existed but could not be attributed to the person who caused them. Covered by apps/api/tests/source-relation.test.ts.

Replay metadata fills its owner in with COALESCE across chunks: identify() usually runs after the recorder has already shipped its first chunk, so the first chunk that knows the user is the one that sets it.

Issues and errors live in different stores

issues is one row per distinct bug, deduplicated on a hash of error type and normalised stack. error_logs is one row per time it happened. If the same TypeError fires 400,000 times that is 400,000 error rows and one issue row.

They are split because they want opposite things:

issues error_logs
Grows with distinct bugs traffic
Written upserted repeatedly inserted once, never updated
Read as a filtered, sorted page aggregates over a window
Store Postgres ClickHouse

Issues stays in Postgres because it needs things ClickHouse does not have: a unique constraint on (app_id, fingerprint) — which is what makes fold-in correct at all — a transactional create-or-bump that reads the id back, an in-place UPDATE when a person resolves or assigns, and a foreign key to users. Size is not the reason: measured at 2M rows a page of the issue list was 0.64 ms and read 15 buffers, because it walks the index backward and stops at 50.

Occurrences moved out because the reverse is true of them. At 5M rows in Postgres the Errors page took 26.3 s: EXPLAIN showed an index scan reading 210,774 blocks to return 162,642 rows, since a row store has to fetch the whole ~520-byte row to aggregate four narrow columns. Widening the window to 7 days made the planner switch to a parallel sequential scan, which returned 7x more data 11x faster — performance that improves as you ask for more is not something you can capacity-plan.

The link is issue_id, carried as a plain column exactly like request_log_id and trace_id already were. There is no cross-store join: ingest upserts the issue first and stamps its id onto the occurrences; the issue list reads Postgres only; issue detail is one Postgres row plus a bounded lookup by issue_id.

The same denormalisation applies as everywhere else here — endpoint_id and consumer_id were foreign keys, and the method/path and consumer identity they resolved to are stored on the occurrence instead.

Sampling (the biggest cost lever)

requestLogging.sampleRate in the plugin keeps 1-in-N successful requests; errors and slow requests (≥ slowRequestThresholdMs) are always kept. Counters are aggregated in-process before sampling, so dashboard metrics stay exact at any rate. Every stored row records its effective sample_rate, and the ClickHouse rollup extrapolates (sum(1/sample_rate)), so even log-derived counts stay honest. Verified: a 0.1-rate row aggregates as 10 requests.

Retention

Read/write separation

Ingest writes always hit the Postgres primary. Dashboard queries use dbRead, which binds to DATABASE_REPLICA_URL when set (managed Postgres providers hand these out; you can also attach a streaming replica) and falls back to the primary otherwise. ClickHouse handles its own read parallelism; add replicas + Distributed tables only at much larger scale.

Scaling ladder

  1. Now — single box, TELEMETRY_STORE=postgres. Fine to ~10 req/s.
  2. First customers — docker-compose.prod.yml: Postgres + ClickHouse, TELEMETRY_STORE=clickhouse, sampling 0.1–1.0 per plan. Fine to ~1000 req/s on one decent host. Add Redpanda (--profile queue) and ingest uses it automatically as a durable log in front of ClickHouse, failing over in both directions — see PIPELINE.md.
  3. Growth — managed Postgres + replica URL; ClickHouse on its own host (or ClickHouse Cloud); API scales horizontally (it's stateless), and the ingest consumer can run as its own process off the same log.
  4. Big — ClickHouse cluster w/ replication, the same log clustered for burst absorption, per-org rate limits.

Operations