With 12 NestJS services talking over HTTP and RabbitMQ, "why was this sign-up slow?" no longer has an answer in a single log. At PokerNetic I built the observability of the whole platform, from the SDK in the services to the alerts. Here is how it works and what I learned along the way.
What I wanted to solve
- Following a request end to end, across different services and through the queue.
- Knowing about a problem before users do: 5xx errors, latency, CPU, memory, disk and restarts.
- Going from a trace to the logs and metrics of that moment in one click.
- Doing all of that at a predictable cost.
flowchart LR
subgraph back["Backend"]
svc["12 NestJS services: OpenTelemetry SDK and Pino logs with trace_id"]
end
subgraph front["Browser"]
browser["Next.js front-ends: traces, Web Vitals and errors, 10% sample"]
end
edge["Edge relay: validates the origin and injects the credential"]
nginx["NGINX: authenticated ingest"]
collector["OpenTelemetry Collector: memory limit, attribute cleanup, 20% sample, batch"]
loki["Loki: logs, 7 days"]
tempo["Tempo: traces, 7 days"]
prom["Prometheus: metrics, 30 days"]
grafana["Grafana: 10 dashboards and 12 alerts"]
chat["Alert channel"]
svc -->|"OTLP with gzip"| collector
browser -->|OTLP| edge
edge --> nginx
nginx --> collector
collector --> loki
collector --> tempo
collector --> prom
tempo -->|"service graph and span metrics"| prom
loki --> grafana
tempo --> grafana
prom --> grafana
grafana -->|alerts| chat
class svc tech-nestjs
class nginx tech-nginxInstrumenting the services
Each service has a tracing file imported on the first line of main that just calls a shared library function with the service name. Since the project is CommonJS, this guarantees the hooks are registered before the code loads pg, ioredis and amqplib.
The library starts the OpenTelemetry SDK with OTLP exporters for traces, metrics and logs, all gzipped, and turns on the automatic instrumentations for HTTP, Express, NestJS, PostgreSQL, Redis, RabbitMQ, Pino and the Node runtime, plus host metrics, because the PaaS is out of cAdvisor's reach.
Resource attributes identify service, version and environment. The environment comes from a dedicated variable, because the PaaS sets NODE_ENV to production in staging too. And the instance id is stable, taken from the replica: with one UUID per boot, every restart created new series and one metric reached thousands of ghost series.
Logs that point to the trace
Logs go out through Pino with trace_id and span_id on every line, to two destinations: readable stdout and a structured OTLP record that goes to Loki. Request and response serializers are lean, because headers used to be most of the log bytes, and fields such as authorization, cookie, password and tokens are masked before they leave the process.
Keeping the thread through the queue
Propagation through RabbitMQ has no manual code: the amqplib instrumentation puts the traceparent in the message headers and the consumer extracts it from the same place. It works because the Nest queue clients run on amqplib. To validate it, a sign-up must show auth, communication and affiliates in the same trace.
The browser, with no credential in the bundle
The Next.js front-ends send load, fetch, click and long-task spans, Web Vitals and unhandled errors, with a 10% sample. Telemetry does not go straight to the ingest: it goes through an edge relay that accepts only the three OTLP paths and injects the credential from a secret of the relay itself. So no credential shows up in the public JavaScript.
The Collector keeps cost and cardinality in check
- Protection: a memory limiter in front of everything.
- Cleanup: process and operating system attributes are removed before they become labels.
- Useful labels: environment and version become metric dimensions.
- Sampling: a stateless 20% probabilistic sampler, by trace_id hash.
- Shipping: batching on every pipeline and, when exporting to Tempo, a bounded queue and retry.
Tempo also generates the service graph and span metrics, sent to Prometheus via remote write, and from a trace Grafana jumps to the logs and metrics.
Two expensive lessons
The first was cardinality. An unbounded label, combined with cumulative metric temporality, multiplied telemetry traffic. The answer was to attack it in several places: few labels in Loki and the rest as metadata, a stable instance id, attributes removed in the Collector, a trimmed cAdvisor, a Loki streams alert and a probe that warns when the number of series goes over the limit. Gzip and lean logs were the biggest volume reductions.
The second was sampling. I started with tail sampling, to always keep traces with errors, but it held traces in memory until the pipeline stalled. I switched to a stateless probabilistic sampler. I lost the guarantee of keeping 100% of error traces, but error logs are still complete in Loki, with the trace_id for when the trace exists.
Dashboards and alerts
There are 10 dashboards: an overview of what is on fire, per-service detail with p50, p95 and p99, container resources, business metrics, the service map, Loki, Tempo and Collector health and two front-end views with Web Vitals and browser errors. The 12 alert rules cover a 5xx rate above 1%, p95 above 500 ms, CPU, memory and restarts of the stack containers, host disk, targets down and the health of the stack itself, and reach a chat channel grouped so they do not become noise.
Infra and delivery
The stack runs self-hosted on a VPS with Docker Compose: Loki, Tempo, Prometheus, the Collector, Grafana, cAdvisor and node-exporter, with the public ingest behind NGINX and authentication. Logs and traces are kept for 7 days and metrics for 30. A push to main triggers a deploy over SSH that only restarts the Collector when its configuration changed, and CI validates the configurations, starts the whole stack and runs a smoke test that sends a trace, a log and a metric and checks that each one arrives.
What is still missing
- Object storage for Loki and Tempo.
- Fine-tuning alerts after the sampling change.
- SSO for Grafana.