Structured Logging That Survives an Incident

The Problem#

Free-form logs feel fine until an incident starts. Searching for "failed payment" across inconsistent strings does not answer which tenant, request, release, or dependency caused the failure.

Why It Matters#

During incidents, engineers ask structured questions: which release introduced this, which tenants are affected, which dependency is failing, and how many requests are impacted. Plain text logs rarely answer those questions quickly.

Most observability platforms, including Datadog, Honeycomb, Grafana Loki, Elasticsearch/OpenSearch, and Cloud Logging, work best when logs have consistent fields.

Project Example#

For an API request, every log line should include stable fields:

{
  "level": "error",
  "requestId": "req_123",
  "tenantId": "acme",
  "service": "checkout-api",
  "operation": "chargePayment",
  "dependency": "payment-gateway",
  "statusCode": 502
}

Now incident queries can group errors by tenant, dependency, route, and release.

Implementation Example#

logger.info({
  event: "payment_attempted",
  requestId,
  tenantId,
  orderId,
  provider: "stripe",
  amountCents: 2499,
});

Prefer event names and fields over long sentences. The human-readable message can exist, but the fields are what make the log queryable.

Implementation Checklist#

• Generate a request id at the edge.
• Pass trace/request ids to downstream services.
• Log JSON with stable field names.
• Redact secrets and tokens.
• Sample noisy success logs but keep errors and audit events.
• Include deploy version, region, service, route, and tenant where relevant.
• Standardize field names across services.
• Add log examples to service templates.

Production Notes#

Do not log full request or response bodies by default. They often contain secrets, tokens, email addresses, addresses, or payment data. Log identifiers and safe diagnostic metadata instead.

Structured logs are strongest when paired with metrics and traces. Logs explain specific events; metrics show shape and volume; traces show cross-service timing.

Common Mistakes#

• Logging full request bodies.
• Changing field names between services.
• Logging errors without tenant, route, or dependency context.
• Treating logs as a replacement for metrics and traces.
• Using different request-id names such as reqId, request_id, and trace.

How To Validate#

Pick one recent incident question and try to answer it using only logs. For example: "which tenants saw payment failures after deploy 2026.08.16?" If the query needs manual grep, missing fields, or tribal knowledge, the logging schema needs work.

Summary#

Structured logs make production failures searchable. Use stable JSON fields, correlation ids, and redaction from the first version of a service.