Monitoring and Logging Best Practices

When production fails, the most embarrassing part is not that it went down, but that you did not know, users found out first, and customer support asked about it before you did. You log into the server and rummage around with tail -f, but the logs are fragmented, have no IDs, and use timestamps that do not line up. Half an hour later, you are still guessing where exactly the order failed. What this kind of team lacks is not a particular tool, but the basic discipline of observability.

This article does not explain how to install Prometheus. It focuses only on practical logging and monitoring techniques: how to write machine-queryable logs, which metrics are useful, how to keep alerts from becoming noise, and which troubleshooting sequence to follow during an incident.

1. A Service Without Observability Is a Black Box

A service without observability is fundamentally a black box: you can see that it is running, but not how well it is running. Machine-centric metrics such as CPU and memory cannot answer user-centric questions: "Is ordering slow right now?" "Are payments failing?" "Did yesterday's release break a particular endpoint?"

As a result, every incident follows the same script: a user calls to say "My order failed" → customer support contacts operations → operations contacts development → development logs into the server and discovers that critical information was never logged → by the time the cause is found, half an hour has passed and the impact has spread.

The core purpose of observability can be stated in one sentence: turn "people discover incidents" into "systems discover incidents," and turn "diagnosis by guesswork" into "diagnosis by data." Logging and monitoring are the two legs of this process; without either one, it cannot stand reliably.

2. Logs: Structured JSON + Request IDs

The first iron rule is: logs must be structured; stop writing prose meant only for humans.

Of the two formats below, the prose may look friendlier, but in practice it is a disaster:

# Anti-pattern: pleasant for humans to read, impossible for machines to query
2026-07-17 10:23:11 INFO User 8821 placed order #55012, amount=199.00, paid by alipay

# Correct approach: structured JSON
{"ts":"2026-07-17T10:23:11.123Z","level":"info","msg":"order_placed","user_id":"8821","order_id":"55012","amount":199.00,"pay_channel":"alipay","request_id":"req-a3f9c2"}

The second format looks verbose, but it is JSON. Any logging backend, including ELK, Loki, or Datadog, can index and filter it directly by field. To find order logs where the amount is greater than 100 and the payment channel is alipay, you need only one query. With the first format, you can only run a full-text search for "order" and manually filter hundreds of results.

The second iron rule is: every request must have a request_id, also called a trace_id, and it must remain attached throughout the entire path.

// Generate a request_id in the ingress middleware and put it in the context
app.use((ctx, next) => {
  ctx.request_id = ctx.headers['x-request-id'] || crypto.randomUUID();
  ctx.logger = logger.child({ request_id: ctx.request_id });
  ctx.set('x-request-id', ctx.request_id);
  return next();
});

The key is to propagate this ID downstream. Include it when accessing the database, calling Redis, or invoking downstream services, whether through the X-Request-ID HTTP header, gRPC metadata, or the message body in a queue. When one request passes through five services, searching the logging backend for request_id=req-a3f9c2 then reveals the complete call chain across all five. Without a request_id, logs are scattered scraps of paper; with one, they become a chain.

3. Log Levels and Sampling: Do Not Log Everything

A level is a log's "priority label." Misusing levels is equivalent to having no levels at all. The following convention is sufficient:

• debug: Used for development and troubleshooting; disabled by default in production.
• info: Important business events such as ordering, payment, and login, plus normal but significant paths.
• warn: Abnormal but recoverable events such as a successful retry, fallback, or rate-limit activation.
• error: A real failure that requires human intervention.

The most common problem is using warn/error as a trash can. Events such as "user input validation failed" and "the third-party API occasionally returned 429" all get logged as errors. The result is hundreds of thousands of error logs every day, with the events that actually matter buried among them. Alerts based on error counts also fire constantly, so eventually everyone mutes them. Validation failure is a user problem, not a system problem; logging it at info is enough, and it may not need to be logged at all.

The second trap is logging everything. An endpoint handling 100,000 QPS that writes one info log per request produces 8.6 billion logs per day, making storage more expensive than the servers themselves. High-QPS paths require sampling: log 1% of normal requests and 100% of failed requests. This preserves a statistical sample of what the overall traffic looks like without missing a single error.

// Sample healthy requests and log every error
if (status >= 500) {
  logger.error({ path, status, duration_ms, ... });  // Log 100%
} else if (Math.random() < 0.01) {
  logger.info({ path, status, duration_ms, ... });    // Sample 1%
}

4. What Logs Should and Should Not Contain

What to log: milestones on critical paths such as order creation and state transitions; errors and their context, including exception type, input summary, and downstream response; slow operations such as database queries or external calls exceeding a threshold; and important business events such as user registration and successful payment.

What not to log (this is even more important than the list above):

• Passwords, tokens, national ID numbers, and card numbers: any PII or secret. Even if you consider them "internal logs," sending them to a logging system creates a disclosure risk and can cause both compliance and security incidents. Redact them instead of "trusting that no one will look."
• Complete request or response bodies: especially large objects such as images, files, and long text. A single log entry can be several MB and overwhelm the logging system within half a day.
• Every loop iteration or every line: this treats logs as print-style debugging and fills production with noise.
• Internal details from third-party libraries: unless you are investigating a bug in the library, set their default level to warn.

A simple standard is: will someone search for and use this log entry in the future? If you cannot imagine a scenario in which you would need it, do not log it.

5. Monitoring Metrics: The RED and USE Methods

The easiest mistake when instrumenting metrics is to "add a pile of useless ones." Two classic methods help maintain focus:

The RED method, for services, especially HTTP/RPC endpoints, uses three metrics:

• Rate: QPS and requests per second.
• Errors: number of failed requests and error rate, including 5xx responses and business failures.
• Duration: latency distribution; examine p50/p95/p99 instead of only the average.

Instrument these three metrics first for every online service and place them on a dashboard to make service health immediately visible. Apply RED to called dependencies such as Redis, MySQL, and third-party APIs as well. If they become slow, your service becomes slow too.

The USE method, for resources such as machines and middleware, also uses three metrics:

• Utilization: CPU%, memory%, and bandwidth%.
• Saturation: queue depth, the number of requests waiting for a connection pool, and disk I/O wait.
• Errors: hardware errors, packet loss, and OOM events.

USE focuses on whether a resource is approaching exhaustion. A CPU at 80% may still operate normally, but continuously increasing queue depth and a connection pool that has begun queueing are the real signs that the system is close to failure. Saturation often provides earlier warning than utilization.

A practical combination is RED for every service and USE for every machine. That establishes the baseline. Do not begin by adding dozens of custom metrics; cover these two methods first, then add whatever is missing.

6. Alert Design: Symptom-Based, Tiered, and Resistant to Alert Fatigue

Alerts are monitoring's "output." Poor alert design makes the entire system useless. Follow three principles:

1. Alert on symptoms, not causes. Alert on "order error rate > 1%" because users are affected, rather than "the payment service Pod restarted," which is only one possible cause and may not have affected users. Cause-based alerts fire whenever a Pod fluctuates or a disk reaches 80% capacity, even when nothing is actually wrong. Alert only on problems users can perceive, and keep possible causes on dashboards for investigation.

2. Use tiers, and take them seriously:

• P0/Emergency: A core function is unavailable; call someone immediately. Examples include ordering or payments being completely unavailable.
• P1/Warning: A localized anomaly affects some users and should be handled during working hours.
• P2/Information: A statistical fluctuation that should be recorded on a dashboard without active notification.

The critical point is that a P0 must truly be an emergency. Once a P0 fires for something that is not serious, people begin ignoring it. That is the cry-wolf effect.

3. Avoid alert fatigue. If a team receives 200 alerts per day, eventually nobody reads any of them, which is equivalent to having no monitoring. The remedy is periodic review: examine every alert from the past 30 days and ask, "Did this event really require human intervention when it fired?" If the answer is no, downgrade or delete it. A good alerting system fires infrequently, and every firing deserves attention.

7. Correlating the Three Pillars: Connect Logs, Metrics, and Traces with request_id

Metrics, logs, and traces are the three pillars of observability, but none is sufficient on its own. Their real power comes from correlation, enabled by the request_id / trace_id discussed earlier.

• Metrics tell you "there is a problem": the payment error rate rose from 0.1% to 5%.
• Traces tell you "which segment has the problem": the call from the payment service to the bank gateway took 8 seconds.
• Logs tell you "exactly why": that gateway call returned a timeout, and the upstream bank API reported a 503 at 10:23:11.

The three are connected through trace_id: a metric exemplar links to a trace, and every span in that trace links to logs from the corresponding service segment. Modern observability stacks such as OpenTelemetry, Datadog, and Grafana Tempo + Loki + Prometheus support this navigation by default. Click an anomalous point on the dashboard to jump directly to its trace, then click again to reach the relevant logs. This click-through workflow cannot be provided by a homegrown logging system, and it is worth the investment.

8. Incident Investigation: Alert → Metrics → Trace → Logs

Finally, consider an actual incident. When an alert fires in the middle of the night, follow this sequence instead of immediately logging into a server and rummaging blindly:

1. Read the alert: Identify the metric, threshold, service, and start time. First assess the scope of impact. Does it affect every user or only one region? Is it on a core path or an edge feature?
2. Use metrics to narrow the scope: Examine RED metrics from the same period. Which endpoint's error rate increased? During which interval did p99 latency rise? Do the USE metrics for related dependencies such as MySQL, Redis, or downstream services show anomalies? This step narrows the search area.
3. Use a trace to locate the node: Select the trace_id of a slow or failed request and inspect the full chain to find the span with abnormal latency or an error. This step identifies the "offending hop."
4. Use logs to identify the cause: Search that segment's logs by trace_id and inspect the specific error message, inputs, and downstream response. This step answers "why."
5. Correlate the timeline: Compare it with release times, configuration changes, and alerts from upstream dependencies. Often the root cause is not in your service; an upstream dependency changed first.

Once this process becomes familiar, a P0 can normally go from initial alert to identified root cause within 10 minutes. In contrast, if you skip metrics and search logs directly, you will become lost among billions of entries. If you inspect only metrics and not traces, you will know that "payments are slow" but not "which hop is slow." Every step has an irreplaceable role.

Summary

• A service without observability is a black box, and learning about incidents from users is a team's greatest embarrassment. The purpose of observability is not to "install monitoring," but to let the system discover problems before they affect users.
• Logs must be structured JSON and include a request_id, which must propagate across the entire call chain. Human-readable sentences are for developers; machine-queryable JSON is what systems can use.
• Use log levels sparingly, and do not treat error as a trash can. Sample high-QPS paths so logging does not cost more than compute. Never log passwords, tokens, or complete request bodies.
• Use RED for services and USE for resources to cover the baseline before adding dozens of custom metrics.
• Alerts should be based on symptoms that affect users, use three tiers, and reserve P0 for genuine emergencies. Manage alert fatigue by regularly deleting alerts: remove those that never fire and those that provide no value when they do.
• Correlate the three pillars with trace_id. During an incident, investigate in the order "alert → metrics → trace → logs." Each step narrows the scope, enabling root-cause identification within 10 minutes.

To put it in one sentence: observability is not a pile of tools; it keeps you from being completely blind when an alert fires in the middle of the night. What truly matters is the ability to answer four questions quickly during an incident: what happened, who is affected, where the slowdown is, and why. The faster you can answer them, the closer your service is to being "trustworthy." The return on investment from logging and monitoring never appears on quiet days; it appears during the next middle-of-the-night outage.