Use Metrics and Logs to Establish a Production Root Cause

Describe how you would use metrics and logs to trace a production issue back to its root cause. Take a concrete incident or a clearly hypothetical scenario and demonstrate how you progress from an observed symptom to evidence supporting a cause.

Constraints

The exact production symptom and the instrumentation you have are left open. Specify your example and the telemetry that is present. You may suggest extra tools such as traces, but do not assume they are already in place. Do not treat a correlation or one error message as definitive proof.

Clarifying Questions

• What changed in the user-visible success rate, latency, or throughput, and when did that change occur?
• Which services and request paths are impacted, and what changed just before the problem began?

Hint — Narrow the affected population: Before gathering more unfocused logs, compare failing and healthy requests across time, release, endpoint, instance, and dependency.

What a Strong Answer Covers

• Defining the impact and time window, mitigating the issue, and preserving evidence.
• Metrics that identify a bottleneck and logs that check particular hypotheses.
• Correlation IDs, limits of the telemetry, and validation of causality.

Follow-up Questions

• What would you do if the logs you need were absent or only sampled?
• How would you tell a downstream failure apart from a retry storm caused by your own service?

Overview: Investigate production incidents using metrics, logs, request correlation, change history, and evidence that separates symptoms from causes.