Logging in the critical path by sampling

From: Logging Wisdom: How to Log

A common reason for not logging is request heavy processes where you couldn’t possibly do a bunch of logging for every request. Here, many just skip logging entirely and instead resort to reporting aggregates — such as periodic check: users_logged_in=10 with mean_session_time=12min. This is sure useful to see the overall health, but it’s meaningless when it comes to spotting bottleneck in individual requests.

A more powerful method is to sample: add log statements to your every request, but only output a X% sample of them. Each log statement could be wrapped in a shouldILog() type of statement that makes it a no-op unless it will be sampled for the particular request. Given enough requests, this will in time be enough to catch at least one instance of almost every single “type” of request — including those hitting tail latencies.