Lean Team
Lean Team blog

One stack trace, 40 billed log entries: the logging cost most teams miss

Most log platforms treat each line of stdout as a separate log entry. So one plain-text Java stack trace arrives as 40 or more entries: each one billed, most with no level or timestamp, and impossible to search as a single error. Python tracebacks, Node errors and .NET exceptions have the same problem.

It rarely shows up as a line item. It just makes the bill bigger and outages harder to debug.

How to tell if it's happening to you

Search your logs for entries that start with:

Another sign: your platform reports far more "errors" than your application actually threw. If a single failed request produces dozens of error entries, you're paying for line splitting.

Fix 1 (best): structured logging

Have the logger serialize the exception into one field of one JSON line:

{"level":"ERROR","message":"payment failed",
 "error.type":"java.net.SocketTimeoutException",
 "error.stack":"java.net.SocketTimeoutException: Read timed out\n\tat ...",
 "order_id":"8812"}

One error, one entry, fully searchable. It fixes levels, timestamps and request context at the same time. Every major language has a JSON encoder for its standard logger: Logback or Log4j2 for Java, pino or winston for Node, python-json-logger or structlog for Python, Serilog for .NET.

Fix 2 (stopgap): join the lines in the collector

When you can't change the application yet, have the log shipper stitch multi-line events back together before they're sent. With Fluent Bit:

[FILTER]
    Name                  multiline
    Match                 kube.*
    multiline.key_content log
    multiline.parser      java, python, go

The CloudWatch agent and Google's Ops Agent have equivalent multi-line settings. It's a stopgap because the joined entry is still unstructured text, but it stops the per-line billing today.

Bonus: AWS log groups that never expire

While you're looking at logging costs: CloudWatch log groups never expire by default. Any group nobody configured keeps growing forever. This lists every group with no retention set:

aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' --output text

Then set one, for example 30 days:

aws logs put-retention-policy --log-group-name <name> --retention-in-days 30

For comparison, Google Cloud's _Default bucket keeps logs for 30 days unless you change it, and Azure Log Analytics defaults to 30 days for most tables.

The short version

  1. Search for stack-trace fragments to see how much of your volume is split lines.
  2. Switch to JSON logging where you can.
  3. Join lines in the collector where you can't yet.
  4. Put a retention policy on every log group.