Understanding InfluxDB Storage Engine

TSM and WAL Architecture

InfluxDB uses a Time Structured Merge Tree (TSM) format for durable, compressed storage. Incoming writes are buffered in a Write Ahead Log (WAL) before being compacted and flushed to TSM files. Over time, TSM files accumulate into shards, which can grow inefficiently if not properly configured or aged out.

Retention Policies and Shards

Data is segmented into shards based on retention policy duration and time interval. A single retention policy may create dozens or hundreds of shards depending on configuration. Misaligned policies can cause excessive shard creation, compaction overhead, and data lingering beyond its useful life.

Common Symptoms of Storage and Write Issues

  • Disk usage grows rapidly despite low active retention periods
  • WAL files do not flush properly or appear to stall compaction
  • High CPU or I/O during background compaction cycles
  • Slow write performance or dropped points under load
  • Queries against older data are unusually slow or timeout

Root Causes

1. Improper Retention Policy Configuration

Setting a long retention duration without data expiration results in unbounded growth. Default policies often retain data indefinitely unless explicitly overridden.

2. High Series Cardinality

Too many unique series (measurement + tag key/value combinations) overload the index, memory, and write path. This is common when unbounded tag values (e.g., user IDs, hostnames) are used improperly.

3. Inefficient Shard Duration

Using very short shard durations (e.g., 1h) creates excessive shard files and compaction events. Each shard has its own index and lifecycle overhead.

4. WAL Accumulation Due to Write Backlog

InfluxDB flushes WAL buffers asynchronously. If disk or CPU is saturated, WAL flush stalls, leading to long startup times and increased memory use.

5. Continuous Queries Without Downsampling

Storing raw data without rollups for long periods puts pressure on both storage and query performance. Without downsampling, queries scan large volumes of fine-grained points unnecessarily.

Diagnostics and Tools

1. Check Shard Sizes and Retention

SHOW SHARDS
SHOW RETENTION POLICIES ON "mydb"

Review shard durations and verify if expired shards are still retained due to default policy misconfiguration.

2. Monitor WAL and TSM Compaction

influx_inspect report -dir /var/lib/influxdb/data

Analyze compaction patterns and WAL size. Look for excessive TSM files or large unflushed WAL buffers.

3. Measure Series Cardinality

SHOW SERIES CARDINALITY
SHOW MEASUREMENT CARDINALITY

Identify measurements or tag sets with exploding series count.

4. Use Internal Diagnostics

Query internal stats from _internal database:

SELECT * FROM "runtime" WHERE time > now() - 10m

Inspect heap usage, goroutines, and flush timings.

5. Profile Compactions

Enable logging at DEBUG level for storage engine to trace compaction intervals, shard activity, and flush delays.

Step-by-Step Fix Strategy

1. Define Explicit Retention Policies

CREATE RETENTION POLICY "one_month" ON "metrics" DURATION 30d REPLICATION 1 DEFAULT

Prevent indefinite data retention. Always define data lifecycle expectations explicitly.

2. Tune Shard Duration per Measurement

Use longer shard durations (e.g., 7d or 30d) for low-frequency or long-term data to reduce overhead.

3. Use Downsampling and Rollups

Define Continuous Queries (CQs) to downsample raw data and store in secondary retention policies:

CREATE CONTINUOUS QUERY "cq_avg" ON "metrics" BEGIN SELECT mean(value) INTO "metrics"."downsampled"."cpu_avg" FROM "cpu" GROUP BY time(10m) END

4. Eliminate High Cardinality Tags

Avoid user-specific or device-specific tag keys that grow unbounded. Replace with numeric IDs or hash grouping.

5. Enable Compaction Tuning

In influxdb.conf, adjust compact-full-write-cold-duration and max-series-per-database to control compaction pressure and memory growth.

Best Practices

  • Always define retention policies with proper duration and default setting
  • Group metrics logically to control shard count and size
  • Use numeric or static tag values for high-ingest pipelines
  • Downsample high-volume measurements to reduce long-term cost
  • Regularly monitor WAL and compaction status via influx_inspect

Conclusion

InfluxDB is optimized for time-series data, but proper configuration is essential to avoid unbounded storage growth and write slowdowns. By controlling retention policy durations, avoiding high-cardinality tag keys, and implementing downsampling strategies, teams can build scalable and predictable time-series pipelines. Monitoring compaction behavior and shard layout is crucial for diagnosing performance anomalies in production deployments.

FAQs

1. Why is my disk usage growing even with a retention policy set?

Check if expired shards are still active or if data is being written to the wrong retention policy.

2. How do I find the worst offenders in series cardinality?

Use SHOW SERIES CARDINALITY and analyze which tags or measurements have the most combinations.

3. Can I delete WAL files manually?

No. Let InfluxDB manage WAL files. Manual deletion can corrupt your database or result in data loss.

4. What’s a safe shard duration for high-volume metrics?

Typically 7d is ideal for daily metrics. Avoid 1h unless your use case requires high temporal resolution with short TTL.

5. Is downsampling required for all data?

Not mandatory, but highly recommended for any data retained longer than a few weeks. It improves performance and reduces cost.