In this article, we will analyze the root causes of compaction storms in Cassandra, explore debugging techniques, and provide best practices for optimizing compaction strategies to ensure cluster stability.

Understanding Compaction Storms in Cassandra

Compaction is the process of merging SSTables (Sorted String Tables) to optimize read performance. However, in high-write workloads, excessive compaction operations can lead to:

• High disk I/O, reducing available bandwidth for read and write operations.
• Increased CPU usage, leading to query latency.
• Storage bloat due to inefficient SSTable management.
• Potential node failures if compaction cannot keep up with write throughput.

Common Symptoms

• High CPU and disk I/O usage without an increase in client requests.
• Slow queries and increased read latency.
• Growing disk space usage despite deletion of old data.
• Frequent warnings in logs related to pending compactions.

Diagnosing Compaction Issues

1. Checking Compaction Statistics

Monitor ongoing compaction activity using:

nodetool compactionstats

If there are too many pending compactions, the system is likely overwhelmed.

2. Monitoring Pending Compactions

Check the pending compaction queue:

nodetool tpstats | grep CompactionExecutor

3. Analyzing SSTable Growth

Identify excessive SSTable growth per table:

nodetool cfstats | grep SSTable

4. Checking Disk I/O Impact

Monitor disk usage using:

iostat -dx 1

If disk I/O is consistently high, compaction may be a bottleneck.

Fixing Compaction Storms in Cassandra

Solution 1: Adjusting Compaction Strategies

Cassandra supports multiple compaction strategies. Switching to a more efficient strategy can reduce compaction overhead.

For write-heavy workloads, use Leveled Compaction Strategy (LCS):

ALTER TABLE my_table WITH compaction = { 
  'class': 'LeveledCompactionStrategy', 
  'sstable_size_in_mb': '20' 
};

For read-heavy workloads, use Size-Tiered Compaction Strategy (STCS):

ALTER TABLE my_table WITH compaction = { 
  'class': 'SizeTieredCompactionStrategy', 
  'min_threshold': '4' 
};

Solution 2: Limiting Concurrent Compactions

Reduce CPU and disk contention by limiting concurrent compactions:

nodetool setcompactionthroughput 32

Lower values reduce disk contention but may slow down the compaction process.

Solution 3: Flushing Data to Reduce SSTable Growth

Manually flush data to reduce memory pressure and prevent SSTable accumulation:

nodetool flush

Solution 4: Using nodetool compact for Manual Compaction

Manually trigger compaction for specific tables:

nodetool compact my_keyspace my_table

Use this carefully to avoid excessive disk I/O.

Best Practices for Compaction Optimization

• Use appropriate compaction strategies based on workload patterns.
• Monitor pending compactions using nodetool compactionstats.
• Adjust compaction_throughput_mb_per_sec to balance performance and disk I/O.
• Perform manual compaction during low-traffic hours.
• Use nodetool flush to prevent SSTable accumulation.

Conclusion

Compaction storms in Cassandra can severely impact performance and node stability. By adjusting compaction strategies, limiting concurrent compactions, and actively monitoring disk I/O, database administrators can optimize Cassandra for high-performance workloads.

FAQ

1. How do I check if my Cassandra node is overwhelmed by compaction?

Use nodetool compactionstats and nodetool tpstats to check for high pending compactions.

2. What compaction strategy is best for high-write workloads?

Leveled Compaction Strategy (LCS) is better suited for write-heavy applications.

3. Can I manually trigger compaction in Cassandra?

Yes, use nodetool compact to manually start compaction on specific tables.

4. How do I reduce high disk I/O caused by compaction?

Lower compaction_throughput_mb_per_sec and limit concurrent compactions.

5. What happens if compaction cannot keep up with writes?

Excessive SSTables accumulate, leading to increased read latency and potential node failures.