Background and Architectural Context

CockroachDB uses a shared-nothing architecture with data replicated across nodes using the Raft protocol. Each range (data shard) elects a leaseholder node, which handles reads and coordinates writes. In multi-region deployments, leaseholder placement directly impacts latency — if a client in one region communicates with a leaseholder in another, every transaction round-trip incurs inter-region latency. This is compounded in high-contention workloads where multiple transactions fight for the same keys.

Where This Occurs in Practice

  • Multi-region clusters with uneven leaseholder distribution
  • High-frequency updates on a small set of rows (hotspots)
  • Global workloads without geo-partitioning of data

Root Causes of the Problem

Leaseholder Misplacement

If leaseholders are not located close to the majority of transaction originators, every read/write incurs added latency.

Unbounded Transaction Retries

Contention on popular keys triggers transaction restarts, especially in serializable isolation. This can spiral under load if retry loops are not controlled.

Improperly Sized Ranges

Overly large ranges increase Raft commit latency and slow down rebalancing, leading to hotspots.

Diagnostics and Detection

Identify Hot Ranges

cockroach debug hot-ranges --url "http://$NODE:8080"

Check for ranges with high QPS and verify leaseholder locality.

Monitor Transaction Retries

SELECT node_id, retries_per_sec FROM crdb_internal.node_statement_statistics
ORDER BY retries_per_sec DESC;

High retries may indicate contention on specific keys or ranges.

Check Latency by Region

cockroach sql --execute "SHOW RANGES"

Verify that ranges are distributed according to the intended locality policy.

Common Pitfalls

  • Deploying without explicit locality settings
  • Assuming automatic balancing will fully optimize leaseholder placement
  • Ignoring contention metrics until retries cause timeouts

Step-by-Step Fixes

1. Configure Locality and Lease Preferences

ALTER RANGE default CONFIGURE ZONE USING
  lease_preferences = '[[+region=us-east]]';

Place leaseholders near the majority of traffic to reduce latency.

2. Implement Geo-Partitioning

ALTER TABLE users PARTITION BY LIST (region)
(PARTITION us_east VALUES IN ('us-east'),
 PARTITION eu_west VALUES IN ('eu-west'));

Ensure data is stored and served in the region closest to its consumers.

3. Limit Transaction Scope

Reduce the number of ranges a transaction touches to minimize Raft coordination overhead.

4. Tune Range Sizes

ALTER RANGE default CONFIGURE ZONE USING range_max_bytes = 67108864;

Smaller ranges can rebalance more quickly and reduce contention.

5. Apply Backoff to Retries

Client drivers should implement exponential backoff for retries to avoid retry storms during contention spikes.

Long-Term Architectural Solutions

  • Design schemas to avoid hotspots (e.g., use UUID keys instead of sequential IDs)
  • Align leaseholder placement policies with traffic distribution
  • Continuously monitor crdb_internal metrics for contention

Performance Optimization Considerations

Proper leaseholder placement alone can reduce p99 transaction latency by over 50% in multi-region setups. Geo-partitioning further eliminates unnecessary cross-region traffic, improving both throughput and availability.

Conclusion

CockroachDB's global consistency guarantees are powerful but demand careful topology and workload planning. Performance degradation from poor leaseholder placement and high contention can cripple otherwise well-provisioned clusters. By leveraging geo-partitioning, tuning range sizes, and applying intelligent retry logic, architects can maintain predictable performance at scale while preserving CockroachDB's consistency and fault tolerance.

FAQs

1. Is leaseholder placement fully automated in CockroachDB?

Yes, but automation may not align with workload patterns. Manual lease preferences often yield better performance for predictable traffic.

2. Can contention be completely eliminated?

No, but it can be reduced by schema design, range tuning, and partitioning strategies that minimize hot keys.

3. Does CockroachDB support region-aware queries?

Yes. With locality settings and geo-partitioning, queries can be served preferentially from the closest region.

4. How do retries affect application logic?

Retries can cause user-visible delays if not handled properly. Applications must be idempotent and implement backoff strategies.

5. Are smaller ranges always better?

Not necessarily. Too-small ranges can increase metadata overhead. The optimal size balances rebalancing speed with storage efficiency.