Fixing Network Throughput and BGP Instability in Equinix Metal Deployments

Details: Category: Cloud Platforms and Services; By Mindful Chase; 21.Apr; Hits: 70

Equinix Metal offers bare-metal infrastructure with cloud-like provisioning, making it ideal for high-performance workloads, custom OS environments, and edge computing. However, teams running production environments often encounter the challenge of "network throughput inconsistencies and packet loss during high concurrency or BGP session instability". These issues manifest as degraded application performance, failed health checks, or intermittent disconnections in Kubernetes, NFV, or custom routing setups. This article explains the architectural underpinnings of Equinix Metal's network stack, investigates root causes, and provides concrete steps to stabilize and scale networking in real-world deployments.

Mindful Chase

Writing Code, Writing Stories

tbd

Experience

tbd

More to Explore

Understanding Equinix Metal Networking Architecture

Layer 2 and Layer 3 Modes

Equinix Metal supports both Layer 2 (VLAN) and Layer 3 (routed) networking. Users can configure multiple bonded interfaces, BGP sessions, and VRFs (Virtual Routing and Forwarding) to support advanced routing and traffic engineering. Misconfiguration or under-observability of these modes can lead to routing black holes, BGP flaps, or dropped packets during failovers or scale events.

NIC and Bonding Configurations

Network interfaces on Metal servers are typically preconfigured using bond0 (LACP or active-backup) for redundancy. Custom configurations require careful tuning, especially under high-traffic or multi-tenant use cases.

Symptoms of Network Instability

High packet loss or retransmits under load
Intermittent drops in HTTP/gRPC traffic or Kubernetes readiness probes
BGP sessions dropping and reconnecting repeatedly
Throughput caps well below expected line rate (e.g., sub-5Gbps on a 10Gbps interface)
Cluster flapping or network policy errors in CNI plugins

Root Causes

1. Suboptimal BGP Session Configuration

BGP sessions without proper keepalive, hold timers, or MTU matching may flap under latency or retransmit conditions, impacting routing and DNS resolution.

2. Missing or Conflicting MTU Settings

Equinix Metal defaults to 1500 MTU. Overlays like Calico or WireGuard require matching MTUs; mismatches result in fragmented or dropped packets.

3. Bonding Misconfiguration (LACP vs Active-Backup)

Improper bonding mode or misaligned hashing algorithms (layer2 vs layer3+4) can cause uneven traffic distribution and degraded throughput on multi-interface hosts.

4. Interrupt Coalescing or NIC Queue Saturation

High packet-per-second rates without NIC tuning (e.g., RX/TX buffer tuning, IRQ balance) can drop packets silently or bottleneck a single queue.

5. Incomplete Firewall or ACL Rules

Security groups, custom iptables, or misconfigured CNIs (e.g., Calico, Cilium) can block required routes or sessions, particularly in hybrid cloud or BGP-peered setups.

Diagnostics and Tools

1. Monitor NIC Stats with `ethtool`

ethtool -S bond0 | grep -i drop

Check for interface-level RX/TX errors or packet drops on bonded and physical interfaces.

2. Verify BGP Health

birdc show protocols all

Inspect BIRD daemon status and session lifetimes. Flapping or down sessions indicate routing instability.

3. Measure Performance with `iperf3`

Run controlled throughput tests between nodes to validate expected bandwidth vs observed throughput.

4. Use `tcpdump` for MTU and Fragment Analysis

tcpdump -i bond0 -n "icmp or tcp"

Look for ICMP "fragmentation needed" messages or excessive retransmits.

5. Cross-check CNI Plugin Logs

Check logs from Calico, Cilium, or Flannel for policy drops, node sync issues, or tunnel interface errors.

Step-by-Step Fix Strategy

1. Optimize BGP Configuration

Set keepalive to 30s, hold timer to 90s, and ensure consistent ASN and address families. Match MTUs with upstream router and use graceful restart if supported.

2. Align MTU Across Stack

Set consistent MTU (e.g., 1450) across bond interfaces, CNI overlays, and host routes to prevent fragmentation. Use ip link show and ip route get to verify.

3. Tune Bonding Mode and Queue Settings

echo 3 > /sys/class/net/bond0/bonding/mode

Use active-backup for simplicity, or LACP with xmit_hash_policy=layer3+4 for better distribution. Adjust RX/TX queue length and enable multiqueue if needed.

4. Enable IRQ Balancing and Increase Buffers

Install irqbalance and tune buffer sizes via ethtool -G to prevent packet loss during peak periods.

5. Segment Traffic via VLANs or VRFs

Isolate control-plane vs data-plane traffic. Use Metal’s support for Layer 2 VLANs and VRF-aware routing to reduce congestion and policy conflicts.

Best Practices

Always baseline NIC performance with iperf3 post-provisioning
Match MTU end-to-end, including tunnels and peer interfaces
Monitor BGP session uptime continuously via Prometheus or Birdwatcher
Use Calico BGP peering for Kubernetes with route reflectors in larger clusters
Deploy network observability tools like Netdata or Grafana + node_exporter

Conclusion

Equinix Metal offers high-performance, cloud-native bare-metal infrastructure, but network tuning is essential for consistency under modern, distributed workloads. Understanding the implications of BGP, bonding modes, MTU sizing, and NIC queue management helps teams avoid common pitfalls. With careful configuration, layered observability, and traffic isolation, Metal users can build resilient and high-throughput environments suitable for demanding edge and core cloud applications.

FAQs

1. What is the default MTU on Equinix Metal servers?

1500 bytes. Custom overlays or tunnels require coordinated MTU adjustments across the stack.

2. How can I detect BGP flapping?

Use birdc show protocols or export metrics to Prometheus to observe session up/down transitions.

3. Can I use LACP bonding on Equinix Metal?

Yes, but ensure both server and upstream switch support LACP. Set the correct hash policy to balance flows efficiently.

4. What causes dropped packets despite low CPU usage?

NIC queue overflow, lack of IRQ balancing, or MTU mismatch are likely causes. Monitor NIC stats and tune buffers accordingly.

5. Is Layer 2 networking supported in Kubernetes on Metal?

Yes. Equinix Metal supports Layer 2 with VLANs, which can be used for flat networking or advanced CNI topologies.

Contact Us