Related Alerts

• sequencer-low-peers
• replica-low-peers

Alert Severity

• SEV2, unless the low peers issue has caused an unsafe head halt in which case refer to ‣ for the appropriate severity.

Alert Summary

This alert indicates that the sequencer or a replica has an abnormally low number of peers. The sequencer’s peer count should always match the number of internal replicas. Replicas should generally have between 10 and 40 peers each.

This error often occurs after a replica restart.

Investigation

1. Look at the Bedrock P2P dashboard to confirm the number of peers the sequencer/replicas actually have. Also look at the bandwidth in/out graphs to make sure that data is actually being communicated between each peer. Healthy graphs should look something like this:

   

2. If the alert is sequencer-low-peers, check to see if any of the replicas or the sequencer were restarted recently. If so, replicas should automatically reconnect to the sequencer after 60 seconds.

3. Search the op-node error logs for errors indicating peer-to-peer failures. Look for errors dialing peers.

4. Search the op-node error logs for logs regarding discovery. A warning peer discovery is disabled means that the node is not actively discovering peers. This should appear on the sequencer, but not replicas.

Mitigation

1. If related to a restart, do nothing - the alert will self-close in a few minutes.
2. Restart any affected replicas. This will cause it them retry any statically connected peers.
3. Escalate if the issue persists.