TODO: Replace ‣ with this runbook once psp_executor is provisioned properly for both Superchain Mainnet and Sepolia.

Background

As deputy guardian of the Superchain, Optimism Foundation has authorized both OP Labs and Coinbase to maintain pre-signd pauses so that authorized personnel can unilaterally execute a Superchain-wide pause to pause withdrawals across all chains in the Superchain.

For OP Labs, the pre-signed pauses are kept in GCP Secret Manager and a psp_executor API server in k8s is responsible for monitoring, retrieving and executing the pre-signed pause when called.

This runbook outlines how an authorized personnel (OP Labs Security Oncall for Superchain Mainnet, and all signers on the Superchain Sepolia guardian multisig for Superchain Sepolia) can leverage the API service to execute the pre-signed pause.

Steps

1. Authenticate with the right k8s cluster:

   $ gcloud container clusters get-credentials oplabs-dev-client-primary --region us-central1 --project oplabs-dev-client
   
   Reauthentication required.
   Please touch your security key.
   Fetching cluster endpoint and auth data.
   kubeconfig entry generated for oplabs-dev-client-primary.
   
   

2. Make sure Tailscale VPN is on.

3. kubie ctx and choose the right cluster.

   1. Production (Mainnet & Sepolia): gke_oplabs-tools-security_us-central1_oplabs-tools-security-primary
   2. Development (Mainnet & Sepolia): gke_oplabs-dev-security_us-central1_oplabs-dev-security-primary

4. kubie ns and choose the op-defender namespace.

5. kubectl get all and keep a note of the pod name to use.

   1. e.g. pod/psp-executor-op-sepolia-dev-5db657c574-m4lws .

6. Port forward the API server to localhost with the correct pod.

   1. Production (Sepolia on port 8080): kubectl port-forward pod/psp-executor-op-**sepolia**-**prod**-5db657c574-m4lws **8080**:8080
   2. Development (Mainnet on port 8888): kubectl port-forward pod/psp-executor-op-**mainnet**-**dev**-4ac887c533-339gp **8888**:8080

7. Use cURL to call the HTTP API psp-executor

   # PROD
   $ curl -X POST <http://localhost:8080/api/psp_execution> \\-H "Content-Type: application/json" \\-d '{"Pause":true,"Timestamp":1596240000,"Operator":"your name"}' 
   
   {"message":"PSP executed successfully","status":200}
   
   # DEV
   $ curl -X POST <http://localhost:8888/api/psp_execution> \\-H "Content-Type: application/json" \\-d '{"Pause":true,"Timestamp":1596240000,"Operator":"your name"}' 
   

8. Verify if the pause is successful onchain

   1. Production (Mainnet):

      cast call 0x95703e0982140D16f8ebA6d158FccEde42f04a4C "paused()(bool)" --rpc-url <https://ethereum.publicnode.com> 
      

   2. Production (Sepolia):

      cast call 0xC2Be75506d5724086DEB7245bd260Cc9753911Be "paused()(bool)" --rpc-url <https://ethereum-sepolia.publicnode.com> 
      
      

   3. Development (Mainnet or Sepolia):

      # Use the RPC the tenderly provided RPC for the forks. 
      cast call 0xC2Be75506d5724086DEB7245bd260Cc9753911Be "paused()(bool)" --rpc-url <https://tenderly.co/[XXXXX...XXXX]>
      

References

Other useful links for the operator to refer to.

1. Code: https://github.com/ethereum-optimism/monitorism/tree/main/op-defender
2. k8s config: https://github.com/ethereum-optimism/k8s/tree/master/kustomize/op-defender
3. Alert config: TODO
4. Grafana Dashboard: https://optimistic.grafana.net/d/adyd382uer9q8d/op-defender?from=now-6h&to=now&timezone=browser&var-cluster=oplabs-dev-security-primary&var-job=op-defender/psp-executor-op-mainnet-dev