Mars 400 Ceph cluster Maintenance SOP

This document is intended to provide a standard operation procedure (SOP) for maintaining Mars400 series systems.


If the client access is slow or stuck, the top reason is OSD slow ops. OSD slow ops are usually caused by one or more OSD nodes on Mars400 being frozen. Please follow the Procedure A to resolve the slow ops. 


If there is an OSD down and doesn't back to the up status, Please follow the Procedure B 


A. Procedure to Solve Slow Request/Ops (Reset the OSD node causes slow ops)

The following steps are intended to solve the problem of slow requests/ops.

  1. Log in to the Ceph mon node.
  2. Execute the ceph -s command.
  3. Check if there are words like "slow ops" in the output.
  4. The lines that contain "slow ops" usually contain multiple OSD IDs.
    For example, the output is like this:
SLOW_OPS 8 slow ops, oldest one blocked for 1314 sec, osd.49 has slow ops
  1. In this case, the osd that has slow ops is osd.49.
  2. Execute ceph osd find <osd id> to get the hostname of the osd, and then get the chassis and node number from the hostname.
  3. Log in to the current chassis's BMC.
  4. Enter advance at the console.
  5. Enter 1 to select the mod reset option.
  6. Enter the node number to reset.
  7. If there are multiple OSD IDs in the list, please wait until the OSD
    goes back to the cluster after each execution.

B. Procedure to Solve OSD Down (Reset node)

The following steps are intended to solve the problem of osd down.

  1. Log in to the Ceph mon node.
  2. Execute the ceph -s command.
  3. Check the osd number and the status of pgs.
  4. Run ceph osd tree down to get the osd id that is down. 
  5. ceph osd tree down
    . Get the chassis and node number from the output.
  6. Log in to the current chassis's BMC.
  7. Enter advance at the console.
  8. Enter 1 to select the mod reset option.
  9. Enter the node number to reset.

C. Procedure to Solve OSD Down After Reboot (Fix OSD bluestore metadata)

The following steps are intended to solve the problem of osd down after reboot.

  1. Log in to the Ceph mon node.
  2. Execute the ceph -s command.
  3. Check the osd number and the status of pgs.
  4. Run ceph osd tree down to get the osd id that is down.
  5. Use ssh to remote access the down osd.
  6. Run the following command to fix the osd:
img=$(ceph config dump -f json | jq -r '.[]|select(.section=="global" and .name=="container_image").value')
sudo cephadm --image $img shell --fsid <ceph fsid>  --name osd.<osd id>  
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<osd id> --command fsck
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-<osd id> --command repair
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-<osd id> --op fsck
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-<osd id> --op repair
  1. After that, restart the ceph-osd service or reboot the system to bring the OSD online.