When you have to remove an up OSD for hardware maintenance, you can do an valuation to see if it is ok to stop an OSD.
You can evaluate this in two ways.
1. Check the OSD usage to see if the cluster has enough available capacity after the OSD is out.
2. Ceph has a command to evaluate if it is ok to stop an OSD.
Example: ceph osd ok-to-stop osd.19
You can see from the following command outputs. It is ok to stop osd.19.
Reference document: https://docs.ceph.com/en/quincy/man/8/ceph/
3. You can use safer approach comparing to remove the up OSD directly. Moving the osd to the recycle bin will gracefully move data to other osds instead of removing it directly to avoid PG degradation. Although this method requires two backfilling times, it ensures the safety of the PG and the overall system.
root@stor-ch1-n8:~# ceph osd df |grep ssd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
2 ssd 3.59499 1.00000 3.6 TiB 796 GiB 691 GiB 99 KiB 2.2 GiB 2.8 TiB 21.61 1.99 149 up
3 ssd 3.59499 1.00000 3.6 TiB 797 GiB 692 GiB 94 KiB 2.6 GiB 2.8 TiB 21.64 1.99 149 up
4 ssd 3.59499 1.00000 3.6 TiB 801 GiB 697 GiB 88 KiB 2.3 GiB 2.8 TiB 21.76 2.00 148 up
5 ssd 3.59499 1.00000 3.6 TiB 799 GiB 694 GiB 110 KiB 2.4 GiB 2.8 TiB 21.70 2.00 148 up
6 ssd 3.59499 1.00000 3.6 TiB 796 GiB 692 GiB 84 KiB 2.2 GiB 2.8 TiB 21.63 1.99 147 up
9 ssd 3.59499 1.00000 3.6 TiB 807 GiB 703 GiB 90 KiB 2.6 GiB 2.8 TiB 21.93 2.02 151 up
10 ssd 3.59499 1.00000 3.6 TiB 803 GiB 699 GiB 82 KiB 2.5 GiB 2.8 TiB 21.82 2.01 152 up
11 ssd 3.59499 1.00000 3.6 TiB 807 GiB 703 GiB 112 KiB 2.7 GiB 2.8 TiB 21.92 2.02 151 up
12 ssd 3.59499 1.00000 3.6 TiB 802 GiB 697 GiB 93 KiB 2.1 GiB 2.8 TiB 21.77 2.00 150 up
13 ssd 3.59499 1.00000 3.6 TiB 803 GiB 699 GiB 97 KiB 2.5 GiB 2.8 TiB 21.82 2.01 148 up
16 ssd 3.59499 1.00000 3.6 TiB 815 GiB 711 GiB 93 KiB 2.6 GiB 2.8 TiB 22.15 2.04 152 up
17 ssd 3.59499 1.00000 3.6 TiB 814 GiB 710 GiB 88 KiB 2.5 GiB 2.8 TiB 22.12 2.04 151 up
14 ssd 3.59499 1.00000 3.6 TiB 814 GiB 709 GiB 88 KiB 2.3 GiB 2.8 TiB 22.11 2.03 152 up
19 ssd 3.59499 1.00000 3.6 TiB 817 GiB 713 GiB 90 KiB 2.3 GiB 2.8 TiB 22.19 2.04 151 up
20 ssd 3.59499 1.00000 3.6 TiB 815 GiB 710 GiB 92 KiB 2.6 GiB 2.8 TiB 22.13 2.04 151 up
23 ssd 3.59499 1.00000 3.6 TiB 797 GiB 692 GiB 86 KiB 2.0 GiB 2.8 TiB 21.64 1.99 148 up
24 ssd 3.59499 1.00000 3.6 TiB 776 GiB 671 GiB 102 KiB 2.4 GiB 2.8 TiB 21.07 1.94 145 up
25 ssd 3.59499 1.00000 3.6 TiB 783 GiB 679 GiB 95 KiB 1.6 GiB 2.8 TiB 21.27 1.96 144 up
26 ssd 3.59499 1.00000 3.6 TiB 779 GiB 675 GiB 85 KiB 2.5 GiB 2.8 TiB 21.17 1.95 143 up
27 ssd 3.49309 1.00000 3.5 TiB 659 GiB 656 GiB 108 KiB 2.3 GiB 2.9 TiB 18.41 1.69 144 up
28 ssd 3.59499 1.00000 3.6 TiB 777 GiB 673 GiB 90 KiB 2.3 GiB 2.8 TiB 21.10 1.94 145 up
root@stor-ch1-n8:~# ceph osd ok-to-stop osd.19
{"ok_to_stop":true,"osds":[19],"num_ok_pgs":151,"num_not_ok_pgs":0,"ok_become_degraded":["4.1","4.6","4.e","4.13","4.17","4.18","4.2a","4.2b","4.38","4.3e","4.41","4.4f","4.51","4.53","4.60","4.6d","4.6e","4.71","4.73","4.80","4.82","4.85","4.88","4.96","4.9a","4.9f","4.aa","4.b1","4.b4","4.ba","4.c8","4.c9","4.d1","4.d4","4.dc","4.e8","4.eb","4.ec","4.104","4.109","4.10e","4.127","4.12b","4.12e","4.12f","4.140","4.142","4.148","4.149","4.14c","4.150","4.151","4.160","4.166","4.18d","4.197","4.19e","4.1a3","4.1ac","4.1c5","4.1c6","4.1c7","4.1ce","4.1cf","4.1d8","4.1e2","4.1e6","4.1f5","4.1f8","4.1fe","4.200","4.203","4.209","4.20a","4.20c","4.20e","4.213","4.215","4.21d","4.229","4.22a","4.235","4.250","4.251","4.259","4.25c","4.25d","4.262","4.264","4.266","4.279","4.27a","4.286","4.289","4.28d","4.294","4.29c","4.29d","4.2a3","4.2a5","4.2a9","4.2c4","4.2c8","4.2c9","4.2cd","4.2e5","4.2e7","4.2ee","4.2ef","4.2f4","4.306","4.308","4.30f","4.316","4.31f","4.32c","4.33f","4.343","4.348","4.349","4.34d","4.353","4.356","4.35a","4.35b","4.365","4.369","4.36b","4.36e","4.372","4.378","4.37a","4.38e","4.392","4.393","4.396","4.3a3","4.3a9","4.3ab","4.3ae","4.3cf","4.3d6","4.3d8","4.3e3","4.3ea","4.3ed","4.3ee","4.3f3","4.3f8","4.3fb","6.12"]}