Let's use the following information of a Ceph cluster as the exampleto explain.


Ceph status

root@p-test-15:~# ceph -s
  cluster:
    id:     75fcf386-1d82-11ef-a874-7924015d89f1
    health: HEALTH_WARN
            Degraded data redundancy: 372519/3096552 objects degraded (12.030%), 32 pgs degraded, 32 pgs undersized
 
  services:
    mon: 2 daemons, quorum mars500-218,mars500-219 (age 3d)
    mgr: mars500-218.xhwghw(active, since 32h), standbys: mars500-219.umczhc
    mds: 1/1 daemons up, 1 standby
    osd: 16 osds: 16 up (since 3d), 16 in (since 3d)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 385 pgs
    objects: 875.25k objects, 1.9 TiB
    usage:   4.9 TiB used, 9.1 TiB / 14 TiB avail
    pgs:     372519/3096552 objects degraded (12.030%)
             353 active+clean
             32  active+undersized+degraded

Ceph usages

--- RAW STORAGE ---
CLASS    SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    14 TiB  9.1 TiB  4.9 TiB   4.9 TiB      34.87
TOTAL  14 TiB  9.1 TiB  4.9 TiB   4.9 TiB      34.87
 
--- POOLS ---
POOL              ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr               1    1  897 KiB        2  2.6 MiB      0    2.6 TiB
rbd_r3             2  128  1.1 TiB  293.45k  3.4 TiB  30.31    2.6 TiB
cephfs_pool.data   3   64  204 GiB   52.31k  613 GiB   7.20    2.6 TiB
cephfs_pool.meta   4   32   92 KiB       22  357 KiB      0    2.6 TiB
erasure_4p2.data   8  128  613 GiB  156.94k  920 GiB  10.42    5.1 TiB


Ceph osd usage

root@p-test-15:~# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS
 0    ssd  0.87329   1.00000  894 GiB  313 GiB  312 GiB    4 KiB   762 MiB  581 GiB  35.00  1.00   96      up
 1    ssd  0.87329   1.00000  894 GiB  323 GiB  323 GiB    6 KiB   980 MiB  571 GiB  36.18  1.04   99      up
 2    ssd  0.87329   1.00000  894 GiB  314 GiB  313 GiB    4 KiB   1.0 GiB  580 GiB  35.10  1.01  101      up
 3    ssd  0.87329   1.00000  894 GiB  324 GiB  323 GiB    6 KiB   899 MiB  570 GiB  36.25  1.04  101      up
 4    ssd  0.87329   1.00000  894 GiB  303 GiB  302 GiB    9 KiB   895 MiB  591 GiB  33.91  0.97   99      up
 5    ssd  0.87329   1.00000  894 GiB  356 GiB  354 GiB    3 KiB   1.1 GiB  539 GiB  39.76  1.14   96      up
 6    ssd  0.87329   1.00000  894 GiB  345 GiB  344 GiB   13 KiB   1.3 GiB  549 GiB  38.63  1.11   99      up
 7    ssd  0.87329   1.00000  894 GiB  310 GiB  309 GiB    3 KiB   874 MiB  585 GiB  34.63  0.99   91      up
 8    ssd  0.87329   1.00000  894 GiB  339 GiB  337 GiB    5 KiB   1.2 GiB  556 GiB  37.86  1.09   93      up
 9    ssd  0.87329   1.00000  894 GiB  253 GiB  253 GiB    2 KiB   515 MiB  641 GiB  28.31  0.81   84      up
10    ssd  0.87329   1.00000  894 GiB  339 GiB  338 GiB    6 KiB   1.2 GiB  555 GiB  37.92  1.09   92      up
11    ssd  0.87329   1.00000  894 GiB  288 GiB  287 GiB    4 KiB   752 MiB  606 GiB  32.23  0.92   88      up
12    ssd  0.87329   1.00000  894 GiB  262 GiB  261 GiB    6 KiB   528 MiB  633 GiB  29.25  0.84   87      up
13    ssd  0.87329   1.00000  894 GiB  343 GiB  342 GiB   12 KiB  1010 MiB  551 GiB  38.35  1.10   99      up
14    ssd  0.87329   1.00000  894 GiB  265 GiB  264 GiB    6 KiB   901 MiB  630 GiB  29.59  0.85   86      up
15    ssd  0.87329   1.00000  894 GiB  312 GiB  311 GiB    5 KiB   859 MiB  582 GiB  34.89  1.00   96      up
                       TOTAL   14 TiB  4.9 TiB  4.9 TiB  102 KiB    15 GiB  9.1 TiB  34.87                   
MIN/MAX VAR: 0.81/1.14  STDDEV: 3.38


  1. There are total of 7 pools. Pool erasure_4p2 is a erasure code pool K+M = 4 +2. The other pools are all replica 3 pools. 
    • Erasure code pool consumes 1.5 times raw space of the original data size.
    • Replica 3 pools consume 3 times raw space of the original data size.
  2. Total raw space of the cluster is 14 TiB. 
  3. Total 4.9 TiB raw space is used by all pools.
  4. 9.1 TiB raw space are available for future use. 


The Max Available column in the section of of the ceph df output is an estimate of the notional amount of data that can be written to this pool. It is the amount of data that can be used before the first OSD becomes full. It considers the projected distribution of data across disks from the CRUSH map and uses the first OSD to fill up as the target.


The estimation of the total available raw space:

  1. The total number of OSD is 16.
  2. Because Ceph data distribution is not perfectly even, the uses raw space of each OSD are not exactly same.
  3. The least available space OSD is 539 GiB (osd.5)
  4. The total available raw space is the multiply of the least OSD space and number of osd. 539 GiB * 16 =  8624 GiB.
  5. 8624 GiB is available for all pool to use. 
  6. Available space after data protection for replica 3 pools is (8624 / 3 ) * 0.95 = 2587 GiB = 2.52 TiB
  7.  Available space after data protection for erasure code pools is (8624 / 1.5 ) * 0.95 = 5462 GiB = 5.3 TiB


Reference

https://www.ibm.com/docs/en/storage-ceph/5?topic=cluster-understanding-storage-clusters-usage-stats