The two 10Gbps upper-link ports of every Mars 400 in-chassis switch are also bonded to provide aggregated bandwidth. So, if you connect two links from the same Mars 400 switch to the Top-of-Rack switch ports, you have to configure the Top-of-Rack switch ports as static link aggregation Group (or static port-channel). Please note, don't configure the ToR switch ports as LACP.
You can use the Mars 400 BMC to configure the bonding mode of two 2.5Gb network interfaces on each Arm node to be a bonded ethernet interface. This bonding configuration provides a redundant mechanism to keep the network communication even one of the in-chassis fails. Therefore, the Ceph in-cluster and public network share the same bonding network.
The Mars 400 BMC supports you to configure the bonding of two network interface Arm node to be mode 6 (adaptive load balancing mode) or mode 2 (balance-XOR). Mode 6 is recommended for new deployments because it is simple for the ToR switch. You don't have to make extra configuration for the bonding mode 6 on Mars 400. Please refer to Figure 1.
The old version of Mars 400 BMC supports only the bonding mode 2 configuration. You need to configure another static port-channel for all the links from the same Mars 400. Please refer to Figure 2.
Figure 1: When Mars 400 nodes are configured as bonding mode 6 (balance-alb).
Figure 2: When Mars 400 nodes are configured as bonding mode 2 (balance-XOR).
