Multipathing

How to properly setup a new SR with multipathing on Xen Orchestra and XCP-ng.

Multipathing uses several independent network or SAN paths between each host and the storage unit. It is the recommended setup for iSCSI and Fibre Channel SRs: a path failure (NIC, cable, switch, controller) doesn't interrupt storage traffic, and total throughput can be higher.

What requires care is when you enable it, not whether you should:

warning

Do not attempt to enable multipathing on a production pool with existing and active iSCSI and/or HBA and/or FC SRs: the device topology changes when multipathing takes over, which must not happen under active I/O. Enable it before creating these SRs.

You can activate it on the "fly", per XCP-ng host (Advanced tab), but it is recommended to do so with XCP-ng hosts that have no VMs running.

🌐 iSCSI

Requirements

Two different network interfaces.
Two different switches.
Multiple targets per LUN on your storage unit.
iSCSI target ports are operating in portal mode.

info

Keep the default MTU (1500) on your storage networks. Non-standard MTUs are a frequent source of hard-to-diagnose storage issues, for little to no gain on modern networks: see MTUs.

warning

We recommend that you do not use bonded network interfaces on the XCP-ng host and on the storage unit for iSCSI interfaces.
We recommend that you do not configure network routes on iSCSI interfaces.

This could have an impact on expected performance.

Target architecture

Configuration example

Path	Vlan	Subnet	XCP-ng Host PIF address	Storage Controller 1 address	Storage Controller 2 address
🔵	421	10.42.1.0/24	10.42.1.11	10.42.1.101	10.42.1.102
🟢	422	10.42.2.0/24	10.42.2.11	10.42.2.101	10.42.2.102

Target architecture diagram

iSCSI multipathing · two independent paths from host to storagepath A · VLAN 421path B · VLAN 422

Operating procedure

1. Prepare XCP-ng hosts

On one of the XCP-ng hosts, make sure that the multipath.conf configuration includes your storage equipment.

This can be found in the file /etc/multipath.xenserver/multipath.conf

warning
Do not modify /etc/multipath.xenserver/multipath.conf directly, as any changes may be overwritten by future system updates

If your equipment is not present, ask the manufacturer for the multipath configuration for GNU/Linux otherwise move on to the next step (Prepare the pool).

Add it to the file /etc/multipath/conf.d/custom.conf

For example:

devices {

  # Configuration for ACME CORP UltraSAN
  # This is an example of syntax; do not use it in production.
  device {
     vendor  "ACME"
     product "UltraSAN"
     path_selector "service-time 0"
     path_grouping_policy group_by_prio
     prio alua
     features "1 queue_if_no_path"
     hardware_handler "1 alua"
     failback immediate
     rr_weight uniform
     rr_min_io 100
     no_path_retry 10
  }
}

info

In this case, the configuration will be kept after updates.

If necessary, migrate the VMs active on the XCP-ng host in question to another one.
Reboot the XCP-ng host.
Do the same for all XCP-ng hosts in the pool (step 2. to 4.).

2. Prepare the pool

Make sure that multipathing is enabled on the pool. To do this, go to the advanced configuration of the pool.

If this is not the case:

Make sure there are no VMs running on an iSCSI and/or HBA SR in the pool.
Activate "Enable multipathing for all XCP-ng hosts.

3. Configure the SR

Proceed with the iSCSI SR configuration as indicated in the storage documentation.

🧵 Fibre Channel (HBA)

Requirements

Check that the Fibre Channel cards model(s) is supported via the HCL.
Two different Fibre Channel ports.
Two different SAN switches.
Multiple targets per LUN on your storage unit.
Zoning performed.

info

Multipath in XCP-ng is managed by the Storage Manager layer. Multipath devices are created only for LUNs that are actively used by Storage Manager. As a result, multipath -ll will show devices only after HBA SR has been created.

Storage Manager also controls /etc/multipath/wwids content. Do not edit this file manually and do not add entries with multipath -a.

To verify that LUN paths are visible on the host before a multipath device is created, use multipathd show paths format "%w %d %p %P %R %r %n".

warning

Make sure not to mix Fibre Channel speeds.

Target architecture

Target architecture diagram

Fibre Channel multipathing · two independent fabrics from host to storagefabric Afabric B

Operating procedure

1. Prepare XCP-ng hosts

On one of the XCP-ng hosts, make sure that the multipath.conf configuration includes your storage equipment.

This can be found in the file /etc/multipath.xenserver/multipath.conf.

warning
Do not modify /etc/multipath.xenserver/multipath.conf directly, as any changes may be overwritten by future system updates.

If your equipment is not present, ask the manufacturer for the multipath configuration for GNU/Linux otherwise move on to the next step (Prepare the pool).

Add it to the file /etc/multipath/conf.d/custom.conf

For example:

devices {

  # Configuration for ACME CORP UltraSAN
  # This is an example of syntax; do not use it in production.
  device {
     vendor  "ACME"
     product "UltraSAN"
     path_selector "service-time 0"
     path_grouping_policy group_by_prio
     prio alua
     features "1 queue_if_no_path"
     hardware_handler "1 alua"
     failback immediate
     rr_weight uniform
     rr_min_io 100
     no_path_retry 10
  }
}

info

In this case, the configuration will be kept after updates.

If necessary, migrate the VMs active on the XCP-ng host in question to another one.
Reboot the XCP-ng host.
Do the same for all XCP-ng hosts in the pool (step 2. to 4.).

2. Prepare the pool

Make sure that multipathing is enabled on the pool. To do this, go to the advanced configuration of the pool.

If this is not the case:

Make sure there are no VMs running on an iSCSI and/or HBA SR in the pool.
Activate "Enable multipathing for all XCP-ng hosts.

3. Configure the SR

Proceed with the HBA SR configuration as indicated in the storage documentation.

🔧 Maintenance operations

Add a new XCP-ng host to an existing multipathing pool

warning

Do not add the new XCP-ng host to the pool without completing these steps.

Prepare the XCP-ng host as specified in this operating procedure for iSCSI or this operating procedure for FC.
Ensure that the iSCSI PIF configuration is completed if you are using iSCSI.
Add the new XCP-ng host to the pool.

🧑‍⚕️ Troubleshooting

Verify multipathing

You can use the command multipath -ll to check if multipathing is active.

3600a098765432100000123456789abcd dm-3 ACME,UltraSAN
size=500G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 8:0:0:1  sdb  8:16   active ready running
| |- 8:0:1:1  sdd  8:48   active ready running
|-+- policy='service-time 0' prio=10 status=enabled
  |- 8:0:2:1  sdf  8:80   active ready running
  |- 8:0:3:1  sdh  8:112  active ready running

info

In this example, we have four active paths: our multipathing is working correctly.

iSCSI

Verify iSCSI sessions

You can use the command iscsiadm -m session to check if iSCSI session is active.

tcp: [1] 10.42.1.101:3260,1 iqn.2024-02.com.acme:ultrasan.lun01 (non-flash)
tcp: [2] 10.42.1.102:3260,1 iqn.2024-02.com.acme:ultrasan.lun01 (non-flash)
tcp: [3] 10.42.2.101:3260,2 iqn.2024-02.com.acme:ultrasan.lun01 (non-flash)
tcp: [4] 10.42.2.102:3260,2 iqn.2024-02.com.acme:ultrasan.lun01 (non-flash)

info

In this example, we have four iSCSI sessions with one LUN.

iSCSI: verify MTU consistency

An MTU mismatch between the host, the switches and the storage unit causes erratic storage behavior. To check for one, connect to a XCP-ng host and ping all IP addresses involved in your iSCSI storage (target and initiator) with a full-size, non-fragmentable packet:

iSCSI: verify MTU consistency

# ping -M do -s 1472 <REMOTE_IP_ADDRESS>

(1472 is for the standard MTU of 1500: packet size minus the 28 bytes of IP and ICMP headers. If your network uses another MTU, subtract 28 from it.)

warning

If you get an error, usually ping: sendmsg: Message too long, your MTU settings are inconsistent along the path, and you need to fix your network configuration.

Was this page helpful?

🌐 iSCSI​

Requirements​

Target architecture​

Configuration example​

Target architecture diagram​

Operating procedure​

1. Prepare XCP-ng hosts​

2. Prepare the pool​

3. Configure the SR​

🧵 Fibre Channel (HBA)​

Requirements​

Target architecture​

Target architecture diagram​

Operating procedure​

1. Prepare XCP-ng hosts​

2. Prepare the pool​

3. Configure the SR​

🔧 Maintenance operations​

Add a new XCP-ng host to an existing multipathing pool​

🧑‍⚕️ Troubleshooting​

Verify multipathing​

iSCSI​

Verify iSCSI sessions​

iSCSI: verify MTU consistency​

🌐 iSCSI

Requirements

Target architecture

Configuration example

Target architecture diagram

Operating procedure

1. Prepare XCP-ng hosts

2. Prepare the pool

3. Configure the SR

🧵 Fibre Channel (HBA)

Requirements

Target architecture

Target architecture diagram

Operating procedure

1. Prepare XCP-ng hosts

2. Prepare the pool

3. Configure the SR

🔧 Maintenance operations

Add a new XCP-ng host to an existing multipathing pool

🧑‍⚕️ Troubleshooting

Verify multipathing

iSCSI

Verify iSCSI sessions

iSCSI: verify MTU consistency