OpenShift PVC Mount failure: already contains mpath_member

A while ago, the VMs that hosted my OpenShift platform were rebooted. For most of it, all nodes and containers came back gracefully. However, there were a few containers that did not start because they had Persistent Volumes Claims to underlying SSD storage that kept failing with the following message:

MountVolume.MountDevice failed for volume "pvc-f6220425-22dd-11e8-ad31-000d3af45c1a" : azureDisk - mountDevice:FormatAndMount failed with failed to mount the volume as "ext4", it already contains mpath_member.

After some digging, I found the following: https://bugzilla.redhat.com/show_bug.cgi?id=1550271

Comment #14 in particular contains some critical steps that helped me to solve this issue.

On each of my app and infra nodes I followed these steps:

1) Add the “find_multipaths yes” line in /etc/multipath.conf:

$ cat /etc/multipath.conf
# LIO iSCSI
# TODO: Add env variables for tweaking
devices {
    device {
        vendor "LIO-ORG"
        user_friendly_names "yes"
        path_grouping_policy "failover"
        path_selector "round-robin 0"
        failback immediate
        path_checker "tur"
        prio "const"
        no_path_retry 120
        rr_weight "uniform"
    }
} 
blacklist {
} 
defaults {
    find_multipaths "yes" #### ADD THIS LINE ####
}

2) Remove the device from the wwids file. Example:

$ multipath -w /dev/sdd
wwid '3600224800e3810d32614a31726daf7c8' removed

3) Restart multipath service:

$ systemctl restart multipathd

Regarding step 2 above, some commands that can help you to find the device that needs to be removed are:

$ lsblk
  sdd                                                                                               8:48   0   10G  0 disk 
  └─3600224808cbb2ebf8b08ee87764f24d8                                                             253:3    0   10G  0 mpath
  sde                                                                                               8:64   0   10G  0 disk 
  └─3600224805e7c9eed650ebf3acdf00000                                                             253:1    0   10G  0 mpath
 
$ multipath -ll

Leave a Reply

Your email address will not be published. Required fields are marked *

*