After an upgrade to a new OVA image Storage Connector is unable to read and write to NFS-mounted storage

Problem description

After a Storage Connector server is upgraded to a new Open Virtual Appliance (OVA) image, the Group/User identifiers (GID/UID) of the syncp-storage user can become different to the ones used in the previous OVA version. The reason for this difference (for example, between CentOS 6 and CentOS 7 based OVAs) is an increase in the range of UID/GID reserved for dynamic allocation by the system. These GID/UID mismatches can make Storage Connectors that use NFS-mounted storage unable to read and write to this storage, because the numerical value of the UID/GID for the syncp-storage user is different from what it was on the previous (e.g. CentOS 6) server.

For example, below is a case where the Storage Connector that is configured with NFS will not be able to read and write to NFS-mounted storage after switching to a new OVA image.

Let us assume that Storage Connector is still deployed on CentOS 6 (e.g. the Storage Connector version is 2.12). 

When NFS was set up, the following command was executed to make the syncp-storage user the owner of the mounted directory:

sudo chown syncp-storage:syncp-storage /mnt/syncp

In our case, the NFS mount point was set up for the syncp-storage user with uid=498 and for the syncp-storage group with gid=499

id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

When switching to a new OVA version (e.g. CentOS 7 with Storage Connector 2.21), the UID and GID for the syncp-storage user become different (uid=996 and gid/groups=994):

id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

Because of this, when you try to reuse NFS mounted storage without re-executing the command ‘sudo chown -R syncp-storage:syncp-storage /mnt/syncp’, the new Storage Connector will not be able to write to the NFS-mounted storage.

The cause for this is that the syncp-storage user with the new GID/UID is not the owner of the Storage Connector directories and the storage mount point directory.

Often it is impractical to run this recursive change in ownership for all files and folders in the NFS mounted storage due to the number of files and folders (which could cause a degradation in quality of service and/or require significant downtime).

Problem resolution

Let us assume that on our old machine, the UID and GID for the syncp-storage user are:

id syncp-storage
uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

while on the new machine, they are:

id syncp-storage
uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

To resolve this problem :

The steps below are applicable for the case of upgrading from a CentOS 6 machine to a new OVA image running CentOS 7. However, you can also use them for troubleshooting whenever there is an UID or GID mismatch between two Storage Connector machines after upgrading.

  1. Verify that the newly deployed Storage Connector is configured to use NFS-mounted storage.

  2. Go to the old machine and check the UID of the syncp-storage user—it should be 498. Otherwise, note the UID and use it in place of 498 in Step 8 below.
    id syncp-storage
    uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

  3. On the old machine, check the GID of the of the syncp-storage group—it should be 499. Otherwise, note the GID and use it in place of 499 in Step 9  below.
    id syncp-storage
    uid=498(syncp-storage) gid=499(syncp-storage) groups=499(syncp-storage)

  4. On the new machine, check the UID of the syncp-storage user. 
    id syncp-storage
    uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

    By default, the UID is 996, but could vary. If the UID matches the UID on the old machine, skip steps 7.a and c, and step 8.
  5. On the new machine, check the GID of the syncp-storage group.
    id syncp-storage
    uid=996(syncp-storage) gid=994(syncp-storage) groups=994(syncp-storage)

    By default, the GID is 994, but could vary. If the GID matches the GID on the old machine, skip steps 7.b and c, and step 9.

  6. On the new machine, stop the Storage Connector service.

    sudo systemctl stop syncp-storage

  7. On the new machine, verify that the UID and GID from the old machine are not already in use.

    Run the following command to check if a user is assigned to the UID that you need.
    getent passwd 498

    Run the following command to check if a user is assigned to the GID that you need.
    getent group 499

    If the query returns a user for any of these IDs, assign that user another UID or GID before you proceed.

    This operation might require an OVA restart. If an important system user was assigned the UID and GID that you want to free up (in this example, 498 and 499), the machine might not operate as expected after the restart, so you might need to redeploy the OVA.


  8. Set the correct UID for the syncp-storage user.

    sudo usermod -u 498 syncp-storage

  9. Set the correct GID for the syncp-storage group.

    sudo groupmod -g 499 syncp-storage

  10. Get a list of all files and directories used by the Storage Connector
    find / -user syncp-storage

  11. Set the correct ownership for the list of files and directories from the previous step. The list may vary depending on the Storage Connector version. Example:

    sudo chown -R syncp-storage:syncp-storage /*/*/syncp-storage/
    sudo chown -R syncp-storage:syncp-storage /*/syncp-storage/
    sudo chown -R syncp-storage:syncp-storage /etc/systemd/system/syncp-storage.service.d/
    sudo chown syncp-storage:syncp-storage /usr/lib/systemd/system/syncp-storage.service
    sudo chown syncp-storage:syncp-storage /usr/bin/checkmount.sh
    sudo chown syncp-storage:syncp-storage /usr/bin/syncp-storage
    sudo chown syncp-storage:syncp-storage /usr/bin/syncpstgconfig
    sudo chown syncp-storage:syncp-storage /usr/bin/syncp-storage*

  12. Set the correct ownership of the configuration directory containing the .secret file (if applicable).

    sudo chown -R syncp-storage:syncp-storage /mnt/config

  13. Unmount the storage mount point (e.g. /mnt/syncp).

    umount /mnt/syncp

  14. Remount the storage to the mount point by reloading the  /etc/fstab  file.

    mount -a

  15. Start the Storage Connector service. 

    sudo systemctl start syncp-storage

This concludes the problem resolution steps.  The Storage Connector running on the new machine should now be able to read and write to the NFS-mounted storage.

Related Links