attachVolume fails with HTTP 500 from vCenter datastore

[mogamal1]

problem

AttachVolume fails with HTTP 500 from vCenter datastore browser when
attaching a detached volume on a PreSetup (DatastoreCluster) storage pool
to a Running VMware VM.

ERROR: Failed to attach volume!
java.io.IOException: Server returned HTTP response code: 500 for URL:
https://vcenter.../folder/i-2422-79-VM%2Fcaa1f96c...vmdk?dcPath=XXXXX&dsName=OShit-BigVol

CloudStack calls VirtualMachineMO.attachDisk() → getVmdkFileInfo() which
does an HTTP GET via vCenter datastore browser. vCenter routes this to a
non-owner ESXi host which cannot serve the locked file of the running VM.

EXPECTED RESULT:
Volume attaches successfully. CloudStack should use the vSphere API
(ReconfigVM_Task with VirtualDiskSpec) directly instead of going through
the vCenter HTTP datastore browser to validate the VMDK.

WORKAROUND:
Power off the VM, attach the volume, power it back on.
Reference: Related issue #10626

versions

COMPONENT NAME: Storage / VMware
CLOUDSTACK VERSION: 4.22.0

The steps to reproduce the bug

1. Create a volume on a PreSetup/DatastoreCluster-type primary storage pool
2. Migrate the volume (migrateVolume API) to BIGVOL (DatastoreCluster)
3. Volume lands correctly on ESXi host VMFS under i-XXXX-XX-VM/ folder
4. Attempt attachVolume to a Running VM on the same cluster

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mogamal1]

Title:
attachVolume fails with HTTP 500 on PreSetup (DatastoreCluster) pool -
VirtualMachineMO.attachDisk uses vCenter HTTP folder browser which doesn't
work for DatastoreCluster datastores

Component: Storage, VMware

CloudStack Version: 4.22.0

Description:
When attaching a detached volume (state=Ready) to a VM (even Stopped) where
the volume resides on a PreSetup/DatastoreCluster-type primary storage pool,
attachVolume fails with:

java.io.IOException: Server returned HTTP response code: 500 for URL:
https://vcenter.../folder/i-XXXX-XX-VM%2F.vmdk
?dcPath=&dsName=
at VirtualMachineMO.attachDisk(VirtualMachineMO.java:1458)

Root Cause:
VirtualMachineMO.attachDisk() calls getVmdkFileInfo() which performs an
HTTP GET to vCenter's /folder/ datastore browser endpoint to determine
VMDK capacity. This HTTP endpoint returns HTTP 500 for DatastoreCluster
(poolType=PreSetup) datastores — vCenter does not support HTTP-based file
browsing for DatastoreCluster paths.

The fix should use vSphere API queryVirtualDiskInfo() instead of HTTP
datastore browser to retrieve VMDK metadata for PreSetup pool types.

Steps to Reproduce:

1. Add a PreSetup/DatastoreCluster primary storage pool (poolType=PreSetup)
2. Create or migrate a detached data volume to that pool
3. Attempt cmk attachVolume id= virtualmachineid=
   (fails whether VM is Running or Stopped)

Expected: Volume attaches successfully
Actual: HTTP 500 from vCenter, attach fails

Environment:

• CloudStack 4.22.0
• VMware vCenter (vcenter.opscloud.eumetsat.int)
• Primary storage type: DatastoreCluster (poolType=PreSetup)
• ESXi hosts: tcebld* cluster

Workaround: None within CloudStack currently.
Related: #10626

[mwaag]

Hi there, hi @DaanHoogland

we run in this exact problem with cloudstack 4.20.3.0 and Vsphere 8.0.3 (0700).

Our insights on this so far:

• Cloudstack 4.20.3.0 creates data-disk with VMWare VirtualDiskStorageObjectManager
• this creates a so called 'first class disk', the descriptor file contains ddb.fcd.-entries.
• when attached, cloudstack manages the disk as conventional vmdk and copies descriptor and -flat.vmdk into directory of target-vm
• cloudstack then calls updateVmdkAdapter() to change controller-type in copied vmdk-descriptor-file
• therefore cloudstack tries to download descriptor-file via vsphere file browser http://vcenter.url/folder/....vmdk
• vcenter 8.0.3 notices the ddb.fcd.-entries, thinks its still an first class disk and tries to manage through VStorageObject
• but in target-dir doesn't exist a VStorageObject, vCenter reports vim.fault.NotFound and delivers HTTP 500
• cloudstack cancels attach-action

We created a request to shapeblue (ticket 7160) just today as this will keep us from migrating to vmware 8 (which is comming more urgent since the newest vmsa: VMSA-2026-0006 ) and are currently working internally on a patch for 4.20.3.0 (which looks promising so far) and after some more testing planing to open a PR for it for more / better review.

[DaanHoogland]

@mwaag can you look at #13773 ? (It might be too simple for your taste. just let me know)