Hyper-V Cluster CSV Volumes Inaccessible After SAN Controller Failure (ReFS / CSVFS Issue)

Shukran 101 Reputation points
2026-04-17T05:39:07.3833333+00:00

Environment:

  • Windows Server Failover Cluster (2 nodes)
  • Hyper-V Cluster (CSV enabled)
  • Storage: Fujitsu ETERNUS DXL (Fibre Channel SAN)
  • SAN Switches: Brocade 6505 (dual fabric)
  • HBA: QLogic QLE2692 (16G FC)
  • File System: CSVFS (ReFS-based volumes)

Issue Description: We are experiencing a critical issue where Cluster Shared Volumes (CSV) have become inaccessible after a storage controller failure.

Controller A of the storage array became unresponsive. Following this, SAN paths were disrupted. Although connectivity was partially restored via Controller B, the cluster volumes remain inaccessible.

The following issues are observed:

  • CSV volumes (Volume1, Volume2) are not accessible
  • Error when accessing CSV path: C:\ClusterStorage\Volume1 is not accessible. The parameter is incorrect.
  • Cluster disks appear online but are not functional as CSV
  • Quorum disk intermittently affected
  • Both nodes are unable to access affected volumes

Cluster / Storage Symptoms:

  1. Event Viewer Errors:
    • Event ID: 5142
      • Message: "Cluster Shared Volume 'Volume2' is no longer accessible from this cluster node because of error (1460). Please troubleshoot this node's connectivity to the storage device and network connectivity."
      1. Disk State:
        • Disks are visible in OS (Get-Disk shows Healthy/Online)
          • Not listed in Get-ClusterAvailableDisk
            • Some disks show as "Reserved"
            1. CSV State:
              • Existing CSV volumes partially visible
                • One or more CSVs missing / inaccessible
                  • Cannot re-add disk to CSV: "The disk is not configured in a way to be used with CSV"
                  1. Filesystem Issue:
                    • Disk partitions exist but filesystem is not mountable
                      • TestDisk shows partition but no NTFS marker
                        • X-Ways Forensics detects ReFS partition

SAN / Connectivity Findings:

  • One FC path (Controller A) failed
  • After swapping SFP/cable, paths via Controller B are active
  • Brocade switch shows active F-ports
  • nsshow confirms visibility of storage WWPNs
  • However, pathing appears inconsistent

HBA Status (QLogic):

  • One port shows:
    • Link Down
    • Other ports:
      • Online (16G)
      • All WWPNs are visible in SAN fabric

Actions Already Taken:

  • Reseated storage controller
  • Swapped SFP modules and fiber cables
  • Verified zoning configuration
  • Restarted host servers
  • Checked cluster resources (all appear online except storage functionality)
  • Attempted to re-add disks to CSV (failed)
  • Attempted DiskPart operations (blocked by cluster control)
  • Ran TestDisk and X-Ways Forensics for filesystem validation

Current Status:

  • SAN connectivity partially restored
  • Disks visible but CSV inaccessible
  • Filesystem appears corrupted or not recognized
  • Active recovery in progress using forensic tools

Request for Microsoft Support:

We need assistance with:

  1. Determining if CSV / ReFS metadata corruption has occurred
  2. Safely reattaching existing disks to the cluster without data loss
  3. Validating whether filesystem repair (ReFS) is possible
  4. Guidance on recovering CSV volumes without formatting
  5. Ensuring cluster stability after SAN path disruption

Impact:

  • All virtual machines stored on affected CSV volumes are offline
  • Production services are impacted
  • Potential data loss risk

Additional Notes:

We suspect that asymmetric SAN path failure (Controller A down) caused inconsistent access to CSV volumes, leading to filesystem inconsistency or corruption.Environment:

  • Windows Server Failover Cluster (2 nodes)
  • Hyper-V Cluster (CSV enabled)
  • Storage: Fujitsu ETERNUS DXL (Fibre Channel SAN)
  • SAN Switches: Brocade 6505 (dual fabric)
  • HBA: QLogic QLE2692 (16G FC)
  • File System: CSVFS (ReFS-based volumes)

Issue Description:
We are experiencing a critical issue where Cluster Shared Volumes (CSV) have become inaccessible after a storage controller failure.

Controller A of the storage array became unresponsive. Following this, SAN paths were disrupted. Although connectivity was partially restored via Controller B, the cluster volumes remain inaccessible.

The following issues are observed:

  • CSV volumes (Volume1, Volume2) are not accessible
  • Error when accessing CSV path:
    C:\ClusterStorage\Volume1 is not accessible. The parameter is incorrect.
  • Cluster disks appear online but are not functional as CSV
  • Quorum disk intermittently affected
  • Both nodes are unable to access affected volumes

Cluster / Storage Symptoms:

  1. Event Viewer Errors:
    • Event ID: 5142
      • Message:
        "Cluster Shared Volume 'Volume2' is no longer accessible from this cluster node because of error (1460). Please troubleshoot this node's connectivity to the storage device and network connectivity."
      1. Disk State:
        • Disks are visible in OS (Get-Disk shows Healthy/Online)
          • Not listed in Get-ClusterAvailableDisk
            • Some disks show as "Reserved"
            1. CSV State:
              • Existing CSV volumes partially visible
                • One or more CSVs missing / inaccessible
                  • Cannot re-add disk to CSV:
                    "The disk is not configured in a way to be used with CSV"
                  1. Filesystem Issue:
                    • Disk partitions exist but filesystem is not mountable
                      • TestDisk shows partition but no NTFS marker
                        • X-Ways Forensics detects ReFS partition

SAN / Connectivity Findings:

  • One FC path (Controller A) failed
  • After swapping SFP/cable, paths via Controller B are active
  • Brocade switch shows active F-ports
  • nsshow confirms visibility of storage WWPNs
  • However, pathing appears inconsistent

HBA Status (QLogic):

  • One port shows:
    • Link Down
    • Other ports:
      • Online (16G)
      • All WWPNs are visible in SAN fabric

Actions Already Taken:

  • Reseated storage controller
  • Swapped SFP modules and fiber cables
  • Verified zoning configuration
  • Restarted host servers
  • Checked cluster resources (all appear online except storage functionality)
  • Attempted to re-add disks to CSV (failed)
  • Attempted DiskPart operations (blocked by cluster control)
  • Ran TestDisk and X-Ways Forensics for filesystem validation

Current Status:

  • SAN connectivity partially restored
  • Disks visible but CSV inaccessible
  • Filesystem appears corrupted or not recognized
  • Active recovery in progress using forensic tools

Request for Microsoft Support:

We need assistance with:

  1. Determining if CSV / ReFS metadata corruption has occurred
  2. Safely reattaching existing disks to the cluster without data loss
  3. Validating whether filesystem repair (ReFS) is possible
  4. Guidance on recovering CSV volumes without formatting
  5. Ensuring cluster stability after SAN path disruption

Impact:

  • All virtual machines stored on affected CSV volumes are offline
  • Production services are impacted
  • Potential data loss risk

Additional Notes:

We suspect that asymmetric SAN path failure (Controller A down) caused inconsistent access to CSV volumes, leading to filesystem inconsistency or corruption.image

Windows for business | Windows Server | Storage high availability | Clustering and high availability
0 comments
Answer accepted by question author
Tracy Le 10,975 Reputation points Independent Advisor
2026-04-20T17:46:26.0266667+00:00

Hi Shukran,

Ah, you hit the exact limitation of that OS version. You are absolutely correct-refsutil was not introduced until Windows Server 2019 (and Windows 10 v1703).

However, you are not out of luck, and you do not need to upgrade your cluster to save your data. Because ReFS is readable across newer OS builds, you can simply use a helper machine to run the salvage operation.

Here is the standard workaround for 2016 environments:

  1. Temporarily unmap/unmask both the corrupted LUN and your new destination LUN from the 2016 cluster nodes.
  2. Present (map) both of those LUNs to a separate machine running Windows Server 2019, 2022, or even a Windows 10/11 Pro workstation.
  3. Use that newer machine to execute the refsutil salvage process. It will read the corrupted 2016 ReFS volume and safely extract your VHDX files onto the new LUN.
  4. Once the salvage is complete, unmap the new LUN from the helper machine, map it back to your 2016 cluster, and re-import your VMs.

Tracy

Was this answer helpful?

1 person found this answer helpful.
Answer accepted by question author
Tracy Le 10,975 Reputation points Independent Advisor
2026-04-23T09:26:08.17+00:00

Hi Shukran,

The error message "A device attached to the system is not functioning" indicates a hardware or connectivity drop. Your new Server 2025 physically lost the connection to the SAN LUN during the heavy data extraction (after running for over an hour).

This is an I/O path failure, not a refsutil software bug. Here is how to fix this and continue:

1. Configure MPIO (Crucial): Did you install the MPIO (Multipath I/O) feature and the Fujitsu DSM on this new Server 2025? If you just attached the fiber cables without configuring MPIO, the heavy read/write load will cause path thrashing and eventually disconnect the drive.

2. Check SAN/HBA Logs: Look at the Windows Event Viewer (System logs) or your Brocade switch logs to see why the FC path dropped at that exact moment.

3. Rerun Salvage: Once the physical connection is stable and MPIO is active, rerun the exact same refsutil salvage command. It will attempt to extract the remaining files.

Secure the physical connection first, then try the extraction again!

Was this answer helpful?

Answer accepted by question author
Tracy Le 10,975 Reputation points Independent Advisor
2026-04-17T06:44:04.8133333+00:00

Hi Shukran,

First, you can ignore the AI-generated answer above. It is giving you PowerShell commands designed for Storage Spaces Direct (S2D) storage pools, which are completely irrelevant to your hardware-based Fujitsu Fibre Channel SAN environment.

The Reality of the Situation: When Controller A hard-crashed during active I/O, the active node could not flush the ReFS metadata cleanly before MPIO paths failed over to Controller B. Right now, you are facing one of two things: either a stuck SCSI-3 Persistent Reservation (PR) lock from the cluster, or a legitimately corrupted ReFS filesystem (RAW state).

Here is the exact, step-by-step method to isolate and recover this without formatting:

Step 1: Break the Cluster Ghost Locks Remove the affected "Reserved" disks from the Failover Cluster Manager configuration entirely. Then, open an elevated PowerShell prompt on a node and run this command to forcibly clear any stuck SCSI-3 PR locks: Clear-ClusterDiskReservation -Disk [DiskNumber] (Replace [DiskNumber] with the actual OS disk number).

Step 2: Test Native OS Mounting Open Disk Management (diskmgmt.msc) on one node only. Bring that disk Online and assign it a drive letter.

Step 3: The ReFS Crossroads

Scenario A (It mounts and data is visible): The filesystem is perfectly fine; the cluster database was just hung up on the path flap. You can safely remove the drive letter, bring it offline, re-add it to the cluster, and promote it back to CSV.

Scenario B (It shows as RAW or prompts to format): The abrupt SAN controller failure corrupted the ReFS metadata. Because ReFS does not support traditional chkdsk repairs, your only Microsoft-supported recovery path is to provision a brand-new LUN on your SAN, map it to the host, and use the built-in refsutil salvage command to extract your VHDX files from the corrupted volume over to the new one.

Do not run any third-party partition repair tools that write to the disk, as they will permanently destroy any chance of refsutil salvaging your VMs. I hope the response provided some helpful insight. If it clarified the issue for you, please consider marking it as Accept Answer so others with the same issue can find the solution. Feel free to leave a comment if you need further information.

Tracy Le.

Was this answer helpful?

0 comments

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Tracy Le 10,975 Reputation points Independent Advisor
    2026-04-19T14:23:04.29+00:00

    Hi Shukran,

    I just wanted to do a quick follow-up to see how the recovery of your CSV volumes is going.

    Did running the Clear-ClusterDiskReservation command help break those cluster ghost locks, or did you ultimately have to provision a new LUN and use refsutil salvage to recover your VHDX files?

    If you are still stuck or need any help interpreting the refsutil output, please don't hesitate to drop a comment. I'm always happy to help!

    TL.

    Was this answer helpful?

  2. AI answer

    2026-04-17T05:39:54.8966667+00:00

    The symptoms match a CSV going offline or becoming inaccessible after a storage component failure, with the disks still visible to the OS but CSV/cluster access failing. The documented guidance focuses on:

    1. Restoring or stabilizing storage connectivity.
    2. Recovering the CSV/cluster resources without reformatting.
    3. Manually repairing the storage pool/CSV if a component cannot be restored.

    Based on the context, the following steps are recommended.

    1. Stabilize storage and cluster connectivity
    • Verify that all storage paths are healthy and consistent on both nodes:
      • Confirm that all FC paths are up and that each node sees the same LUNs in Disk Management.
      • Validate MPIO configuration and that all paths are Online; resolve any HBA, switch, or cabling issues first.
    • Check for storage‑related events (11, 129, 153, 157) and CSV events 5120/5142/153 in the System and FailoverClustering logs and address any underlying disk, HBA, or switch faults.
    • Ensure sufficient capacity on the CSVs and that no hardware or filter‑driver issues (for example, antivirus) are affecting I/O.
    1. Handle CSVs that went offline after the controller failure

    The behavior where a CSV goes into Failed state after a node/storage component goes offline during active I/O is by design in newer Windows Server builds. When metadata limits are reached, I/O fails and the CSV is taken offline instead of hanging.

    If the failed controller or path can be restored (Method 1):

    • After restoring the offline node or storage path, the cluster will periodically attempt to bring the CSV online.
    • When the CSV comes online successfully, automatic repair processes start and the volume becomes available again.
    • After recovery, manually start any VMMS‑managed VMs that use the CSV, because VMMS stops retrying after 30 minutes of cluster downtime.
    1. Manual recovery if the failed storage component cannot be restored (Method 2)

    If one controller/path or a storage component cannot be brought back and the CSV remains inaccessible, use the documented manual recovery procedure on a node with full access to the storage pool:

    1. Open an elevated PowerShell session on a cluster node that has full access to the storage pool.
    2. Get the properties of the storage pool resource: 
         Get-ClusterResource <Pool>
      Use the pool name and owner group information for later steps.
    3. Enumerate physical disks, virtual disks, and CSVs and identify resources in Failed state: 
         Get-ClusterResource | Where-Object { $_.ResourceType -eq "Physical Disk" }
   Get-ClusterSharedVolume
    4. Replace or add any required disks to the storage pool and allow repair jobs to start.
    5. Monitor repair jobs: 
         Get-StorageJob
      When the repair percentage is greater than 0, continue.
    6. Return the storage pool to cluster management: 
         Get-CimInstance -Namespace "root\MSCluster" -ClassName "MSCluster_AvailableStoragePool" | Invoke-CimMethod -MethodName AddToCluster
    7. Restore all non‑failed virtual disks to cluster management. For each virtual disk that was previously a CSV, re‑add it and convert back to CSV. Example: 
         $virtualdiskname = "ClusterPerformanceHistory"
   $virtualdiskid   = "603bb5d0-9c4d-4fc6-9c25-eec92a478733"
   (Get-ClusterAvailableDisk | Where-Object { $_.Id -eq $virtualdiskid } | Add-ClusterDisk).Name = $virtualdiskname
      Then use Add-ClusterSharedVolume to make it a CSV again.
    8. Monitor virtual disks and wait until OperationalStatus = InService for all of them: 
         Get-VirtualDisk
    9. Bring virtual disks online and clear read‑only/offline flags: 
         Get-VirtualDisk | Get-Disk | Where-Object { $_.IsReadOnly -eq $true } | Set-Disk -IsReadOnly $false
   Get-VirtualDisk | Get-Disk | Where-Object { $_.IsOffline  -eq $true } | Set-Disk -IsOffline  $false
    10. Monitor retired physical disks and their VirtualDiskFootprint: 
          Get-PhysicalDisk -Usage Retired | ft DeviceId, Usage, VirtualDiskFootprint
      When the footprint reaches zero, restore previously failed virtual disks to cluster management, convert them back to CSVs if needed, and bring them online as read/write.
    11. CSV/cluster stability and further checks
    • Use the general storage troubleshooting checklist for Hyper‑V and failover clusters:
      • Confirm hardware compatibility and that all drivers, firmware, DSM/MPIO software, and Windows updates are current.
      • Verify that all cluster nodes see the shared storage and that Hyper‑V and Failover Clustering are correctly configured.
      • Ensure no conflicts from antivirus or unsupported filter drivers; remove or adjust them if they interfere with CSV I/O.
      • Review event logs for any remaining storage, clustering, or disk warnings.
    1. Notes on filesystem/metadata corruption and repair

    The provided documentation describes how CSVs behave and how to recover cluster/CSV resources after a node or storage component goes offline during active I/O, including automatic and manual repair of virtual disks and storage pools. It does not provide a procedure for low‑level ReFS metadata repair beyond these storage‑pool and CSV recovery steps. If, after completing the above and stabilizing storage, the CSV volumes still cannot be mounted or brought online, engage Microsoft Support directly for advanced ReFS/CSVFS analysis.

    After the cluster and CSVs are recovered and stable, manually start affected VMs and consider configuring monitoring/alerts on CSV capacity and storage health to detect future issues earlier.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.