Hyper-V GPU Virtualization

readme.md

Configuring GPU-PV on Hyper-V

This works on a Windows Pro 10 version 2004 or newer machine, confirmed to function with nVidia GPUs and recent AMD GPUs. Intel should work as well, but why do you want that?

I am not in any way responsible for working this out, that credit goes to Nimoa at cfx.re and reddit users on r/hyperv

1. Make sure you have Hyper-V enabled (no way!) - there's a list of other features below that you might need, enable those if it doesn't work.
2. Create a new VM using New-GPUPVirtualMachine.ps1 below and install Windows 10 on it.
3. Gather driver files for your guest using one of the two methods below:

Using the driver gathering script (recommended)

Download the New-GPUPDriverPackage.ps1 script from this gist, it will gather the various files for you - it must be run as admin, make sure you either specify a destination path or run it from the folder you'd like the .zip created in.

1. Run the script on your host, in an admin PowerShell session. It will create GPUPDriverPackage-[date].zip in the current directory, or a path specified with -Destination <path>.
2. Copy the .zip to your guest VM and extract it.
3. Copy the contents of the extracted GPUPDriverPackage folder into C:\Windows\ on the guest VM
4. Reboot the guest, and enjoy your hardware acceleration!

This has been tested on nVidia, AMD, and Intel GPU drivers.
(Most Intel iGPUs have support for GPU-P, though I've not tested for Quick Sync Video support in guests yet)

Gathering driver files manually

This only covers nVidia drivers, but the process is very similar for Intel and AMD.

1. On your host system:
   1. Browse to C:\Windows\system32\DriverStore\FileRepository
   2. Find the nvdispsi.inf_amd64_<guid> and/or nvltsi.inf_amd64_<guid> folders, and copy them to a temporary folder
   3. View driver details in device manager on your host, and copy all the files you see listed in System32 or SysWOW64 into matching folders inside your temporary folder.
2. On your guest system:
   1. Browse to C:\Windows\system32\HostDriverStore\FileRepository
      (You will likely need to create the HostDriverStore and FileRepository directories)
   2. Copy the two driver folders you collected from your host to this path.
   3. Copy nvapi64.dll (and the other to C:\Windows\system32\ on the guest as well
3. Shut down the guest VM,
4. Make sure the VM's checkpoints are disabled, and automatic stop action is set to 'Turn Off'
   (The VM creation script covers this, but never hurts to be sure)
5. Boot your VM, and enjoy your hardware acceleration!

---

Windows features that must be enabled:

• Hyper-V
• Windows Subsystem for Linux*
• Virtual Machine Platform

* WSL/WSL2 is probably not necessary, but this functionality is present in Windows 10 to allow for CUDA support in WSL2, so it's probably a good idea to turn it on.

New-GPUPDriverPackage.ps1

<#
.SYNOPSIS
    Create a GPU-P Guest driver package.
.DESCRIPTION
    Gathers the necessary files for a GPU-P enabled Windows guest to run.
.EXAMPLE
    New-GPUPDriverPackage -DestinationPath '.'
.EXAMPLE
    New-GPUPDriverPackage -Filter 'nvidia' -DestinationPath '.'
.INPUTS
    None.
.OUTPUTS
    A driver package .zip
.NOTES
    This has some mildly dodgy use of CIM cmdlets...
.COMPONENT
    PSHyperTools
.ROLE
    GPUP
.FUNCTIONALITY
    Creates a guest driver package.
#>

[CmdletBinding(
    SupportsShouldProcess = $true,
    PositionalBinding = $true,
    DefaultParameterSetName = 'NoPathProvided',
    HelpUri = 'http://www.microsoft.com/',
    ConfirmImpact = 'Low')]
[Alias()]
[OutputType([String])]
Param (
    # Path to output directory.
    # If no file name is specified the filename will be GPUPDriverPackage-YYYYMMMDD.zip
    [Parameter(
        Mandatory = $false,
        ParameterSetName = 'PathProvided',
        HelpMessage = "Path to one or more locations.")]
    [Alias("PSPath", "Path")]
    [ValidateNotNullOrEmpty()]
    [string]
    $DestinationPath,

    # Device friendly name filter.
    # Only devices whose friendly names contain the supplied string will be processed
    [Parameter(
        Mandatory = $false,
        HelpMessage = "Only add drivers for devices whose friendly names contain the supplied string.")]
    [ValidateNotNullOrEmpty]
    [String]
    $Filter
)

process {
    try {
        # make me a temporary folder, and assemble the structure
        $fTempFolder = Join-Path -Path $Env:TEMP -ChildPath "GPUPDriverPackage"
            (New-Item -ItemType Directory -Path "$fTempFolder/System32/HostDriverStore/FileRepository" -Force -ErrorAction SilentlyContinue | Out-Null)
            (New-Item -ItemType Directory -Path "$fTempFolder/SysWOW64" -Force -ErrorAction SilentlyContinue | Out-Null)

        # Set default archive name
        $ArchiveName = ('GPUPDriverPackage-{0}.zip' -f $(Get-Date -UFormat '+%Y%b%d'))

        # Check if DestinationPath has been provided
        if ($PSCmdlet.ParameterSetName -eq "PathProvided") {
            switch ($DestinationPath) {
                { (Split-Path -Path $_ -Leaf) -match '(\.zip)' } {
                    # Path we've been provided is a full file path, so we'll just use it
                    $ArchiveFolder = Split-Path -Path $DestinationPath -Parent
                    $ArchiveName = Split-Path -Path $DestinationPath -Leaf
                    break
                }
                { Test-Path -Path $_ -PathType Container } {
                    # Path exists and is a directory, so we place our file in it with the default name.
                    $ArchiveFolder = $DestinationPath
                    break
                }
                Default {
                    # Path doesn't end in .zip and doesn't exist, so we're going to assume it's a directory and place the file in it with the default name.
                    $ArchiveFolder = $DestinationPath
                        (New-Item -ItemType Directory -Path $ArchiveFolder -Force -ErrorAction SilentlyContinue | Out-Null)
                    break
                }
            }
        } else {
            # if DestinationPath not supplied, use current directory and default name
            $ArchiveFolder = (Get-Location).Path
        }

        # just double check that one
        if (-not $ArchiveFolder) { $ArchiveFolder = (Get-Location).Path }

        Write-Output -InputObject ('Creating GPU-P driver package for host {0}' -f $Env:COMPUTERNAME)
        Write-Output -InputObject ('Destination path: {0}' -f (Join-Path -Path $ArchiveFolder -ChildPath $ArchiveName))

        <#
            Determine which cmdlet we should use to gather the list of GPU-P capable GPUs.
            On Windows builds before Server 2022/21H2, the cmdlet is 'Get-VMPartitionableGpu'
            On later builds, it's 'Get-VMHostPartitionableGpu', and the old cmdlet just prints an error.
            So, we check if Get-VMHostPartitionableGpu is a valid cmdlet to determine whether we should use it.
        #>
        Write-Output -InputObject "Getting all GPU-P capable GPUs in the current system..."
        if (Get-Command -Name 'Get-VMHostPartitionableGpu' -ErrorAction SilentlyContinue) {
            Write-Verbose -Message 'Using new Get-VMHostPartitionableGpu cmdlet'
            $PVCapableGPUs = Get-VMHostPartitionableGpu
        } else {
            Write-Verbose -Message 'Using old Get-VMPartitionableGpu cmdlet'
            $PVCapableGPUs = Get-VMPartitionableGpu
        }

        # if we found no GPU-P capable GPUs, throw an exception
        if ($PVCapableGPUs.Count -lt 1) {
            throw [System.Management.Automation.ItemNotFoundException]::new('Did not find any GPU-P capable GPUs in this system.')
        } elseif ($PvGPUs.Count -gt 1) {
            Write-Warning -Message (
                    ("You have {0} GPU-P capable GPUs in this system. `n" -f $PvGPUs.Count) +
                "         At present, there is no way to control which one is assigned to a given VM.`n" +
                "         Unless one of the available GPUs is an intel IGP, it is highly recommended`n" +
                "         that you disable the GPU(s) you do not wish to use.`n")
            $choices = '&Yes', '&No'
            $question = 'Do you wish to proceed without disabling the extra GPU(s)?'
            if ($Host.UI.PromptForChoice('', $question, $choices, 1) -eq 1) {
                throw [System.Management.Automation.ActionPreferenceStopException]::new('User requested to cancel.')
            }
        }

        # Map each PVCapableGPU to the corresponding PnPDevice. Regex (mostly) extracts the InstanceId from the VMPartitionableGpu 'name' property.
        Write-Output -InputObject ('Mapping GPU-P capable GPUs to their corresponding PnPDevice objects...')
        $InstanceExpr = [regex]::New('^\\\\\?\\(.+)#.*$')
        $TargetGPUs = $PVCapableGPUs.Name | ForEach-Object -Process {
            # I'm not proud of this dirty regex trick, but it works.
            Get-PnpDevice -InstanceId $InstanceExpr.Replace($_, '$1').Replace('#', '\')
        }

        # OK, now that we have some actual device names, we can filter them if we've been asked to
        if ($null -ne $Filter) {
            Write-Output -InputObject ('Applying filter "{0}" to device list...' -f $Filter)
            $TargetGPUs = $TargetGPUs | Where-Object { $_.FriendlyName -like ('*{0}*' -f $Filter) }
        }
        Write-Output -InputObject ('Will create driver package for {0} GPUs:' -f $TargetGPUs.Count)
        $TargetGPUs.FriendlyName | ForEach-Object { Write-Output -InputObject ('  - {0}' -f $_) }
    } catch { throw $PSItem }

    # Last chance to turn back, traveler. Are you sure?
    if ($pscmdlet.ShouldProcess("Driver Package", "Create")) {
        try {
            Write-Output -InputObject ('The next few steps may take some time, depending on how many devices & driver packages are installed.')
            Write-Output -InputObject ('If the script appears hung, please give it a few minutes to complete before terminating.')
            # Get display class devices
            Write-Output -InputObject ('Gathering display device CIM objects...')
            $PnPEntities = Get-CimInstance -ClassName 'Win32_PnPEntity' | Where-Object { $_.Class -like 'Display' }
            Write-Verbose -Message ('Found {0} display devices' -f $PnPEntities.Count)
            ($PnPEntities | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message ('    {0}' -f $_) }

            # Get display class drivers
            Write-Output -InputObject ('Gathering display device driver CIM objects...')
            $PnPSignedDrivers = Get-CimInstance -ClassName 'Win32_PnPSignedDriver' -Filter "DeviceClass = 'DISPLAY'"
            Write-Verbose -Message ('Found {0} display device drivers' -f $PnPSignedDrivers.Count)
            $PnPSignedInfo = ($PnPSignedDrivers | Select-Object -Property DeviceName,DriverProviderName,InfName,DriverVersion,Description | Format-Table -AutoSize | Out-String).Trim().Split("`n")
            $PnPSignedInfo | ForEach-Object -Process { Write-Verbose -Message ('    {0}' -f $_) }

            # next we have to get every PnPSignedDriverCIMDataFile, because Get-CimAssociatedInstance doesn't wanna play ball
            Write-Output -InputObject ('Gathering all driver file objects... (this is the slow one. Blame Microsoft.)') # or me not understanding CIM i guess?
            $SignedDriverFiles = Get-CimInstance -ClassName 'Win32_PNPSignedDriverCIMDataFile'
            Write-Output -InputObject ('Found {0} files across all system drivers.' -f $SignedDriverFiles.Count)

            foreach ($GPU in $TargetGPUs) {
                Write-Output -InputObject ('Getting driver package for {0}' -f $GPU.FriendlyName)
                $PnPEntity = $PnPEntities | Where-Object { $_.InstanceId -eq $GPU.InstanceId }[0]
                Write-Verbose -Message ('Device PnP Entity:')
                ($PnPEntity | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message ('    {0}' -f $_) }

                $PnPSignedDriver = $PnPSignedDrivers | Where-Object { $_.DeviceId -eq $GPU.InstanceId }
                Write-Verbose -Message ('Device PnPSignedDriver:')
                ($PnPSignedDriver | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message ('    {0}' -f $_) }

                $SystemDriver = Get-CimAssociatedInstance -InputObject $PnPEntity -Association Win32_SystemDriverPNPEntity
                Write-Verbose -Message ('Device SystemDriver:')
                ($SystemDriver | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message ('    {0}' -f $_) }

                Write-Verbose -Message ('SystemDriver main/anchor file: {0}' -f $SystemDriver.PathName)
                $DriverStoreFolder = Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)
                while ((Get-Item (Split-Path $DriverStoreFolder)).Name -notlike 'FileRepository') {
                    $DriverStoreFolder = Get-Item -Path (Split-Path -Path $DriverStoreFolder -Parent)
                }
                Write-Verbose -Message ('Device DriverStoreFolder: {0}' -f $DriverStoreFolder.FullName)

                Write-Output -InputObject ('Found package {0}, copying DriverStore folder {1} to temporary directory' -f $PnPSignedDriver.InfName, (Split-Path $DriverStoreFolder -Leaf))
                $TempDriverStore = ('{0}/System32/HostDriverStore/FileRepository/{1}' -f $fTempFolder, $DriverStoreFolder.Name)
                $DriverStoreFolder | Copy-Item -Destination $TempDriverStore -Recurse -Force
                Write-Output -InputObject ('Copied {0} of {1} files to temporary directory' -f (Get-ChildItem -Path $TempDriverStore -Recurse).Count, (Get-ChildItem -Path $DriverStoreFolder -Recurse).Count)

                # Get driver files from system32 etc and copy
                Write-Output -InputObject ('Gathering files from System32 and SysWOW64')
                $DriverFiles = ($SignedDriverFiles | Where-Object { $_.Antecedent.DeviceID -like $GPU.DeviceID }).Dependent.Name | Sort-Object
                $NonDriverStoreFiles = $DriverFiles.Where{$_ -notlike '*DriverStore*'}

                Write-Output -InputObject ('Found {0} files, copying to temporary directory...' -f $NonDriverStoreFiles.Count)
                $NonDriverStoreFiles | ForEach-Object -Process {
                    $TargetPath = Join-Path -Path $fTempFolder -ChildPath $_.ToLower().Replace(('{0}\' -f $Env:SYSTEMROOT.ToLower()),'')
                    # make sure the parent folder exists
                    (New-Item -ItemType directory -Path (Split-Path -Path $TargetPath -Parent) -Force -ErrorAction SilentlyContinue | Out-Null)
                    Write-Output -InputObject ('  - {0} -> {1}' -f $_, $TargetPath)
                    Copy-Item -Path $_ -Destination $TargetPath -Force -Recurse
                }
                Write-Output -InputObject ('Finished gathering files for {0}' -f $GPU.FriendlyName)
            }
            Write-Output -InputObject ('All driver files have been collected, creating archive file')
            $Location = (Get-Location).Path
            Set-Location -Path (Split-Path -Path $fTempFolder -Parent)
            Compress-Archive -Path $fTempFolder -DestinationPath (Join-Path -Path $ArchiveFolder -ChildPath $ArchiveName) -CompressionLevel Fastest -Confirm:$false
            Set-Location -Path $Location
            Write-Output -InputObject ('GPU driver package has been created at path {0}\{1}' -f $ArchiveFolder, $ArchiveName)
        } catch {
            throw $PSItem
        } finally {
            Write-Output -InputObject ('Cleaning up temporary directory {0}' -f $fTempFolder)
            Remove-Item -Recurse -Force -Path $fTempFolder
        }
    }
    Write-Output -InputObject ('Driver package generation complete.')
    Write-Output -InputObject ('Please copy it to your guest and extract the archive into C:\Windows\')
}

New-GPUPVirtualMachine.ps1

$Config = @{
    VMName      = 'GPU-VM'  # Edit this to match your existing VM. If you don't have a VM with this name, it will be created.
    VMMemory    = 8192MB    # Set appropriately
    VMCores     = 4         # likewise
    MinRsrc     = 80000000  # We don't really know what these values do - my GPU reports 100,000,000 available units...
    MaxRsrc     = 100000000 # I suspect in the current implementation they do nothing, but this is known to work - play around if you like!
    OptimalRsrc = 100000000
}

### actual execution
try {

    # Get VM host capabilities
    $VMHost = Get-VMHost
    # Get highest VM config version available
    $VMVersion = $VMHost.SupportedVmVersions[$VMHost.SupportedVmVersions.Count - 1]

    # Get existing VM if it exists
    $VMObject = (Get-VM -Name $Config.VMName -ErrorAction SilentlyContinue)

    # Create VM if it doesn't already exist
    if (-not $VMObject) {
        $NewVM = @{
            Name               = $Config.VMName
            MemoryStartupBytes = $Config.VMMemory
            Generation         = 2
            Version            = $VMVersion
        }
        New-VM @NewVM
    }

    # Enable VM features required for this to work
    $SetParams = @{
        VMName                    = $Config.VMName
        GuestControlledCacheTypes = $true
        LowMemoryMappedIoSpace    = 1Gb
        HighMemoryMappedIoSpace   = 32GB
        AutomaticStopAction       = 'TurnOff'
        CheckpointType            = 'Disabled'
    }
    Set-VM @SetParams
    # Disable secure boot
    Set-VMFirmware -VMName $Config.VMName -EnableSecureBoot 'Off'

    # Parameters for vAdapter
    $GPUParams = @{
        VMName                  = $Config.VMName
        MinPartitionVRAM        = $Config.MinRsrc
        MaxPartitionVRAM        = $Config.MaxRsrc
        OptimalPartitionVRAM    = $Config.OptimalRsrc
        MinPartitionEncode      = $Config.MinRsrc
        MaxPartitionEncode      = $Config.MaxRsrc
        OptimalPartitionEncode  = $Config.OptimalRsrc
        MinPartitionDecode      = $Config.MinRsrc
        MaxPartitionDecode      = $Config.MaxRsrc
        OptimalPartitionDecode  = $Config.OptimalRsrc
        MinPartitionCompute     = $Config.MinRsrc
        MaxPartitionCompute     = $Config.MaxRsrc
        OptimalPartitionCompute = $Config.OptimalRsrc
    }

    # Get adapter if it exists
    $VMAdapter = (Get-VMGpuPartitionAdapter -VMName $Config.VMName -ErrorAction SilentlyContinue)

    # Add adapter if not present, update if present
    if ($VMAdapter) {
        Set-VMGpuPartitionAdapter @GPUParams
    } else {
        Add-VMGpuPartitionAdapter @GPUParams
    }

} catch {
    Write-Error "Something went wrong with creation. Error details below:"
    Write-Error $PSItem.ErrorDetails
    throw $PSItem
}

@mmikeww that's... weird. It shouldn't be missing anything, it grabs that entire folder.

Which means there's a typo in there, again...

And, yep, I didn't do the folder selection and file gathering logic very well the first time around. Fixed now, I believe.

this line
$DriverStoreFolder = Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)
should be changed to
$DriverStoreFolder = Get-Item -Path ((Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)).Parent.FullName)

since we need the root folder not only the folder includes the drivers

this line $DriverStoreFolder = Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent) should be changed to $DriverStoreFolder = Get-Item -Path ((Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)).Parent.FullName)

since we need the root folder not only the folder includes the drivers

Can either of you confirm whether $SystemDriver.PathName for that driver is:

C:\Windows\system32\DriverStore\FileRepository\u0376884.inf_amd64_6831c6ef75046906\B376870\u0376884.inf

or

C:\Windows\system32\DriverStore\FileRepository\u0376884.inf_amd64_6831c6ef75046906\u0376884.inf

please? because from @mmikeww's screenshot above, and the behaviour they've described, it looks like it should be the second one. If it is the second one, I don't need to do anything and this script is working just fine.

If it is the first one, then this is a problem that's specific to AMD drivers - possibly ASUS's OEM AMD drivers, even - where for some reason they've decided to put the main .inf a folder deeper than everyone else does.

Your solution may fix the issue for your specific AMD machine/driver, but it will break the script for everyone else, since their drivers don't have the extra subdirectory. All your solution would end up doing is grabbing the host's entire FileRepository, which is completely unnecessary and kind of insane.

There's a better solution to the 'two folders' problem, if it even is a problem:

                $DriverStoreFolder = Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)
                while ((Get-Item (Split-Path $DriverStoreFolder)).Name -notlike 'FileRepository') {
                    $DriverStoreFolder = Get-Item -Path (Split-Path -Path $DriverStoreFolder -Parent)
                }
                Write-Verbose -Message ('Device DriverStoreFolder:')
                Write-Verbose -Message ('  - Driver store folder: {0}' -f $DriverStoreFolder.FullName)

That'll recurse up the directory tree until it finds the folder whose parent is $Env:SYSTEMROOT\System32\DriverStore\FileRepository.

Technically, I don't need to bother with the DriverStoreFolder stuff at all, since the contents are all tagged to the driver's DriverFiles. You could replace $NonDriverStoreFiles = $DriverFiles.Where{$_ -notlike '*DriverStore*'} with just $NonDriverStoreFiles = $DriverFiles and that'd work, it's just cleaner to copy the whole subdirectory tree.

Its neither of those two. When I output $SystemDriver.PathName right before that line, I get this:

C:\Windows\system32\DriverStore\FileRepository\u0376884.inf_amd64_6831c6ef75046906\B376870\amdkmdag.sys

The updated version of the script does get some extra files and folders that were not there previously (the inf and amd folders in the screenshot below, and which the Easy-GPU-PV script that I linked above also grabs).

But it is still missing that u0376884 folder level and the 3 files inside there:

Its neither of those two. When I output $SystemDriver.PathName right before that line, I get this:

C:\Windows\system32\DriverStore\FileRepository\u0376884.inf_amd64_6831c6ef75046906\B376870\amdkmdag.sys

Right, that's the first of the two options, so there is an actual problem here. If you ask me, the problem is with whoever put that driver together; u0376884.inf is supposed to be the root file for the SystemDriver, that's why the folder shares a name with it...

Anyway.

Fix is fairly simple, it's that change I mentioned before - have pushed updated script which should find the right directory no matter how many levels deep the root file is, please give it a try, should work fine 😃

I meant its not the first option because it says amdkmdag.sys rather than u0376884.inf

But anyway yes it looks like the update now works, and gives the same results as that other script 👍

I meant its not the first option because it says amdkmdag.sys rather than u0376884.inf

Okay, okay, technically, yes, but the filename isn't important, just how many directories deep it is 😝

But anyway yes it looks like the update now works, and gives the same results as that other script 👍

Woo! Thanks for confirming, glad it's working your end 😄

I'd like to expand this to automatically transfer the driver bundle to the guest (using Copy-VMFile) and unpack it into the right place (using PowerShell Direct) but that's a whole other nightmare to deal with... One day...

I'd like to expand this to automatically transfer the driver bundle to the guest (using Copy-VMFile) and unpack it into the right place (using PowerShell Direct) but that's a whole other nightmare to deal with... One day...

That Easy-GPU-PV script does this too. Instead of using Copy-VMFile, it mounts the VHD so that it gets a normal drive letter, and then performs the copy. See this file:

https://github.com/jamesstringerparsec/Easy-GPU-PV/blob/main/Update-VMGpuPartitionDriver.ps1

The full repo creates brand new VMs, but I don't use it for that. I just use these two files I've linked to manually perform the driver copy into my existing VMs

gpu appears in VM's device manager, but nvidia-smi says:
Failed to initialize NVML: Unknown Error

It seems they (Nvidia) have changed something again.
I installed Nvidia drivers (527.41) on host machine via nvidia cuda toolkit 12.0 if it helps somehow.

Configuration:
Host:
Windows 10 22H2
2x RTX 2070

VM:
Windows 10 22H2
RTX 2070

list of all files in prepared by script zip:
https://gist.github.com/RomaKoks/420bd9e06ed0af2e88c7c1b62cc66ee7