This is a continuation of January’s post about the upcoming secure boot certificate expiration. I’ve put together a script assisted strategy for patching our fleet of VMware vSphere machines by clearing out the NVRAM files in mass.
The first pass on the script has been uploaded to my GitHub as VMW_MSUEFICA2023Patch.ps1 and has so far been successful in test and pre-production scenarios to work through and patch a batch of VMs.
To reiterate on the issue at hand. In June of this year, the Microsoft signing certificates that are part of the secure boot process that have been in place since 2011 are expiring. Replacement of those certificates in the virtual environment cannot be handled from within the operating system because of an invalid platform key in the VMware vSphere virtual BIOS. So far, Broadcom has published two potential solutions. Either, (1) deleting the NVRAM file and allowing the UEFI variables to reset to new defaults on boot or, (2) fixing the invalid platform key through some very manual console work. I’ve opted to implement the first option.
Graceful, orderly, and approved reboot cycles in an enterprise environment across thousands of virtual machines being what they are, I put together this script to at least handle most of the tedious legwork.
Notice!
Broadcom has revoked the public guidance for the removal of the NVRAM file as a mitigation to the issue. This article makes use of that original guidance. Refer to the reverting instructions at the end if the change needs to be undone.
Requirements
- A list of virtual machines to operate against.
- Sufficient vSphere permissions to query, power off, and power on virtual machines; and to delete files from datastores.
- At least one functional ESXi 8.0-Update 2 or later host.
- GitHub: VMW_MSUEFICA2023Patch.ps1
Process
Start by assembling a list of virtual machines that should be checked or acted on. If they are Windows virtual machines, the script will attempt to use WinRM to query registry keys for the system status and only make changes if required.
WinRM Note: If non-Windows, lacking credentials to the OS, or the process otherwise fails, use -SkipStatusCheck to ignore the result from the check.
Ensure you’re connected to the vCenter server(s) where your VMs are running using PowerCLI. The -Name parameter passed in the script is expected to match the name of the VM object in vCenter. For the WinRM queries to work, this must also match the VM hostname. FQDNs are also allowable, and the script will attempt to convert them to shortname if the vCenter query fails.
Pass the list of machines to the script, either by pipeline or with the -Name parameter. Once confident with the detections, remove the -WhatIf parameter to commit changes.
For example,
.\VMW_MSUEFICA2023Patch.ps1 -Name WSNOCCA10,WSNOCCA20 -Verbose -WhatIf
During execution, the script will gracefully power down and then power back on the VM. The NVRAM file is renamed to *.nvram.bak in case the change needs to be reverted for a particular VM.
Reverting Changes
The script will rename the existing NVRAM file from <vmname>.nvram to <vmname>.nvram.bak within the VM’s home folder on the datastore. If the original file needs to be restored, it can simply be renamed to the original name. If your VM fails to boot due to a rollback of the secure boot certificate due to an unsupported host version, restoring the original file will allow boot to occur.
In testing, NVRAM files could not be transferred between VMs.
Observations from Testing
- Remediation only required that the host operating system version be 8.0-Update 2 or later. The virtual hardware version was not relevant, and replacing the UEFI variables was successful for virtual hardware as old as HWv15.
- The first boot of the virtual machine after removing the NVRAM file must take place on an 8.0-Update2 host. Once this completes, the VM can vMotion, restart, or power on from an older host without issue.
- If the boot loader is updated to boot from the new certificate chain, the UEFI variables are reset, and the first boot occurs on a host older than 8.0-Update2, startup will fail. (Expected)
References
- https://knowledge.broadcom.com/external/article/421593/missing-microsoft-corporation-kek-ca-202.html
- https://knowledge.broadcom.com/external/article/423919/manual-update-of-the-secure-boot-platfor.html
- https://knowledge.broadcom.com/external/article/423893/secure-boot-certificate-expirations-and.html