And it broke. I mean not just a little bit broke - my Mac kernel panicked reliably every 15/20 minutes or so.

And my heart sank, as I really didn’t want to have to rebuild my machine from scratch - but also wanted to limit any troubleshooting time… I didn’t want to spend weeks without my Mac - so really only had an afternoon to do some digging.

Kextpocalypse?

Apple have made it increasingly clear over the releases of their last few operating systems that they don’t like kernel extensions. Kernel extensions are broadly similar to drivers in Windows world - allowing additional functionality to be added at a low level to your operating system.

Apple being Apple, it’s also something they can’t control - which almost certainly drives them crazy. So starting in High Sierra, kernel extensions need manual approval by end users in order to work. Or, if you have MDM you can pre-approve the kernel extensions you want to always load in your environment - something we’ve done for Sophos, Cisco AnyConnect, Google Drive FileStream and Zoom in our SimpleMDM deployment at ThoughtWorks.

LMGTFY

So my Mac is rebooting every 15 minutes - with the cryptic message

panic(cpu 0 caller 0xffffff80098a2722): "attempting to register a sysctl at previously registered slot : 101"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.11.26/bsd/kern/kern_newsysctl.c:250

and further down the kernel panic logs:

      Kernel Extensions in backtrace:
     com.vmware.kext.vmhgfs(725.95.39)[D09FBF14-1356-3D6F-BCFF-EEB9E0808639]@0xffffff7f8cc38000->0xffffff7f8cc43fff
     
     <snip>
     
     	System uptime in nanoseconds: 767275846683
	last loaded kext at 767273659785: com.vmware.kext.vmhgfs	0725.95.39 (addr 0xffffff7f8cc38000, size 49152)
	last unloaded kext at 756526237452: com.vmware.kext.vmmemctl	0725.95.39 (addr 0xffffff7f8cc32000, size 24576)

so I did what any Mac admin would do. Panic. No, wait, I Googled.

It looks like those two VMWare kernel extensions com.vmware.kext.vmhgfs and com.vmware.kext.vmmemctl are the culprits… but I don’t want to remove VMWare if I can help it, at least not straight away - it’s a tool I use all the time.

Google suggests there are a limited number of virtual device slots available for kernel extensions. And that the operating system works around this by loading and unloading kernel extensions (kexts) on demand.

Now, I don’t think I have any particularly exciting kernel extensions loaded. I use VMWare Fusion, Google Drive FileStream, Cisco AnyConnect and Keybase

So what to do?

My first thought was to start removing all the applications that used kernel extensions one by one to free up some of those slots.

So - to start with I removed Sophos. After 15/20 minutes or so there was another kernel panic.

So that kills the lack of slots idea.

Now what?

How do I get a proper sense of what kexts are running on my machine? They are supposed to be in /System/Library/Extensions and also /Library/Extensions/ - but looking in those folders shows some, but clearly not all of the kexts on my system - as it doesn’t seem to contain the VMWare kexts that might be crashing my Mac.

Erik Gomez (he of installapplications fame) to the rescue with a python script that will scan your system for ALL the kexts, everywhere.

Back to the kernel panic logs - there were two kexts that specifically referenced VMWare. When I ran Erik’s script I could see some kernel extensions didn’t live in either of the standard locations - they appeared to be bundled inside the VMWare Fusion app. (Although I’ve no idea what the /Library/StagedExtensions folder is all about…)

    {
        "Identifier": "com.vmware.kext.vmnet", 
        "KextPath": "/Library/StagedExtensions/Applications/VMware Fusion.app/Contents/Library/kexts/vmnet.kext", 
        "Version": "1463.49.96"
    }, 
    {
        "Identifier": "com.vmware.kext.vmx86", 
        "KextPath": "/Library/StagedExtensions/Applications/VMware Fusion.app/Contents/Library/kexts/vmmon.kext", 
        "Version": "1463.49.96"
    }, 
    {
        "Identifier": "com.vmware.kext.vmioplug.18.7.0", 
        "KextPath": "/Library/StagedExtensions/Applications/VMware Fusion.app/Contents/Library/kexts/vmioplug.kext", 
        "Version": "18.7.0"
    }, 

So - with a heavy heart (I spend ALL my time testing in VMWare Fusion) I decided the best thing to do was to uninstall Fusion - as that would remove those VMWare kernel extensions… Right?

VMWare removed… Mac rebooted, fingers crossed…

… 20 minutes pass …

Kernel panic. “Oh bother!” I said (or words to that effect…)

Back to the drawing board

So I’m seeing VMWare kernel extension issues, but I’ve uninstalled VMWare Fusion completely so what the hell is going on?

Back to Erik’s script…

What’s this… turns out I’d not been looking at the output properly! There are MORE VMWare kernel extensions on my system.

    {		            
        "Identifier": "com.vmware.kext.vmhgfs", 
        "KextPath": "/Library/Application Support/VMware Tools/vmhgfs.kext", 
        "Version": "0725.95.39"
    }, 
    {
        "Identifier": "com.vmware.kext.vmmemctl", 
        "KextPath": "/Library/StagedExtensions/Library/Application Support/VMware Tools/vmmemctl.kext", 
        "Version": "0725.95.39"
    }, 
    {
        "Identifier": "com.vmware.kext.vmhgfs", 
        "KextPath": "/Library/StagedExtensions/Library/Application Support/VMware Tools/vmhgfs.kext", 
        "Version": "0725.95.39"
    }, 

/Library/Application Support/VMware Tools/ ?? So here are additional VMWare kernel extensions that aren’t part of the core application… yet here they are, still lurking on disk, being loaded and unloaded and CRASHING MY MAC

There’s actually a “Remove VMWare tools” package in that /Library/Application Support/VMware Tools folder. So I ran that. And crossed my fingers…

I think I may have fixed it…

Well, this computer has been up all the time I’ve been writing this blog post… which is a huge relief. I really didn’t want to have to reinstall from scratch!

I was running the latest version of VMWare Fusion - 11.5 - but I’ve been using VMWare for a long time, so I don’t know if the kernel extensions in that folder are left over components from older versions of VMWare Fusion.

When I finally reinstalled VMWare Fusion 11.5 - no additional VMWare tools were installed - so it looks like my previous suspicion was correct.

Now to get on with enjoying Catalina… (looks at non-Sidecar compatible iPad and sobs quietly…)