Editor’s Note: Once again, this is a moment frozen in time, designed to educate about a passing moment in time. This post is one of a series, so please be sure to read the other posts in this series, and recognize that things are changing constantly.

Related posts

macOS 10.14.5 beta 2: Kernel Extension Notarization, UAMDM, Whitelisting & You

Recap

Last time on this blog, I talked about a new requirement that is present in the early betas of macOS 10.14.5. Kernel Extensions that are installed on a 10.14.4 system that is upgraded to 10.14.5 may not operate correctly if they are not notarized by Apple. In this situation, if the kernel extension is whitelisted (aka UAKEL) by a user-accepted MDM (aka UAMDM), you have nothing to worry about for now. If you’re not using UAKEL and UAMDM, and you are installing kernel extensions that are not signed and notarized by Apple, you’re going to have a bad time. These extensions will not load, and the applications that depend on them will not operate, if they are built and signed after the demarcation date, which is currently 11 March 2019, but may change in the future.

An Example

Recently, DisplayLink released a new version of their kernel extension:

The release notes state:

Software package notarized by Apple as required for macOS 10.14.5 onwards.

DisplayLink Release Notes

However, should one download the software, and inspect it, one might find that things are lacking:

Persephone:Downloads tom$ stapler validate -v DisplayLink\ USB\ Graphics\ Software\ for\ macOS\ 5.1.1.dmg 
Processing: /Users/tom/Downloads/DisplayLink USB Graphics Software for macOS 5.1.1.dmg
Properties are {
    NSURLIsDirectoryKey = 0;
    NSURLIsPackageKey = 0;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Disk Image";
    NSURLTypeIdentifierKey = "com.apple.disk-image-udif";
    "_NSURLIsApplicationKey" = 0;
}
Creating synthetic cdHash for unsigned disk image, DisplayLink USB Graphics Software for macOS 5.1.1.dmg. Humanity must endure.
Signing information is {
    cdhashes =     (
        <fd2d35b7 cea70fab 2e22850b 3f39070d a7fa0f52>
    );
    "cdhashes-full" =     {
        2 = <fd2d35b7 cea70fab 2e22850b 3f39070d a7fa0f52 781113f0 7b8686a8 7803c116>;
    };
    cms = <>;
    "digest-algorithm" = 2;
    "digest-algorithms" =     (
        2
    );
    flags = 2;
    format = "disk image";
    identifier = ADHOC;
    "main-executable" = "file:///Users/tom/Downloads/DisplayLink%20USB%20Graphics%20Software%20for%20macOS%205.1.1.dmg";
    source = "explicit detached";
    unique = <fd2d35b7 cea70fab 2e22850b 3f39070d a7fa0f52>;
}
Stored Codesign length: 12 number of blobs: 0
Total Length: 12 Found blobs: 0
DisplayLink USB Graphics Software for macOS 5.1.1.dmg does not have a ticket stapled to it.

Well, they didn’t staple the DMG file, how about the kext itself?

Persephone:Extensions tom$ stapler validate -v DisplayLinkDriver.kext/
Processing: /Library/Extensions/DisplayLinkDriver.kext
Properties are {
    NSURLIsDirectoryKey = 1;
    NSURLIsPackageKey = 1;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Kernel Extension";
    NSURLTypeIdentifierKey = "dyn.ah62d4qmuhk2x445ftb4a";
    "_NSURLIsApplicationKey" = 0;
}
Props are {
    cdhash = <c90f6a0c 1076a443 e73cf694 9fe11422 f63f383e>;
    digestAlgorithm = 2;
    flags = 65536;
    secureTimestamp = "2019-04-12 09:34:45 +0000";
    signingId = "com.displaylink.driver.DisplayLinkDriver";
    teamId = 73YQY62QM3;
}
DisplayLinkDriver.kext does not have a ticket stapled to it.

Nope, no joy there, either. How about the package inside the DMG?

Persephone:Extensions tom$ stapler validate -v /Volumes/DisplayLink\ Installer/DisplayLink\ Software\ Installer.pkg 
Processing: /Volumes/DisplayLink Installer/DisplayLink Software Installer.pkg
Properties are {
    NSURLIsDirectoryKey = 0;
    NSURLIsPackageKey = 0;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Installer package";
    NSURLTypeIdentifierKey = "com.apple.installer-package-archive";
    "_NSURLIsApplicationKey" = 0;
}
Sig Type is RSA. Length is 3
Sig Type is CMS. Length is 3
Package DisplayLink Software Installer.pkg uses a checksum of size 20
We do not know how to deal with trailer version 41376. Exepected 1
DisplayLink Software Installer.pkg does not have a ticket stapled to it.

Well, if they notarized any of the parts, they didn’t actually complete the process in a way that allows us to verify the process offline.

When I ran the installer package on my machine, I did receive a UAKEL alert during install that indicates that the payload was being blocked until I accepted the kext, which means that the kext was notarized, just not stapled.

So, what would lead a developer to think that they have notarized their kernel extension successfully, but the operating system would believe otherwise? I can’t be sure of what happened in DisplayLink’s case, but there’s a possibility that it was built on an airgapped system where Xcode could compile the code, and then when it was submitted to Apple for signing and notarization, the final step of stapling the returned ticket to the application was not completed. If the ticket isn’t stapled, Gatekeeper will recognize the unstapled object, because Gatekeeper can talk with Apple and ask for a check based on other factors.

Apple’s Developer Documentation says:

Notarization produces a ticket that tells Gatekeeper that your app is notarized. After notarization completes successfully, the next time any user attempts to run your app on macOS 10.14 or later, Gatekeeper finds the ticket online. This includes users who downloaded your app before notarization.

So, if you deliver an unstapled object, as DisplayLink has, it may still pass muster, but that requires your machine to be able to talk with Apple at the time of install. If you are operating a network which embraces 802.1X user certificates, and you install software at the login window (with Munki, say) you may run into a circumstance where the software is actually notarized by Apple, but without that stapled ticket, you’re stuck if you can’t talk to Apple to prove it. This will result in a failed install.

So, Who Do You Need To Talk To?

According to Apple:

In addition, stapler uses CloudKit to download tickets, which requires access to the following IP address ranges, all on port 443:
17.248.128.0/18
17.250.64.0/18
17.248.192.0/19

If you can’t open up your network to those segments, consider that failure to do so will mean you cannot run what you need to run to make your Mac endpoints successful.

So, What Can I Do?

Well, you might be able to try stapling on your own. If it’s been validated by Apple during a notarization process, but the distributed resources are unstapled, you may be able to “fix” that by trying to staple the necessary objects yourself. They’re notarized, after all, just not by you! You can attempt this yourself.

xcrun stapler staple /path/to/DisplayLinkDriver.pkg

This results in a different result when you review the dmg file:

Persephone:Extensions tom$ stapler validate -v ~/Desktop/DisplayLink\ Software\ Installer.pkg 
Processing: /Users/tom/Desktop/DisplayLink Software Installer.pkg
Properties are {
    NSURLIsDirectoryKey = 0;
    NSURLIsPackageKey = 0;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Installer package";
    NSURLTypeIdentifierKey = "com.apple.installer-package-archive";
    "_NSURLIsApplicationKey" = 0;
}
Sig Type is RSA. Length is 3
Sig Type is CMS. Length is 3
Package DisplayLink Software Installer.pkg uses a checksum of size 20
Terminator Trailer size must be 0, not 2073
{magic: t8lr, version: 1, type: 2, length: 2073}
Found expected ticket at 7812133 with length of 2073
JSON Data is {
    records =     (
                {
            recordName = "2/1/5362032c46062ca6e74bab1bf6ce672f6a578989";
        }
    );
}
 Headers: {
    "Content-Type" = "application/json";
}
Domain is api.apple-cloudkit.com
Response is <NSHTTPURLResponse: 0x7f85265134a0> { URL: https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup } { Status Code: 200, Headers {
    "Apple-Originating-System" =     (
        UnknownOriginatingSystem
    );
    Connection =     (
        "keep-alive"
    );
    "Content-Encoding" =     (
        gzip
    );
    "Content-Type" =     (
        "application/json; charset=UTF-8"
    );
    Date =     (
        "Thu, 18 Apr 2019 20:23:51 GMT"
    );
    Server =     (
        "AppleHttpServer/70a91026"
    );
    "Strict-Transport-Security" =     (
        "max-age=31536000; includeSubDomains;"
    );
    "Transfer-Encoding" =     (
        Identity
    );
    Via =     (
        "xrail:st13p00ic-zteu25223401.me.com:8301:18H164:grp60",
        "icloudedge:sv05p01ic-ztde010811:7401:19RC85:San Jose"
    );
    "X-Apple-CloudKit-Version" =     (
        "1.0"
    );
    "X-Apple-Request-UUID" =     (
        "95f1738a-0da3-441e-abe4-982d57970d51"
    );
    "X-Responding-Instance" =     (
        "ckdatabasews:16302401:st42p63ic-ztfb18181201:8201:1906B425:3cafa700202"
    );
    "access-control-expose-headers" =     (
        "X-Apple-Request-UUID, X-Responding-Instance",
        Via
    );
    "apple-seq" =     (
        0
    );
    "apple-tk" =     (
        false
    );
} }
Size of data is 3377
JSON Response is: {
    records =     (
                {
            created =             {
                deviceID = 2;
                timestamp = 1555062296808;
                userRecordName = "_d28c74d190a3782e89496b0a13437fef";
            };
            deleted = 0;
            fields =             {
                signedTicket =                 {
                    type = BYTES;
                    value = "snipped for simplicity.";
                };
            };
            modified =             {
                deviceID = 2;
                timestamp = 1555062296808;
                userRecordName = "_d28c74d190a3782e89496b0a13437fef";
            };
            pluginFields =             {
            };
            recordChangeTag = judvxvj5;
            recordName = "2/1/5362032c46062ca6e74bab1bf6ce672f6a578989";
            recordType = DeveloperIDTicket;
        }
    );
}
Downloaded ticket has been stored at file:///var/folders/tk/qhvvt21x7z3fzt125dpgjlym0000gp/T/95f1738a-0da3-441e-abe4-982d57970d51.ticket.
The validate action worked!

This will mean that, as admins, if we want to install notarized software in a circumstance where network access won’t permit a conversation with the Apple CloudKit servers, you’re going to want to make sure the notarization ticket is stapled to the installer. This may require changes to our workflows, and now’s a good time to start thinking about what that will mean for automatic download and interpretations of installers.

Thanks as always to the gang from #notarization on the Mac Admins Slack for providing good discussion of a difficult topic.

Editor’s Note: This is an evolving topic and by the time you come across this in a search engine, circumstances may have changed. Treat this post as a frozen moment in time, things may have evolved for better or worse in the intervening weeks.

BLUF: If you are whitelisting kernel extensions on Macs with UAMDM, by Team ID, or by Team ID and Bundle ID, notarization is not necessarily required as of beta 2 of macOS 10.14.5. Those without UAMDM-defined kernel extension whitelists will need to make sure that kernel extensions are installed with both valid signatures and a correct notarization secureTimestamp.

Kernel Extension Signing in macOS 10.14.5 beta 2

Let’s begin with the recitals: beginning with macOS 10.14.5’s release, kernel extension signing is no longer sufficient. Kernel extensions updated after March 11th, 2019, or created for the first time after that date, will need to be notarized as well as signed. This means that your application and all attendant parts must have been signed and notarized by Apple. Here is how Apple explains this:

Notarization gives users more confidence that the Developer ID-signed software you distribute has been checked by Apple for malicious components. Notarization is not App Review. The Apple notary service is an automated system that scans your software for malicious content, checks for code-signing issues, and returns the results to you quickly. If there are no issues, the notary service generates a ticket for you to staple to your software; the notary service also publishes that ticket online where Gatekeeper can find it.

Notarizing Your App Before Distribution, Apple Developer Documentation

We had two easy tests for how this operated. Once macOS 10.14.5 beta 2 was installed on my daily driver, I downloaded updates to two of the apps we use that have kernel extensions and had been updated after March 11th: VMware Fusion Pro 11.0.3 and Kerio’s VPN Client 9.3.0.

On install of the new VPN Client, I received the following dialog:

Kerio’s VPN Client was now dead in the water and not functional, no matter what I could do to follow up. An inspection (which requires Xcode 10.2 and not just the command line tools) of the kvnet.kext file in /Library/Extensions indicated I did not have a valid kernel extension any longer:

Persephone: tom$ stapler validate -v /Library/Extensions/kvnet.kext/
Processing: /Library/Extensions/kvnet.kext
Properties are {
    NSURLIsDirectoryKey = 1;
    NSURLIsPackageKey = 1;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Kernel Extension";
    NSURLTypeIdentifierKey = "dyn.ah62d4qmuhk2x445ftb4a";
    "_NSURLIsApplicationKey" = 0;
}
Props are {
    cdhash = <5bf723ec 9f7a0027 4592266d 0514db04 5f1760bb>;
    digestAlgorithm = 1;
    flags = 0;
    secureTimestamp = "2019-04-08 12:34:03 +0000";
    signingId = "com.kerio.kext.kvnetnew";
    teamId = 7WC9K73933;
}
kvnet.kext does not have a ticket stapled to it.

Without a valid ticket stapled to the kext, I was going to have a problem running it, as the secureTimestamp value is after 2019-03-11.

Well crap. I need that kernel extension to work for my VPN to client locations to work, so how am I going to get around it? Thanks to #notarization on the Mac Admins Slack, and Allen Golbig at NASA Glenn, Graham Pugh, and the help of others, the answer was already in our hands: User-Accepted Mobile Device Management and Team ID Whitelisting in the Kernel Extensions Whitelisting payload in MDM.

If you have a Mac with UAMDM (either via actual user acceptance, or via implied acceptance through Automated Enrollment), and you are specifying the Team ID of kernel extensions that you want to be whitelisted the new requirement of kernel extension whitelisting is transitive, meaning checks are not made to the notarization of the kernel extension, as the signing of the kernel extension is sufficient to its privileged execution.

This year’s MacADUK Conference is in the books, and I’ve made it back to the States in one piece. It was a busy week, full of socializing and engaging with colleagues, as well as learning about new topics in client management and deployment workflows, encryption details, and security philosophy. My sincerest thanks to Ben Toms and James Ridsdale from Datajar who chaired this year’s conference, and to the team at Amsys that handled logistics and details.

Highlights

This year’s conference had some great sessions. When the videos are out, I would strongly recommend seeing the following sessions on the small screen:

Armin Briegel, Deployment: Modern deployment workflows for business

Deployment is a source of opportunity for every IT out there. It’s literally your coworkers’ first impression of your operation, so why aren’t you putting your best foot forward with customized deployment via Automated Enrollment and Mobile Device Management. Figuring out how to replace the older technologies of ASR-based imaging with new deployment strategies is a challenge worth embracing.

Chris Chapman, macOS in a Docker Container for Development

There’s no question that Docker and Kubernetes are key components of modern software development stacks, especially for web-oriented applications. Chris Chapman of MacStadium has taken this to a whole new level, by writing a boot loader for Kubernetes and Docker for Apple hardware, allowing you to deploy a macOS image through orchestration and docker. The more I think about this, the crazier it is, but it demonstrates a flexibility that wasn’t possible before. I’m sure this is completely unsupported, but what a phenomenal way to think about the underlying tool chains we build from. It’s called Orka, and MacStadium is looking for beta sites.

David Acland, All about digital signatures

We spend a whole lot of our admin life making sure that signatures align and are approved, but how does that process actually happen? What’s the working relationship between a hash and a signature? What’s the actual cryptographic process used to take a hash and sign it as a measure of identification integrity? David took us through the details, and it was a real pleasure. And my head didn’t explode.

Ben Goodstein, Working with APIs: Power up your scripts with real time information

APIs as part of scripts is table stakes for adminry these days, and where better to get a refresher than with a low-stakes custom API that Ben wrote for accepting data from a script. He also told us about Insomnia, a GUI app for practicing with, in order to review what’s come down from an API call, and help better gather information. It was a great session, and I learned a lot of useful things to iterate against.

Takeaways

I had a few big thought lines that came back a few times during the conference, and lead to some noodling in my head on walks through London. We’re once again at an inflection point in macOS Administration, much as we were in the 10.8/10.9 period, the 10.7 period, and the 10.5 period. There are changes to our management structures that are no longer flashes in a pan:

MDM is not optional.

Deployment should be automated.

Manage as little as you need to retain a presence on the platform.

Managing more than you need to results in Shadow IT and Loopholes.

IT Operations relies on trust. Not just mechanized and automated trust chains established through TLS certificates and certificate authorities, it relies on a human trust that is implicit between Management and IT, and IT and the end users, your coworkers. For any IT policy to succeed, it must come with buy-in from your coworkers, not just in your department, but in your whole organization. Systems that are deemed too complicated will be ignored. Systems that are deemed too cumbersome to be operated will spark grudges. Systems that are deemed to be unpersonalizeable will result in shadow IT usage on personal equipment.

The balance between security, usability, and management philosophy remains the single most important challenge of any IT environment, large or small. If you have a bad balance, your coworkers will fight with you, resent you, and eventually work around you and cut you out.

Having a light hand on your workstations will be fought by internal and external security guidelines, though, and you’ll need to be ready with justifications based on feedback in the event that your choices are questioned. Obviously, there are some guidelines you can’t ignore. But, the security of the platform needs to be part of your process, not bolted on, not thought of after, but holistically part of your deliberations and choices. Self-healing management is a part of that, as is centralized reporting mechanisms designed to track the state of a machine.

If IT isn’t working to enhance the culture of your organization by extending and embracing systems of participation and training, your value will be subsumed by internal groups that are doing these things. That means providing good guard rails, but also providing knowledge and power at the workstation level to enhance your colleagues’ ability to do their jobs.

IT is a human-facing department in 2019. We serve the people. We just also serve the machines they use.

As an Apple-focused Admin, one of the tools that most belongs in your toolbox is an understanding of how Wi-Fi works on a fundamentals level. You need to know how devices interact with this network layer as much or more than you do how they interact with TCP/IP and other Layer 3 technologies. In our day-long workshop, Chris Dawe and I are going to be talking about how Wi-Fi works, from a history of the technology (now in its 3rd decade!), to how your Apple devices interact with Wi-Fi, to how to troubleshoot networks and design better ones.

We’ll be doing this workshop at X World in Sydney in late June, and then again at Mac Admins in July. We’re thrilled to be working with two of the Apple Admin world’s best conferences.

We’ll be starting with a thorough history of Wi-Fi, moving to the nuts and bolts of how Wi-Fi works between a client and access point, and introductory sections on network design, network troubleshooting, and network security, and then an overview of the survey and analysis tools. We’ll wrap it all up with how Apple devices interact with all of the above, and all the specialized knowledge you’ll need to make sure that your networks are tuned properly for your fleets.

Physics Always Wins is the name of the workshop, and a stern reminder that there are good rules of the road to follow for good Wi-Fi. Discerning what the best settings are for your environment feels like an artisan’s job these days, given the layers of marketing speak and incomplete understandings of the radio frequency world. We’re hoping to give more Apple Admins a firmer understanding of how it all works, so you can make the right engineering choices.

One of Technolutionary’s first purchases in 2006 was this Mac mini, which today returned to our offices after 13 years of duty in co-location at Solid Space in North Carolina. It’s been a file server, a mail server, and more, and we absolutely, positively got our money’s worth on this beauty.

Thanks for the service, Mac mini, your retirement awaits. Thank you Apple, for building technology we can depend on.