There are a lot of complaints about the Star Spangled Banner that I’ll hear and accept. It is a song with a massive vocal range, at just over an octave. It is a song often performed at a dirge pace, unlike the drinking song it steals its melody from. It is frequently embellished by those who maybe shouldn’t.
His rendition weighs in at 84 seconds, is sung in an approachable key, instead of one that will strain the basses on top or the sopranos on bottom. It is powerful and confident, and it understands the lyrics’ impact without overdramatizing them. It finishes on the final strains with such force as to singlehandedly keep that flag flying over Baltimore’s Fort McHenry amid the tumult.
If every rendition could be so good, perhaps we might not have to fall back on the execrable God Bless America now seemingly required in the 7th inning.
I’m glad the world got to see DC perform last night, and I hope they get to again next week.
Today is World Mental Health Day. My name is Tom Bridge, and I suffer from periodic depression. It’s not a formal diagnosis, it’s just something that I’ve known about myself going back to, I think, junior high school. I worked through all of this stuff on my own, mostly. In college, I saw a therapist, who gave me a lot of tools for managing my seasonal affection.
Over the last few winters, it’s gotten worse, and being around me in the winter has been difficult for my family and for my colleagues and for my friends. I finally started to seek professional attention for this issue about 18 months ago, and things have gotten a bit better. I’ve found a medication that my body seems to like (after finding one it really did not like) and I’m hopeful that this winter isn’t like the last few!
I gave a lightning talk at X World this year about mental health, because I want to destigmatize this disease that is stunningly common. Our society often treats mental illness like a moral failing, when it’s not that at all. Mental illness is like cancer, or like any other chronic condition that you have to fight through. Depression is real, I have it, and sometimes it makes life terrifically difficult in ways that are hard to describe and cope with.
So, all my sysadmin friends, and all my real life friends, it’s important to take time to troubleshoot your own troubleshooting self. Your brain may not respond to sudo the way that your computers do, and your life may not order itself around obvious log files and audit trails. Brains are more nuanced. But just like having trouble with anything in your IT world, you’ve got resources to call on.
And if you don’t, look me up, and we’ll help find you some together.
Long as I’ve got a job, you’ve got a job. Let’s get through this together.
I noticed my watch battery wasn’t lasting as long as it once did. Instead of going to bed at 11pm with 40-50% battery, I would run out at 10pm, then 9pm. I started to see the red glyph on my watch that says it’s disconnected from my phone more often, at peak 10-20 times a day.
I began to troubleshoot:
Unpair my watch & restore from backup
Wipe phone & restore from backup
Unpair my watch & setup as new.
Nothing has brought my watch anywhere close to back to normal.
I’m not using a lot of apps on my watch. I have a couple complications I use from 3rd party apps (Carrot weather mostly!) and they update infrequently.
I am trying to determine what I can do to bring my battery life back to where it was for the first six months of life. I have no idea how to do this, and strangely, neither does Apple.
My Support Journey
Back in May when I first noticed this, I went to Apple’s Carnegie Library location in DC to get this fixed. Diagnostics were done, nothing was found, and I was offered their only service option: we can send it out for a week and get it serviced. No loaner, no replacement unit, just an all expenses paid week for the watch at the Austin service depot. Too much, I thought. I could deal with a 10% alert at bedtime.
Now that alert arrives at 8pm.
In June, I shattered the glass on my iPhone and had that repaired, while I waited in a comfortable atrium working on fast Wi-Fi. 90 minutes later, I had a good as new phone. Even if I’d bent the case or done more damage, I would’ve left the store with a working phone under my AppleCare+ arrangement with Apple. Service units exist for the iPhone, why not for the Watch, too?
Two weeks ago, I started to poke at this again, knowing full well the complications the beta cycle would add to my support experience. I asked informally at the store if they had advice while picking up another repair. They advised the app would be a good way to approach this.
I used the Apple Support app and got a responsive and polite but ultimately unsuccessful support engineer who tried to help but was unable to do so, despite diagnostics and support efforts. They didn’t want to leave me unhelped, though, and arranged a callback from someone who could help with devices that had the beta installed.
I spoke with the phone support agent and repeated the process, but got no further than being asked if I could send them the watch for a week. Instead, I arranged with the phone support agent to go to the store for the diagnostics instead, since at least there I’d have a person who could tell me what was up.
Today was the visit to the Carnegie Library store, back where I started. I sat with a Genius for 20 minutes who confirmed I was having a problem, but couldn’t explain it, or help resolve it any further. I was offered a mail-in repair on the spot, no loaner. I could call if I wanted to setup an advance replacement for $30, but the only way to do that was over the phone.
My Self-Diagnosis
There’s a clear, demonstrable and repeatable problem with my watch. It doesn’t stay connected to my phone all the time, even when it’s close by. This is the symptom, there’s an underlying problem with the bluetooth stack on one or both devices that is causing dropouts in the connection. I cannot repeat this with my AirPods, Car stereo, or bluetooth speaker, but I am getting odd latency on my Tile reports.
I don’t have the tools to diagnose this, as far as I can tell, and apparently neither does Apple. What they do have, as device manufacturers, and the makers of the software that runs them, is unlimited latitude to make things right by switching devices out. They have opted not to do that, citing a lack of damage.
I have a broken watch, or a broken phone. I’ll find out in ten days when my new phone arrives. I am frustrated that the diagnostics cannot explain what’s going on with the disconnects between the Watch and the iPhone.
What Apple Watch Means To Me
Apple Watch has been my constant companion since the Series 0. I owned the first Apple Watch, then the Series 2, and now a Series 4. While I find it awkward to wear a watch — I’m no horophile(1) — the taps on my wrist have allowed me a greater freedom to be in touch with my profession while being distinct from my phone. I can keep distractions at bay in meetings, but still be reachable for emergencies. I can keep focus on my clients without losing track of an emergency. I’ve written at length about how my Apple Watch saved me thousands of dollars at the emergency room and cardiologist this year. Apple is definitely right when they call it their most personal device. So, why won’t they treat it like that in their stores? Why not support trade-ins for repair on the spot? There is an incredibly healthy refurbished market for these units that Apple participates in, after all.
Apple talks a big game, and often delivers. I was delighted with the video that lead off the iPhone 11 announcement event, and I was doubly delighted that Apple understands how to market how helpful the Watch is as a product. Apple says: Give people wonderful tools and they will do wonderful things. They are right. I can do marvelous, magical, incredible things with the technical Apple sells.
When it stays healthy.
You might think this would be chronophilia. A quick Google set me back on the right path. Whew. Glad I checked.
You’re going to want to read Part 1 of this piece first. It covered managing the installation of packages that are not notarized. This section of the article covers the operational limits of doing the same thing for applications, using the operation:execute verb. There are several caveats to all of this that may make this procedure non-viable. But, there’s also a way to make it work, so here we are.
The Mechanism
Last time, we got a package installer whitelisted. That’s a simple operation: whitelist the certificate of the installer with the right Requirements payload, and the installer package will pass through quarantine checks unobstructed.
But what about a non-notarized application?
There are some software manufacturers who have challenging installers (which is to say: non-pkg, app-based installers) that require a different sort of whitelisting. In this example, we’re going to be reviewing the Cisco Webex Add-in. The installer is an application, intended to be run by a user. It’s a signed application, thankfully, and that gives us everything we need to build a whitelist to recover from the lack of a notarization process.
The Requirement payload in this this profile might look familiar. The identifier and certificate language looks a lot like the Privacy Preferences Policy Control payload that was introduced with macOS 10.14 Mojave. Much as you would with with one of those payloads, deriving the contents is as simple as interrogating the application with codesign:
codesign -dr - /path/to/Application.app
This will return what you need to embed:
identifier "com.cisco.webex.Cisco-WebEx-Add-On" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = DE8Y96K9QP
Once again, this is adding entries to the /var/db/SystemPolicy database, which Gatekeeper uses at inspection time to determine whether or not something passes. This is your pass past the security desk.
There Are Big Caveats
Okay, here’s where it gets less ideal.
There is only one road to a successful deployment.
This profile cannot be installed before macOS 10.15 Catalina is installed. Not because the profile can’t be interpreted by Mojave – it can – but because the installation process for macOS 10.15 might reset the SystemPolicy database.
And, because profiles are only ever interpreted at the time of install, you now have a profile that is installed on the machine, but no longer in the database that Gatekeeper uses.
One such failed case is here.
Once the profile’s installed, it’s installed, and it can’t or won’t be reinterpreted by the System. This is a huge flaw in the configuration profiles system overall, and we’ve seen it come back to bite us in the behind each of the last two revisions of the OS, with no revision in the system. The lack of defined-state management with configuration profiles, coupled with management-via-UDP profile delivery, means that we’re kinda stuck.
So, let’s talk about how this can go poorly.
Some MDMs have a method for reinstalling certain types of profiles after an operating system upgrade to make sure that profiles get reinterpreted, but none of those mechanisms are foolproof. There’s almost always a little lag between those mechanisms hitting and the upgrade completing.
What happens then?
Well, if the application is launched, and it won’t pass the Quarantine check, then the file gets marked as a failed application, and that file is, for all intents and purposes, blackballed.
Normally, that would mean the only recourse at that time is to uninstall and reinstall the application. If you have a good uninstall script, this could be a solution. There are some applications that aren’t such good citizens about containerization and slop their resources all over your filesystem.
But here’s the rub, even if you do catch all the resources, now you’re in an exorcism situation. There are some applications that just won’t be bypassed this way. I was able to get this working, but a colleague with another MDM has not been successful as of press time in making this work 100% of the time.
Well, That’s Bad. Now What?
Well, for starters, if you can, deploy these tools with other means. Remember that passing these checks are a lot like getting pass the security desk. Tools like the Jamf binary and root agents like Munki can build an Employees Entrance for key personnel. Installing with these tools bypasses the Quarantine process, which means notarization checks won’t apply.
If you’re expecting for users to directly download and manually install software that isn’t signed and notarized, you’re going to need to either have a robust help desk that can handle the volume, you’ll need to put pressure on your software vendor to do the responsible thing and fix their installer or fix their deployment mechanism(1).
Whitelisting is going to be a game of whackamole. If you have to play it, you’re going to need to plan for two things:
Deliver your profiles with an MDM that can rapidly re-deliver the necessary profiles after a major version upgrade.
Consider delivering your major version upgrade along-side a pre-install script that scrubs out packages that aren’t properly notarized.
Whatever you do, this is a use-case you will need to prepare your service desk for.
First up, this post is a direct response to my previous posts on this summer’s talk about notarization. Notarization is a subject of much discussion, and there’s a lot happening out there. If you are looking for an exhaustive summary of notarization through many, many links, might I recommend this compilation post on Mr. Macintosh. If you are looking for the TL;DR, here it is: macOS 10.15 requires that software be notarized to pass the Gatekeeper checks. If it is not notarized, it will not pass these checks unless you can manipulate the System Policy Database to whitelist a Team’s certificate.
I didn’t do this alone, and I want to say a huge, huge thank you to everyone who helped me out as we tinkered through this somewhat opaque process.
In the Beginning, there was Gatekeeper and spctl
Before we go digging into how all of this works, you need to understand the importance of Gatekeeper in the process of reviewing the notarization of individual items. The spctl binary that is part of macOS’s command line interface, and has been for a very long time, are responsible for controlling what Gatekeeper looks at. These both write to a sqlite3 database stored at /var/db/SystemPolicy, and think of it a lot like a database of ID cards that the security guard at the desk will review. If your card is recognized, you pass through security without more than a passing hello at the barrier. If you card is not recognized, your ID is checked, your destination cleared, your name jotted down, and you’re granted a card if you belong.
This is how spctl whitelists applications for LaunchServices’ purposes.
This system can be directly manipulated via configuration profile, and those configuration profiles can be delivered by a capable MDM. Moreover, this has been the case since macOS 10.12. Hidden away in Apple’s documentation is the SystemPolicyRule payload type, which can allow you to embed whitelisted objects in an MDM Profile.
Anatomy of a SystemPolicyRule Payload
The key properties of the SystemPolicyRule payload
There are four interesting elements of the payload, and I’m going to take them in reverse order, from the bottom up:
Requirement is the policy requirement for the Code Signing information, and must match the syntax described by the CSRL. If you are familiar with writing PPPC policies in various tools, this will not be unfamiliar. The Requirement for our example profile is:
In this case, the value of the string contains a reference to the hash of a specific certificate, here called DeveloperCertificate. We’ll come back to that certificate in a bit, but it corresponds to the signing certificate used by the developer to sign the package. Here, the $HASHCERT_ prefix is really a command to the MDM. It will derive a SHA hash value of the attached certificate and replace it in the string, so that the final value is then interpreted as:
Priority is the weighted value of the policy requirement for the payload. Rules can have varying priority level, and in our test profile, we’re using a Float value of 100.0.
Operation Type will tell you what kind of action you want govern. The operation type we care about for this example profile is an install. We want Gatekeeper to skip the notarization check during installation, so we’re going to add an OperationType of "operation:install"
LeafCertificate is the sticky point and that’s why I left it for last. You have to define the LeafCertificate of the signer. In our example profile that follows, the entirety of the public key of the signer’s certificate is embedded in the profile. This is going to require the most legwork, but there are shortcuts to get there. Flat packages in macOS have an XML Table of Contents, and you can use the xar command to extract it to a file where you can read it:
xar -t -f /path/to/your/package.pkg --dump-toc=toc.xml
You can now read the toc.xml file in BBEdit or any other fine text editors to review the contents of the XML table of contents, which will contain the signing certificate required for what follows:
I’ve tested this profile with SimpleMDM and a current macOS 10.15 Catalina system with a signed-but-not-notarized package that would fail to install otherwise, and I was no longer warned while trying to install this otherwise-quarantined package.
The profile was delivered via MDM and now shows a System Policy Rule with detail that matches our payload content. Now, keep in mind, the /var/db/SystemPolicy database only cares about the Requirement item, which is just a hex hash of the certificate. You can, if you choose, just include the hex hash in the Requirement item, but if you choose to do that, you will have to know what that hash represents. If you want to be kind to your users, which you should, you should include the Certificate itself, which will be decoded in the display to show you which signer and which certificate authority the certificate resolves against.
After the certificate is installed, you can check your work with a review of the /var/db/SystemPolicy sqlite3 database’s authority table. It should show a new entry that matched the newly whitelisted certificate:
The red box shows the newly added rule that arrived via MDM Profile.
This appears to be how you can whitelist individual signing certificates for notarization checks and distribute that whitelist to clients with a Mobile Device Manager using a payload that’s been around since macOS 10.12.
How Do I Do This For My Environment?
Start with a package that is signed, but not notarized. Using the xar command, extract the X509 Certificate of the package. You will need this for your profile.
Second, use uuidgen to create new UUIDs for the profile. You’ll need at least two. Replace the two UUIDs in your copy of the profile. UUIDs are paired in the PayloadIdentifier and PayloadUUID fields.
Customize your PayloadIdentifier and Comment fields with your organization and a description of the policy.
Postscript: Using This To Run Non-Notarized Applications
While all of the above is intended for the operation:install key, operation: execute would allow you to run non-notarized Applications without Gatekeeper dialogs for those applications that are downloaded in their entirety without an installer package. You will need a separate profile if you want to whitelist both an installer and an application.
To make this transition easier and to protect users on macOS Catalina who continue to use older versions of software, we’ve adjusted the notarization prerequisites until January 2020.
You can now notarize Mac software that: • Doesn’t have the Hardened Runtime capability enabled. • Has components not signed with your Developer ID. • Doesn’t include a secure timestamp with your code-signing signature. • Was built with an older SDK. • Includes the com.apple.security.get-task-allow entitlement with the value set to any variation of true.
Make sure to submit all versions of your software. While Xcode 10 or later is still required to submit, you don’t need to rebuild or re-sign your software before submission.
This represents a substantial change over the existing guidelines. This is a positive development, in my eyes, which allows more developers to submit their packages, disk images and zip files for notarization in their current form, and to work over a longer term to get the Hardened Runtime enabled, as well as find replacements for third-party pre-compiled frameworks that are submitted with another developer’s signature embedded.
This does still mean you need to get notarized packages, zips and disk images for your environment if you intend to have 3rd party non-AppStorer software installed directly by end users. If you are installing tools via Munki’s LaunchDaemons or Jamf’s framework, this doesn’t apply yet.
Any issues encountered during notarization (like not enabling the hardened runtime or not validly signing binaries) will still be warnings in the logs, so make sure to read and address them!
In the early days of Web 2.0, in the short time after the dot com crash, there arose a common standard for syndicating your blog across the web, into RSS Readers. Google Reader was a big damn deal in those days, but before Google Reader hit the market, there was NetNewsWire.
Brent Simmons’ app was the RSS Reader for the Mac for a good long time, and an app that I lived and died by. In the days where Twitter (blessedly) did not yet exist, getting your news meant going to a website manually, like some kind of animal. NetNewsWire could read the secret code that held these sites together and produce a feed of articles that you could pay attention to directly, without having to remember which sites you needed to see.
In the post-social world, where suddenly everything got dumped out to the feeds full of our friends’ quick thoughts and longer form rants, RSS began to die a bit. Google saw that Reader was cannibalizing their own ads, and rapidly pushed it to the ash-heap of history, and in-so-doing, wrecked a whole lot of models for publishing. Suddenly, we were back to depending on people to go to browser-based reading habits, which came with a ton of terrible ads, tracking that was just full of garbage, or a social existence defined by the hellscape that Twitter and Facebook have become.
All this set the table for the return of NetNewsWire, which exited beta last week, and returned to my dock shortly thereafter. The base metaphor of NetNewsWire (NNW) is unchanged: feeds, grouped according to your choices, contain stories, which can view feed by feed, or in a timeline. Anything that can be served up as RSS can be shunted over into NNW’s hopper to await your attention.
For the last few years, I have used the #blog-feed channel on the Mac Admins Slack as my version of a professional RSS reader. I’m moving all those feeds to NetNewsWire so I can better track what I’ve read and what I haven’t. Now I’ve got a great view into what I’m up to date on, and what I’m yet to cover.
A view into my RSS feeds
This is about to be heavy season for Mac Admins, if it’s not already. We’re in the waning days of the beta period before macOS Catalina 10.15 drops and iOS 13 is released. We’ve got a lot of work ahead of us. How about helping yourself with a whole list of helpful feeds that are there to keep you up to date?
