Making Big Decisions With Scoring Rubrics

About a year ago, as my car crossed the 75,000-mile mark, I started to think it might be time for a new car. My Ford Fusion Hybrid, bought the summer before Charlie was born, has been a real performer, but the edges were definitely starting to show. Fuel economy has been a bit on the decline, the lack of good car interface is showing, and, frankly, I was feeling like it was time.

My search was, for a long time, pretty casual. I started looking at car news sites as new models came out, and I had a Notes file going for which cars I liked and why. At the same time, I setup a Digit savings goal (Disclaimer: we both get $5 if you sign up with that link) to help me set aside money for a down payment.

There were setbacks and fits-and-starts as I had to reprioritize other monetary priorities, but over the winter I hit a high enough target to make a car payment affordable again. That’s when things got real. I started an AirTable to keep track of my different options.

Comparing two options is pretty easy. You start with a Pro/Con list and work it out based on a weighting of options. When you start with a larger field, the decision-making matrix gets a lot more complicated. That got me to thinking about the way we make product decisions, generally, in IT operations. We frequently help organizations compare MDMs from a set group, compare and contrast features, and help the client weight the choices accordingly.

When faced with a similar situation, my wife and I used AirTable to make a weighting system for prospective schools for Charlie. AirTable is highly flexible, and allowed us to easily build a system for understanding our options.

We specified tags and star rankings, and used that, along with distance ratings, created a scoring system. Montessori schools added points, same as schools in our part of town, and our likelihood of getting in.

We ended up with this formula:

IF(1/Distance > 1, 3, 1)+{⭐️}+IF(Montessori = "Yes", 4, 0) + IF(Uni = "Yes", 0, 1) + IF(Ward = "Ward 5", 2, 0) + IF(Ward = "Ward 4", 1, 0) + {Waitlist Score}

It gave us a guide for handling a complex decision. Sure, it doesn’t beat going with your gut. At the end of the process, we didn’t just go in score order, because there was no adjustment in our formula for intangibles. That changed with the car search this time around.

The Looking Phase

You can compare cars, and rank their cost, speed, power, fuel economy, cargo space, passenger load, wheel size and more, but that’s only part of the process. There are subjective factors, like how it corners, what it feels like to sit in the seats, and how comfortable your passengers might feel, and that gives you extra dynamics. The same is true of MDMs, Cloud Directories, Solutions Providers, and more. Having room in your own rubrics for scoring for the intangibles like design UX, code extensibility, and more, will give you a better feel for what it all comes down to.

When it came to the search, I didn’t know what I didn’t know, so I went to the Washington Auto Show at the DC Convention Center in early January. I took my best tester, Charlie, and we went and sat in just about every compact crossover from Toyota and Hyundai up to Alfa Romeo and Jaguar. I had made an entry form to rank a bunch of key stats, and a picture field to help remind me of a car’s aesthetic features.

I knew I wanted to see the Subarus, but on my way through to them, I encountered the new RAV4 and Highlander from Toyota. I enjoyed their aesthetics enough to put them on the test drive. Though they ended up outside of my price range, I was impressed with the build quality and cabin construction of the Buick Enclave models they had on-site. I had no idea they were so nice! I never thought much of the Buick badge as a whole, but their product was quite nice. Charlie thought they were his favorite of the day.

Sitting in an Alfa Romeo Stelvio will always remind me of the week I spent with one in the Alps this winter, so that was a bit of a sentimental, if completely impractical (they’re sold and serviced at Maserati dealers! Come on!) entry on the list. The Jaguar E-PACE was a dizzying build of practically every little option on their website, but the interior was quite luxe and comfortable. Onto the list it went.

Sadly, Audi, BMW, Mercedes and Mini Cooper all passed on this year’s auto show. While I liked the look of the BMW X1 and X2, I couldn’t get over how terribly rated their console system is. I couldn’t drive a Mercedes and feel good about myself. Mini Cooper sounds more fun than its practical reality. But Audi gave me some thoughts, so I ended up seeking out an Audi dealer to try their new Q3 SUV.

Several friends recommended I check out Hyundais as well. The Auto Show exhibit was great for sitting in just about every version of the Hyundai lineup, from the Kona to the Palisade. I was surprised, though, that you had to go all the way to the top of the line before leather seats made an appearance. I never quite felt at home in the interiors, sadly, because their pricing was quite attractive.

Evaluating Your Formula After The First Round

The Auto Show was a great place to be a car buyer. No sales people, just company reps who can’t sell you anything but the dream. It was a great place to evaluate vehicles without any pressure. I sat in car after car, asked question after question, and didn’t have to deal with a pushy rep who wanted to be elsewhere. This wouldn’t be true once I sat down in a showroom.

I spent the weeks following the Auto Show in heavy research mode, watching videos from various car reviewers (Alex on Autos was a regular, as was Doug DeMuro, the Kelley BlueBook team, and others) and tuning my ratings.

Figuring out what’s working for you and what’s not becomes the challenge after the first round. Initially, I had a few variables I was considering: Passenger Experience, Intangibles, Cargo Space, and Overall Interior. A five-star rating system was an easily-configured option in AirTable, so that’s where I started.

The original version of the scoring formula put the Hyundais on top of the pack due to cargo space and fuel economy. These were two important features, undoubtedly, but I realized that they were floor features. I cared about what the floor was, not what the peak meant above that. I iterated the formula again to give a floor value, and enhanced scoring for features that actually delighted me, instead of just met a bare minimum.

But how do you rate something when you can only look at its features, watch videos, instead of using it yourself? You can’t get the full picture of anything until you click the buttons, or push the pedals. On to the test drive phase.

Test Drives

This is where things get real. You need to be ready for the dealership experience, which is to say, you need to be ready to deal with car salespeople. They are professionally trained to get you to buy something massively expensive the day they first meet you.

This is not an ideal outcome for a number of reasons.

What I did was setup another email box just for dealers, and a Google Voice phone number just for the dealers, and use those as a way of keeping these salespeople at bay. There are good salespeople and bad salespeople. The good ones will respect your time and privacy. The bad ones, well, there’s a reason that car salespeople have a particular reputation.

Since you need to surrender your license to a dealer in order to get a test drive, there’s no point in an alias, so just give up your real name.

I test drove two Hyundais, two Subarus, an Alfa Romeo (well, technically…), an Audi, a Toyota, and a Volvo. Here were my general thoughts, if you are in the market for cars.

Toyota RAV4: Toyota updated the looks on the new RAV4 last year, and it really showed. Toyota caught me entirely by surprise at the Auto Show, and the exterior of the car made me want to test drive it. Unfortunately, this was absolutely the worst car I drove. As much as cars are comprised of a cabin, a driving experience, and a driver experience, Toyota’s changes are entirely superficial. The RAV4 was a total dud. Sluggish off the mark, soft on the steering, and sloppy in the cabin, I kept wanting more from the actual vehicle. Looks great on the outside, but that’s all.

Alfa Romeo Stelvio: When I rented an SUV at the Marseille airport for my trip up into the Alps, imagine my surprise when they handed me the keys to a fairly new Alfa Romeo. Ooh la la! I loved the way the Stelvio handled those Alpine roads, and the smaller diesel engine had plenty of capable power. The infotainment system was a bit of a disappointment, as it wasn’t a touch-sensitive screen. The cargo area was a little bit of frustration, as while we got eight bags in there, the compartment wasn’t straight-forward to work with.

The Subarus – Forester & Outback: I enjoyed the Subaru build quality, and the Boxer engine is a solid driver, but I found myself wanting a hybrid of the two. Either give me an Outback with the moonroof of the Forester, or give me the Forester with the tech stack of the Outback. Each felt like it was missing something.

Hyundai Tucson and Santa Fe: While Hyundai is making a quality product that is comfortable to drive, it’s hard for me to balance the generic cabin experience with the car that it is part of. Zippy engines, interesting technology choices. A cabin that feels like a sanitized doctor’s office.

Audi Q3: This was the car that I almost bought. And maybe if the dealer had told us what it would actually cost to own one, I would’ve, but his sales games were a total turn-off. The tech stack in this car was the best in class, with wireless CarPlay and a really interesting cockpit display that had live traffic data in the gauge cluster, the Q3 was at the top of my list. What it came down to was worse fuel economy and twenty fewer horsepower in the engine. The off-the-block feeling of the Q3 didn’t come anywhere close to matching our final choice.

Volvo XC40: The winner was the XC40 T5 Inscription in Denim Blue. 247 hp under the hood, a stellar infotainment cluster (sadly, no wireless CarPlay to be found yet), a roomy and comfortable cabin, plenty of storage space that was well constructed and laid out, and the best looking exterior of the bunch. Couple that with Volvo’s safety record and the best Pilot Assist system short of a Tesla and you have the car that won my brain and my heart.

So How’d The Scoring System Work?

Here’s the final formula:

{Interior Rating}+{Cargo Space Rating}+2*{Intangibles Rating}+2*{Drive Test Rating}+{Passenger Experience Rating}+IF({Mileage}>25,3,0)+IF({Cargo Space}>23,3,1)+IF({MSRP Price w/ Options}<42000,3,1)+{Car Tech Stack}+IF({Approx Payment}<450,2,1)

The beginning of the formula is all about 5-star ratings. A rating of the interior and a rating of the cargo space begin the process. I decided that I actually did care about how the car made me feel, and how I felt about its driving ability, and each of those ratings were doubled in the final score (up from a 1x value in the initial version). If the mileage was good, I wanted that to count for a significant bump, same as the cargo space. I wanted to penalize highly expensive vehicles, same with the financing situation, and lastly, I wanted to include the tech stack’s rating separate from the rest.

And so it came down to the XC40 and the Q3 in the final appraisal. We let both dealers know we wanted a final price, we started the finance paperwork, and set that process in motion. Audi failed pretty spectacularly in the closing of the deal. Our salesperson wouldn’t tell us the cost of a Prestige model or try and find one for us, and it was a huge frustration. Volvo was the opposite. Shawn Hirsch at Volvo Cars Silver Spring was excellent to work with. I never felt any pressure, except around the matter of our trade-in, which was a garbage fire of a process.

What Worked With The Rubric? What Didn’t?

Like any testing scale, there are going to be some intangible items, and some tangible items. You can compare numeric values when they exist, but what happens when they don’t? How do you decide how to weight these factors?

You’ve got to establish your own priorities for the product you are in the market for. If you’re looking at MDMs, and you want a robust REST API, you’re going to need to plan for that and evaluate it based on your use-case criteria. You’re going to want to weight that value based on two factors: how important it is to your use-case, and how good the result is. That’s going to give you what amounts to a vector: a scalar value for how good it is, and a directional value for how closely the product aligns to your needs. It can be the best API in the world, but if it doesn’t allow you to, say, easily remap the owner of a device from the API, you’re going to tank it.

What I found worked and didn’t with the rubric was interesting. I found that I liked all the cars that I drove while I was driving them. Well, except that Toyota. I knew I didn’t like that one almost before we got out of the dealer parking lot. But the rest of them all felt good immediately. I had to revise my ratings over time. An immediate evaluation would often lead to a less critical result because I wanted to like them. I would overlook some things on initial test — again, except for the Toyota, it really was that bad, y’all — that I would come back to in the days following my initial test drive. The important parts of this process can’t be the result of snap judgments during or immediately after your time with the sales folks and the product. You’re going to need to stew on this. I changed ratings multiple times to weight different cabin features and drive experiences. I checked and double-checked statistics.

I made a change toward the end of the process to the scoring system. This is the sort of thing that usually feels like a no-no, right? Like changing the rules after the game has started? It feels wrong to do that. But it’s not wrong at all, in fact, you should absolutely revisit your scoring rubric if you start to be shown patterns that don’t match your experience. What I found out was: all of the Hyundais were scoring the same as the Q3 and the XC40, when my opinions around them were clearly a tier apart.

Why was my formula doing that?

Well, the Hyundais both had more cargo room AND better fuel economy than Volvo and Audi models. They were getting 6 more points, and that was making up for a substantial difference in intangible and cabin ratings. In short, though they weren’t achieving on individual stats, they were making it up due to a single over-sized category boost.

I had to change my rubric to match what the emotional decisions as well as the logical side of the decision. In software, this comes down to how a piece of software looks and operates. Design is how it works, and part of how it works is what it looks like.

I am thrilled to own a new Volvo XC40, and a month after purchase, I’m still as excited as I was on day one. I love the driving experience, the pilot assist is a life-saver in heavy freeway traffic, the stereo and infotainment system are magical, and the cargo space is everything I was looking for.

If you’re interested in a copy of my AirTable, happy to arrange one. Here’s the final version in read-only format.

Virtual Happy Hour

I’ve been feeling the weight of the crisis for the last two weeks. Being in an operations job means knowing that what’s about to hit is gonna throw everyone for a loop. Being the owner of a company with a lot of small business clients means knowing that not every company’s coming out the other side of this. I’m a little worried about my company not making it if this lasts for a while.

I’ve been making it, but just.

Today, I accidentally started a Zoom meeting happy hour for some old friends, and we all sat and talked while we worked from our houses. We haven’t all been in the same place since… 2002?

It was transformative. It was light. It was just good to care with some other folks who’ve seen the same stuff you’ve all seen.

Camaraderie amidst the challenges of the pandemic is going to be a hard thing to live without. But thanks to our friends at Slack, Zoom, WebEx, Skype, Teams, Hipchat and others, we don’t have to do it alone. Technology can bring us closer, it can bridge us together from all over the world. It doesn’t always. It can be used to divide us, to put a wedge in between us. But it can also bridge that divide, span that chasm, and lift us up in the midst of crisis.

Quarantine and social distancing are lonely things. Care for yourself. Care for others. Reach out and talk to people you haven’t seen in a while. We can use our technology to help push back that loneliness.

The Best Anthem in the Biz: DC Washington

There are a lot of complaints about the Star Spangled Banner that I’ll hear and accept. It is a song with a massive vocal range, at just over an octave. It is a song often performed at a dirge pace, unlike the drinking song it steals its melody from. It is frequently embellished by those who maybe shouldn’t.

I get that.

Now listen to Dwight Clyde “DC” Washington sing it.

I’ll wait.

His rendition weighs in at 84 seconds, is sung in an approachable key, instead of one that will strain the basses on top or the sopranos on bottom. It is powerful and confident, and it understands the lyrics’ impact without overdramatizing them. It finishes on the final strains with such force as to singlehandedly keep that flag flying over Baltimore’s Fort McHenry amid the tumult.

If every rendition could be so good, perhaps we might not have to fall back on the execrable God Bless America now seemingly required in the 7th inning.

I’m glad the world got to see DC perform last night, and I hope they get to again next week.

Troubleshooting The Troubleshooter

Today is World Mental Health Day. My name is Tom Bridge, and I suffer from periodic depression. It’s not a formal diagnosis, it’s just something that I’ve known about myself going back to, I think, junior high school. I worked through all of this stuff on my own, mostly. In college, I saw a therapist, who gave me a lot of tools for managing my seasonal affection.

Over the last few winters, it’s gotten worse, and being around me in the winter has been difficult for my family and for my colleagues and for my friends. I finally started to seek professional attention for this issue about 18 months ago, and things have gotten a bit better. I’ve found a medication that my body seems to like (after finding one it really did not like) and I’m hopeful that this winter isn’t like the last few!

I gave a lightning talk at X World this year about mental health, because I want to destigmatize this disease that is stunningly common. Our society often treats mental illness like a moral failing, when it’s not that at all. Mental illness is like cancer, or like any other chronic condition that you have to fight through. Depression is real, I have it, and sometimes it makes life terrifically difficult in ways that are hard to describe and cope with.

So, all my sysadmin friends, and all my real life friends, it’s important to take time to troubleshoot your own troubleshooting self. Your brain may not respond to sudo the way that your computers do, and your life may not order itself around obvious log files and audit trails. Brains are more nuanced. But just like having trouble with anything in your IT world, you’ve got resources to call on.

And if you don’t, look me up, and we’ll help find you some together.

Long as I’ve got a job, you’ve got a job. Let’s get through this together.

Battery Adventure with my Series 4 Watch

This started four months ago.

I noticed my watch battery wasn’t lasting as long as it once did. Instead of going to bed at 11pm with 40-50% battery, I would run out at 10pm, then 9pm. I started to see the red glyph on my watch that says it’s disconnected from my phone more often, at peak 10-20 times a day.

I began to troubleshoot:

  1. Unpair my watch & restore from backup
  2. Wipe phone & restore from backup
  3. Unpair my watch & setup as new.

Nothing has brought my watch anywhere close to back to normal.

I’m not using a lot of apps on my watch. I have a couple complications I use from 3rd party apps (Carrot weather mostly!) and they update infrequently.

I am trying to determine what I can do to bring my battery life back to where it was for the first six months of life. I have no idea how to do this, and strangely, neither does Apple.

My Support Journey

Back in May when I first noticed this, I went to Apple’s Carnegie Library location in DC to get this fixed. Diagnostics were done, nothing was found, and I was offered their only service option: we can send it out for a week and get it serviced. No loaner, no replacement unit, just an all expenses paid week for the watch at the Austin service depot. Too much, I thought. I could deal with a 10% alert at bedtime.

Now that alert arrives at 8pm.

In June, I shattered the glass on my iPhone and had that repaired, while I waited in a comfortable atrium working on fast Wi-Fi. 90 minutes later, I had a good as new phone. Even if I’d bent the case or done more damage, I would’ve left the store with a working phone under my AppleCare+ arrangement with Apple. Service units exist for the iPhone, why not for the Watch, too?

Two weeks ago, I started to poke at this again, knowing full well the complications the beta cycle would add to my support experience. I asked informally at the store if they had advice while picking up another repair. They advised the app would be a good way to approach this.

I used the Apple Support app and got a responsive and polite but ultimately unsuccessful support engineer who tried to help but was unable to do so, despite diagnostics and support efforts. They didn’t want to leave me unhelped, though, and arranged a callback from someone who could help with devices that had the beta installed.

I spoke with the phone support agent and repeated the process, but got no further than being asked if I could send them the watch for a week. Instead, I arranged with the phone support agent to go to the store for the diagnostics instead, since at least there I’d have a person who could tell me what was up.

Today was the visit to the Carnegie Library store, back where I started. I sat with a Genius for 20 minutes who confirmed I was having a problem, but couldn’t explain it, or help resolve it any further. I was offered a mail-in repair on the spot, no loaner. I could call if I wanted to setup an advance replacement for $30, but the only way to do that was over the phone.

My Self-Diagnosis

There’s a clear, demonstrable and repeatable problem with my watch. It doesn’t stay connected to my phone all the time, even when it’s close by. This is the symptom, there’s an underlying problem with the bluetooth stack on one or both devices that is causing dropouts in the connection. I cannot repeat this with my AirPods, Car stereo, or bluetooth speaker, but I am getting odd latency on my Tile reports.

I don’t have the tools to diagnose this, as far as I can tell, and apparently neither does Apple. What they do have, as device manufacturers, and the makers of the software that runs them, is unlimited latitude to make things right by switching devices out. They have opted not to do that, citing a lack of damage.

I have a broken watch, or a broken phone. I’ll find out in ten days when my new phone arrives. I am frustrated that the diagnostics cannot explain what’s going on with the disconnects between the Watch and the iPhone.

What Apple Watch Means To Me

Apple Watch has been my constant companion since the Series 0. I owned the first Apple Watch, then the Series 2, and now a Series 4. While I find it awkward to wear a watch — I’m no horophile(1) — the taps on my wrist have allowed me a greater freedom to be in touch with my profession while being distinct from my phone. I can keep distractions at bay in meetings, but still be reachable for emergencies. I can keep focus on my clients without losing track of an emergency. I’ve written at length about how my Apple Watch saved me thousands of dollars at the emergency room and cardiologist this year. Apple is definitely right when they call it their most personal device. So, why won’t they treat it like that in their stores? Why not support trade-ins for repair on the spot? There is an incredibly healthy refurbished market for these units that Apple participates in, after all.

Apple talks a big game, and often delivers. I was delighted with the video that lead off the iPhone 11 announcement event, and I was doubly delighted that Apple understands how to market how helpful the Watch is as a product. Apple says: Give people wonderful tools and they will do wonderful things. They are right. I can do marvelous, magical, incredible things with the technical Apple sells.

When it stays healthy.

  1. You might think this would be chronophilia. A quick Google set me back on the right path. Whew. Glad I checked.

Manipulating the System Policy Database with Configuration Profiles, Part 2

You’re going to want to read Part 1 of this piece first. It covered managing the installation of packages that are not notarized. This section of the article covers the operational limits of doing the same thing for applications, using the operation:execute verb. There are several caveats to all of this that may make this procedure non-viable. But, there’s also a way to make it work, so here we are.

The Mechanism

Last time, we got a package installer whitelisted. That’s a simple operation: whitelist the certificate of the installer with the right Requirements payload, and the installer package will pass through quarantine checks unobstructed.

But what about a non-notarized application?

There are some software manufacturers who have challenging installers (which is to say: non-pkg, app-based installers) that require a different sort of whitelisting. In this example, we’re going to be reviewing the Cisco Webex Add-in. The installer is an application, intended to be run by a user. It’s a signed application, thankfully, and that gives us everything we need to build a whitelist to recover from the lack of a notarization process.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>PayloadDescription</key>
            <string>Configures Gatekeeper to accept developer certificate</string>
            <key>PayloadDisplayName</key>
            <string>System Policy Rule</string>
            <key>PayloadIdentifier</key>
            <string>com.apple.systempolicy.rule.5B8D7EE6-199A-48BC-B317-F223FB036552</string>
            <key>PayloadType</key>
            <string>com.apple.systempolicy.rule</string>
            <key>PayloadUUID</key>
            <string>5B8D7EE6-199A-48BC-B317-F223FB036552</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
            <key>Requirement</key>
            <string>anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = DE8Y96K9QP</string>
            <key>OperationType</key>
            <string>operation:execute</string>
            <key>Priority</key>
            <real>100.0</real>
            <key>Comment</key>
            <string>Test configuration 5 - OperationType: operation:execute</string>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>Gatekeeper Config</string>
    <key>PayloadIdentifier</key>
    <string>com.example.156ED537-CB5E-4AC9-80D6-376234F2DF60</string>
    <key>PayloadOrganization</key>
    <string>Example</string>
    <key>PayloadScope</key>
    <string>System</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>156ED537-CB5E-4AC9-80D6-376234F2DF60</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>

The Requirement payload in this this profile might look familiar. The identifier and certificate language looks a lot like the Privacy Preferences Policy Control payload that was introduced with macOS 10.14 Mojave. Much as you would with with one of those payloads, deriving the contents is as simple as interrogating the application with codesign:

codesign -dr - /path/to/Application.app

This will return what you need to embed:

identifier "com.cisco.webex.Cisco-WebEx-Add-On" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = DE8Y96K9QP

Once again, this is adding entries to the /var/db/SystemPolicy database, which Gatekeeper uses at inspection time to determine whether or not something passes. This is your pass past the security desk.

There Are Big Caveats

Okay, here’s where it gets less ideal.

There is only one road to a successful deployment.

This profile cannot be installed before macOS 10.15 Catalina is installed. Not because the profile can’t be interpreted by Mojave – it can – but because the installation process for macOS 10.15 might reset the SystemPolicy database.

And, because profiles are only ever interpreted at the time of install, you now have a profile that is installed on the machine, but no longer in the database that Gatekeeper uses.

One such failed case is here.

Once the profile’s installed, it’s installed, and it can’t or won’t be reinterpreted by the System. This is a huge flaw in the configuration profiles system overall, and we’ve seen it come back to bite us in the behind each of the last two revisions of the OS, with no revision in the system. The lack of defined-state management with configuration profiles, coupled with management-via-UDP profile delivery, means that we’re kinda stuck.

So, let’s talk about how this can go poorly.

Some MDMs have a method for reinstalling certain types of profiles after an operating system upgrade to make sure that profiles get reinterpreted, but none of those mechanisms are foolproof. There’s almost always a little lag between those mechanisms hitting and the upgrade completing.

What happens then?

Well, if the application is launched, and it won’t pass the Quarantine check, then the file gets marked as a failed application, and that file is, for all intents and purposes, blackballed.

Normally, that would mean the only recourse at that time is to uninstall and reinstall the application. If you have a good uninstall script, this could be a solution. There are some applications that aren’t such good citizens about containerization and slop their resources all over your filesystem.

But here’s the rub, even if you do catch all the resources, now you’re in an exorcism situation. There are some applications that just won’t be bypassed this way. I was able to get this working, but a colleague with another MDM has not been successful as of press time in making this work 100% of the time.

Well, That’s Bad. Now What?

Well, for starters, if you can, deploy these tools with other means. Remember that passing these checks are a lot like getting pass the security desk. Tools like the Jamf binary and root agents like Munki can build an Employees Entrance for key personnel. Installing with these tools bypasses the Quarantine process, which means notarization checks won’t apply.

If you’re expecting for users to directly download and manually install software that isn’t signed and notarized, you’re going to need to either have a robust help desk that can handle the volume, you’ll need to put pressure on your software vendor to do the responsible thing and fix their installer or fix their deployment mechanism(1).

Whitelisting is going to be a game of whackamole. If you have to play it, you’re going to need to plan for two things:

  1. Deliver your profiles with an MDM that can rapidly re-deliver the necessary profiles after a major version upgrade.
  2. Consider delivering your major version upgrade along-side a pre-install script that scrubs out packages that aren’t properly notarized.

Whatever you do, this is a use-case you will need to prepare your service desk for.

1.

Manipulating the System Policy Database with Configuration Profiles

First up, this post is a direct response to my previous posts on this summer’s talk about notarization. Notarization is a subject of much discussion, and there’s a lot happening out there. If you are looking for an exhaustive summary of notarization through many, many links, might I recommend this compilation post on Mr. Macintosh. If you are looking for the TL;DR, here it is: macOS 10.15 requires that software be notarized to pass the Gatekeeper checks. If it is not notarized, it will not pass these checks unless you can manipulate the System Policy Database to whitelist a Team’s certificate.

I didn’t do this alone, and I want to say a huge, huge thank you to everyone who helped me out as we tinkered through this somewhat opaque process.

In the Beginning, there was Gatekeeper and spctl

Before we go digging into how all of this works, you need to understand the importance of Gatekeeper in the process of reviewing the notarization of individual items. The spctl binary that is part of macOS’s command line interface, and has been for a very long time, are responsible for controlling what Gatekeeper looks at. These both write to a sqlite3 database stored at /var/db/SystemPolicy, and think of it a lot like a database of ID cards that the security guard at the desk will review. If your card is recognized, you pass through security without more than a passing hello at the barrier. If you card is not recognized, your ID is checked, your destination cleared, your name jotted down, and you’re granted a card if you belong.

This is how spctl whitelists applications for LaunchServices’ purposes.

This system can be directly manipulated via configuration profile, and those configuration profiles can be delivered by a capable MDM. Moreover, this has been the case since macOS 10.12. Hidden away in Apple’s documentation is the SystemPolicyRule payload type, which can allow you to embed whitelisted objects in an MDM Profile.

Anatomy of a SystemPolicyRule Payload

Properties of the SystemPolicyRule Payload
The key properties of the SystemPolicyRule payload

There are four interesting elements of the payload, and I’m going to take them in reverse order, from the bottom up:

Requirement is the policy requirement for the Code Signing information, and must match the syntax described by the CSRL. If you are familiar with writing PPPC policies in various tools, this will not be unfamiliar. The Requirement for our example profile is:

<key>Requirement</key>
<string>certificate leaf = $HASHCERT_DeveloperCertificate$</string>

In this case, the value of the string contains a reference to the hash of a specific certificate, here called DeveloperCertificate. We’ll come back to that certificate in a bit, but it corresponds to the signing certificate used by the developer to sign the package. Here, the $HASHCERT_ prefix is really a command to the MDM. It will derive a SHA hash value of the attached certificate and replace it in the string, so that the final value is then interpreted as:

certificate leaf = H"ca9284955c38aa337610c78e2bc1e532ef82ca2d"

Priority is the weighted value of the policy requirement for the payload. Rules can have varying priority level, and in our test profile, we’re using a Float value of 100.0.

Operation Type will tell you what kind of action you want govern. The operation type we care about for this example profile is an install. We want Gatekeeper to skip the notarization check during installation, so we’re going to add an OperationType of "operation:install"

LeafCertificate is the sticky point and that’s why I left it for last. You have to define the LeafCertificate of the signer. In our example profile that follows, the entirety of the public key of the signer’s certificate is embedded in the profile. This is going to require the most legwork, but there are shortcuts to get there. Flat packages in macOS have an XML Table of Contents, and you can use the xar command to extract it to a file where you can read it:

xar -t -f /path/to/your/package.pkg --dump-toc=toc.xml

You can now read the toc.xml file in BBEdit or any other fine text editors to review the contents of the XML table of contents, which will contain the signing certificate required for what follows:

XML File with X509 Certificate XML

This leads us to the sample profile.

Sample Profile

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>PayloadDescription</key>
            <string>Configures Gatekeeper to accept developer certificate</string>
            <key>PayloadDisplayName</key>
            <string>System Policy Rule</string>
            <key>PayloadIdentifier</key>
            <string>com.apple.systempolicy.rule.1496C06B-32CC-4725-9648-D310B45D78AB</string>
            <key>PayloadType</key>
            <string>com.apple.systempolicy.rule</string>
            <key>PayloadUUID</key>
            <string>1496C06B-32CC-4725-9648-D310B45D78AB</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
            <key>DeveloperCertificate</key>
            <data>MIIFcTCCBFmgAwIBAgIIBS/BS5wUWZUwDQYJKoZIhvcNAQELBQAweTEtMCsGA1UEAwwkRGV2ZWxvcGVyIElEIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQLDB1BcHBsZSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTETMBEGA1UECgwKQXBwbGUgSW5jLjELMAkGA1UEBhMCVVMwHhcNMTcwNjI5MTE1MDM1WhcNMjIwNjMwMTE1MDM1WjCBkTEaMBgGCgmSJomT8ixkAQEMCjQ2SlE2NTM1ODgxOjA4BgNVBAMMMURldmVsb3BlciBJRCBJbnN0YWxsZXI6IERleXNvbiwgSW5jLiAoNDZKUTY1MzU4OCkxEzARBgNVBAsMCjQ2SlE2NTM1ODgxFTATBgNVBAoMDERleXNvbiwgSW5jLjELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJl9BeLVXpspNA+ZQV8PcJ1zs0m3UZ9kW0PWGoHIL7qF1ngNnrd4+Lj62+DgasoVhZumWEL+GooYOZDQiswdAkF1qG7EPKXYmO6sfLNSKhVruha80LS/PpegqDJNsOw+o2ns3tNj7TTZ5BpSuyQlkiAO4mo92v+Qfhvqu1vmQHJi4nVF1wIxg+LFchkQJIElReb6+g1Y0xOIbxC3JM+wIMfDDKd1sz22zhP6i/t+EGDWoEnTyOV1dmCNgQloQcgetJmODJCEsj9h1sa4+FpQmbHdYxSQIHwnIfYOPQjoc4FPl7Da8dW5XSuTvLIzzfIn5rdmARwk6mq+rXSCM2AvfhAgMBAAGjggHiMIIB3jA+BggrBgEFBQcBAQQyMDAwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmFwcGxlLmNvbS9vY3NwLWRldmlkMDIwHQYDVR0OBBYEFP8Ryi3QibMrmNCHi5WAEtFQW6LWMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUVxftos/cfJihEOD8voctLPLjF1QwggEOBgNVHSAEggEFMIIBATCB/gYJKoZIhvdjZAUBMIHwMCgGCCsGAQUFBwIBFhxodHRwOi8vd3d3LmFwcGxlLmNvbS9hcHBsZWNhMIHDBggrBgEFBQcCAjCBtgyBs1JlbGlhbmNlIG9uIHRoaXMgY2VydGlmaWNhdGUgYnkgYW55IHBhcnR5IGFzc3VtZXMgYWNjZXB0YW5jZSBvZiB0aGUgdGhlbiBhcHBsaWNhYmxlIHN0YW5kYXJkIHRlcm1zIGFuZCBjb25kaXRpb25zIG9mIHVzZSwgY2VydGlmaWNhdGUgcG9saWN5IGFuZCBjZXJ0aWZpY2F0aW9uIHByYWN0aWNlIHN0YXRlbWVudHMuMA4GA1UdDwEB/wQEAwIHgDAXBgNVHSUBAf8EDTALBgkqhkiG92NkBA0wEwYKKoZIhvdjZAYBDgEB/wQCBQAwDQYJKoZIhvcNAQELBQADggEBABoQkblcJ67ogJCItUxk/TyScFgx5Ln3PJt1OwOTy4GqriL2T+bI+bxd206pcIjN0T7DMfvHLv0vsUQuENo5uPFHQGQeTo1WLd6Ys99MkjhqbLpVJLVch/AVRExXzFLkpnTWAk0l4ApVScSIRUswUeONhRY7Mc+7dFPoX2oOtsjwIR/98QnCykwBd01c4dhgg10BQ4bHAZjmHj8kXiI+yEJAT9YcuEw4PfSN6BgDViZVdsRLZD8z2UViyzYnwtbbKvCE2dx302SJ/ka+AVbnZdH8ZLAb09rgO+U6fdvDidiWtLf2Itehf21w0pnU9bRIe/IgqGFFJehhLDTP/O5+vmc=</data>
            <key>Requirement</key>
            <string>certificate leaf = $HASHCERT_DeveloperCertificate$</string>
            <key>OperationType</key>
            <string>operation:install</string>
            <key>Priority</key>
            <real>100.0</real>
            <key>Comment</key>
            <string>Test configuration 3 - OperationType: operation:install</string>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>Gatekeeper Config</string>
    <key>PayloadIdentifier</key>
    <string>com.example.934CF679-5ABA-444C-BCE1-22BA582182AD</string>
    <key>PayloadOrganization</key>
    <string>Example</string>
    <key>PayloadScope</key>
    <string>System</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadUUID</key>
    <string>934CF679-5ABA-444C-BCE1-22BA582182AD</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>

I’ve tested this profile with SimpleMDM and a current macOS 10.15 Catalina system with a signed-but-not-notarized package that would fail to install otherwise, and I was no longer warned while trying to install this otherwise-quarantined package.

The profile was delivered via MDM and now shows a System Policy Rule with detail that matches our payload content. Now, keep in mind, the /var/db/SystemPolicy database only cares about the Requirement item, which is just a hex hash of the certificate. You can, if you choose, just include the hex hash in the Requirement item, but if you choose to do that, you will have to know what that hash represents. If you want to be kind to your users, which you should, you should include the Certificate itself, which will be decoded in the display to show you which signer and which certificate authority the certificate resolves against.

After the certificate is installed, you can check your work with a review of the /var/db/SystemPolicy sqlite3 database’s authority table. It should show a new entry that matched the newly whitelisted certificate:

The red box shows the newly added rule that arrived via MDM Profile.

This appears to be how you can whitelist individual signing certificates for notarization checks and distribute that whitelist to clients with a Mobile Device Manager using a payload that’s been around since macOS 10.12.

How Do I Do This For My Environment?

Start with a package that is signed, but not notarized. Using the xar command, extract the X509 Certificate of the package. You will need this for your profile.

Second, use uuidgen to create new UUIDs for the profile. You’ll need at least two. Replace the two UUIDs in your copy of the profile. UUIDs are paired in the PayloadIdentifier and PayloadUUID fields.

Customize your PayloadIdentifier and Comment fields with your organization and a description of the policy.

Postscript: Using This To Run Non-Notarized Applications

While all of the above is intended for the operation:install key, operation: execute would allow you to run non-notarized Applications without Gatekeeper dialogs for those applications that are downloaded in their entirety without an installer package. You will need a separate profile if you want to whitelist both an installer and an application.