Everything I Know (Now) About The 13-inch MacBook Pro (non Touch Bar) Solid-State Drive Service Program

This Fall, Apple announced a service program for the non-Touch Bar MacBook Pros (also known as the MacBook Escape, for the hardware Esc key that they still have), specifically around the solid state drive that stores the operating system and user data. Think of a service program a lot like a car’s technical service bulletin program: designed to identify a potential failing of a given make and model of machine, and resolve that defect before it turns serious.

The Apple documentation for this repair is clear: the machine will have all of its data wiped during the firmware fix. Apple states: “Prior to service, it’s important to do a full back up of your data because your drive will be erased as part of the service process.” This means that you must backup the data before you take the machine to Apple. In our case, where Time Machine backups exist, we will perform a final update to the backup before the machine goes in. Where one does not exist, we will use Carbon Copy Cloner to backup to a disk image.

Today, I got to watch as a technician completed this process on a client computer, and I wanted to catalog what happened, as there’s not a step-by-step guide available for admins. In this case, I had three affected machines, and a Genius Bar appointment. Two of the machines failed the diagnostic portion of the firmware fix, and one was successful, which gave me a look at both cases of the SSD Firmware Update.

The Basics of the Solid-State Service Program

Before the process began, each of our machines was inspected and made sure to be in operating condition. After a brief check to determine OS level and functioning status, the machine was restarted, its PRAM zapped, and then it was run through standard onboard diagnostics (ie, hold Shift-D at boot). Our friendly Genius also reminded us for the third time that all data should be backed up at this point, or forever hold your peace. Now the machine was ready for the next step.

The firmware update process was handled in a NetBoot environment, as these machines are not T2 machines, and thus can be NetBooted. A specifically-created NBI was used by the Genius to boot the machine to a single-use tool. The appearance of this tool was very similar to booting into recovery, where a standard window appears and offered a single tool, the SSD Firmware Update.

The actual process of running the SSD Firmware Update is quick. I clocked it at well less than three minutes. If there’s a failure, it’s even faster.

In The Event of a Failure

If the mechanism doesn’t pass muster, a failure dialog is displayed, and it advises that the machine’s SSD needs to be replaced. This is not something Apple was ready to do on the spot, and said it would need to go to depot for repairs. There was a silver lining here: the existing volume was preserved with its information. This allowed us to take the machine back and do a direct transfer of data to an alternate loaner machine and schedule the depot repair at our convenience. In short, the machine’s ready to go back to use for the time being, and you’ve got a good backup.

In The Event of Success

If the mechanism does pass muster, you will get one last confirmation before everything is wiped from the drive. This is the fourth time I was asked if there was a backup of the volume. There was, we proceeded.

After a short period — three to five minutes by my recollection — the firmware was updated and we could proceed. It was then booted into Internet Recovery, and we used Disk Utility to create a new APFS volume on the otherwise-vacant SSD. After the firmware update, there was nothing on the disk, not even an empty volume. In order for the OS to be reinstalled, a volume had to be created first.

Once that was completed, the OS was reloaded, and twenty minutes later we had a working machine again.

Summary And Opinions

The process here was, thankfully, fairly painless. The machines that failed the upgrade weren’t erased and can go gingerly into the hands of their users until we can identify sufficient loaners. The machine that succeeded is now deemed cured and shouldn’t have this problem again. But that takes us to the problem’s mere existence. We had 40 MacBook Pros that fit the description of the warranty program, and something like 22 of them have to go to Apple in the coming months. I feel particularly awful about the company where 11 of their 18 machines have to go in.

The effect of this service program occasionally requiring a depot repair is also deeply unfortunate, because how many loaners is a 15-person company supposed to keep around? In this case, it should be possible for an org to arrange to just have these machines replaced in their entirety. Machines that have this defect can just stop working in their entirety, leaving a trusted member of your staff facing a nightmare scenario of recovery. Worse, depot repair is 5-7 days.

To bolster good will, I would hope that Apple would consider a new machine swap for these machines to get them transferred in a way that was more respectful of the time of Mac Admins and Apple customers in general. It is also quite frustrating to arrange with Apple to do these firmware fixes en masse. It takes an hour to prepare the machine, an hour to transport it to Apple and wait in the store, and then another hour to two hours to restore the operating system and user data to the machines. In addition, this service program requires Apple to participate. For shops that are using internal technicians who are Apple-certified, this tool is apparently not available via Global Service Exchange or GSX. That means you either have to find an AASP who will help you, but still require you to bring in the machines to their bench, or you have to make Genius Bar appointments for these machines.

All of them.

This isn’t a good experience for the companies that pay to be part of GSX, or the organizations that can’t participate on that scale. And these machines are fairly popular, as they represented a good balance between cost and functionality in a world where the Touch Bar is still a bit of an unknown quantity.

Yes, this is a special situation. It’s unlikely that any future machine will need this fix, due to the migration of the storage controller into the T2 silicon that Apple uses for its storage controllers. That, however, underscores the need for a better customer experience to fix this issue in the longterm.

We now have to go back to users and request their permission to disrupt them again in the future, and that’s not a fun experience. Just swap out the defective hardware for new, and populate the refurb store with the difference. It’s the least Apple could do.

Supraventricular Tachycardia: Or, A Trip to the ER with my Apple Watch

Overall, I’m a pretty healthy person. My blood pressure’s normal, my resting heart rate is in the low 70s, my cholesterol is normal, my blood sugar is normal, and I can go for a good long bike ride or walk without feeling winded. I’m heavy — my BMI is obese — but I’m in good health overall. (General reminder that BMI is BS.)

I bought my Apple Watch Series 4 when Apple announced it this summer, an upgrade from my Series 2. I was attracted by the fall detection (I’m an award-winning accident prone fellow) and also by the new ECG feature. I have a family history of atrial fibrillation, and I’m now 40, so some precautions seemed wise.

This afternoon, I was helping a client move offices, mostly just deconstructing a simple network rack and moving access points into new space. I was doing some physical work, but nothing anyone would mistake for exercise. But, then I felt it. My heart was pounding. I got dizzy. Tunnel vision. I had to sit down.

heart rate city

I took my heart rate on the watch and it was over 200. I spent five years as a competitive swimmer, and to my knowledge I never got above 195. Even riding up Box Hill on Zwift didn’t get me over 170 this winter. 200 is scary territory. I remembered the ECG functionality, and googled how it worked. I took a reading.

img_1630

I didn’t know how to read it, and I knew I was in a bit of trouble, so I had a coworker take me up to MedStar Washington Hospital Center, a mile or two away. Triage saw me rapidly, and I unlocked my phone to show the nurse. She was setting up a more complicated EKG, but because my heart rate had dropped back toward normal, it might not have any clear result they could read beyond just normal operation.

As soon as the tele-doc came on screen, the nurse rotated my phone and put it up to the camera to show the doctor the rapid rhythm from half an hour earlier.

“Oh, that’s an SVT,” he said immediately.

I didn’t see what it had to do with Ford’s Special Vehicle Team, but he clarified that he meant Supraventricular Tachycardia. They wanted to make sure labs were taken, and that nothing abnormal in my blood work showed a more troubling cause. But the diagnosis was there in an instant, thanks to my wrist watch.

Both the attending and her supervisor wanted a look before the day was done, and I was sent home with instructions to go see my doctor (don’t worry, I’m going on Thursday), but now I’ve got something to show my medical team, as well.

Sure, a lot of the time it feels like we live in a dystopian version of the future, and I’m still not sure where the flying cars are, but today I used my wrist computer — list price $399 — to take an ECG before arriving at the emergency room, where a doctor, appearing in my room via video conference, was able to read that medical diagnostic and make a snap judgment that I was probably going to be alright for now.

Apple remains a company that exists five to ten years into the future, building bridges back to the present. Touch ID and Face ID. Secure Enclave. Device Enrollment Program. Apple Watch Series 4 Health Tools. Perfect? No. Better than the rest? By miles and miles.

Thanks, Apple. My heart is in your hands, it seems.

img_1591

2018: Arbitrary Boundary Condition Met

Sunset at Asilomar Beach, December 29th, 2019

The problem with linear time — well, one of them — is that you don’t always know when your personally meaningful boundary conditions have been met. Life is uneven, some chapters are long and interesting, some short but sweet, some arduous and never-ending. 2018 fell into a lot of those categories. So we’ve met our arbitrary boundary condition prescribed by the journey of the Earth around the Sun. Let’s look at what happened?

Crash Migration

2018 began with a crash migration for one of our clients. We had 26 days to handle their office move, and brought them into new digs on time and with complete operating functionality, despite the short timeline. I’m thankful to, of all people, Comcast Enterprise for bringing their might to bear and they brought their A game and got us a gigabit circuit in almost no time at all. Crash Migrations always feel like a bit of a trial-by-fire, and this one was no exception.

41 Episodes

The Mac Admins Podcast had an incredible year, and I couldn’t be prouder of the team. We produced 41 episodes of the podcast, including The One With Apple, live episodes at JNUC and MacDevOps YVR. We talked with Apple Luminary Sal Soghoian, Fraser Speirs about the State of iOS in Education, Thomas Reed about Graykey and iOS, and Tim Perfitt about Secure Boot and the best way to kill a chicken. 2018 brought more than 175,000 downloads of the podcast!

Here’s to more and more episodes in 2019! I’m hopeful our conversation with Apple gets a sequel. If you want to see us do a live show, come on out to MacADUK in March 2019!

Cloud Living

With the, um, active retirement of macOS Server, I sunset the code for Munki in a Box, but not content to just abandon the idea, I also released Munki in a Cloud which works with AWS. If I were a better coder, I’d be combining it with Graham Gilbert’s excellent Munki Terraforming project. I guess I just found my first 2019 goal.

Frontal Boundaries

We also moved our primary software distribution platform from an on-premise Munki server into AWS’ CloudFront. We’ve moved almost a third of our clients to it already, and we’ve got planned migrations for a bunch of the rest. Serving client updates via CloudFront was a really great experience for us from a budget perspective. We centrally manage manifests and applications from a workstation on our network, do QA, then push to production. We’ve got a secure distribution system that I’m pretty proud of. And it cost us much less per client than even our wildest dreams.

What’s The Future Bring?

2019 I’m getting on the stick about two specific things: Python and the SimpleMDM API. I’ve said it before about Python, but I’ve actually accomplished some small tasks this way, so I’m excited to tweak a few more things into Amazon Lambda, Python and the SimpleMDM API as part of our goal to make better touch-less workflows in 2019. I’ve got some ideas for an open library of Python scripts for using the SimpleMDM API, but I need to get some tasks genericized and working, first.

I’m looking forward to 2019, to whatever macOS version ships in beta form come the summer, the demise of kexts and 32-bit applications, and more MDM options.

If the last three years have taught me anything — as a father, as a business person, and as a Mac Admin — it’s that being ready for anything means approaching everything like it’s an angry emo porcupine with lethal quills: carefully, thoroughly, and with as much empathy for the problem as you can muster. Everything’s always changing. We’re always building new things, always undoing old mistakes and making new ones. We’re going to keep that up. The constant is change.

Holding strong opinions loosely is one way to avoid the ossification to orthodoxy that can keep you from seeing where the Future is. Getting stuck doing things one way because it’s how you’ve always done it is a great way to miss what’s coming. Chasing the Future means being willing to abandon work that you’ve done, and that can hurt, because a problem solved elegantly comes with it a certain satisfaction in overcoming entropy through clever application of technical knowledge. But staying with an old solution when a new one avoids the problem entirely is unwise in an ever-changing situation.

Go forth, my friends, and solve new problems in 2019. Solve them together, toward making computing a more seamless and enjoyable task for all the participants. The simultaneous promotion of all interests — usability, security, and repetition — is possible.

Sardines at The Kelp Forest, Monterey Bay Aquarium, December 30th, 2018

Exporting S/MIME Certificate Identities For iOS Use

Cryptographically-signed email is a complicated subject. There’s keys and certificates, there’s signing authorities, all of the wonderful PKI structures that allow us to communicate securely. Securely signed email is an easy way to indicate to other parties that you’re taking some precautions with your email that authenticates the sender in an end-to-end way. A typical free commercial certificate, say, from Comodo, will affirm that the original certificate creator can receive email at a given address. This allows you to sign outgoing emails, which wraps the message in a secure envelope that confirms that you are who you represented yourself to the certifying authority to be.

Generally, these certificates are commissioned on a desktop computer, like any Mac. You end up with a certificate stored in your Login keychain that Mail.app or Outlook use to sign your outgoing messages, or decrypt messages signed with your public certificate. If you open Keychain Access, select your Login Keychain, and then set it to filter by certificates, you will see your email signature.

If you want to sign email with your iPhone or iPad (and you do), you’ll need to move this certificate to your device in a way that your device will be able to work with it.

keychain-access-cert

Normally, you might just drag your certificate out to the desktop and embed it in an MDM Profile, or something similar. Your certificate also contains a private key, and that is a critical element. The drag and drop method won’t work this time. What you need to do is export the certificate in .p12 format (also known as PCKS #12). To do this, right-click on the Certificate and select Export.

keychain-access-export

Pick a location for the file. I recommend the Desktop, since we’re going to be emailing this file.

pick-a-location

You’ll be prompted to pick a password for this .p12 file. You’re going to need this when the certificate gets to the iOS device. This is what lets you securely move the certificate and private key together in a safe package.

enter-a-password

Pick a password that you’ll remember and that isn’t just password. If your email is compromised, an attacker could take this .p12, and with suitable equipment, some good luck, and a super computing farm, sign email on your behalf, unless you revoke the certificate at the Certificate Authority. Note the password down carefully, you’ll need it in a moment.

pick-a-good-password

You may be asked to allow access to the private key by the system. You’ll need to allow access in order to export. I think this step might be unnecessary in most cases. If it doesn’t present, don’t worry about it.

allow-access-to-key

Now, attach the .p12 file to an email that your device can receive. Now, what follows is instructions for use with the built-in Mail app on the iPhone. There may be ways to work with S/MIME in other mail clients, but this post will not cover them.

attached-email

Once you have the .p12 in Mail on your iOS device, tap on the attachment to open it. The Settings app on your iOS device will now open and you’ll go through the standard profile installation process. If you have an iOS Device that is paired with an Apple Watch, you will get prompted to pick whether you want to install the certificate on your Watch or your iPhone. You want it on your iPhone.

location-picker

Keep in mind – you aren’t working with a signed standard identity certificate, but that doesn’t mean the payload won’t certify up the trust chain.

confirm-cert-details

install-warning

Accept these dialogs by tapping Install, and continue. You will now have to enter the password for the .p12 container that you wrapped around your certificate and private key. Enter it and tap Next when you’re ready.

certificate-password

Lastly, you’ll need to finalize the profile and confirm the install.

confirm-install

Your certificate is now resident on the iOS device, and it’s time to go turn on S/MIME in your Mail Account. Go to Mail Settings, and select your account, and then head to the Advanced Settings. Turn on S/MIME, and turn on the signing settings.

smime-account-settings

You can confirm that your identity is selected, or select which identity your device is using to sign messages, by tapping on Sign.

signing-settings

You can tap the i at the end of the line to review the signing identities that are configured for your account. If you start using S/MIME certificates, be ready to keep old expired certificates around in the event that you are not just signing messages, but encrypting them. Messages encrypted with your public certificate by other people will only be decrypted by that old, expired certificate and its private key.

If you want to review your S/MIME certificates, you can do so in the Profiles section of Settings. Tap on Settings, then General, then Profiles.

profiles-list

You can get detail on an individual certificate and see more information surrounding the certificate, which should be on your calendar.

cert-identity

Link

MacAdmins Podcast Episode 6: Dreyer, Rhymes with Slayer

We got the chance recently to sit down with Arek Dreyer, author of so, so many books, in time for the release of his new 3rd Edition of Managing Apple Devices. We talked about WWDC, writing books like Managing Apple Devices, as well as nearly getting arrested in a Chicago Server Room, and the first apps we bought. Co-hosts Charles Edge and Emily Kausalik were awesome, as was our mixing engineer Aaron Lippincott, who made us sound amazing.

Link

Techno Bits vol. 60: Packaging Isn’t (Quite) Dead

This week in Techno Bits vol. 60: Packaging Isn’t (Quite) Dead yet, some feedback on last week’s issue that sparked a lot of commentary. There are updates to the idea of a future without packages and why we might not be there just yet that you should catch up on. I’ve also got a download of my favorite talks from MacADUK, as well as some commentary on the nature of getting ahead vs. doing good.

Techno Bits vol. 59: What if Packages Went Away Tomorrow?

This week’s newsletter contains highlights from the MacADUK conference, put on by Amsys in London, England this week. It was an incredible show where I got to talk with a lot of really great admins, kick around good ideas, ponder appropriate security changes necessary for our production environments, and plan for a better tomorrow. One particular discussion at the pub on Tuesday night lead to the longest section of this week’s newsletter: what if the end of the .pkg as we know it is upon us? What if the tool we use for deployment every day was suddenly curtailed by a change at Apple?

Read up and see why it might not be as awful as you think.