A Hero’s Return

One of Technolutionary’s first purchases in 2006 was this Mac mini, which today returned to our offices after 13 years of duty in co-location at Solid Space in North Carolina. It’s been a file server, a mail server, and more, and we absolutely, positively got our money’s worth on this beauty.

Thanks for the service, Mac mini, your retirement awaits. Thank you Apple, for building technology we can depend on.

A sunset over Puget Sound, from the offices of Dropbox Seattle

Mac Admins Podcast 112: Live From Seattle

This past week, I traveled to Seattle, Washington to join the Apple Admins of Seattle and the Greater Northwest at their monthly meetup, and to record an episode of the Mac Admins Podcast with my friends Chris Dawe, Jonathan Spiva and Vi Lynk, as well as my new friend Ashley Smith. The topic was fairly simple: let’s talk about career paths and career trajectories and all the crazy things a life in IT can bring.

We talk a lot about technology in our jobs, but we don’t talk a lot about our jobs in technology, and it was great to sit down and chat about how we’ve gotten to where we are, where we’re headed, and what we’re learning about working with people, machines, and applications. In particular, I found Vi’s conversation about relationships mattering in IT to be illuminating. How we fit our departments and businesses into each other is so important. It made me go watch her talk from Penn State again, to remind myself of who my internal folks are, who my external folks are, and so I can close the loop with so many of those people again in the near term.

Ashley Smith reminded me of the importance of being willing to do the legwork on a topic when you don’t know the answer, and that the best response to a question you don’t know the answer to is “I don’t know, but I’ll find out.” We grind so many people through the grist mill of Tier 1 support, but we don’t spend time letting them learn, in favor of metrics that likely don’t have a good backing in objective reality. As part of managing service desks, we need to make sure that we’re not blindly adhering to metrics over the development of our people.

This week’s episode is a break from the minutiae of the job, in favor of some of the bigger picture. It’s worth your time to listen in the browser, or out on Overcast.

Seattle was a marvelous city to visit, even in the midst of winter, and I had so many incredible meals (Seven Stars Pepper! Harbor City! Jack’s! Arctic Club! Beer Star!) and conversations that it will remain in my heart. My thanks to organizers of the event, and to everyone that I got to see while I was out there for three days. Getting out to meet people in Mac IT all over the country is the best part of my job, and I can’t wait to do it again in March.

Everything I Know (Now) About The 13-inch MacBook Pro (non Touch Bar) Solid-State Drive Service Program

This Fall, Apple announced a service program for the non-Touch Bar MacBook Pros (also known as the MacBook Escape, for the hardware Esc key that they still have), specifically around the solid state drive that stores the operating system and user data. Think of a service program a lot like a car’s technical service bulletin program: designed to identify a potential failing of a given make and model of machine, and resolve that defect before it turns serious.

The Apple documentation for this repair is clear: the machine will have all of its data wiped during the firmware fix. Apple states: “Prior to service, it’s important to do a full back up of your data because your drive will be erased as part of the service process.” This means that you must backup the data before you take the machine to Apple. In our case, where Time Machine backups exist, we will perform a final update to the backup before the machine goes in. Where one does not exist, we will use Carbon Copy Cloner to backup to a disk image.

Today, I got to watch as a technician completed this process on a client computer, and I wanted to catalog what happened, as there’s not a step-by-step guide available for admins. In this case, I had three affected machines, and a Genius Bar appointment. Two of the machines failed the diagnostic portion of the firmware fix, and one was successful, which gave me a look at both cases of the SSD Firmware Update.

The Basics of the Solid-State Service Program

Before the process began, each of our machines was inspected and made sure to be in operating condition. After a brief check to determine OS level and functioning status, the machine was restarted, its PRAM zapped, and then it was run through standard onboard diagnostics (ie, hold Shift-D at boot). Our friendly Genius also reminded us for the third time that all data should be backed up at this point, or forever hold your peace. Now the machine was ready for the next step.

The firmware update process was handled in a NetBoot environment, as these machines are not T2 machines, and thus can be NetBooted. A specifically-created NBI was used by the Genius to boot the machine to a single-use tool. The appearance of this tool was very similar to booting into recovery, where a standard window appears and offered a single tool, the SSD Firmware Update.

The actual process of running the SSD Firmware Update is quick. I clocked it at well less than three minutes. If there’s a failure, it’s even faster.

In The Event of a Failure

If the mechanism doesn’t pass muster, a failure dialog is displayed, and it advises that the machine’s SSD needs to be replaced. This is not something Apple was ready to do on the spot, and said it would need to go to depot for repairs. There was a silver lining here: the existing volume was preserved with its information. This allowed us to take the machine back and do a direct transfer of data to an alternate loaner machine and schedule the depot repair at our convenience. In short, the machine’s ready to go back to use for the time being, and you’ve got a good backup.

In The Event of Success

If the mechanism does pass muster, you will get one last confirmation before everything is wiped from the drive. This is the fourth time I was asked if there was a backup of the volume. There was, we proceeded.

After a short period — three to five minutes by my recollection — the firmware was updated and we could proceed. It was then booted into Internet Recovery, and we used Disk Utility to create a new APFS volume on the otherwise-vacant SSD. After the firmware update, there was nothing on the disk, not even an empty volume. In order for the OS to be reinstalled, a volume had to be created first.

Once that was completed, the OS was reloaded, and twenty minutes later we had a working machine again.

Summary And Opinions

The process here was, thankfully, fairly painless. The machines that failed the upgrade weren’t erased and can go gingerly into the hands of their users until we can identify sufficient loaners. The machine that succeeded is now deemed cured and shouldn’t have this problem again. But that takes us to the problem’s mere existence. We had 40 MacBook Pros that fit the description of the warranty program, and something like 22 of them have to go to Apple in the coming months. I feel particularly awful about the company where 11 of their 18 machines have to go in.

The effect of this service program occasionally requiring a depot repair is also deeply unfortunate, because how many loaners is a 15-person company supposed to keep around? In this case, it should be possible for an org to arrange to just have these machines replaced in their entirety. Machines that have this defect can just stop working in their entirety, leaving a trusted member of your staff facing a nightmare scenario of recovery. Worse, depot repair is 5-7 days.

To bolster good will, I would hope that Apple would consider a new machine swap for these machines to get them transferred in a way that was more respectful of the time of Mac Admins and Apple customers in general. It is also quite frustrating to arrange with Apple to do these firmware fixes en masse. It takes an hour to prepare the machine, an hour to transport it to Apple and wait in the store, and then another hour to two hours to restore the operating system and user data to the machines. In addition, this service program requires Apple to participate. For shops that are using internal technicians who are Apple-certified, this tool is apparently not available via Global Service Exchange or GSX. That means you either have to find an AASP who will help you, but still require you to bring in the machines to their bench, or you have to make Genius Bar appointments for these machines.

All of them.

This isn’t a good experience for the companies that pay to be part of GSX, or the organizations that can’t participate on that scale. And these machines are fairly popular, as they represented a good balance between cost and functionality in a world where the Touch Bar is still a bit of an unknown quantity.

Yes, this is a special situation. It’s unlikely that any future machine will need this fix, due to the migration of the storage controller into the T2 silicon that Apple uses for its storage controllers. That, however, underscores the need for a better customer experience to fix this issue in the longterm.

We now have to go back to users and request their permission to disrupt them again in the future, and that’s not a fun experience. Just swap out the defective hardware for new, and populate the refurb store with the difference. It’s the least Apple could do.

Point to Point Wireless with LiteBeam

From time to time, we get asked a question like “Hey, I need to get signal to a building that’s not part of our regular building. Can you do that?” and the answer is usually, “Sure, we could bury a fiber, or fly a cable,” mostly because we haven’t felt the loss in speed and signal makes sense. We recently had a situation that called out for a wireless point to point link, though, and that got us thinking.

Our client took a new space on an upper floor of a warehouse building, across the loading dock from their storage space. They have a staff of two or three on the far side of the gap, and they wanted to extend their current connection to this space without paying for a second internet connection, relying on cellular hotspots, and the building is such that a flown cable or a trenched fiber was impractical.

They’re a Ubiquiti shop, and so we looked at our options. There are the NanoStation and NanoBeam options, but our reseller house of choice was badly backordered, so we ended up with a LiteBeam AC Gen2 setup. I think, given what we found regarding our mounting situation, it’s fortunate we ended up with the antenna geometry and power pairing that was present in the LiteBeam.

The LiteBeam gear is powered by 24V passive injectors, or, if your switch is capable, it can take 24V passive POE directly off a switch. Most places aren’t going to have switches capable of 24V power, and it’s a real bummer that’s what this requires. I’m still scratching my head why this won’t just take standard 802.3af.

When we toured the space, the client suggested that we could mount the warehouse dish on the exterior of the building and “easily” plumb the cable into their space. On the office side, we could position the dish in the north-facing window. There was no roof access, and definitely no exterior penetrations permitted in their space. So through the looking glass we went.

The LiteBeam antennas are parabolic reflector dishes approximately 14″ wide by 10″ tall by 10″ deep. They come with adjustable mounting equipment, including a super helpful hoseclamp mount.

Specifications of the LiteBeam Gen2

Assembly is fairly rapid. The dish ships in three panels which slot together nicely, then screwed together, the feed receiver attaches via tension tab mounts, and the antenna feed snaps into place. From there, you can attach the elevation and azimuth mounts, and which then attach to the pole mount kit.

But, what if we don’t have a pole to mount to?

It was off to the hardware store to talk to my friend neighborhood Annie’s Ace Hardware folks about ways to handle this. What we settled on was a set of galvanized flanges and pipe joints, which easily allowed us to mount an elbowed pipe to the vertical wall of the warehouse, and an offset pipe mounted to a piece of 2×4 with lag bolts for screwing into the window frame. This gave us superb stability at a cost of less than $50.

Two LiteBeam dishes with attached mounting kits, resting on a dining room table. A LiteBeam dish hanging from a pipe mount beneath a 2x4

Having mounted the office side, we went to mount the warehouse side. After several broken concrete anchors, and a trip for a bigger drill and better anchors, and a lot of creative cabling, we were able to get the second dish properly mounted. Time had come to setup and test.

Now, we’d laid the groundwork ahead of time, and everything had been firmware updated and tested and prepared from inside the warm office, before heading out into the cold. We knew these things should easily sync up, we just had to get there, and get the dishes aligned.

LiteBeam Wireless Link mounted in its final position

If we were smart, I’d have picked up a green laser pointer to help with the alignment of the two dishes, but Mark I Eyeball still does the job pretty well. On our first attempt we got the wireless link close enough to register without having to futz with the positioning, we’d gotten close enough for a functioning link:

An image from the setup up showing functional links

The patient lives! We were getting about 20Mbps through the link, on a connection that is often twenty times that fast, so we knew we had work to do. We were able to get the signal up to 40dB of signal, and that was about as good as we could get. With the LiteBeam good for kilometers, we knew we should be doing better at a distance of under 200 feet.

To test our theory, we unmounted the dish and stood outside with it, and sure enough, signal strength spiked back up to the top of the range. The window’s coating was messing with our signal. There was, unfortunately, no fix for that, as glaziers weren’t in the budget for the move, but we did get service on the far side of the link up to 50Mbps on our speed test, more than adequate for a staff of two primarily doing light streaming and office work.

Lessons Learned:

Building penetrations are never as easy as they say they are.

Window glass can be a tougher barrier to signal than you’d think.

A laser sight of some sort is required for point to point wireless.

Sometimes $50 at the hardware store is going to be plenty for creative mounting solutions.

The LiteBeam Gear is pretty awesome, but you need 24V Passive POE to power it, which is not awesome.

Supraventricular Tachycardia: Or, A Trip to the ER with my Apple Watch

Overall, I’m a pretty healthy person. My blood pressure’s normal, my resting heart rate is in the low 70s, my cholesterol is normal, my blood sugar is normal, and I can go for a good long bike ride or walk without feeling winded. I’m heavy — my BMI is obese — but I’m in good health overall. (General reminder that BMI is BS.)

I bought my Apple Watch Series 4 when Apple announced it this summer, an upgrade from my Series 2. I was attracted by the fall detection (I’m an award-winning accident prone fellow) and also by the new ECG feature. I have a family history of atrial fibrillation, and I’m now 40, so some precautions seemed wise.

This afternoon, I was helping a client move offices, mostly just deconstructing a simple network rack and moving access points into new space. I was doing some physical work, but nothing anyone would mistake for exercise. But, then I felt it. My heart was pounding. I got dizzy. Tunnel vision. I had to sit down.

heart rate city

I took my heart rate on the watch and it was over 200. I spent five years as a competitive swimmer, and to my knowledge I never got above 195. Even riding up Box Hill on Zwift didn’t get me over 170 this winter. 200 is scary territory. I remembered the ECG functionality, and googled how it worked. I took a reading.

img_1630

I didn’t know how to read it, and I knew I was in a bit of trouble, so I had a coworker take me up to MedStar Washington Hospital Center, a mile or two away. Triage saw me rapidly, and I unlocked my phone to show the nurse. She was setting up a more complicated EKG, but because my heart rate had dropped back toward normal, it might not have any clear result they could read beyond just normal operation.

As soon as the tele-doc came on screen, the nurse rotated my phone and put it up to the camera to show the doctor the rapid rhythm from half an hour earlier.

“Oh, that’s an SVT,” he said immediately.

I didn’t see what it had to do with Ford’s Special Vehicle Team, but he clarified that he meant Supraventricular Tachycardia. They wanted to make sure labs were taken, and that nothing abnormal in my blood work showed a more troubling cause. But the diagnosis was there in an instant, thanks to my wrist watch.

Both the attending and her supervisor wanted a look before the day was done, and I was sent home with instructions to go see my doctor (don’t worry, I’m going on Thursday), but now I’ve got something to show my medical team, as well.

Sure, a lot of the time it feels like we live in a dystopian version of the future, and I’m still not sure where the flying cars are, but today I used my wrist computer — list price $399 — to take an ECG before arriving at the emergency room, where a doctor, appearing in my room via video conference, was able to read that medical diagnostic and make a snap judgment that I was probably going to be alright for now.

Apple remains a company that exists five to ten years into the future, building bridges back to the present. Touch ID and Face ID. Secure Enclave. Device Enrollment Program. Apple Watch Series 4 Health Tools. Perfect? No. Better than the rest? By miles and miles.

Thanks, Apple. My heart is in your hands, it seems.

img_1591

2018: Arbitrary Boundary Condition Met

Sunset at Asilomar Beach, December 29th, 2019

The problem with linear time — well, one of them — is that you don’t always know when your personally meaningful boundary conditions have been met. Life is uneven, some chapters are long and interesting, some short but sweet, some arduous and never-ending. 2018 fell into a lot of those categories. So we’ve met our arbitrary boundary condition prescribed by the journey of the Earth around the Sun. Let’s look at what happened?

Crash Migration

2018 began with a crash migration for one of our clients. We had 26 days to handle their office move, and brought them into new digs on time and with complete operating functionality, despite the short timeline. I’m thankful to, of all people, Comcast Enterprise for bringing their might to bear and they brought their A game and got us a gigabit circuit in almost no time at all. Crash Migrations always feel like a bit of a trial-by-fire, and this one was no exception.

41 Episodes

The Mac Admins Podcast had an incredible year, and I couldn’t be prouder of the team. We produced 41 episodes of the podcast, including The One With Apple, live episodes at JNUC and MacDevOps YVR. We talked with Apple Luminary Sal Soghoian, Fraser Speirs about the State of iOS in Education, Thomas Reed about Graykey and iOS, and Tim Perfitt about Secure Boot and the best way to kill a chicken. 2018 brought more than 175,000 downloads of the podcast!

Here’s to more and more episodes in 2019! I’m hopeful our conversation with Apple gets a sequel. If you want to see us do a live show, come on out to MacADUK in March 2019!

Cloud Living

With the, um, active retirement of macOS Server, I sunset the code for Munki in a Box, but not content to just abandon the idea, I also released Munki in a Cloud which works with AWS. If I were a better coder, I’d be combining it with Graham Gilbert’s excellent Munki Terraforming project. I guess I just found my first 2019 goal.

Frontal Boundaries

We also moved our primary software distribution platform from an on-premise Munki server into AWS’ CloudFront. We’ve moved almost a third of our clients to it already, and we’ve got planned migrations for a bunch of the rest. Serving client updates via CloudFront was a really great experience for us from a budget perspective. We centrally manage manifests and applications from a workstation on our network, do QA, then push to production. We’ve got a secure distribution system that I’m pretty proud of. And it cost us much less per client than even our wildest dreams.

What’s The Future Bring?

2019 I’m getting on the stick about two specific things: Python and the SimpleMDM API. I’ve said it before about Python, but I’ve actually accomplished some small tasks this way, so I’m excited to tweak a few more things into Amazon Lambda, Python and the SimpleMDM API as part of our goal to make better touch-less workflows in 2019. I’ve got some ideas for an open library of Python scripts for using the SimpleMDM API, but I need to get some tasks genericized and working, first.

I’m looking forward to 2019, to whatever macOS version ships in beta form come the summer, the demise of kexts and 32-bit applications, and more MDM options.

If the last three years have taught me anything — as a father, as a business person, and as a Mac Admin — it’s that being ready for anything means approaching everything like it’s an angry emo porcupine with lethal quills: carefully, thoroughly, and with as much empathy for the problem as you can muster. Everything’s always changing. We’re always building new things, always undoing old mistakes and making new ones. We’re going to keep that up. The constant is change.

Holding strong opinions loosely is one way to avoid the ossification to orthodoxy that can keep you from seeing where the Future is. Getting stuck doing things one way because it’s how you’ve always done it is a great way to miss what’s coming. Chasing the Future means being willing to abandon work that you’ve done, and that can hurt, because a problem solved elegantly comes with it a certain satisfaction in overcoming entropy through clever application of technical knowledge. But staying with an old solution when a new one avoids the problem entirely is unwise in an ever-changing situation.

Go forth, my friends, and solve new problems in 2019. Solve them together, toward making computing a more seamless and enjoyable task for all the participants. The simultaneous promotion of all interests — usability, security, and repetition — is possible.

Sardines at The Kelp Forest, Monterey Bay Aquarium, December 30th, 2018
Image

Using Prey with SimpleMDM to Recover a Stolen Laptop

Monday morning, I got the call that no one wants to get: “The lock was jimmied. They got some of our computers.”

Immediately, we sprang into action. We’re big fans of SimpleMDM and Watchman Monitoring, and both of those tools came in handy. The first thing we did was check the logs from Watchman Monitoring’s client agent on the machine to see if it had checked in over the weekend.

One of the machines had checked in on Sunday! We set both to alert us if they checked in again, and logged into SimpleMDM to see if the device was checking in there, as well. We could see the one device, which gave us a couple different options: We could lock or wipe the machine and hope that it wouldn’t just end up in a landfill, or we could try to get the machine back by giving some data to the police.

I know from experience that just giving them an IP Address isn’t likely to get a good result, so we started to think what else we could we do to get the machine back? What if we could give them a location, and more information?

Screen Shot 2018 10 17 at 12 28 27 PM

Enter the Prey Project. The Prey tool works as a behind-the-scenes agent on your behalf. When it’s in regular mode, it’s not doing much. But, when you turn on Missing Mode, things get a lot more interesting. Your Mac will now check-in with nearby Wi-Fi networks, perform a full location scan and give the police something to work with. It will also take pictures with the FaceTime Camera on the Computer, and capture screenshots, giving you more material to work with:

Prey Screenshot with Wireless Networks NearbyPrey Screenshot with Map Detail

This post isn’t here to get you to buy Prey, but it’s to tell you how we got Prey installed when we didn’t have the machine in our full control.

By default, Prey requires an API key to register new machines, and their method is just “Hey! Install that at the Command Line by SSH’ing into the machine!” Which, okay, fine, that might work if you can get that far, but how’s about we do something a little bit different?

What we opted to do was to repackage the Prey installer, so that the package installer they built is stored in a common directory, (in our case, /Users/Shared) and then a postinstall script tied to the package handles the install with our API Key:

#!/bin/bash

API_KEY=0xdeadbeef /usr/sbin/installer -pkg /Users/Shared/prey-mac-1.8.1-x64.pkg -target /

To build this package, we used Packages from WhiteBox. I created a new project, gave it a name (Black Widow), Included our Prey installer package to a known directory, and then add a Post Install script to invoke it using our API Key.

Screen Shot 2018 10 17 at 3 09 08 PM

Packages Post Install Script Screen

This gave me a functioning package that installed Prey and keyed it to our instance, which was great! But, how do I get it onto the stolen machine?

Enter SimpleMDM. You can use SimpleMDM to install a package onto a device, but only if you have a properly signed distribution package. The Black Widow package I made in Packages was unsigned, so now I just had to properly sign it using the prodsign command:

 

Screen Shot 2018 10 17 at 3 10 12 PM

This gave us a properly signed package with a valid signature:

Screen Shot 2018 10 17 at 3 17 52 PM

After uploading the package to our SimpleMDM instance, we scoped it to the machine, and waited for its next check-in:

Screen Shot 2018 10 17 at 3 19 16 PM

From there, it was a waiting game until the person who had the laptop now was back in range of the internet. Sure enough, they came back online today:

Screen Shot 2018 10 17 at 3 20 58 PM

The machine’s location and positioning information, as well as some additional detail, gave the police something to use to be a little more active on the case. We’re now waiting to hear if they’ll be able to repatriate the laptop to its owner.