Work Stories

Build repeatable provisioning and operational workflows that reduce drift and improve deployment confidence.

Define SLIs/SLOs, build dashboards that drive action, and create alerting that catches real risk without overwhelming teams.

Operate platform services used by engineering orgs, with emphasis on uptime, cost control, and production readiness.

Customer Success Engineer - L2 Team Lead

05/2021 – 06/2022

Nike had a problem. Nike had literally over 1,000 technical teams, and they generally needed similar things. Every team needed a Kubernetes cluster (for example), so every team spun up a Kubernetes cluster, and we ended up with 1,000 Kubernetes clusters all configured differently, leading to uncertainty in areas like cost and day-to-day operations.

Nike's answer was to create a management interface so when a team needs a Kubernetes cluster, they click through a form and a cluster gets spun up for them. This cluster starts in compliance with all the corporate policies, and it's maintained centrally so it remains in compliance. We called this a "platform". My team supported a set of these.

Support is part of any mature service offering, and that's where I came in. They created a new team that was half customer support and half SRE. I created a handbook of how to be an engineer on our team: training and documentation aimed specifically at our engineers. I documented policies about how to interact with customers and dev teams, but first I had to create those policies. I met with dev teams and worked out SLIs and SLOs and SLAs, and made dashboards to visualize them.

I interviewed over 100 candidates, and we hired about 40 engineers in 6 time zones. The other major focus was training and mentoring: not just the handbook, but how to work through problems we hadn't seen before.

This support work was later moved to a vendor. I handed it off and moved on to my second role at Nike.

In this role I used ▼

Lead Software Engineer - Build and Agile Tools

06/2022 – 04/2026

After a short stint automating housekeeping in GitHub Actions, I landed on the team that runs Jira, Confluence, and related Atlassian tools. My main focus was the Jenkins pipelines that roll those apps out.

Over the next year I added features to these pipelines: blue/green deployments, proper tagging, unit tests and code coverage, and of course optimizing for reduced cost. I brought the pipelines in line with corporate standards and cut a lot of wasted spend.

Once I implemented code scanning, I found AI very useful in working through the findings.

Our team also picked up a few whiteboarding SaaS tools. There were no processes yet, so the first job was figuring out ownership and escalations. I used Cursor AI to build dashboards for availability and usage.

In this role I used ▼

Vectra brought me in on a 4-month project to improve monitoring of both their enterprise products and corporate systems.

So I set them up with Zabbix. Why Zabbix? Because Zabbix was 20 years old and thoroughly tested, and because it has alerting and trending in one package. Because it's common, they should have no problem hiring someone to manage it after me. Because it's free and open source.

I created a Salt State File to spin up a Zabbix server, and ran it to create the server, and a different state file to install the Zabbix client, and integrated that into the existing server provisioning system. I created dashboards that showed trends for various metrics, and I created alerts to email the team if any of these metrics breached defined thresholds. I documented the whole thing and called it a job well done.

In this role I used ▼

Bloomfire's infrastructure had been stable for years, which also meant it had not moved. SSH, Chef, and Ubuntu were stuck as a set.

• The version of ssh was so old that it didn't work with a lot of modern software.
• The version of chef-solo they used depended on that old ssh, so they couldn't upgrade ssh. Also it wouldn't run on modern Ubuntu because of version incompatibilities and other technical issues.
• The product apps were set up to start from Upstart, which was unavailable in modern versions of Ubuntu.

We called this the chicken-egg-egg problem, because each of these 3 parts (ssh, ubuntu, chef-solo) could easily be upgraded on its own, but that would break the system unless the other 2 were also upgraded at the same time.

First I switched the product apps from Upstart to systemd. This theoretically allowed our apps to run on Ubuntu 18, if everything was set up properly. It was set up by Chef, which wouldn't run on modern Ubuntu.

So I re-implemented it in Ansible. I spent months with 2 windows up on my screen: reading Chef on the left, writing Ansible on the right. This got rid of both the ssh problem and the Chef version problem at once.

Once I had a minimum viable subset of Chef re-implemented in Ansible, I did the Ubuntu upgrade in production so ssh, Chef, and Ubuntu could all move together. Then I finished the Chef-to-Ansible re-implementation.

I completed one other major task here. The SaaS product was running in a bespoke VPC hand-crafted years ago and added to manually ever since. I created a series of Rake tasks and Terraform modules to roll out a new instance of the product in a different region.

In this role I used ▼

I was hired to midwife a shift from startup operations to something that would scale:

• Introduce and establish agile processes and sprint cycles
• Promote Infrastructure as Code and modern DevOps practices
• Go through the whole SRE process of defining metrics, agreeing SLIs and SLOs, creating graphs and dashboards, all the way through updating architecture and actively improving response times and user experiences.
• Move the entire installation. Where? That had not yet been decided when I was hired.

SambaTV ran production in several physical datacenters. The datacenters themselves had been stood up by hand, so no two looked quite the same, and each had its own stubborn quirks. Early on I spent a week at a new site they were evaluating, mostly on network switching: VLANs, spanning tree, that kind of thing.

Remote hands installed the physical servers and I set up preseed to install Ubuntu. Next came automation to install our app and register each server with the ZooKeeper cluster, load balancer, etc.

Nagios configs were being maintained by hand, and that doesn't scale. So one project was to replace Nagios with Sensu, which included updating and rewriting plenty of check scripts. We ended up ripping out the entire monitoring system top to bottom: a Prometheus exporter for metrics, Grafana graphs and dashboards, and an ELK stack in place of Splunk.

One open question was whether the app could run fast enough in the cloud. Static EC2 was easy; autoscaling the whole stack was the real test. I spent 3 months making that work. AWS was plenty fast enough, but the datacenter cost less.

This whole time I was mentoring the team on Infrastructure as Code: Ansible playbooks we could commit to Git and run from a pipeline, not one-off commands across a hundred servers.

In this role I used ▼

MachineZone makes a game with millions of active players. Success had outrun the metrics stack, which is why I was there. The cluster could not keep up with the volume. I upgraded every step in that software chain.

Metrics wound up on a Graphite server and got stored in Whisper files that were not keeping up with the volume. I migrated all that data to modern InfluxDB, and configured our servers to send their metrics there as well. The next bottleneck was Python-based carbon-relay: it wasn't keeping up with the volume. Replaced that with carbon-relay-ng in C. Next bottleneck was rsyslogd. Replaced that with collectd. That also meant updating a hundred scripts to the new collectd format. When I say I replaced these things, really I was authoring and updating Puppet modules and abstracting them into Puppet classes.

MachineZone was a real turning point in my career because my first week there someone told me: The deliverable for this project is not the servers running the app. The deliverable for this project is the automation, so we can press go and the servers will spin up. Once we have that, then we just press go. That part is easy. That one statement instantly made me understand Infrastructure as Code and what Site Reliability Engineering is all about.

In this role I used ▼

Senior Linux System Administrator

02/2014 – 11/2015

CNE Media was a small video production company that ran the entire process in-house, with several studios on site: casting, shooting, writing, directing, marketing, and pre- and post-production. That full stack is what made the infrastructure interesting. How do you store and back up multiple petabytes of video going back 20+ years? (We settled on ZFS snapshots).

There were many processes that were undocumented and poorly understood, so my first task was software archaeology: read the code, find out what it does, and re-implement it in a modern, highly available and reliable configuration. Implement all this in Puppet.

Since I was the infrastructure engineer I was also tasked with migrating the VM infrastructure from VMware to Citrix, as well as generally tuning anything that felt slow. Often MySQL, PostgreSQL, MongoDB, Redis, or Jenkins.

In this role I used ▼

Site Reliability Engineer

12/2015 – 11/2016

I read about Google's idea of Site Reliability Engineering and convinced my manager to try it. I officially became a Site Reliability Engineer. I was tasked to implement enterprise-wide backups with Bacula supplemented by custom Bash scripts around mysqldump and pg_dump. Implemented asset management software GLPI and OCSinventory; deployed agents with Puppet. Spun up and maintained virtual servers in VMware and Citrix XenCenter as necessary. Installed and maintained what services they needed, mostly LAMP. Did it all with Puppet. Infrastructure as Code is just so much easier and better than the old way of doing things and I was sold on the idea.

In this role I used ▼

This was a front-line firefighting role. Alerts reached me after another team had already confirmed there was a real problem. I looked up the associated runbook and followed it. If there was no runbook, I read the docs and code, wrote one, then followed it.

MobiTV streams television to phones. The product had to stay up the way broadcast television stays up: 24x7x365, with outages handled inside tight SLA timeframes.

This was largely a firefighting role. I administered, monitored, and supported the media services. When something broke I troubleshot it, fixed it, or escalated. Day to day that also meant LDAP users and groups, backups, clearing logs and full partitions, restarting services, and helping install hardware and software.

I set monitoring thresholds and triggers, wrote the response procedures, and trained NOC staff to follow them. I also mentored junior NOC folks in Unix and network knowledge — the lore, not just the runbook.

I wrote custom Icinga plugins in bash and Python for the application suite and the operational infrastructure. The fleet was 600+ Linux servers running Apache and Tomcat, managed with Puppet.

This role was also my second experience as a manager. One big client required that we have multiple engineers in a room checking things every 15 minutes. I interviewed and hired 4 engineers, trained them on the product and the processes, and acted as the link between this team and my own manager.

In this role I used ▼

Experience across Mozilla, Trulia.com, InTicketing, Penguin Computing, and Servepath.

This period built the systems foundation that shaped later work in platform engineering and SRE leadership.