Via a Push Queue


Topic of the #push channel

Push Queue

mittens> .join
pushbot> You’re up: mittens
  sasha> .join with mittens
pushbot> You’re up: mittens, sasha
  sasha> .good
mittens> .good
pushbot> mittens, sasha: Everyone is ready
 nassim> .join
pushbot> You’re now on staging
  sasha> .good
mittens> .good
pushbot> You’re now on prod
  sasha> .good
mittens> .good
mittens> .done
  sasha> Thanks for driving, mittens!
pushbot> You’re up, nassim

My colleague Hannah (nick: mittens) has joined the queue She’s indicating to everyone she has code she wants to deploy Because the queue was empty before she joined She becomes the driver Pushbot lets Hannah know she can push her changes Now I join the queue Because I have code I need to deploy too Notice me typing dot-join in Slack modified the topic That’s pushbot doing that Both hannah and I push our changes We let others know we’re ready by marking .good Pushbot let’s us know we’re ready to deploy to staging with both our changes Hannah deploys to preprod She uses a tool called Deployinator It’s basically a button for a bunch of shell scripts While that staging deploy is happening Nassim joins the queue We have a more interesting topic now so let’s understand what these things mean This is the whole push queue These are the push trains it consists of And these are the drivers of their respective trains There can be up to 8 people in one train Anyone can take over driving a train Trains can also be specified as special by designating them as a hold, which prevents others from joining There’s a bunch of commands to manipulate the queue, all via Slack If problems arise, you can manually change the topic as a last resort Ok, looks like our staging deploy is finished so Hannah and I check our changes Look at graphs / logs Confirm staging tests passed Manually test our changes and some key pages (like search, cart, listing page) Great, our changes are good If they hadn’t been good we’d redeploy a hotfix to staging or revert that person’s changes and they’d leave the queue

So now we deploy to production Hannah hits the button

The deploy is complete, so we test our changes

Our changes look good on prod too, so we end our train Pushbot hands off to the next train If the changes had issues A Make a hotfix Push Deploy to staging, then to prod B Push a revert Deploy revert to staging and then to prod Ask me later about what happens if deploys fail, Or if the site goes down

Web Deploys

Are pretty fast (~20 min)
Happen many times a day (up to 50)
Frequent, small changes
Everyone responsible for monitoring
Driver responsible for pushing buttons and herding
Easy to revert and back out

Deploy Culture

Transparent
Engineer-driven
Encourages collaboration
Reduces single points of failure

App Deploys

Are pretty slow (~6 hours)
Happen once every two weeks
Periodic, medium-large changes
Everyone responsible for monitoring?
Release manager responsible for pushing buttons and herding
Hard to revert and back out

Web Deploys

App Deploys

App Deploy Tasks & Responsibilities

mittens + sasha | nassim

release managers

Release Managers, Sasha & Jen

More release managers?

no

Better app deploy process?

yes!

Everyone installs the testing build and indicates their status

A cron branches the release and creates a release candidate.

Great, now we hit branching time, and a cron generates a release candidate This candidate is sent out to a testing team, which performs a bunch of manual testing

final testing finds no show-stopping issues

A cron submits v4.64.0 to iTunes Connect for review.

Ship sends an email letting folks know the release is approved.

A cron releases v4.64.0.

Ship sends an email letting folks know the release is out on prod.

Ship emails a report of top crashes to all the subscribers of the release

I actually don’t have any screenshots for the remaining phases, so we’ll just have to imagine together ;-) The testing team finds no issues, and they hit a “tests pass” button A cron submits the build for review Ship sends a status update email when the build is approved Or when it’s rejected The release date arrives, And if the build was approved, Ship releases it IT sends another email at release time notifying people And then one 24 hours later with a list of top crashes

Before Ship, all of this would’ve been performed manually But you’ll notice that release managers are missing from the above process have we replaced release managers with all the automations in Ship?

bit.ly/how-etsy-ships-apps