Fun with Deployments and Zero Downtime (ZDT)

Background concepts:

1. To deploy ZDT: upload_capi_release && ~/workspace/capi-release/scripts/deploy -o ~/workspace/capi-ci/cf-deployment-operations/temporary/add-deployment-updater.yml

   If you ssh to the scheduler vm, sudo su then monit summary you should see:

   Process 'cc_deployment_updater' running

2. That process was invoked from /var/vcap/jobs/cc_deployment_updater/bin/cc_deployment_updater by running rake deployment_updater:start and this starts a loop that sleeps for cc.deployment_updater.update_frequency_in_seconds (default 5) and then runs one orchestration step on all active deployments (e.g. ones in state DEPLOYING).

To make ZDT happen:

1. First, cf v3-push dora && cf v3-scale dora -i 4 an app

2. Stage a new package. (These commands come from How to Create an App Using V3 of the CC API):

   perl -pi -e 's/(Hi.*Dora.*\#)(\d+)/$1 . (int($2) + 1)/e' dora.rb
   
   CF_API_ENDPOINT=$(cf api | grep -i "api endpoint" | awk '{print $3}')
   
   APP_GUID=$(cf v3-app dora --guid  2>/dev/null)
   
   PACKAGE_GUID=$(cf curl /v3/packages -X POST -d "$(printf '{"type":"bits", "relationships": {"app": {"data": {"guid": "%s"}}}}' "$APP_GUID")" | tee /dev/tty | jq -r .guid)
   
   zip -r my-app-v2.zip * # ( -x *.zip if there are old zip files hanging around)
   
   curl -k "$CF_API_ENDPOINT/v3/packages/$PACKAGE_GUID/upload" -F bits=@"my-app-v2.zip" -H "Authorization: $(cf oauth-token | grep bearer)"
   
   # Wait for the package to go from PROCESSING_UPLOAD to READY
   while : ; do state=$(cf curl /v3/packages/$PACKAGE_GUID | tee /dev/tty | jq -r .state) ; if [ "$state" == "READY" ] ; then break; fi; sleep 5 ; done
   
   rm my-app-v2.zip  # clean up after yaself
   
   BUILD_GUID=$(cf curl /v3/builds -X POST -d "$(printf '{ "package": { "guid": "%s" }}' "$PACKAGE_GUID")" | tee /dev/tty | jq -r .guid)
   
   # Wait for staging to complete
   
   while : ; do state=$(cf curl /v3/builds/$BUILD_GUID | tee /dev/tty | jq -r .state); if [ "$state" != "STAGING" ] ; then break; fi; sleep 2 ; done
   
   

3. Get the new package's droplet and associate it with the app:

   At some point the build should be STAGED. If it's FAILED or still staging after an unreasonable amount of time, you have a problem more severe than trying to ZDT-update an app. Otherwise, stop watch and continue:

   DROPLET_GUID=$(cf curl /v3/builds/$BUILD_GUID | jq -r '.droplet.guid')
   
   cf curl /v3/apps/$APP_GUID/relationships/current_droplet -X PATCH -d "$(printf '{"data": {"guid": "%s"}}' "$DROPLET_GUID")"
   
   

4. Finally, create a deployment for the app with the new droplet:

   cf curl /v3/deployments -d "$(printf '{ "relationships":{ "app": { "data": { "guid": "%s" }}}}' $APP_GUID)"
   
   

5. Repeatedly hit the app and verify that the output contains an ever-increasing proportion of the new output until the old output completely disappears and the webish process instances increase and then drop away so you're back to N web instances.

   for x in {1..10000} ; do curl "dora.${CF_API_ENDPOINT#*.}"; sleep 0.5; done
   
   

Verifying the v2 app doesn't blink

1. Add a sleep 60 to the updater with log statements

2. Tail logs in one window

3. When prompted, run commands like

   `cf curl /v2/apps/[APP_GUID]

The WHOLE Script

perl -pi -e 's/(Hi.*Dora.*\#)(\d+)/$1 . (int($2) + 1)/e' dora.rb

CF_API_ENDPOINT=$(cf api | grep -i "api endpoint" | awk '{print $3}')

APP_GUID=$(cf app dora --guid  2>/dev/null)

PACKAGE_GUID=$(cf curl /v3/packages -X POST -d "$(printf '{"type":"bits", "relationships": {"app": {"data": {"guid": "%s"}}}}' "$APP_GUID")" | tee /dev/tty | jq -r .guid)

zip -r my-app-v2.zip * # ( -x *.zip if there are old zip files hanging around)

curl -k "$CF_API_ENDPOINT/v3/packages/$PACKAGE_GUID/upload" -F bits=@"my-app-v2.zip" -H "Authorization: $(cf oauth-token | grep bearer)"

# Wait for the package to go from PROCESSING_UPLOAD to READY
while : ; do state=$(cf curl /v3/packages/$PACKAGE_GUID | tee /dev/tty | jq -r .state) ; if [ "$state" == "READY" ] ; then break; fi; sleep 5 ; done

rm my-app-v2.zip  # clean up after yaself

BUILD_GUID=$(cf curl /v3/builds -X POST -d "$(printf '{ "package": { "guid": "%s" }}' "$PACKAGE_GUID")" | tee /dev/tty | jq -r .guid)

# Wait for staging to complete

while : ; do state=$(cf curl /v3/builds/$BUILD_GUID | tee /dev/tty | jq -r .state); if [ "$state" != "STAGING" ] ; then break; fi; sleep 2 ; done

DROPLET_GUID=$(cf curl /v3/builds/$BUILD_GUID | jq -r '.droplet.guid')

cf curl /v3/apps/$APP_GUID/relationships/current_droplet -X PATCH -d "$(printf '{"data": {"guid": "%s"}}' "$DROPLET_GUID")"

cf curl /v3/deployments -d "$(printf '{ "relationships":{ "app": { "data": { "guid": "%s" }}}}' $APP_GUID)"

for x in {1..10000} ; do curl "dora.${CF_API_ENDPOINT#*.}"; sleep 0.5; done