Investigate & mitigate evicted GitLab pods

[zackgalbreath]

While poking around with kubectl we noticed the GitLab web pod had been evicted multiple times in short succession.

Investigation:
Gain a better understanding of why this happened. It must've been resource pressure, but we haven't found a "smoking gun" in the logs yet. One idea is to add CPU monitoring to our watchdog pod:

awk '{u=$2+$4; t=$2+$4+$5; if (NR==1){u1=u; t1=t;} else print ($2+$4-u1) * 100 / (t-t1) "%"; }' <(grep 'cpu ' /proc/stat) <(sleep 1;grep 'cpu ' /proc/stat)

(from here)

Mitigation options:

• Migrate GitLab to a larger EC2 instance
• Pursue a high availability strategy for deploying GitLab

cc @opadron