GitHub Enterprise Server Administration Best practices from GitHub Support

[djdefi]

Every day, the GitHub Enterprise Support team assists GitHub Enterprise Server administrators in managing, securing, and supporting the most critical and central hub of their organization's development lifecycle.

We've been keeping track of best practices for GitHub Enterprise Server Administrators that you can apply in your environment.

High Availability and Disaster Recovery (Backups)

• GitHub Enterprise Backup Utilities should be configured and backups scheduled to run regularly against your GitHub Enterprise Servers. Testing restores to a staging or test environment is also equally important.
• Configuring at least one High Availability Replica is advised for redundancy and ease of recovery in the event of a primary appliance failure.
• Placing replicas and GitHub Enterprise Backup Utilities hosts in different regions or data centers from the primary appliance can help to ensure geographic redundancy.
• Geo-replicas or Repository Cache nodes should be added only as a response to a specific identified need. Low latency and high bandwidth links are important so as to not create additional bottlenecks or management overhead.

Test / Staging environments

• Keep your GitHub Enterprise Server appliance up to date with the latest release to take advantage of security updates and bug fixes. Testing upgrades in a staging environment is important, particularly when upgrading to a new feature release.
• Make use of one or more GitHub Enterprise Server Staging Environments to test and familiarize yourself with procedures such as Upgrading the appliance, Configuring and failing over in High Availability scenarios, and Restoring backups
• Your GitHub Enterprise Server license can be used to set up any number of non-production environments, and their use is strongly encouraged when preparing for upgrades or other changes to the environment.

Implement policies for rate limiting and abuse

• API and Git rate limiting is only effective if you also set enforceable, policy-backed limits. Exceptions to your rate-limiting policy should be rare and used for specific integrations and service accounts that are outliers to your normal user API usage.
• Use log forwarding and collectd metrics forwarding to spot increases in request rates, bad actors, and patterns.
• CLI usage reporting tools such as ghe-governor-summary, ghe-btop, should be regularly reviewed for outlying or abusive behaviors.
• Suspend bad actors, or block them at a network or firewall level.

Be a good CI or API citizen

• Use webhooks rather than cron-style polling
• If Cron-style polling has to be done, spread out and skew the fetches to reduce parallel requests.
• Utilize conditional requests with the GitHub API, and ensure that CI plugins are up to date, with caching features enabled as well.
• Build artifacts, logs, or even data backups should not be stored in git (especially in refs/heads). Git is not designed to be a database. Use Git LFS when dealing with large files to keep Git performing great.
• Frequent automated pushes can be even more impactful on overall performance than frequent fetches. Limit usage of cron style pushes, and batch multiple commits into a single push where possible if automating pushes on a schedule.

Provision hardware for growth

• Provision or expand your root volume to be larger than the default. To allow for the natural growth of your deployment, provisioning up to 2x or 4x or more than the 200GB default is recommended for busy systems.
• Pay close attention to disk IO performance and metrics such as burst bucket volumes and throughput maximums. (One example is the published performance expectations of EBS volume types). Fast SSD-based, low latency storage is required for optimal performance. Git operates on vast numbers of generally very small files meaning disk latency, and by extension IOPS capability, is extremely important. A fast, locally attached SSD-based disk is highly recommended for optimal performance.
• More memory = more room for disk cache and additional GitHub worker and service scaling.
• Also provide sufficient CPU and resources along with a fast disk for your GitHub Enterprise Backup Utilities system to increase backup and restore speed.

Join the conversation!

Now you have some best practices to manage your GitHub Enterprise Server environment and tips to make it more secure and reliable. At the end of the day, maintaining a healthy and reliable environment is up to how you and your team manage it. What are some of your go-to practices? Please feel free to share in the comments below!

[MattSnow-amd]

@djdefi - Given that GHES is a virtual machine deployed on a few different hypervisors, can you share any more details for specific hardware that is used with success?

I would love to be able to use GHES on Linux w/ Libvirt or something like oVirt, but this hypervisor is not listed.
Additionally are there any recommendations or experiences you can speak to on hyperconverged infrastructure, such as VMware vSAN or oVirt+GlusterFS ? I feel like HCI is the likely way to go for on-premise GHES environments for quicker recovery in the event of hardware failure.

[djdefi]

Hey @MattSnow-xlnx

In support we often have quite a bit more insight into the hardware cloud platform than on-prem deployments just due to those instance types having more readily available documentation and benchmark data to reference. In the supported cloud platforms, we find that "memory optimized" instances which reflect a 1CPU to 8GB RAM ratio often fit workloads the best. Additionally, faster CPU clock speed offerings often perform better during heavy Git operations than ones that advertise power saving or other CPU stepping features.

For on premises hardware, we actually see a lot of variance in what is used, and what can be successful. In general you want the fastest CPU clock speeds, a fast SSD disk, lots of RAM, and the fewest possible NUMA nodes for the CPU+RAM combination.

For both cloud and on premises, in many cases we have found that large, active systems with a smaller number of NUMA nodes (ideally 2) perform better than systems that have 4 or more nodes. Also enabling Performance mode / disabling power saving features in the BIOS may be recommended to maximize and stabilize the potential hardware performance.

We don't have a lot of specific benchmarks with regards to VMware vSAN or oVirt+GlusterFS currently, but in general we do see that high storage latency can become a limiting factor or bottleneck. For this reason our documentation recommends fast locally attached SSD, but many modern storage vendors and systems can likely provide equivalent performance. One caveat is that we often we do find network type storage overhead, such as with NFS can often perform poorly with GitHub Enterprise and many tiny files contained within repositories on the appliance. For that reason, we would recommend making sure that any software based storage solution has roughly equivalent latency and throughput as a local SSD to the extent possible.

While we don't specifically mention Libvirt or oVirt in our documentation (just OpenStack KVM), we are fairly confident that these platforms would work and be easily supportable given enough interest. We use a vanilla KVM / oVirt based hypervisor for a large portion of CI/CD, so this is already is rather well tested path. If you are interested in official Libvirt or oVirt support specifically, I would suggest raising a support ticket or contacting your GitHub account manager to "officially" log a feature request.

I hope this helps!