Automate a multi machine ArcGIS Enterprise deployment on Linux using Chef Server
This workflow describes how to deploy ArcGIS Enterprise on a set of Ubuntu machines using Chef Server. It requires basic knowledge of ArcGIS Enterprise and of Linux. Familiarity with Chef is recommended.
To follow this workflow for Windows machines, see Automate a multi machine ArcGIS Enterprise deployment on Windows using Chef Server.
The Chef Server model is optimal for a multi-machine deployment like this because all of the work is initiated from your workstation, using a Chef Server as a repository.
Note: The main goal of this workflow is to demonstrate the deployment process. Chef cookbook can configure any ArcGIS Enterprise deployment pattern. Please refer to the ArcGIS Enterprise documentation to pick the ideal deployment architecture for your organization.
- Five Ubuntu machines with access to the internet. (If you are in a disconnected intranet, you must have the Chef SDK and ArcGIS Chef resources available on your intranet.)
- Setups for ArcGIS Enterprise for Linux
- Software authorization files for ArcGIS Enterprise (Portal for ArcGIS and ArcGIS GIS Server) and for ArcGIS GeoAnalytics Server. All software and authorization files should be for the same version.
- A SSL certificate file for the domain name of the base ArcGIS Enterprise machine.
- Starter Kit for Chef Server. See the options below under Prepare Chef Server.
Note: In this workflow, you will need to upload files from local or shared locations to Linux servers multiple times. Before you begin, download an appropriate client (such as putty, winscp, scp, etc.) to connect with and move files to Linux servers.
Below is the intended mapping for our five machines and deployment roles.
- Machine 1 -> Chef workstation
- Machine 2 -> Base ArcGIS Enterprise (baseenterprise)
- Machine 3 -> Big Data Store (bds)
- Machine 4 -> File server for GeoAnalytics (fileserver)
- Machine 5 -> GeoAnalytics Server (gaserver)
Most organizations using Chef Server to automate deployment already have a Chef Server set up. If you don’t have a Chef Server, there are multiple options available. You can download the setup from the Chef website and install it on your machine. Alternatively, you can use Hosted Chef Server provided by Chef (SaaS) or Chef Automate. If you have an AWS account, then you can use AWS opsworks for Chef to set up Chef Server.
For this workflow, we are using Hosted Chef Server.
- Sign up for a free trial of Hosted Chef if you don't already have an account.
- Sign in to the Chef Server management console, create an organization and download the starter kit for the organization (Administration > Organizations > Starter Kit).
If you don’t have a Chef workstation set up, the following steps can be used to set it up as a reference.
2. On the Chef workstation machine, install Chef Workstation.
wget https://packages.chef.io/files/stable/chef-workstation/21.2.278/ubuntu/20.04/chef-workstation_21.2.278-1_amd64.deb
dpkg -i chef-workstation_21.2.278-1_amd64.deb
3. Unzip the 'chef-repo' directory from your Chef Server starter kit into the home directory of your Chef workstation machine (Machine 1).
Download the cookbooks and roles zip files corresponding to your version of ArcGIS Enterprise. Unzip the contents of the cookbooks file into the ~/chef-repo/cookbooks directory on your Chef workstation machine.
knife cookbook upload --all
Unzip your arcgis-enterprise-linux-roles.tar.gz file and move its two folders into ~chef-repo/roles and ~chef-repo/environments folders, respectively.
Edit this file in any editor. The file has a number of sample values meant to be replaced. Substitute values for these nodes specific to your deployment, such as MACHINE2PRIVATEIP (the Private IP address of Machine 2), yourdomainname (the FQDN for mywebsite.mydomain.com), REPOMACHINEIP (the IP address of the Linux machine serving as your repository), and so on.
knife environment from file environments/demo.json
Switch back to your Chef workstation machine.
A bootstrap is a process that installs the chef-client on a target system so that it can run as a chef-client and communicate with a Chef server. Run the following command, with some changes: Replace HOST2PRIVATEIP and ssh_key in the command below with those specific to your deployment. The --bootstrap-version parameter allows you to state the version of the Chef Client you want to use; recommended versions can be found in the Esri-cookbooks-and-Chef-Infra-client-version-compatibility wiki page.
knife bootstrap HOST2PRIVATEIP -N baseenterprise -i ~/.ssh/ssh_key.pem -x ubuntu --sudo --bootstrap-version '15.14.0'
HOST2PRIVATEIP refers to the private IP address of Machine 2. ssh_key refers to the SSH key used to connect with Machine 2.
Update values for nodes like REPOMACHINEIP, yourdomainname, file paths, user names, and passwords, and upload it to the Chef server.
knife upload roles/baseenterprise.json
knife node run_list set baseenterprise 'role[baseenterprise]'
knife node environment set baseenterprise demo
5. Run the Chef client on nodes in the 'baseenterprise' role. Replace ssh_key with your specific key.
knife ssh 'role:baseenterprise' 'sudo chef-client' -i ~/.ssh/ssh_key.pem -x ubuntu
The Chef run takes about an hour. After the Chef run finishes, you have a base ArcGIS Enterprise deployment, with Portal, Server and Data Store, installed on Machine 2 with the appropriate configuration, as depicted in the above architecture diagram. You can access your ArcGIS Enterprise portal using the URL format https://<yourdomainname>/portal/home.
Run the following command, replacing the HOST3PRIVATEIP and ssh_key parameters with your own as with the previous machine.
knife bootstrap HOST3PRIVATEIP -N bds-i ~/.ssh/ssh_key.pem -x ubuntu --sudo --bootstrap-version '15.14.0'
2. Modify node values in the file ~/chef-repo/roles/bds.json' available on your Chef workstation machine with values specific to your deployment, and upload it to the Chef server.
knife upload roles/bds.json
knife node run_list set bds 'role[bds]'
knife node environment set bds demo
knife ssh 'bds' 'sudo chef-client' -i ~/.ssh/ssh_key.pem -x ubuntu
After the Chef run finishes, a spatiotemporal big data store is installed on Machine 3 and registered with the ArcGIS Server installed on Machine 2, as depicted in the architecture diagram.
Most organizations probably already have a file server in place. If you do, then you can use your existing file server by providing its file path to the GeoAnalytics Server you set up. Refer to the Chef parameter list on how to define it. The following steps are just an example to configure a file server for you if you don’t already have one.
As before, substitute the sample parameters for your own, specific to Machine 4, and run the command.
knife bootstrap HOST4PRIVATEIP -N bds-i ~/.ssh/ssh_key.pem -x ubuntu --sudo --bootstrap-version '15.14.0'
2. Modify node values in the ~/chef-repo/roles/fileserver.json' file on your Chef workstation machine with values specific to your deployment and upload them to the Chef server.
knife upload roles/fileserver.json
knife node run_list set fileserver 'role[fileserver]
knife node environment set fileserver demo
knife ssh 'fileserver' 'sudo chef-client' -i ~/.ssh/ssh_key.pem -x ubuntu
After the Chef run finishes, a file server is configured on Machine 4, as depicted in the architecture diagram.
Run the following command, replacing the sample parameters with your own as before.
knife bootstrap HOST5PRIVATEIP -N bds-i ~/.ssh/ssh_key.pem -x ubuntu --sudo --bootstrap-version '15.14.0'
2. Modify node values in the ~/chef-repo/roles/gaserver.json file on your Chef workstation machine with values specific to your deployment and upload them to the Chef server.
knife upload roles/gaserver.json
knife node run_list set gaserver 'role[gaserver]'
knife node environment set gaserver demo
knife ssh 'gaserver' 'sudo chef-client' -i ~/.ssh/ssh_key.pem -x ubuntu
After the Chef run finishes, ArcGIS Server is installed on Machine 5 and licensed as ArcGIS GeoAnalytics Server, as depicted in the architecture diagram.
Note: This sample workflow only installs ArcGIS GeoAnalytics Server on one node, but to truly take advantage of the GeoAnalytics distributed processing power, we recommend you install the server on at least three nodes. To install it on additional machines, run the same commands, but use the 'gaserver_node' role file and assign that role to each new node.
You can verify deployments by accessing the URLs in format https://yourdomainname/portal and https://yourdomainname/server.
1. Modify the ~/chef-repo/roles/cleanarcgisenterprise.json file on your Chef workstation,and upload it to your Chef server.
knife upload roles/cleanarcgisenterprise.json
knife node run_list set baseenterprise 'role[cleanarcgisenterprise]'
knife node run_list set fileserver 'role[cleanarcgisenterprise]'
knife node run_list set gaserver 'role[cleanarcgisenterprise]'
knife node run_list set bds 'role[cleanarcgisenterprise]'
3. Run the Chef client on the nodes identified with the 'cleanarcgisenterprise' role. Replace the SSH key path as before.
knife ssh 'cleanarcgisenterprise' 'sudo chef-client' -i ~/.ssh/ssh_key.pem -x ubuntu
After the Chef run finishes, all ArcGIS software will be uninstalled from respective nodes. This will also clean up content directories like arcgisserver, arcgisportal, and arcgisdatastore.
The above workflow was designed to make the deployment process easier to understand and recover from mistakes. In production deployments, instead of deploying one machine at a time, it is recommended to split and script the process into three phases: