Ansible

This repository is a collection of ansible playbooks implemented by Vertilux’s IT dept to perform some server automations.

Getting Started

Make sure ansible is installed on your machin, to install ansible:

Install ansible on Mac

brew install ansible

Install ansible on Windows

Probably the best option is to installed Ubuntu on WSL (Windows Subsytem for Linux).

• Search for Windows features in the search box. And when the “Turn Windows features on or off ” appears, click on that.
• A window will open with a bunch of features. Scroll down and check the box of Windows Subsystem for the Linux option. And after that, click on the OK button.
• Open the Microsoft Store and search for Ubuntu to install the latest version.
• After the installation, you will see a launch button, use that to open the Ubuntu bash.
• On Ubuntu bash, it will ask you to set the username and password for the default user. You can also set the root account password from here by typing sudo passwd root

Now install ansible and press Y when ask for.

sudo apt-get update
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible

Clone the repository.

git clone git@github.com:vertilux/ansible.git

Then cd into the ansible directory: $ cd ansible

VTX API

Deploy

VTX API is a set of programming code that enables data transmission between one software product and another, based on REST architecture.
It allows simple queries againts Vertilux Databases. Our API has predictable resource-oriented URLs, returns JSON-encoded responses, and uses standard HTTP response codes.

Playbook

ansible-playbook entry_point.yml -i inventory

ERP

Deploy

This playbook contains multiples roles to deploy new versions or ERP updates to production.

To avoid password authetication generate a public key and copy to the remote servers.
To create the SSH keys, type ~ssh-keygen -t rsa -C “your_email@example.com”~. This will create both id_rsa and id_rsa.pub files. Copy the generated public key to the remote host

cat ~/.ssh/id_rsa.pub | ssh user@remote_host 'cat >> .ssh/authorized_keys'

Additioanlly disable password requirement for sudo. Create a file in /etc/sudoers.d/deploy

deploy ALL=(ALL) NOPASSWD:ALL

Entry point

This playbook execute the different roles that will run

ansible-playbook entry_point.yml -i inventory

Roles

description	commands
app-update	Will check if directory exist then is going to clone or pull from version control (Github)
	yarn install
	bundle
	RAILS_ENV={{env}} rake assets:precompile
run-migrations	RAILS_ENV={{env}} rake db:migrat
restart-sidekiq	sudo service sidekiq-{{env}} restart
restart-nginx	sudo service sidekiq-{{env}} restart

Playbooks

A collections o simple playbooks to perform different tasks for the ERP Web Application.

Ping

Try to connect to host, verify a usable python and return pong on success.

ansible-playbook ping.yml -i inventory

Sidekiq

Background processing for ruby. Sidekiq uses threads to handle jobs at the same time in the same process. It is integrated tightly with Rails to make background processing dead simple.

Provision Service

Running sidekiq as a system service with systemd, this will ensure the process is restarted if Sidekiq crashes. Here is an example systemd file:

[Unit]
Description=sidekiq
After=syslog.target network.target

[Service]
Type=simple
WorkingDirectory=/home/deploy/APP_LOCATION/current
ExecStart=/home/deploy/.rbenv/shims/bundle exec sidekiq -e ENVIRONMENT
User=deploy
Group=deploy
UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target

With the following playbook the service is provision on all servers.

ansible-playbook sidekiq_service.yml -K -i inventory.yml

Restart Sidekiq

ansible-playbook restart-sidekiq.yml -K -i inventory.yml

Nginx Webserver

Nginx is a web server that can be used as a reverse proxy, load balancer and HTTP cache.

Append Nginx Configuration

This playbook will append a configuration block to the default configuration with the env passed. This is the easy way to when deploying a new distribution center.

ansible-playbook append-nginx-config.yml -K -i inventory.yml

Restart Nginx

Simply restart nginx on all servers excluding load labancer.

ansible-playbook restart-nginx.yml -K -i inventory.yml

Restart passenger

Phusion Passenger is an open source web application server. It handles HTTP requests, manages processes and resources, and enables administration, monitoring and problem diagnosis.

ansible-playbook restart-passenger.yml -i inventory.yml

This playbook displays a prompt to enter the environment to be restarted, with this option you can restart only one environment (distribution center app).

Logrotate

Configuring Logrotate For Rails Production Logs. Running ansible-playbook append-logrotate.yml --extra-vars "env=ENV" -K -i inventory.yml will edit sudo vim /etc/logrotate.conf, and at the bottom of the file the following block of code (change the ENV, ex: accltd):

/home/deploy/erp-ENV/current/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}

ansible-playbook append-logrotate.yml -K -i inventory.yml

How it works:

• daily – Rotate the log files each day.
• missingok – If the log file doesn’t exist, ignore it.
• rotate 7 – Only keep 7 days of logs.
• compress – GZip the log file on rotation.
• delaycompress – Rotate the file one day, then compress it the next day so we can be sure that it won’t interfere with the Rails server.
• notifempty – Don’t rotate the file if the logs are empty.
• copytruncate – Copy the log file and then empties it.

Running Logrotate: Then the playbook will run: `sudo /usr/sbin/logrotate -f /etc/logrotate.conf`

Maintenance

Sometimes there is a need to set the ERP Web application into maintenance mode; when an item change needs to be run or when the we need to upgrade Sage. For this scenarios the folloging playbook is the easier way to set all instances in maintenance mode.

Set only one distribution center to maintenance mode.

ansible-playbook maintenance-mode.yml -K -i inventory.yml

Put everything into maintenance.

ansible-playbook maintenance-mode-all.yml -K -i inventory.yml

Update DB IP Address

Imagine an scenario where you cannot get a ISP service ofering a static IP address and you need to change your config periodically. With this playbook you can easily update your config in all servers at once.

ansible-playbook update-database-ip.yml -i inventory

Append Env Variable

Environment variables are a set of dynamic named values, stored within the system that are used by applications launched. In simple words, an environment variable is a variable with a name and an associated value.

Persistent Env Variables

To make Environment variables persistent you need to define those variables in the bash configuration files. In most Linux distributions when you start a new session, environment variables are read from the following files:

• /etc/environment: Use this file to set up system-wide environment variables. Variables in this file are set in the following format:

  FOO=bar
  VAR_TEST="Test Var"
      

• /etc/profile: Variables set in this file are loaded whenever a bash login shell is entered. When declaring environment variables in this file you need to use the export command:

  export FOO="bar"
  export VAR_TEST="Test Bar"
      

For our apps we used /etc/profile, to add (append) a new env variables just need to run the following playbook:

ansible-playbook append-env-variable.yml -K -i inventory.yml

Provision

Provision ERP is a collection of ansibles roles to make easy the setup of multiples environments. Every environemnt is a distribution center, this approach makes us use the same code for all distribution centers and easy to maintain.

Running playbooks

ansible-playbook entry_point.yml -i inventory.yml -k -K

Entry point is the main playbook, this is going to ask for:

• User name.
• Environment (Sage300 SQL server name lowecase).
• URL (DNS url for the application, Ex: erp.vertilux.com).
• Port (SQL server port set on firewall rule).

Roles

• set-config-files
• append-nginx-config
• set-sidekiq-service
• logrotate

Tower

A collections of playbooks to perform tasks with the tower web interface.

QNAP

Usefull playbook collection to perform tasks on the QNAP NAS.

Remove old SQL backups

ansible-playbook remove-old-sql-backups.yml -i

Windows Management

Ansible Windows module uses winrm connection, in order to execute commands in the Windows machine.

Why Automate Windows Hosts?

A few of the many things you can do for your Windows hosts with Ansible.Windows include:

• Starting, stopping and managing services
• Pushing & executing custom PowerShell scripts
• Reboot multiples machines simultaneously

For a complete list of Ansible.Windows modules visit this link.

What’s Required?

First, the control machine (where Ansible will be executing your Windows modules from) needs to run Linux. Second, Windows support has been evolving rapidly, so make sure to use the newest possible version of Ansible to get the latest features!

For the target hosts, you should be running at least Windows 7 SP1 or later or Windows Server 2008 SP1 or later.

Lastly, since Ansible connects to Windows machines and runs PowerShell scripts by using Windows Remote Management (WinRM) (as an alternative to SSH for Linux/Unix machines), make WinRM is activated.

Trobleshooting WinRM

Sometimes we get an error plaintext: the specified credentials were rejected by the server and the reason it’s not the credentials as I know they are correct. It must be some other authentication issue.

The problem is that it is sending credentials in plaintext over the unencrypted WinRM port 5985. The fix is a couple of WinRM configuration changes in the Windows servers. In an elevated command prompt type the following commands:

Sets WinRM to Basic Authentication

winrm set winrm/config/service/auth '@{Basic="true"}'

Auth
    Basic = true
    Kerberos = true
    Negotiate = true
    Certificate = false
    CredSSP = false
    CbtHardeningLevel = Relaxed

Sets WinRM to allows unencrypted traffic

winrm set winrm/config/service '@{AllowUnencrypted="true"}'

Service
    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = true
    Auth
        Basic = true
        Kerberos = true
        Negotiate = true
        Certificate = false
        CredSSP = false
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = *
    IPv6Filter = *
    EnableCompatibilityHttpListener = false
    EnableCompatibilityHttpsListener = false
    CertificateThumbprint
    AllowRemoteAccess = true

Aditional Issues

Because WinRM can be configured in so many different ways, errors that seem Ansible related can actually be due to problems with host setup instead.
Some examples of WinRM errors that you might see include:

• HTTP 401 error
• HTTP 500 error
• Timeout issues
• Connection refusal.

Visit the Common WinRM Issues section of our Ansible.Windows Setup documentation page.

Install pywinrm

pywinrm dependencies aren’t shipped by default with Ansible, make sure you install the pywinrm library on the machine that Ansible is installed on. The simplest method is to run:

pip install pywinrm

References

• The Inside Playbook
• Ansible Documentation

Playbooks

Restart Services

This playbook wil use win_service – Manage and query Windows services to restart a chosen service. For beter UX select an application (country or distribution center) or SQL Service from the prompt menu.

Granite

ansible-playbook restart-granite-services.yml -i inventory --ask-vault_pass

SQL Service

Make sure to properly set target machine.

ansible-playbook restart-sql-service.yml -i inventory --extra-vars="target=VXACCPAC2" --ask-vault_pass

Restart Machine

Reboot a Windows machine, wait for it to go down, come back up, and respond to commands. In order to safely limiting to run this playbook on a single machine we using --extra-vars to especify the host.

Running the playbook:
If target isn’t defined, the playbook does nothing. A group from the hosts file can also be passed through if need be.

Playbook targeting a single host:

ansible-playbook restart-server.yml --extra-vars="target=granite_dr" --ask-vault-pass -i inventory

Playbook with a group of hosts:

ansible-playbook restart-server.yml --extra-vars="target=granite" --ask-vault-pass -i inventory

Get Disk Info

Retrieve and output detailed information about the attached disks of the target and its volumes and partitions if existent.

ansible-playbook disck-facts.yml --extra-vars="target=granite" --ask-vault-pass -i inventory

Listing hosts

If you want to list all available hosts in the inventory, run:

ansible all --list-hosts -i inventory

If you want to list that match a pattern you can use list-hosts.yml playbook and pass the option --list-hosts.

ansible-playbook list-hosts.yml -i inventory --list-hosts --extra-vars="target=granite"

Secure sensitive data

Use Ansible vault to protect sensitive data witin playbooks and roles. Ansible Vault performs various operations. Specifically, it can:

• Encrypt a file
• Decrypt a file
• View an encrypted file without breaking the encryption
• Edit an encrypted file

Create an encrypted file

The ansible-vault create command is used to create the encrypted file.

ansible-vault create vault.yml

This commland will ask for a password and then ask where to put your content.

$ANSIBLE_VAULT;1.1;AES256
32373330386330643061613237393466393363333031303965383063316338393261616134353233
6364623038303966323834396636646463613837373461380a303662373161393466326139383565
64396538393562343464666337633337353130306365666637373266393965633766366436623836
3430396234626533340a613637393563333131626665626535393462653139346563383062343535
6335

Editing the encrypted file

If the file is encrypted and changes are required, use the edit command.

ansible-vault create vault.yml

Decrypting a file

The ansible-vault decrypt command is used to decrypt the encrypted file.

ansible-vault decrypt vault.yml

Reset the file password

Use the ansible-vault rekey command to reset the encrypted file password.

Help

Other ansible-vault commands available are seen via:

ansible-vault -h

Wrapper script

Bash wrapper script to simplefy playbooks execution.

Help

Dislpay list of options and arguments:

./acli -h

Help examples:

__     _______ ____ _____ ___ _    _   ___  __
\ \   / / ____|  _ \_   _|_ _| |  | | | \ \/ /
 \ \ / /|  _| | |_) || |  | || |  | | | |\  /
  \ V / | |___|  _ < | |  | || |__| |_| |/  \
   \_/  |_____|_| \_\|_| |___|_____\___//_/\_\

acli - Ansible CLI wrapper writen in bash.

acli [options] [arguments]

Instructions for Windows Playbooks
options:
-r, --reboot-host       Restart host of group of hosts.
-d, --disk-space        Display total disck size.
    If you know the host you can pass it as an argument.
    Or simply run it as: ./acli -d, this will promt the iventory.
-l, --list-hosts        List inventory hosts.
    If you know the host group you can pass it as an argument.
    Or run it as ./acli -l to list all available hosts.
-p, --ping              Ping windows servers
-s, --restart-sql       Restart SQL service.
    Make sure pass the required arguments: host name and service.

Instructions for QNAP NAS
options:
--del                   Delete old backups (older than 3 days).

Instructions for ERP Web Application
options: erp
-s, --restart-sidekiq   Restart sidekiq queue processing job.
-m, --maintenance       Set/Unset ERP to maintenance mode.
-p, --restart-passenger Restart Phusion Passenger application server.
-n, --restart-nginx     Restart Nginx web server.

Windows Modules

Restart a server (host)

win_reboot – Reboot a windows machine.

./acli -r

Restart SQL Service

win_service – Manage and query Windows services

Arguments	Description
target	Machine name, Ex: VXACCPAC2
service	Service name, Ex: MSSQLSERVER

./acli -s <target> <service>

Disk facts

win_disk_facts – Show the attached disks and disk information of the target host.

If you dont know the host or group run the command without passing any argument.

./acli -d

If you know the host or group you can pass it as an argument.

./acli -d granite

List hosts

List all hosts listed in the windows inventory.

./acli -l

If yor want to list only hosts that belogns to an specific group, pass the group as an argument.

./acli -l granite

QNAP

We use QNAP as our main Network-attached storage (NAS).

Remove old SQL Backups

./acli --del

VTX API

Vertilux RESTful API is a Flask application that allows simple queries againts Vertilux Databases. Our API has predictable resource-oriented URLs, returns JSON-encoded responses, and uses standard HTTP response codes, right now it is only accesible withint our main servers.

Deployment

./acli api --deploy

ERP Web App

Vertilux ERP (Enterprise Resource Planning) is a web base application developed to facilitate our customers, vendors and employees information flow between all business functions. Our ERP provides an integrated and continuously updated view of core business processes.

Restart Sidekiq

We use Sidekiq as background job processing for our applications. Sidekiq uses threads to handle many jobs at the same time in the same process. It is integrated tightly with Rails to make background processing dead simple.

./acli erp --restart-sidekiq

Maintenance mode

Setting up the ERP web application into maintenance mode.

./acli erp --maintenance

Restart Phusion Passenger

Passenger is an application server that is in integrated with Nginx and supported our Ruby on Rails app.

./acli erp --restart-passenger

Restart Nginx

Nginx is our default web server and load balancer for the ERP infrastructure.

./acli erp --restart-nginx

Update DB IP address

./acli erp --update-ip