Merge pull request #15 from osanchez42/collection

Refactored into the Ansible Collections format.
device42 · Apr 16, 2020 · 2afdede · 2afdede
2 parents 4953ff5 + cdf248b
commit 2afdede
Show file tree

Hide file tree

Showing 13 changed files with 583 additions and 46 deletions.
diff --git a/README.md b/README.md
@@ -1,37 +1,176 @@
 [Device42](http://www.device42.com/) is a Continuous Discovery software for your IT Infrastructure. It helps you automatically maintain an up-to-date inventory of your physical, virtual, and cloud servers and containers, network components, software/services/applications, and their inter-relationships and inter-dependencies.
 
-This repository has 2 different scripts that you can use with ansible.
-1. `d42_ansible_inventory_hostfile.py` can be used to create a static inventory file for ansible. You can group hosts by tags, customers, building or service level from Device42 data.
-2. `d42_ansible_dynamic_inventory.py` can be used to dynamically by ansible to get hosts from Device42 based on certain filters.
+This Ansible Collection contains the inventory and lookup plugins for Device42. 
+1. Inventory (device42.d42.d42) - can be used to dynamically generate Ansible inventory from Device42 devices.
+2. Lookup (device42.d42.d42)- This lookup plugin allows a playbook to fetch passwords or ip addresses for hosts from Device42.
+3. Lookup (device42.d42.d42_prompt) - This lookup plugin allows a playbook to fetch passwords or ip addresses for hosts from Device42 with prompt. 
 
-## Assumptions
------------------------------
-    * This script works with Device42 12.0.0 and above
+In the `contrib` directory you will find the legacy inventory scripts. Please favor the plugin over the legacy scripts:
+1. `contrib\inventory\d42_ansible_inventory_hostfile.py` can be used to create a static inventory file for ansible. You can group hosts by tags, customers, building or service level from Device42 data.
+2. `contrib\inventory\d42_ansible_dynamic_inventory.py` can be used to dynamically by ansible to get hosts from Device42 based on certain filters.
 
+-----------------------------
+## Assumptions
+- This script works with Device42 12.0.0 and above
 ### Requirements
+- ansible 2.9+
+- python 3.6.x+
+- requests (you can install it with pip install requests or apt-get install python-requests)
+- Ansible must have an available connection to your Device42 instance in order to collect devices for inventory
+
+### Installation Methods
+
+#### Galaxy 
+```bash
+ansible-galaxy collection install device42.d42
+```
+
+#### Automation Hub
+To consume content from hub as part of your automation workflows the content can also be accessed via CLI. 
+For this an offline token is required which can be obtained via the web UI at [automation hub](https://cloud.redhat.com/ansible/automation-hub/token), 
+and needs to be added to the configuration file as follows:
+
+```bash
+[galaxy]
+server_list = automation_hub, galaxy
+
+[galaxy_server.automation_hub]
+url=https://cloud.redhat.com/api/automation-hub/
+auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token
+token=AABBccddeeff112233gghh...
+
+[galaxy_server.galaxy]
+url=https://galaxy.ansible.com/
+```
+
+Once configured the collection can be installed by running the following command
+```bash
+$ ansible-galaxy collection install device42.d42
+```
+
+#### Ansible Tower:
+* Create a collections requirement file in the SCM 
+```bash
+touch requirements.yml
+```
+
+* add the following to your file and save it
+```bash
+collections:
+- name: device42.d42
+version: 1.0.0
+source: https://galaxy.ansible.com/device42/d42
+```
+
+* Install the Ansible collection by running command
+```bash
+ansible-galaxy collection install -r requirements.yml -p <job tmp location>
+```
+Starting with Ansible Tower 3.6, the project folder will be copied for each job run. This allows playbooks to make 
+local changes to the source tree for convenience, such as creating temporary files, 
+without the possibility of interference with other jobs.
+
+* Set the following environmental variables (optional)
+```
+D42_USER = 'device42 user'
+D42_PWD = 'device42 password'
+D42_URL = 'https:// device42 server IP address'
+D42_SKIP_SSL_CHECK = False
+```
+
+For more information on installing collections please follow documentation here https://docs.ansible.com/ansible/latest/user_guide/collections_using.html
+
 -----------------------------
-    * python 2.7.x
-    * requests (you can install it with pip install requests or apt-get install python-requests)
+## Inventory Plugin
+
+### Configuration
+Define an inventory file (`*.d42.yml`) 
+
+View documentation using `ansible-doc -t inventory device42.d42.d42`
+
+Environmental variables not defined:
+```
+plugin: device42.d42.d42
+url: https://10.10.10.10
+username: admin
+password: password
+ssl_check: False
+keyed_groups:
+    - key: d42_service_level
+      prefix: ''
+      separator: ''
+```
+
+Environmental variables defined:
+```
+plugin: device42.d42.d42
+keyed_groups:
+    - key: d42_service_level
+      prefix: ''
+      separator: ''
+```
+See [Ansible documentation](https://docs.ansible.com/ansible/latest/plugins/inventory/constructed.html) for more constructed examples.
+
+### How to run
+from the directory of your newly created file run the following command.
+
+```bash
+ansible-inventory -i *.d42.yml --graph
+```
 
-### Usage
 -----------------------------
+## Lookup Plugin
 
-    * rename conf.sample to conf
-    * in conf add D42 URL/credentials ( also instead of conf file, possible to use environment variables )
+### Configuration
+
+If using environmental variables skip this step, if not create a d42.py or d42_prompt.py in ansible/lib/ansible/plugins/lookup/
+with the following information
 ```
-# ====== Device42 upload settings ========= #
 D42_USER = 'device42 user'
 D42_PWD = 'device42 password'
 D42_URL = 'https:// device42 server IP address'
-D42_SKIP_SSL_CHECK = True
+D42_SKIP_SSL_CHECK = False
+```
+
+### How to run
+```
+To get password call: lookup('d42', 'device_name', 'password', 'username')
+```
+device_name and username need to be filled in
+```
+To run any doql request: lookup('d42', 'SELECT ... FROM ...', 'doql', 'list')
+```
+doql query need to be filled in + we need to set data type of returned result ( 'string', 'list', 'list_dicts' )
+* string - return single string without column headers
+* list - return list of string ( we split DOQL result line by line without column headers )
+* list_dicts - return list of dicts with column headers
+
+All above works the same for the `prompt` version, we just add 3 more arguments in the yaml file, please check reference in promt example.
+
+The following was tested in a playbook using the included example template `example_playbook.yaml`
+playbooks can be run by using the following command
+
+```bash
+ansible-playbook *.yaml -f 10
 ```
+## Legacy Inventory Usage
+-----------------------------
 
-    * in conf add DOQL group settings
+    * rename conf.sample.ini to conf.ini
+    * in conf add D42 URL/credentials ( also instead of conf file, possible to use environment variables )
 ```
-# ====== Ansible settings ========= #
-GROUP_BY_QUERY = 'select name, service_level from view_device_v1' # DOQL QUERY, POSSIBLE TO GROUP BY ANY FIELD
-GROUP_BY_FIELD = 'service_level' # GROUP BY FIELD
-GROUP_BY_REFERENCE_FIELD = 'name' # FIELD THAT COMES AS REFERENCE NAME
+# ====== Device42 upload settings =========
+[DEFAULT]
+D42_USER = admin
+D42_PWD = adm!nd42
+D42_URL = https://10.10.10.10
+D42_SKIP_SSL_CHECK = True
+
+# ====== Ansible settings =========
+[DOQL]
+GROUP_BY_QUERY = select name, service_level from view_device_v1
+GROUP_BY_FIELD = service_level
+GROUP_BY_REFERENCE_FIELD = name
 SPLIT_GROUP_BY_COMMA = False
 ```
 
@@ -58,7 +197,3 @@ and much more! [Please read Ansible docs.](https://ansible-tips-and-tricks.readt
 If you have any questions - feel free to reach out to us at support at device42.com
 
 
-
-### Compatibility
------------------------------
-    * Script runs on Linux and Windows
diff --git a/conf.sample b/conf.sample
diff --git a/contrib/inventory/conf.sample.ini b/contrib/inventory/conf.sample.ini
@@ -0,0 +1,13 @@
+# ====== Device42 upload settings =========
+[DEFAULT]
+D42_USER = admin
+D42_PWD = adm!nd42
+D42_URL = https://10.10.10.10
+D42_SKIP_SSL_CHECK = True
+
+# ====== Ansible settings =========
+[DOQL]
+GROUP_BY_QUERY = select name, service_level from view_device_v1
+GROUP_BY_FIELD = service_level
+GROUP_BY_REFERENCE_FIELD = name
+SPLIT_GROUP_BY_COMMA = False
diff --git a/d42_ansible_dynamic_inventory.py → ...nventory/d42_ansible_dynamic_inventory.py b/d42_ansible_dynamic_inventory.py → ...nventory/d42_ansible_dynamic_inventory.py
@@ -4,6 +4,8 @@
 import sys
 from lib import *
 
+from contrib.inventory.lib import get_conf
+
 try:
     import json
 except ImportError:

diff --git a/d42_ansible_inventory_hostfile.py → ...ventory/d42_ansible_inventory_hostfile.py b/d42_ansible_inventory_hostfile.py → ...ventory/d42_ansible_inventory_hostfile.py
@@ -3,14 +3,16 @@
 import requests
 from lib import *
 
+from contrib.inventory.lib import get_conf
+
 if __name__ == '__main__':
     conf = get_conf()
     ansible = Ansible(conf)
     groups = ansible.get_grouping(Device42(conf).doql())
 
     if ansible.write_inventory_file(groups) is True:
-        print '[!] Done!'
+        print('[!] Done!')
     else:
-        print '[!] Can\'t write to file'
+        print('[!] Can\'t write to file')
 
     sys.exit()
diff --git a/lib.py → contrib/inventory/lib.py b/lib.py → contrib/inventory/lib.py
@@ -1,47 +1,69 @@
-from StringIO import StringIO
+from io import StringIO
 import requests
 import codecs
 import base64
-import imp
+import configparser
 import csv
 import sys
 import os
+
+
 try:
     import json
 except ImportError:
     import simplejson as json
 
+
 def get_conf():
+
+    try:
+        conf_file = configparser.ConfigParser()
+        conf_file.read('conf.ini')
+    except Exception:
+        print('Failed to read config file, have you changed the config file name to conf.ini?')
+        sys.exit(1)
+
     try:
-        conf = imp.load_source('conf', 'conf').__dict__
-        if 'D42_SKIP_SSL_CHECK' in conf and conf['D42_SKIP_SSL_CHECK'] == True:
+        if conf_file.get('DEFAULT', 'D42_SKIP_SSL_CHECK') == 'True':
             requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)
-    except:
+
+        conf = {
+            'D42_URL': conf_file.get('DEFAULT', 'D42_URL'),
+            'D42_USER': conf_file.get('DEFAULT', 'D42_USER'),
+            'D42_PWD': conf_file.get('DEFAULT', 'D42_PWD'),
+            'GROUP_BY_QUERY': conf_file.get('DOQL', 'GROUP_BY_QUERY'),
+            'GROUP_BY_FIELD': conf_file.get('DOQL', 'GROUP_BY_FIELD'),
+            'GROUP_BY_REFERENCE_FIELD': conf_file.get('DOQL', 'GROUP_BY_REFERENCE_FIELD'),
+            'SPLIT_GROUP_BY_COMMA': conf_file.get('DOQL', 'SPLIT_GROUP_BY_COMMA')
+        }
+
+    except Exception as e:
+        print(e)
         if 'D42_SKIP_SSL_CHECK' in os.environ and os.environ['D42_SKIP_SSL_CHECK'] == 'True':
             requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)
 
         if 'D42_URL' not in os.environ:
-            print 'Please set D42_URL environ.'
+            print('Please set D42_URL environ.')
             sys.exit()
 
         if 'D42_USER' not in os.environ:
-            print 'Please set D42_USER environ.'
+            print('Please set D42_USER environ.')
             sys.exit()
 
         if 'D42_PWD' not in os.environ:
-            print 'Please set D42_PWD environ.'
+            print('Please set D42_PWD environ.')
             sys.exit()
 
         if 'GROUP_BY_QUERY' not in os.environ:
-            print 'Please set GROUP_BY_QUERY environ.'
+            print('Please set GROUP_BY_QUERY environ.')
             sys.exit()
 
         if 'GROUP_BY_FIELD' not in os.environ:
-            print 'Please set GROUP_BY_FIELD environ.'
+            print('Please set GROUP_BY_FIELD environ.')
             sys.exit()
 
         if 'GROUP_BY_REFERENCE_FIELD' not in os.environ:
-            print 'Please set GROUP_BY_REFERENCE_FIELD environ.'
+            print('Please set GROUP_BY_REFERENCE_FIELD environ.')
             sys.exit()
 
         conf = {
@@ -110,7 +132,7 @@ def get_grouping(self, objects):
                         groups[object_[self.conf['GROUP_BY_FIELD']]] = []
                     groups[object_[self.conf['GROUP_BY_FIELD']]].append(object_[self.conf['GROUP_BY_REFERENCE_FIELD']])
             except Exception as e:
-                print object_
+                print(object_)
                 sys.exit()
         return groups
 
@@ -128,3 +150,6 @@ def write_inventory_file(groups):
         f.close()
 
         return True
+
+if __name__ == "__main__":
+    print(get_conf())
diff --git a/galaxy.yml b/galaxy.yml
@@ -0,0 +1,15 @@
+namespace: "device42"
+name: "d42"
+version: "1.0.1"
+readme: "README.md"
+authors:
+    - "Will Tome (@willtome)"
+    - "Anatolii (@achmykhalo)"
+    - "Omar Sanchez (@Osanchez42)"
+license:
+    - "Apache-2.0"
+tags:
+    - device42
+    - integration
+    - collection
+repository: "https://github.com/device42/ansible_device42"