python3 [Ansible] #2019

jvonau · 2019-10-24T07:59:35Z

sample for #1810

jvonau · 2019-10-25T18:31:12Z

not a sample now

holta · 2019-10-25T21:41:10Z

This looks like it might be ready?

What OS's has this been tested on?

holta

Let's offer more clarifying explanation as to what ansible -m setup ... is doing?
Looks like a bug in runrole ?

runrole

iiab-support

jvonau · 2019-10-28T03:45:42Z

My intentition for '-m setup | grep python' to be an in your face feedback as to which python ansible is running while demo'ing/debugging ansible running under python3. Would also make a great question as to deduce the age of the user's install.

holta

@jvonau can we remove line 154 from iiab-install here?

It seems to be a duplicate of the line above it.

(Or if it serves a purpose, please explain, thanks.)

iiab-install

holta · 2019-10-30T22:37:45Z

FYI this cannot be merged as it fails every time when running ./iiab-install or ./runrole mysql on Ansible 2.9.0rc5 on Python 3:

Running local playbooks....Stage 0 will now run....followed by Stages 1-9
"iiab_branch": "jvonau-python3",
"ansible_python": {
"executable": "/usr/bin/python3",
"type": "cpython",
"ansible_python_version": "3.6.8",
"ansible_selinux_python_present": false,

...

TASK [mysql : Update MySQL root password for localhost root accounts, if mysql_enabled] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: (1698, "Access denied for user 'root'@'localhost'")"}

PLAY RECAP *********************************************************************
127.0.0.1 : ok=136 changed=29 unreachable=0 failed=1 skipped=53 rescued=0 ignored=0

In contrast, the same Ubuntu 18.04.3 VM with the same Ansible 2.9.0rc5 runs to completion when on Python 2.7 (i.e. iiab/iiab master branch).

Either way, ansible --version run all alone shows this output: (on Ubuntu 18.04.3 here)

root@box:~# ansible --version
ansible 2.9.0rc5
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.15+ (default, Oct 7 2019, 17:39:04) [GCC 7.4.0]

jvonau · 2019-10-30T23:15:55Z

Think you either forgot to manually install python3-pymysql or didn't test the other part of the PR that adds python3-pymysql to scripts/ansble.

holta · 2019-10-30T23:36:30Z

Think you either forgot to manually install python3-pymysql or didn't test the other part of the PR that adds python3-pymysql to scripts/ansble.

I did run scripts/ansible and FWIW python3-pymysql is installed, as confirmed on the Ubuntu 18.04.3 machine here: (10.8.0.30)

root@box:~# apt -a list python3-pymysql
Listing... Done
python3-pymysql/bionic,bionic,now 0.8.0-1 all [installed]

And yet ./runrole mysql fails...am not sure why...anything else I should check?

jvonau · 2019-10-31T00:38:33Z

Don't think ansible --version can be relied upon, on rpi:

ansible --version
ansible 2.8.5
config file = /opt/iiab/iiab/ansible.cfg
configured module search path = [u'/home/pi/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.16 (default, Apr 6 2019, 01:42:57) [GCC 8.2.0]

yet we have

pi@box:/opt/iiab/iiab $ sudo ./runrole mysql
"ansible_python": {
"executable": "/usr/bin/python3",
"type": "cpython",
"ansible_python_version": "3.7.3",
"ansible_selinux_python_present": false,
PLAY [all] ********************************************************************************************
TASK [Gathering Facts] ********************************************************************************

And continues to runs to completion.
I'll note that python-3.6.8 and python-2.7 are installed on 18.04.2, I'll bet that 3.7 python would let this work, the other alternative is just to let 18.04 use python-2.7 for ansible. I'll add that to this PR.

19.10 does work with python-3.7 out of the box once and python3-distutils and python3-pymysql is added with ansible version 2.8.3 supplied by Ubuntu so 2.9 should work there.

holta · 2019-10-31T01:56:01Z

This works on Ubuntu 19.10 with Ansible 2.9.0rc5 on Python 3.7.5rc1 for whatever reason:

root@box:/opt/iiab/iiab# ./runrole mysql
"iiab_branch": "jvonau-python3",
"ansible_python": {
"executable": "/usr/bin/python3",
"type": "cpython",
"ansible_python_version": "3.7.5rc1",
"ansible_selinux_python_present": false,

Context:

root@box:~# ansible --version
ansible 2.9.0rc5
config file = /opt/iiab/iiab/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.17rc1 (default, Oct 10 2019, 10:26:01) [GCC 9.2.1 20191008]

jvonau · 2019-10-31T02:18:44Z

I'm seeing a random flip in python usage with

pi@box:/opt/iiab/iiab $ sudo ./runrole mysql test.log
[WARNING]: Platform linux on host 127.0.0.1 is using the discovered Python
interpreter at /usr/bin/python, but future installation of another Python
interpreter could change this. See https://docs.ansible.com/ansible/2.8/referen
ce_appendices/interpreter_discovery.html for more information.
"ansible_python": {
"executable": "/usr/bin/python",
"ansible_python_version": "2.7.16",
"ansible_selinux_python_present": false,
"discovered_interpreter_python": "/usr/bin/python",

Can you edit ansible.cfg to be and test:
[defaults]
interpreter_python=/usr/bin/python3

I think -e on the cmdline can go away if the above works, 18.04 being the problem child at the moment.

holta · 2019-10-31T02:59:01Z

Uh-oh to clarify the above post from 1hr ago ./iiab-install --reinstall actually failed to complete on Ubuntu 19.10:

TASK [munin : Establish username/password Admin/changeme in /etc/munin/munin-htpasswd] ***
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'passlib'
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (passlib) on box.lan's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

PLAY RECAP *********************************************************************
127.0.0.1 : ok=268 changed=71 unreachable=0 failed=1 skipped=131 rescued=0 ignored=0

iiab-install

holta · 2019-10-31T03:19:16Z

Results from the new commit (aff014a) added to this PR ~40min ago:

./runrole mysql still fails on Ubuntu 18.04.3:

TASK [mysql : Update MySQL root password for localhost root accounts, if mysql_enabled] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: (1698, "Access denied for user 'root'@'localhost'")"}

PLAY RECAP *********************************************************************
127.0.0.1 : ok=30 changed=9 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
./iiab-install --reinstall still fails on Ubuntu 19.10:

TASK [munin : Establish username/password Admin/changeme in /etc/munin/munin-htpasswd] ***
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'passlib'
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (passlib) on box.lan's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

PLAY RECAP *********************************************************************
127.0.0.1 : ok=268 changed=70 unreachable=0 failed=1 skipped=131 rescued=0 ignored=0

jvonau · 2019-10-31T04:36:55Z

passlib is new.. I have python3-passlib installed

holta · 2019-10-31T05:15:06Z

Results from the new commit (7035bb3) added to this PR ~30min ago:

./runrole mysql still fails on Ubuntu 18.04.3:

TASK [mysql : Update MySQL root password for localhost root accounts, if mysql_enabled] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: (1698, "Access denied for user 'root'@'localhost'")"}
./iiab-install --reinstall now works on Ubuntu 19.10:

ASIDE: Should python3-passlib be added within roles/munin/tasks/main.yml#L1-L12 instead?

jvonau · 2019-10-31T10:01:23Z

Can you check if python*-mysqldb is installed please.
Installing python3-mysqldb gives the exact same error on rpi then requires the below 'use mysql.sock' commit to succeed on what was working prior to installing python3-mysqldb

ansible/ansible#47736

Suggests to fix Ubuntu
socket=/var/lib/mysql/mysql.sock in ~/.my.cnf
or login_unix_socket: /var/lib/mysql/mysql.sock

I've added a commit here to ease your testing but should not really be part of this PR.
I've added socket=/run/mysqld/mysqld.sock to the /root/.my.cnf template in the mysql role the above link mentions and would be required if python3-mysqldb is installed.

jvonau · 2019-10-31T10:33:46Z

   ASIDE: Should `python3-passlib` be added within [roles/munin/tasks/main.yml#L1-L12](https://github.com/iiab/iiab/blob/master/roles/munin/tasks/main.yml#L1-L12) instead?

No ansible is generating the traceback due to the missing library, looking to keep all ansible's dependencies in one place.

jvonau · 2019-10-31T11:29:06Z

Can you check if python*-mysqldb is installed please.
Installing python3-mysqldb gives the exact same error on rpi then requires the below commit to succeed on what was working prior to installing python3-mysqldb

Looks like the problem child has:

iiab-admin@box:~$ apt list python*-mysqldb
Listing... Done
python-mysqldb/bionic,now 1.3.10-1build1 amd64 [installed]
python3-mysqldb/bionic 1.3.10-1build1 amd64
apt list python*-pymysql
Listing... Done
python-pymysql/bionic,bionic 0.8.0-1 all
python3-pymysql/bionic,bionic,now 0.8.0-1 all [installed]

From the above link

I don't know where that was, but some time (weeks? months?) ago I looked into a similar problem, and the reason was that MySQLdb (which is based on libmysql) was using Unix sockets to connect by default, while PyMySQL was connecting via TCP to localhost by default. Depending on how root access to your MySQL server is configured, this might help you work around this problem.

I'd prefer one approach for the role..

roles/2-common/tasks/packages.yml

roles/mysql/tasks/main.yml

holta · 2019-10-31T14:33:38Z

FYI the very latest version of this PR still fails on Ubuntu 18.04.3:

TASK [mysql : Update MySQL root password for localhost root accounts, if mysql_enabled] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: (1698, "Access denied for user 'root'@'localhost'")"}

(Is this expected?) Output if it helps...

Running local playbooks....Stage 0 will now run....followed by Stages 1-9
"iiab_branch": "jvonau-python3",
"ansible_python": {
"executable": "/usr/bin/python3",
"type": "cpython",
"ansible_python_version": "3.6.8",
"ansible_selinux_python_present": false,

root@box:~# ansible --version
ansible 2.9.0rc5
config file = /opt/iiab/iiab/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.15+ (default, Oct 7 2019, 17:39:04) [GCC 7.4.0]

jvonau · 2019-10-31T15:39:07Z

In the past ansible's optional modules that need external libraries were intermixed with the roles' dependencies making finding all of ansible's full dependencies disjointed, but be much clearer now.

holta · 2019-10-31T16:13:47Z

In the past ansible's optional modules that need external libraries were intermixed with the roles' dependencies making finding all of ansible's full dependencies disjointed, but be much clearer now.

If a little heavier now forcing everyone to install all 7 packages {python3-distutils python3-mysqldb python3-passlib python3-pip python3-setuptools python3-virtualenv python3-psycopg2}. Making it harder to see which of the 7 above dependencies relates to which playbook (example: if we decide to remove munin altogether in future because nobody's using it, who will know that package python3-passlib can then be removed?) Likewise who can remember which of the above was really only needed for rarely used Moodle etc?

I'm now trying this PR as a fresh install on Debian 10.1 where Ansible 2.9.0rc5 is using Python 3.7.3 and running /opt/iiab/iiab/scripts/ansible is now slower of course, if we make these 7 packages mandatory (so be it, if that's the way we choose to go?)

PS This PR obviously doesn't modify https://github.com/iiab/iiab-admin-console/blob/master/install but are there any considerations there to keep in mind for when that happens? (Similar to this PR's 1-lin [optional?] mods to iiab-install, iiab-network, iiab-support, runrole ?)

PPS this PR has been tested with ./iiab-install --reinstall on Ubuntu 18.04 (BIG-sized) and Ubuntu 19.10 (MEDIUM-sized, currently being rebuilt as BIG-sized) and @jvonau tested an earlier version of this PR on Raspbian. If Debian 10.1 (BIG-sized) also works, are we good to go merging this?

holta · 2019-10-31T16:55:47Z

FYI a fresh install of this PR (#2019, BIG-sized) failed on Debian 10.1 as follows:

TASK [kalite : Install KA Lite static to /usr/local/kalite/venv, using pip (debuntu)] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": ["/usr/local/kalite/venv/bin/pip3", "install", "--no-cache-dir", "ka-lite-static==0.17.5"], "msg": "stdout: Already using interpreter /usr/bin/python3\nUsing base prefix '/usr'\nNew python executable in /usr/local/kalite/venv/bin/python3\nAlso creating executable in /usr/local/kalite/venv/bin/python\nInstalling setuptools, pkg_resources, pip, wheel...done.\nCollecting ka-lite-static==0.17.5\n Downloading https://files.pythonhosted.org/packages/86/7a/86603e8e993fdb2103474f4f6709c4d49cf22aa212304bf5d8b9a15368bd/ka-lite-static-0.17.5.tar.gz (41.2MB)\n\n:stderr: ERROR: Command errored out with exit status 1:\n command: /usr/local/kalite/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-nmz86khr/ka-lite-static/setup.py'"'"'; file='"'"'/tmp/pip-install-nmz86khr/ka-lite-static/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-nmz86khr/ka-lite-static/pip-egg-info\n cwd: /tmp/pip-install-nmz86khr/ka-lite-static/\n Complete output (7 lines):\n Traceback (most recent call last):\n File "", line 1, in \n File "/tmp/pip-install-nmz86khr/ka-lite-static/setup.py", line 17, in \n import kalite\n File "/tmp/pip-install-nmz86khr/ka-lite-static/kalite/init.py", line 4, in \n from version import *\n ModuleNotFoundError: No module named 'version'\n ----------------------------------------\nERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\n"}

Context upon re-running sudo iiab

Running local playbooks....Stage 0 will now run....followed by Stages 7-9
"iiab_branch": "python3",
"ansible_python": {
"executable": "/usr/bin/python3",
"type": "cpython",
"ansible_python_version": "3.7.3",
"ansible_selinux_python_present": false,

iiab-admin@deb10:~$ ansible --version
ansible 2.9.0rc5
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/iiab-admin/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.16 (default, Apr 6 2019, 01:42:57) [GCC 8.3.0]

iiab-admin@deb10:~$ more /etc/issue.net
Debian GNU/Linux 10

iiab-admin@deb10:~$ more /etc/debian_version
10.1

iiab-admin@deb10:~$ uname -a
Linux box.lan 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux

jvonau · 2019-10-31T17:20:47Z

In the past ansible's optional modules that need external libraries were intermixed with the roles' dependencies making finding all of ansible's full dependencies disjointed, but be much clearer now.

If a little heavier now forcing everyone to install all 7 packages {python3-distutils python3-mysqldb python3-passlib python3-pip python3-setuptools python3-virtualenv python3-psycopg2}. Making it harder to see which of the 7 above dependencies relates to which playbook (example: if we decide to remove munin altogether in future because nobody's using it, who will know that package python3-passlib can then be removed?) Likewise who can remember which of the above was really only needed for rarely used Moodle etc?

Moodle is rarely used but part of Medium so it's going to be installed anyway. Having ansible's needs intermixed with what is being installed for the role's needs doesn't really help. You can do your thing and add the notes in scripts/ansible.

I'm now trying this PR as a fresh install on Debian 10.1 where Ansible 2.9.0rc5 is using Python 3.7.3 and running /opt/iiab/iiab/scripts/ansible is now slower of course, if we make these 7 packages mandatory (so be it, if that's the way we choose to go?)

They would only end up being installed at a later point anyway, now we pay upfront instead of over time during the install. Looking to isolate python2 related actions for future porting/removal.

PS This PR obviously doesn't modify https://github.com/iiab/iiab-admin-console/blob/master/install but are there any considerations there to keep in mind for when that happens? (Similar to this PR's 1-lin [optional?] mods to iiab-install, iiab-network, iiab-support, runrole ?)

With no ansible.cfg file to enable exclusive use of python3 ansible will run python2 there, and should be fine as an upgrade because at somepoint all the dependencies are met. Fresh installs should be ok also as python-pip is being installed if needed in those plays.

jvonau · 2019-10-31T17:27:46Z

"/usr/local/kalite/venv/bin/pip3", "install", "--no-cache-dir", "ka-lite-static==0.17.5"], "msg": "stdout: Already using interpreter /usr/bin/python3
Command errored out with exit status 1:\n command: /usr/local/kalite/venv/bin/python3

That was sort of anticipated on what was a python3 only machine, the fix is in #2024 with the venv updates contained there.

holta mentioned this pull request Oct 24, 2019

Python 2 -> Python 3 IIAB strategic planning to prepare for 2020+ #1810

Closed

holta added this to the 7.1 milestone Oct 24, 2019

holta added enhancement question labels Oct 24, 2019

jvonau closed this Oct 25, 2019

jvonau force-pushed the python3 branch from af72f4d to 58b2e9d Compare October 25, 2019 18:15

jvonau added 2 commits October 25, 2019 13:21

add python3 debs

896186f

cherry-pick 57d7777

c2ca70b

jvonau reopened this Oct 25, 2019

holta changed the title ~~python3~~ python3 [Ansible] Oct 25, 2019

jvonau mentioned this pull request Oct 27, 2019

Ah pi gen holta/iiab#308

Closed

holta reviewed Oct 27, 2019

View reviewed changes

runrole Outdated Show resolved Hide resolved

iiab-support Outdated Show resolved Hide resolved

revisons

dfd2547

jvonau force-pushed the python3 branch from 438683f to dfd2547 Compare October 28, 2019 03:38

holta reviewed Oct 30, 2019

View reviewed changes

iiab-install Outdated Show resolved Hide resolved

drop -e and just use ansible.cfg

aff014a

holta reviewed Oct 31, 2019

View reviewed changes

iiab-install Show resolved Hide resolved

add python3-passlib

7035bb3

jvonau added 4 commits October 31, 2019 07:34

use mysql.sock

7fe2e22

don't install python-passlib

37beab1

use python3-mysqldb in place of python3-pymysql

199d1f3

use installed python3-mysqldb in place of python-mysqldb

890e955

jvonau force-pushed the python3 branch from f213151 to 890e955 Compare October 31, 2019 12:35

add python3 versions of pip, setuptools, virtualenv

7bf4e94

holta reviewed Oct 31, 2019

View reviewed changes

roles/2-common/tasks/packages.yml Outdated Show resolved Hide resolved

holta reviewed Oct 31, 2019

View reviewed changes

roles/mysql/tasks/main.yml Outdated Show resolved Hide resolved

requested changes

e3d0d45

jvonau added 2 commits October 31, 2019 09:48

supply my.cnf file before accessing DBs

d68644b

python-psycopg2 -> python3-psycopg2

3179a53

jvonau mentioned this pull request Oct 31, 2019

Python3-2 [Ansible] #2024

Merged

holta merged commit 3179a53 into iiab:master Oct 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3 [Ansible] #2019

python3 [Ansible] #2019

jvonau commented Oct 24, 2019 •

edited

jvonau commented Oct 25, 2019

holta commented Oct 25, 2019

holta left a comment •

edited

jvonau commented Oct 28, 2019

holta left a comment

holta commented Oct 30, 2019

jvonau commented Oct 30, 2019 •

edited

holta commented Oct 30, 2019 •

edited

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019 •

edited

holta commented Oct 31, 2019

holta commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019

holta commented Oct 31, 2019

jvonau commented Oct 31, 2019

jvonau commented Oct 31, 2019

python3 [Ansible] #2019

python3 [Ansible] #2019

Conversation

jvonau commented Oct 24, 2019 • edited

jvonau commented Oct 25, 2019

holta commented Oct 25, 2019

holta left a comment • edited

Choose a reason for hiding this comment

jvonau commented Oct 28, 2019

holta left a comment

Choose a reason for hiding this comment

holta commented Oct 30, 2019

jvonau commented Oct 30, 2019 • edited

holta commented Oct 30, 2019 • edited

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019 • edited

jvonau commented Oct 31, 2019 • edited

holta commented Oct 31, 2019

holta commented Oct 31, 2019 • edited

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019 • edited

jvonau commented Oct 31, 2019 • edited

jvonau commented Oct 31, 2019

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019

jvonau commented Oct 31, 2019

holta commented Oct 31, 2019

holta commented Oct 31, 2019

jvonau commented Oct 31, 2019

jvonau commented Oct 31, 2019

jvonau commented Oct 24, 2019 •

edited

holta left a comment •

edited

jvonau commented Oct 30, 2019 •

edited

holta commented Oct 30, 2019 •

edited

holta commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019 •

edited

holta commented Oct 31, 2019 •

edited

holta commented Oct 31, 2019 •

edited

jvonau commented Oct 31, 2019 •

edited