module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… #47809

timorunge · 2018-10-30T13:34:23Z

…n from MySQLdb to PyMySQL

SUMMARY

With Ansible 2.7.1 and the migration from MySQLdb to PyMySQL all my MySQL related tasks failed since the call db_connection.cursor(cursorclass=mysql_driver.cursors.DictCursor) is trying to trigger https://github.com/PyMySQL/PyMySQL/blob/v0.9.2/pymysql/connections.py#L483 which has no argument cursorclass.

<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616 `" && echo ansible-tmp-1540904667.96-27683954372616="` echo /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616 `" ) && sleep 0'
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/database/proxysql/proxysql_backend_servers.py
<localhost> PUT /root/.ansible/tmp/ansible-local-4588xiOIZd/tmpPPDmmb TO /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/ /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1540904667.96-27683954372616/AnsiballZ_proxysql_backend_servers.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_proxysql_backend_servers_payload_3rvJlX/__main__.py", line 504, in <module>
  File "/tmp/ansible_proxysql_backend_servers_payload_3rvJlX/__main__.py", line 450, in main
  File "/tmp/ansible_proxysql_backend_servers_payload_3rvJlX/ansible_proxysql_backend_servers_payload.zip/ansible/module_utils/mysql.py", line 81, in mysql_connect
TypeError: cursor() got an unexpected keyword argument 'cursorclass'

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

module_utils/mysql.py

ANSIBLE VERSION

ansible --version
ansible 2.7.1
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]

ansibot · 2018-10-30T13:38:48Z

Hi @timorunge, thank you for submitting this pull-request!

click here for bot help

esamattis · 2018-11-29T13:14:11Z

@gundalow Oh, sorry I didn't notice this PR earlier. But unfortunaly no. I'm getting the same error as in #47736

Not familiar how Ansible dev versions are best installed but I installed it manually into a Python 3 virtual env using the setup.py file. Directly from this commit https://github.com/timorunge/ansible/commit/1b4c66a8adf595139674fbdaef1575ba41fbc80e

ansible --version
ansible 2.8.0.dev0
  config file = /Users/esamatti/code/playbooks/ansible.cfg
  configured module search path = ['/Users/esamatti/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/esamatti/Envs/ansibleenv/lib/python3.7/site-packages/ansible-2.8.0.dev0-py3.7.egg/ansible
  executable location = /Users/esamatti/Envs/ansibleenv/bin/ansible
  python version = 3.7.0 (default, Oct  2 2018, 09:18:58) [Clang 10.0.0 (clang-1000.11.45.2)]

anniemelen · 2018-11-30T13:33:00Z

Applying patch from #47809 (https://patch-diff.githubusercontent.com/raw/ansible/ansible/pull/47809.patch) not worked for me.

ansible 2.7.1
  config file = /etc/ansible/ansible.cfg
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

fatal: [node03]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "append_privs": false,
            "check_implicit_admin": false,
            "config_file": "/root/.my.cnf",
            "connect_timeout": 30,
            "encrypted": false,
            "host": "localhost",
            "host_all": true,
            "login_host": "localhost",
            "login_password": "",
            "login_port": 3306,
            "login_unix_socket": null,
            "login_user": "root",
            "name": "root",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "priv": "*.*:ALL,GRANT",
            "sql_log_bin": true,
            "ssl_ca": null,
            "ssl_cert": null,
            "ssl_key": null,
            "state": "present",
            "update_password": "always",
            "user": "root"
        }
    },
    "msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: (1698, u\"Access denied for user 'root'@'localhost'\")"
}
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_mysql_user_payload_6FXkHz/__main__.py", line 588, in main
    connect_timeout=connect_timeout)
  File "/tmp/ansible_mysql_user_payload_6FXkHz/ansible_mysql_user_payload.zip/ansible/module_utils/mysql.py", line 78, in mysql_connect
    db_connection = mysql_driver.connect(**config)
  File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 699, in __init__
    self.connect()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 936, in connect
    self._request_authentication()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1156, in _request_authentication
    auth_packet = self._read_packet()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1018, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 384, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)

timorunge · 2018-11-30T13:48:56Z

@epeli: Can you send me some verbose output?

@anniemelen: If this is the actual output I think your problem is now different to the one which is described in this PR:
"msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: (1698, u\"Access denied for user 'root'@'localhost'\")"

anniemelen · 2018-11-30T13:54:19Z

@timorunge Yes, it's actual and related to #47736.

timorunge · 2018-11-30T13:58:03Z

@anniemelen: Sorry, didn't notice that. Can you send me some verbose output?

anniemelen · 2018-11-30T14:13:00Z

@timorunge Unfortunately, this is all. Other information in '-vvvvv' mode is related to establishing SSH connection.
I can provide module list which Ansible using during a task:

Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/mysql.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/database.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/basic.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/_text.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/six/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/parsing/convert_bool.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/common/_collections_compat.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/common/file.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/parsing/__init__.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/pycompat24.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/common/process.py
Using module_utils file /usr/lib/python2.7/dist-packages/ansible/module_utils/common/__init__.py
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/database/mysql/mysql_user.py

Task:

   - name: Setting Root Password
      mysql_user:
        login_user: '{{ mysql_db_user }}'
        login_password: ''
        login_host: 'localhost'
        name: '{{ mysql_db_user }}'
        password: '{{ mysql_db_pass }}'
        priv: '*.*:ALL,GRANT'
        host_all: yes
        state: present

esamattis · 2018-11-30T15:20:12Z

@timorunge Here's the verbose output with -vvvv

https://gist.github.com/epeli/cad6e4c7b4f380816c63f164b865eaa7

timorunge · 2018-11-30T16:38:02Z

@anniemelen, @epeli: I've rechecked everything on my side locally - before applying the patch and afterwards. Everything is working fine for me in a similar setup (just testing the MySQL connection itself):

python 2.7:

# ansible --version
ansible 2.7.1
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]

python 3.6:

# ansible --version
ansible 2.7.1
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]

Since both of you are facing problems with setting a root password for MySQL itself:

Can you connect via command line without a password defined and without the config file?
Can you connect via command line without a password defined and with a defined config file?
Have you tried to disable check_implicit_admin: yes?
Have you tried to add / change another existing user with Ansible?

esamattis · 2018-11-30T17:34:24Z

Can you connect via command line without a password defined and without the config file?

Actually yes. The intention was to require one... I must re-check this on monday.

Can you connect via command line without a password defined and with a defined config file?

Yes.

I'll get back to the other questions on monday as well.

anniemelen · 2018-12-04T14:17:54Z

Can you connect via command line without a password defined and without the config file?

Yes.

Can you connect via command line without a password defined and with a defined config file?

Yes.

Have you tried to disable check_implicit_admin: yes?

As you can see above, it's already disabled. #47809 (comment)

Have you tried to add / change another existing user with Ansible?

Yes, and it works. But the point of task is to initiate standard mysql_secure_installation procedure and set root password when it's not set by default.
I should mention, that task was succeded with Ansible 2.4.4.

timorunge · 2018-12-04T18:57:22Z

@anniemelen: If adding different mysql users is possible with this patch / not failing with the message which is described in this ticket the issues are not related.

You're also mentioning Ansible 2.4.4 (here) and 2.7.0 (in 47736). If you can provide a Dockerfile to re-produce this issue I'm happy to help you further but I think this PR is the wrong place for this. :)

cahlchang · 2019-01-28T08:47:15Z

This fix was very helpful for me.
Now that part of the mysql driver is broken, I want to agree to capture this PR.
So many things go well.

Xyon · 2019-02-11T21:23:57Z

Code LGTM - I couldn't reproduce the initial issue in the first place, so can't test the fix here, but the logic seems reasonable. shipit

dagwieers

LGTM, did not test.

bmalynovytch · 2019-02-15T08:08:00Z

+label shipit

ansibot · 2019-02-15T08:11:49Z

cc @dagwieers @michaelcoburn @Oneiroi @tolland
click here for bot help

dagwieers · 2019-02-15T09:37:45Z

bot_status

gundalow · 2019-02-15T15:37:58Z

Ansibullbot is current taking a nap.

Thanks to everybody that's help with this, merged.

gundalow · 2019-02-15T15:38:37Z

If you'd like this fixed in Ansible 2.7 feel free to raise a backport PR.

dsjagoda · 2019-03-07T20:28:15Z

Github newbie trying to understand what's going on here. Is this fix going to make it into 2.7.9 (if there is one) and/or 2.8? I applied the code to my Ansible installation and it did fix the error I was receiving.

felixfontein · 2019-03-07T20:35:00Z

@dsjagoda it will definitely be in 2.8; it will only be in 2.7.9 (or later) if someone creates a backport PR for this. I don't know how active the mysql WG is, can someone from them comment on this?

timorunge · 2019-03-07T20:38:03Z

It should go along with #53445 (which needs to be merged).

bmalynovytch · 2019-03-07T21:08:13Z

@felixfontein I seem to be the only one for now

felixfontein · 2019-03-07T21:15:50Z

@timorunge are you interested in becoming part of the mysql WG? (https://github.com/ansible/community/wiki/MySQL)

timorunge · 2019-03-07T21:27:36Z

If nobody is taking it, I took the 5 minutes: #53483 :)

timorunge · 2019-03-07T21:48:45Z

@felixfontein I rarely have time (or basically just some spare time).
I can help if required but I see that we already have 5 people in the MySQL WG. :)

felixfontein · 2019-03-07T22:01:23Z

@timorunge that's a known problem, I guess most people here have it ;) Also, the number of people in a WG doesn't say that much, it depends on how many people are actively doing something. It looks that for the mysql WG, it's mainly @bmalynovytch. So you could also become another idling member ;) Being in the WG also has the advantage that you're pinged for all MySQL-related PRs and issues. If you can comment from time to time, or help reproducing something or can test PRs whether they do as promised, that would probably already be a great help.

If you don't want to spend that time on this, I understand, I was just asking since you seem to be somewhat interested :)

timorunge · 2019-03-08T10:57:34Z

New Backport PR: #53519 (in combination with #53445).

…error after migratio… (#47809) * Fixing unexpected keyword argument 'cursorclass' error after migration from MySQLdb to PyMySQL * Adoptions for mysql.py as suggested by felixfontein. * Adding changelog fragment. (cherry picked from commit 4719008)

jborean93 removed the needs_triage Needs a first human triage before being processed. label Nov 1, 2018

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Nov 9, 2018

gundalow changed the title ~~Fixing unexpected keyword argument 'cursorclass' error after migratio…~~ module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… Nov 23, 2018

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Nov 23, 2018

This was referenced Nov 27, 2018

AttributeError: 'module' object has no attribute 'cursors' bmildren/proxysql_ansible_modules#2

Closed

Module ProxySQL - Backend_servers error #49191

Closed

gundalow mentioned this pull request Nov 29, 2018

mysql_user broken in 2.7.1 when using /root/.my.cnf #47736

Closed

This comment has been minimized.

Sign in to view

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Dec 4, 2018

timorunge mentioned this pull request Feb 14, 2019

MySQL community pinboard ansible/community#439

Closed

dagwieers approved these changes Feb 14, 2019

View reviewed changes

ansibot added shipit This PR is ready to be merged by Core and removed community_review In order to be merged, this PR must follow the community review workflow. stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Feb 15, 2019

gundalow merged commit 4719008 into ansible:devel Feb 15, 2019

timorunge mentioned this pull request Mar 7, 2019

Backport PR to Ansible 2.7 for #47809 & #53445 #53483

Closed

This was referenced Mar 8, 2019

Backport PR 47809 & 53445 #53519

Merged

Fix for "AttributeError: 'module' object has no attribute 'cursors'" (#49191) #53445

Merged

ansible locked and limited conversation to collaborators Jul 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… #47809

module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… #47809

timorunge commented Oct 30, 2018 •

edited

ansibot commented Oct 30, 2018

This comment has been minimized.

This comment has been minimized.

esamattis commented Nov 29, 2018

anniemelen commented Nov 30, 2018

timorunge commented Nov 30, 2018

anniemelen commented Nov 30, 2018

timorunge commented Nov 30, 2018

anniemelen commented Nov 30, 2018

esamattis commented Nov 30, 2018

timorunge commented Nov 30, 2018

esamattis commented Nov 30, 2018

anniemelen commented Dec 4, 2018

timorunge commented Dec 4, 2018

cahlchang commented Jan 28, 2019 •

edited

Xyon commented Feb 11, 2019

dagwieers left a comment •

edited

bmalynovytch commented Feb 15, 2019

ansibot commented Feb 15, 2019

dagwieers commented Feb 15, 2019

gundalow commented Feb 15, 2019

gundalow commented Feb 15, 2019

dsjagoda commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 7, 2019

bmalynovytch commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 7, 2019

timorunge commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 8, 2019

module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… #47809

module_utils/mysql: Fixing unexpected keyword argument 'cursorclass' error after migratio… #47809

Conversation

timorunge commented Oct 30, 2018 • edited

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

ansibot commented Oct 30, 2018

This comment has been minimized.

This comment has been minimized.

esamattis commented Nov 29, 2018

anniemelen commented Nov 30, 2018

timorunge commented Nov 30, 2018

anniemelen commented Nov 30, 2018

timorunge commented Nov 30, 2018

anniemelen commented Nov 30, 2018

esamattis commented Nov 30, 2018

timorunge commented Nov 30, 2018

esamattis commented Nov 30, 2018

anniemelen commented Dec 4, 2018

timorunge commented Dec 4, 2018

cahlchang commented Jan 28, 2019 • edited

Xyon commented Feb 11, 2019

dagwieers left a comment • edited

Choose a reason for hiding this comment

bmalynovytch commented Feb 15, 2019

ansibot commented Feb 15, 2019

dagwieers commented Feb 15, 2019

gundalow commented Feb 15, 2019

gundalow commented Feb 15, 2019

dsjagoda commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 7, 2019

bmalynovytch commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 7, 2019

timorunge commented Mar 7, 2019

felixfontein commented Mar 7, 2019

timorunge commented Mar 8, 2019

timorunge commented Oct 30, 2018 •

edited

cahlchang commented Jan 28, 2019 •

edited

dagwieers left a comment •

edited