Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

helix-loop · 2021-11-14T20:20:59Z

Environment

How did you install GRR? [e.g. from release DEB, HEAD DEB, PIP package, source]
PIP package/source
What GRR version are you running?: [e.g. 3.1.2.3]
3.4.5.1
What operating system does the GRR server run on? [e.g. Ubuntu 18.04]
Debian 11 (Bullseye) (ppc64le)
What operating system does the affected GRR client run on, if applicable? [e.g. Windows 10]

Describe the issue
Trying to install GRR Server from pip packages or via source, I am not able to install fleetspeak-client-bin version 0.1.11 on a ppc64le architecture.

Steps to reproduce:
pip3 install fleetspeak-client-bin

Error logs

ERROR: Could not find a version that satisfies the requirement fleetspeak-client-bin
ERROR: No matching distribution found for fleetspeak-client-bin

Additional context
There is a working (old) version of GRR Server (ppc64el) at https://packages.ubuntu.com/bionic/grr-server for Ubuntu 18.04.

The text was updated successfully, but these errors were encountered:

mbushkov · 2021-11-15T23:51:50Z

Quick question: do you plan to run the clients on ppc64le as well?

fleetspeak-client-bin contains precompiled Fleetspeak binaries for various platforms and is needed to build and run latest generation of GRR clients. Unfortunately, we don't distribute the ppc64le Fleetspeak binaries in the fleetspeak-client-bin package.

At the moment, in order to install GRR and then build client templates for ppc64le, you'd need to effectively build and install your own ppc64le-specific fleetspeak-client-bin package version. To do that, you'd essentially need to follow the steps done by Fleetspeak CI.

Note that even if you do that, you'd still need to build the custom GRR client template for pcc64le too: https://grr-doc.readthedocs.io/en/latest/maintaining-and-tuning/building-custom-client-templates.html

helix-loop · 2021-11-21T21:32:51Z

Answer: I do plan to run the clients on ppc64le as well.

I think I was able to follow the steps done by Fleetspeak CI and created fleetspeak-client_0.1.11_ppc64el.deb and fleetspeak-server_0.1.11_ppc64el.deb. I was able to install them on the server, however following https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html#installing-python-3-grr-version-from-source I run into a few errors with dependencies (fleetspeak-client-bin and fleetspeak-server-bin) listed in several setup.py. I commented them out one by one and got further. I am now getting the following errors:

Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-x34f4rh2
       cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
  Complete output (81 lines):
  /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
    warnings.warn(
  /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
    warnings.warn(
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-ppc64le-3.9
  copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
  creating build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
  creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
  running build_ext
  building '_mysql' extension
  creating build/temp.linux-ppc64le-3.9
  powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
  In file included from _mysql.c:29:
  /usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
      3 | #warning This file should not be included by clients, include only <mysql.h>
        |  ^~~~~~~
  _mysql.c: In function ‘_mysql_server_init’:
  _mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    266 |    s = PyUnicode_AsUTF8(item);
        |      ^
  _mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    296 |    s = PyUnicode_AsUTF8(item);
        |      ^
  _mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
    588 |   _stringsuck(ca, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
    589 |   _stringsuck(capath, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
    590 |   _stringsuck(cert, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
    591 |   _stringsuck(key, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
    577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
        |                ^
  _mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
    592 |   _stringsuck(cipher, value, ssl);
        |   ^~~~~~~~~~~
  _mysql.c: In function ‘_mysql_ConnectionObject_ping’:
  _mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
   1911 |  if ( reconnect != -1 ) self->connection.reconnect = reconnect;
        |                                         ^
  _mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
  _mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
   2603 |  cname = PyUnicode_AsUTF8(name);
        |        ^
  error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for mysqlclient
  Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient, grr-response-server
  Attempting uninstall: mysqlclient
    Found existing installation: mysqlclient 2.1.0
    Uninstalling mysqlclient-2.1.0:
      Successfully uninstalled mysqlclient-2.1.0
    Running setup.py install for mysqlclient ... error
    ERROR: Command errored out with exit status 1:
     command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient
         cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
    Complete output (83 lines):
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
      warnings.warn(
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
      warnings.warn(
    running install
    /path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.linux-ppc64le-3.9
    copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
    creating build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
    creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
    running build_ext
    building '_mysql' extension
    creating build/temp.linux-ppc64le-3.9
    powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
    In file included from _mysql.c:29:
    /usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
        3 | #warning This file should not be included by clients, include only <mysql.h>
          |  ^~~~~~~
    _mysql.c: In function ‘_mysql_server_init’:
    _mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      266 |    s = PyUnicode_AsUTF8(item);
          |      ^
    _mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      296 |    s = PyUnicode_AsUTF8(item);
          |      ^
    _mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
      588 |   _stringsuck(ca, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
      589 |   _stringsuck(capath, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
      590 |   _stringsuck(cert, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
      591 |   _stringsuck(key, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      577 |         if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
          |                ^
    _mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
      592 |   _stringsuck(cipher, value, ssl);
          |   ^~~~~~~~~~~
    _mysql.c: In function ‘_mysql_ConnectionObject_ping’:
    _mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
     1911 |  if ( reconnect != -1 ) self->connection.reconnect = reconnect;
          |                                         ^
    _mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
    _mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     2603 |  cname = PyUnicode_AsUTF8(name);
          |        ^
    error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
    ----------------------------------------
  Rolling back uninstall of mysqlclient
  Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/MySQLdb/
   from /path/xyz/INSTALL/lib/python3.9/site-packages/~ySQLdb
  Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/mysqlclient-2.1.0.dist-info/
   from /path/xyz/INSTALL/lib/python3.9/site-packages/~ysqlclient-2.1.0.dist-info
ERROR: Command errored out with exit status 1: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient Check the logs for full command output.

Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh) Is there another missing dependency?

Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.

mbushkov · 2021-11-24T16:42:08Z

Hey, our docs on the matter were a little outdated, I created a PR (google/grr-doc#144) to fix this.

Before the PR gets submitted, the temp version of the docs is available here: https://grr-doc.readthedocs.io/en/sphinx-update/installing-grr-server/from-source.html

Commenting on your questions/notes:

Answer: I do plan to run the clients on ppc64le as well.

Is it a hard requirement to run the server on the ppc64le architecture? That's not impossible but is much more work, as ppc64le is not a tested scenario and we don't provide prebuilt server Debs for this architecture. A possible option might be: run the server on x86-64 Ubuntu 18 and build a custom client template for ppc64le (building a custom client template doesn't involve installing mysql and building the server pip, thus making the process much simpler).

Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh) Is there another missing dependency?

The error that you see in the build are not a sign of a missing dependency, but rather a build error inside the mysql-client package. I found similar mentions of the same problem: DefectDojo/django-DefectDojo#407

You don't need to have an initialized database, but you need to have the headers/libraries (which you seem to have, otherwise the build would fail totally).

Which database did you install - MySQL or MariaDB?

Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.

Yes, Ubuntu 18.04 is the server environment that we support and test on.

* Updated Sphinx. Migrated from recommonmark to myst-parser. Removed a few hack from conf.py. * Removed most mentions of AFF4 from the docs. * Removed Python 2-specific sections. * Fixed the docs to suggest to change mysql server config file to set max_allowed_packet to 40M (see google/grr#947). * Largely rewritten "Install from source": now telling how to build Fleetspeak wheels and specifically mentioning the case when we install from source to build a client template (see google/grr#948). * Made minor changes to "Build custom client templates". You can see the version of the docs from this PR at https://grr-doc.readthedocs.io/en/sphinx-update/index.html.

max-vogler assigned mbushkov Nov 15, 2021

mbushkov mentioned this issue Nov 24, 2021

Minor documentation overhaul google/grr-doc#144

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

helix-loop commented Nov 14, 2021 •

edited

mbushkov commented Nov 15, 2021

helix-loop commented Nov 21, 2021

mbushkov commented Nov 24, 2021

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

Unable to install fleetspeak-client-bin (0.1.11) on ppc64le #948

Comments

helix-loop commented Nov 14, 2021 • edited

mbushkov commented Nov 15, 2021

helix-loop commented Nov 21, 2021

mbushkov commented Nov 24, 2021

helix-loop commented Nov 14, 2021 •

edited