AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment #1544

sxa · 2020-09-15T13:43:11Z

Ref: https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/master/ansible/playbooks/aix.yml#L735

This can break certain default AIX startup scripts which require the AIX grep command.

We need to determine the impact to the adoptopenjdk scripts of changing this.

@Haroon-Khel Possibly somethign we can do while switching to the invididual Role-based playbooks?

The text was updated successfully, but these errors were encountered:

aixtools · 2020-09-16T21:45:44Z

The tactic I have seen most, over the years, is to either add PATH at the beginning of a script (as opposed to making something a system default), OR having variables defined to execute a specific command, e.g. GREP=/opt/freeware/bin/grep and then calling it using ${GREP}.

And after glancing at the playbook - I see you are 'installing', i.e., attempting to install, a dated version of openSSL (1.0.2.1601) while the version 1.0.2.2100 is already installed.

FYI:

p.s. - while I do not know what version yum.sh is demanding version 1601 may, or may not be sufficient.

p.p.s.:
$ openssl version
OpenSSL 1.0.2u 20 Dec 2019

sxa · 2020-09-17T22:39:49Z

@aixtools I think we need to be clear on why that PATH statement was there in the first case really (Doesn't appear to be commented in the playbook...) I suspect we should undo it on one of the machines and see what breaks :-)

Haroon-Khel · 2020-09-28T14:33:12Z

@sxa Do we know which AIX start up scripts fail as a result of this? And is the issue here that the PATH variable is simply at the start of /etc/environment, or that its there at all?

aixtools · 2020-09-28T14:39:41Z

One thing it breaks is rc.nfs. I wanted to do some AIX maintenaince (using NFS) - and fell over this. It takes too much time to find and document all the things that break because the syntax of 'coreutils' and/or 'binutils' is different than the AIX versions. So, it is not that it is in the PATH, but that it is first. Note: if a nodejs command is requiring a coreutils/binutils command syntax - and it is not providing the PATH - then that will break. In short, the 'bad' is that the systems are unmanageable. Updates cannot be applied (at least not on the system I first saw this, and it had not changed the PATH - it had overwritten /usr/bin/grep. Hope this helps, Michael (aka aixtools)

…

On 28/09/2020 16:33, Haroon Khel wrote: @sxa <https://github.com/sxa> Do we know which AIX start up scripts fail as a result of this? And is the issue here that the PATH variable is simply at the *start* of |/etc/environment|, or that its there at all? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1544 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSZR5OVWW3WFLRIOQYSGN3SICNDTANCNFSM4RNFA6DQ>.

aixtools · 2020-10-01T07:28:47Z

Smarter now: I see where it happens in aix.yml


      ######################################
      # Add bash to available login shells #
      ######################################
        - name: Add bash to available login shells
          replace:
            path: /etc/security/login.cfg
            regexp: 'shells = '
            replace: 'shells = /bin/bash,'
          tags: login_shell

        - name: Add bash to available login shells
          blockinfile:
            dest: /etc/shells
            block: |
              /bin/bash
          tags: login_shell

        - name: Set variables for global environment
          blockinfile:
            dest: /etc/environment
            block: |
              AIXTHREAD_HRT=true
              PKG_CONFIG_PATH=/opt/freeware/lib64/pkgconfig:/opt/freeware/lib/pkgconfig
              PERL5LIB=/opt/freemarker/lib/perl5
          tags: login_shell

        - name: Add freeware and xlc to PATH for global environment
          replace:
            path: /etc/environment
            regexp: 'PATH=/usr/bin'
            replace: 'PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin'
          tags: login_shell

As I mention in #1581 there are some assumptions here.

Classically, tools such as compilers - and java - are appended to the PATH, not pre-pended.
This code does not check if the admins had already placed xlC (must it be C++, or does xlc (no C++) suffice?
This code combines several, I would consider not related, under the heading "bash is included in login_shells". Again, it does not check if bash is already included, it just prepends it.

Haroon-Khel · 2021-04-06T12:37:58Z

Related pr #2057

sxa · 2021-09-23T12:17:20Z

Related pr #2057

Superceded by #2288

aixtools · 2022-03-16T11:36:04Z

OK - currently, they are almost all the same - with:

==============
root@nim.bak:[/home/root]dsh-adopt.ksh "echo \$PATH"
adopt01:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt02:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt03:
/usr/bin:/opt/IBM/xlC/13.1.3/bin:/opt/freeware/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt04:
/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin
==============
adopt05:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
==============
adopt06:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin:/opt/bin
==============
adopt07:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin:/opt/bin
==============
adopt08:
/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin

However, going to change the default to have xlC at the end, and modify jenkins user so that PATH is modified during ssh login.

sxa · 2022-05-30T12:12:16Z

Quick note on this - if /usr/bin is put ahead, then the GNU make is not picked up correctly with the current setup, so that would need to be resolved before we can put this change in.

aixtools · 2022-06-23T18:13:35Z

Maybe adding an alias for make will be a quick fix (jenklins .profile), or we 'mangle' the current /usr/bin/make to be a symbolic link to GNU make.

aixtools · 2023-06-06T11:15:39Z

I am thinking there are two ways to handle this:

PREFIX $PATH on jenkins user with /opt/freeware/bin
Add an alias in /etc/profile for make so that it uses /opt/freeware/bin/make rather than /usr/bin/make.

sxa · 2023-06-06T11:51:20Z

I think most of the invocations of make already have some switches to invoke something else on Solaris. It's quite likely we could do the same on AIX ... If you search the temurin-build and aqa-tests repositories for gmake you can probably find most (maybe all?) of the places it would need to be overridden to /opt/freeware/bin/make on AIX.

aixtools · 2023-06-07T14:07:04Z

ok. when I am not enjoying a sunny day. ie, let the rain come (the yard is begging for rain!)

aixtools · 2023-07-21T11:26:13Z

I think most of the invocations of make already have some switches to invoke something else on Solaris. It's quite likely we could do the same on AIX ... If you search the temurin-build and aqa-tests repositories for gmake you can probably find most (maybe all?) of the places it would need to be overridden to /opt/freeware/bin/make on AIX.

Based on this: created the PR: adoptium/temurin-build#3441

This looks to be all that is needed.

And then we can remove the addition of /opt/freeware/bin from PATH.

aixtools · 2023-08-01T14:38:49Z

As adoptium/temurin-build#3441 has been merged, want to work on a PR cleaning up PATH, but waiting for #3139 to be merged. I hate to rebase stuff.

aixtools · 2023-08-14T15:29:52Z

I've been playing around with options - and, imho - the best path will include:

PR to remove the assignment of /opt/freeware/bin (and xlc13 paths)
manual fix of the PATH variable on the current systems - rather than use playbook editing to remove it.
Once done, basically, the playbook takes a handsoff attitude to global PATH (in /etc/environment and/or /etc/profile). If an admin wants something different - that is their call. And if it breaks java build or runtime - that is their responsibility.
So, discussion first (on above points) - and then I'll take the required steps (including making sure the PATH is set via jenkins parameters/profile).

aixtools · 2023-08-15T13:20:21Z

As a reminder: the current setup breaks the PATH for system administrators - and amazingly, despite the recent refresh of all systems using the playbooks - not all systems display the same behavior:

root@osunim:[/root]dsh-adopt "df -g /tmp"
adopt01:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt02:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt03:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt04:
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3           4.00      3.40   16%      252     1% /tmp
==============
adopt05:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt06:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt07:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt08:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============
adopt10:
df: invalid option -- 'g'
Try 'df --help' for more information.
==============

sxa · 2023-08-15T16:21:12Z

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

aixtools · 2023-08-16T12:38:17Z

adopt04 comes close - but I would ALSO remove the xlc13 compiler related path variable. adopt03 has duplicity issues.

root@osunim:[/root]dsh-adopt /usr/bin/egrep ^PATH /etc/environment
adopt01:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt02:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt03:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/opt/IBM/xlC/13.1.3/bin:/opt/freeware/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/opt/bin
==============
adopt04:
PATH=/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin
==============
adopt05:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt06:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt07:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt08:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============
adopt10:
PATH=/opt/freeware/bin:/opt/IBM/xlC/13.1.3/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin:/opt/bin
==============

aixtools · 2023-08-23T12:03:30Z

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

And - how did that work out?

aixtools · 2023-12-05T13:59:50Z

I'm guessing it works out (on adopt04) - since it has not been changed.
Looking at two systems - build-1 and build-2 - I see that build-1 has the PATH set as a parameter for jenkins, but build-2 does not - so noone has been updating the jenkins settings (and as long as the global PATH prefixes GNU applications (ie, /opt/freeware/bin) we can never know if the jenkins parameters are doing the job.
That said - on adopt04 jenkins it is set: https://ci.adoptium.net/computer/test%2Dosuosl%2Daix72%2Dppc64%2D2/configure - so I hope that is conclusive enough to make this acceptable.
This cannot, really, be resolved via a PR. ie, once accepted that /opt/bin/freeware should not be in the PATH - a PR change can be done (I recommend a new issue to remove a lot of noise from here) and remove the ansible step that includes it.
And, that someone closes this when jenkins has been updated to include PATH for all agents.

sxa · 2023-12-05T17:26:56Z

Thanks for pushing this forward. So I guess adopt04 (test-osuosl-aix72-ppc64-2) is in the state you want at the moment, so we should run a build of a couple of versions (probably 8 and 20) to ensure that jenkins and the build scripts work ok there and then roll this out.

And - how did that work out?

I don't think it's been explicitly done yet ... It's just a matter of kicking off the JDK8u and JDK21u jobs so here they are (Both with ENABLE_TESTS: false since they shouldn't be too important for this, and NODE_LABEL set to test-osuosl-aix72-ppc64-2

sxa · 2023-12-14T11:18:07Z

Noting that the jdk8u build failed, but it doesn't look related to the PATH.
Re-running at https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/411/console

sxa · 2023-12-14T11:46:50Z

Needs further investigation although I can't spend any time on it. The build in the last comment hit the same issue. Maybe it's perl related:

11:18:46  mk-ca-bundle.pl generates 144 certificates
11:18:46  Subject: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:46  Generated alias: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:46  Processing certificate with alias: CN=GlobalSign_Root_CA,OU=Root_CA,O=GlobalSign_nv-sa,C=BE
11:18:47  keytool error: java.lang.Exception: Input not an X.509 certificate

aixtools · 2023-12-21T19:36:57Z

It's been awhile. I forgot what the problem was - and how it got solved.

sxa added bug os:aix labels Sep 15, 2020

sxa added this to the September 2020 milestone Sep 15, 2020

karianna added this to TODO in infrastructure via automation Sep 17, 2020

karianna moved this from TODO to In Progress in infrastructure Oct 1, 2020

karianna assigned aixtools Oct 1, 2020

Haroon-Khel modified the milestones: September 2020, October 2020 Oct 5, 2020

Haroon-Khel modified the milestones: October 2020, November 2020 Nov 9, 2020

Haroon-Khel modified the milestones: November 2020, December 2020 Dec 10, 2020

Haroon-Khel modified the milestones: December 2020, January 2021 Jan 4, 2021

Haroon-Khel modified the milestones: January 2021, February 2021 Feb 1, 2021

Haroon-Khel modified the milestones: February 2021, March 2021 Mar 2, 2021

Haroon-Khel modified the milestones: March 2021, April 2021 Apr 6, 2021

aixtools mentioned this issue Apr 13, 2021

AixPb: Put /usr/bin to the front of the PATH #2057

Closed

Haroon-Khel modified the milestones: April 2021, May 2021 May 18, 2021

Haroon-Khel removed this from the May 2021 milestone Jun 21, 2021

sxa added this to the September 2021 milestone Sep 23, 2021

sxa modified the milestones: September 2021, December 2021 Dec 1, 2021

sxa modified the milestones: December 2021, 2022-01 (January) Jan 6, 2022

sxa modified the milestones: 2022-01 (January), 2022-02 (February) Feb 10, 2022

sxa removed this from the 2022-02 (February) milestone Mar 3, 2022

aixtools mentioned this issue Jul 14, 2022

Ansible request for AIX 7 #2641

Closed

aixtools mentioned this issue Jul 21, 2023

aix: use fullpath for (g)make so PATH does not need to be modified adoptium/temurin-build#3441

Merged

aixtools mentioned this issue Dec 5, 2023

Add steps to install OpenSSL 3 on AIX 7.2 machines #3274

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment #1544

AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment #1544

sxa commented Sep 15, 2020 •

edited

aixtools commented Sep 16, 2020 •

edited

sxa commented Sep 17, 2020

Haroon-Khel commented Sep 28, 2020

aixtools commented Sep 28, 2020 via email

aixtools commented Oct 1, 2020

Haroon-Khel commented Apr 6, 2021

sxa commented Sep 23, 2021

aixtools commented Mar 16, 2022

sxa commented May 30, 2022

aixtools commented Jun 23, 2022

aixtools commented Jun 6, 2023

sxa commented Jun 6, 2023

aixtools commented Jun 7, 2023

aixtools commented Jul 21, 2023 •

edited

aixtools commented Aug 1, 2023

aixtools commented Aug 14, 2023

aixtools commented Aug 15, 2023

sxa commented Aug 15, 2023

aixtools commented Aug 16, 2023 •

edited

aixtools commented Aug 23, 2023

aixtools commented Dec 5, 2023

sxa commented Dec 5, 2023

sxa commented Dec 14, 2023

sxa commented Dec 14, 2023

aixtools commented Dec 21, 2023

AIX playbook should not put /opt/freeware/bin at start of /etc/environment #1544

AIX playbook should not put /opt/freeware/bin at start of /etc/environment #1544

Comments

sxa commented Sep 15, 2020 • edited

aixtools commented Sep 16, 2020 • edited

sxa commented Sep 17, 2020

Haroon-Khel commented Sep 28, 2020

aixtools commented Sep 28, 2020 via email

aixtools commented Oct 1, 2020

Haroon-Khel commented Apr 6, 2021

sxa commented Sep 23, 2021

aixtools commented Mar 16, 2022

sxa commented May 30, 2022

aixtools commented Jun 23, 2022

aixtools commented Jun 6, 2023

sxa commented Jun 6, 2023

aixtools commented Jun 7, 2023

aixtools commented Jul 21, 2023 • edited

aixtools commented Aug 1, 2023

aixtools commented Aug 14, 2023

aixtools commented Aug 15, 2023

sxa commented Aug 15, 2023

aixtools commented Aug 16, 2023 • edited

aixtools commented Aug 23, 2023

aixtools commented Dec 5, 2023

sxa commented Dec 5, 2023

sxa commented Dec 14, 2023

sxa commented Dec 14, 2023

aixtools commented Dec 21, 2023

AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment #1544

AIX playbook should not put `/opt/freeware/bin` at start of /etc/environment #1544

sxa commented Sep 15, 2020 •

edited

aixtools commented Sep 16, 2020 •

edited

aixtools commented Jul 21, 2023 •

edited

aixtools commented Aug 16, 2023 •

edited