Maximo Manage install with OneClick failure during apply-db2cfg-settings.sh

[ReneOro97]

Summary

Attempting to install Maximo Manage is unsuccessful by using the OneClick AddManage playbook

There seems to be a Db2 related issue, specifically with apply-db2cfg-settings.sh script

Before hitting this problem, I faced the same issue described in #1077 and I followed the workaround for reruning the command which seems to have worked for that issue.

Environment

MAS Core v8.11.5
Maximo Manage 8.7.3
VPC in IBM Cloud

Steps to reproduce

1. Install MAS Core with Oneclick Installer
2. Install Maximo Manage with Oneclick Installer
3. The installer initially fails due to issue described in 1077. Before rerun installer, trigger the following command:
   oc exec -n db2u c-db2w-shared-db2u-0 -- su -lc '/tmp/setupdb.sh | tee /tmp/setupdb2.log' db2inst1
4. Rerun the installer, this time, no issue related to setupdb.sh
5. In apply-db2cfg-settings.sh steps, getting the following error:

SQL1668N  The operation failed because the operation is not supported with
    this environment. Reason code: "9".

Relevant Logs

DB2 pod log of Maximo Manage:
c-db2w-shared-db2u-0.log

{"level":"info","result":"stdout: Waiting for up to 180 sec until a change is detected in the relevant configmaps... change detected\n(*) Applying Db2 registry variables\ndb2set DB2AUTH=OSAUTHDB,ALLOW_LOCAL_FALLBACK,PLUGIN_AUTO_RELOAD\ndb2set DB2FODC=DUMPSHM=ON CORESHM=OFF\ndb2set DB2_4K_DEVICE_SUPPORT=ON\ndb2set DB2_ATS_ENABLE=YES\ndb2set DB2_BCKP_PAGE_VERIFICATION=TRUE\ndb2set DB2_CDE_DCC=\"\"\ndb2set DB2_CDE_REDUCED_LOGGING=REDUCED_REDO:NO\ndb2set DB2_DCC_BINARY_FILE=\"\"\ndb2set DB2_DCC_FILE_CHUNKSZ=\"\"\ndb2set DB2_DCC_FILE_DEL_THRES=\"\"\ndb2set DB2_DCC_FILE_INS_THRES=\"\"\ndb2set DB2_DCC_FILE_PATH=\"\"\ndb2set DB2_EVALUNCOMMITTED=YES\ndb2set DB2_FMP_COMM_HEAPSZ=65536\ndb2set DB2_FMP_RUN_AS_CONNECTED_USER=NO\ndb2set DB2_INLIST_TO_NLJN=YES\ndb2set DB2_LOAD_COPY_NO_OVERRIDE=COPY YES TO /mnt/bludata0/db2/copy\ndb2set DB2_MINIMIZE_LISTPREFETCH=Y\ndb2set DB2_OBJECT_STORAGE_LOCAL_STAGING_PATH=/mnt/backup/staging\ndb2set DB2_OBJECT_TABLE_ENTRIES=65532\ndb2set DB2_REMOTE_EXTTAB_PIPE_PATH=/db2u/tmp\ndb2set DB2_RESTORE_GRANT_ADMIN_AUTHORITIES=ON\ndb2set DB2_SKIPDELETED=ON\ndb2set DB2_SKIPINSERTED=ON\ndb2set DB2_USE_ALTERNATE_PAGE_CLEANING=ON\ndb2set DB2_WORKLOAD=ANALYTICS\ndb2set DB2_OVERRIDE_NUM_CPUS=6\ndb2set DB2_OVERRIDE_THREADING_DEGREE=2\ndb2set DB2COMM=TCPIP,SSL\nWolverine HA management state was disabled successfully.\nRunning db2stop force\n01/24/2024 19:43:44     0   0   SQL1064N  DB2STOP processing was successful.\nSQL1064N  DB2STOP processing was successful.\n \nApplication ipclean: Removing all IPC resources for db2inst1(500)\nc-db2w-shared-db2u-0.c-db2w-shared-db2u-internal: ipclean -a completed ok\nRunning db2start\n01/24/2024 19:43:57     0   0   SQL1063N  DB2START processing was successful.\nSQL1063N  DB2START processing was successful.\nDb2 was started successfully\nWolverine HA management state was enabled successfully.\n hostname: c-db2w-shared-db2u-0","caller":"[13]:db2.go:306:ApplyCfg()","timestamp":"2024-01-24T19:44:03Z","message":"executed apply Db2 configuration cmd for setting regvar"}
{"level":"error","caller":"[13]:k8s_v2.go:542:cleanUp()","error":"update failed: the server rejected our request due to an error in our request","timestamp":"2024-01-24T19:50:16Z","message":"failed to remove expired locks"}
{"level":"error","component":"server","subComponent":"GetDb2Health","caller":"[13]:db2.go:262:GetDb2Health()","timestamp":"2024-01-24T19:56:11Z","message":"1 MLNs found in MLN dist file, but only 0 db2sysc processes match"}
{"level":"error","caller":"[13]:k8s_v2.go:542:cleanUp()","error":"update failed: the server rejected our request due to an error in our request","timestamp":"2024-01-24T20:10:16Z","message":"failed to remove expired locks"}
{"level":"error","caller":"[13]:k8s_v2.go:542:cleanUp()","error":"update failed: the server rejected our request due to an error in our request","timestamp":"2024-01-24T20:20:16Z","message":"failed to remove expired locks"}
{"level":"error","caller":"[13]:k8s_v2.go:542:cleanUp()","error":"update failed: the server rejected our request due to an error in our request","timestamp":"2024-01-24T21:00:16Z","message":"failed to remove expired locks"}

MAS Manage Maxinst pod log:
https://ibm.box.com/s/b5cp5forgrd81tu9rzx3cn2fgp5g18pn

Preparing DBC's
delserversession.dbc
====================
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE script SYSTEM "script.dtd"><script author="ManageOperator" scriptname="deleteserversess"><description></description>
  <statements><freeform description="get maxvar"><sql target="all">delete from serversession</sql></freeform>
  </statements></script>
Reduced downtime type:regularUpgrade
Failure control type:retry
Starting database validation for maxinst or updatedb...
MAXVAR return code = 0
Running querycount utility with where clause: varname = 'MAXUPG' and (varvalue like 'V761%' or varvalue like 'V76010%' or varvalue like 'V8%')
DB check return code = 0 (zero is good)
Query count is 0
Maximo version not supported...
Forcing job to error so deployment initContainer does not release.

Ansible Playback for Manage OneClick Installer:
Full log: playbook_install.log

TASK [ibm.mas_devops.suite_db2_setup_for_manage : Run script to make changes take effect] *****************************************************************************************************
changed: [localhost] => changed=true 
  cmd: oc exec -n db2u c-db2w-shared-db2u-0 -- su -lc '/db2u/scripts/apply-db2cfg-settings.sh --setting all | tee /tmp/apply-db2cfg-settings.log' db2inst1
  delta: '0:03:52.534422'
  end: '2024-01-24 19:52:30.277098'
  msg: ''
  rc: 0
  start: '2024-01-24 19:48:37.742676'
  stderr: 'Defaulted container "db2u" out of: db2u, init-labels (init), init-kernel (init)'
  stderr_lines: <omitted>
  stdout: |-
    Waiting for up to 180 sec until a change is detected in the relevant configmaps... change not detected, continuing anyways
    Applying all settings
    (*) Applying Db2 dbm cfg parameters
    update dbm cfg using  AGENT_STACK_SZ 1024 AUTHENTICATION SERVER_ENCRYPT CONN_ELAPSE 60 DIAGPATH '/mnt/blumeta0/db2/log $N' DIAGSIZE 300 DISCOVER DISABLE DISCOVER_INST DISABLE FENCED_POOL 50 HEALTH_MON OFF INSTANCE_USAGE DASHDB JDK_PATH /mnt/blumeta0/home/db2inst1/sqllib/java/jdk64 KEEPFENCED NO MAX_CONNRETRIES 5 MON_HEAP_SZ AUTOMATIC NUMDB 1 PYTHON_PATH /usr/bin/python RQRIOBLK 65535
    DB20000I  The UPDATE DATABASE MANAGER CONFIGURATION command completed
    successfully.
  
    DB20000I  The TERMINATE command completed successfully.
    (*) Applying Db2 db cfg parameters for BLUDB
    update db cfg for BLUDB using  APPLHEAPSZ 8192 AUTOMATIC AUTHN_CACHE_DURATION 10 AUTHN_CACHE_USERS 100 AUTO_DB_BACKUP OFF AUTO_DEL_REC_OBJ ON AUTO_MAINT ON AUTO_REORG OFF AUTO_REVAL DEFERRED AUTO_RUNSTATS ON AUTO_TBL_MAINT ON CATALOGCACHE_SZ 800 CHNGPGS_THRESH 40 CUR_COMMIT ON DATABASE_MEMORY AUTOMATIC DBHEAP AUTOMATIC DDL_CONSTRAINT_DEF YES DEC_TO_CHAR_FMT NEW DFT_QUERYOPT 5 DFT_SCHEMAS_DCC NO DFT_TABLE_ORG ROW EXTBL_LOCATION /mnt/blumeta0/db2/load;/mnt/blumeta0/home;/mnt/bludata0/scratch;/mnt/external;/mnt/backup EXTBL_STRICT_IO NO INDEXREC ACCESS LOCKLIST AUTOMATIC LOCKTIMEOUT 300 LOGARCHMETH1 DISK:/mnt/bludata0/db2/archive_log/ LOGBUFSZ 1024 LOGFILSIZ 32768 LOGPRIMARY 100 LOGSECOND 156 LOG_APPL_INFO NO LOG_DDL_STMTS NO MAXFILOP 61440 MIRRORLOGPATH /mnt/backup/MIRRORLOGPATH NUM_DB_BACKUPS 60 NUM_IOCLEANERS AUTOMATIC NUM_IOSERVERS AUTOMATIC PCKCACHESZ AUTOMATIC REC_HIS_RETENTN 60 SHEAPTHRES_SHR automatic SOFTMAX 0 SORTHEAP automatic STAT_HEAP_SZ AUTOMATIC STMTHEAP 20000 STMT_CONC LITERALS TRACKMOD YES WLM_ADMISSION_CTRL NO
    SQL1668N  The operation failed because the operation is not supported with
    this environment. Reason code: "9".
  
    Wolverine HA management state was disabled successfully.
    Running db2stop force
    01/24/2024 19:52:11     0   0   SQL1064N  DB2STOP processing was successful.
    SQL1064N  DB2STOP processing was successful.
  
    Application ipclean: Removing all IPC resources for db2inst1(500)
    c-db2w-shared-db2u-0.c-db2w-shared-db2u-internal: ipclean -a completed ok
    Running db2start
    01/24/2024 19:52:23     0   0   SQL1063N  DB2START processing was successful.
    SQL1063N  DB2START processing was successful.
    Db2 was started successfully
    Wolverine HA management state was enabled successfully.

Currently the installation is still running but there is no indication it will succeed.

ManageWorkspace:

oc describe ManageWorkspace

Name:         cpst2-masdev
Namespace:    mas-cpst2-manage
Labels:       mas.ibm.com/applicationId=manage
              mas.ibm.com/instanceId=cpst2
              mas.ibm.com/workspaceId=masdev
Annotations:  
API Version:  apps.mas.ibm.com/v1
Kind:         ManageWorkspace
Metadata:
  Creation Timestamp:  2024-01-23T22:07:09Z
  Finalizers:
    manageworkspace.apps.mas.ibm.com/finalizer
  Generation:        1
  Resource Version:  17009680
  UID:               05809fc7-84a3-4a43-9e0f-6448284b69eb
Spec:
  Bindings:
    Jdbc:  system
  Components:
    Base:
      Version:  latest
  Settings:
    Aio:
      Install:  true
    Customization List:
    Db:
      Db Schema:  maximo
      Maxinst:
        Bypass Upgrade Version Check:  false
        db2Vargraphic:                 true
        Demodata:                      false
        Index Space:                   MAXINDEX
        Table Space:                   MAXDATA
    Deployment:
      Build Tag:    latest
      Default JMS:  false
      Mode:         up
      Persistent Volumes:
      Server Bundles:
        Bundle Type:          all
        Is Default:           true
        Is Mobile Target:     true
        Is User Sync Target:  true
        Name:                 all
        Replica:              1
        Route Sub Domain:     all
      Server Timezone:        GMT
    Languages:
      Base Lang:  EN
      Secondary Langs:
Status:
  Bindings:
    Jdbc:  system
  Components:
    Acm:
      Enabled:  false
      Version:  8.1.9
    Anywhere:
      Enabled:  false
      Version:  8.0.2
    Aviation:
      Enabled:  false
      Version:  8.1.11
    Base:
      Enabled:  true
      Version:  8.7.3
    Civil:
      Enabled:  false
      Version:  8.6.3
    Envizi:
      Enabled:  false
      Version:  8.1.0
    Health:
      Enabled:  false
      URL:      /maximo/oslc/graphite/relengineer
      Version:  8.9.2
    Hse:
      Enabled:  false
      Version:  8.1.7
    Icd:
      Enabled:  false
      Version:  8.1.3
    Nuclear:
      Enabled:  false
      Version:  8.1.6
    Oilandgas:
      Enabled:  false
      Version:  8.1.7
    Oracleadapter:
      Enabled:  false
      Version:  8.2.0
    Sapadapter:
      Enabled:  false
      Version:  8.2.0
    Serviceprovider:
      Enabled:  false
      Version:  8.1.9
    Spatial:
      Enabled:  false
      Version:  8.7.2
    Strategize:
      Enabled:  false
      Version:  8.0.0
    Transportation:
      Enabled:  false
      Version:  8.1.8
    Tririga:
      Enabled:  false
      Version:  8.1.0
    Utilities:
      Enabled:  false
      Version:  8.1.6
    Workday:
      Enabled:  false
      Version:  8.1.0
  Conditions:
    Last Transition Time:  2024-01-23T22:35:44Z
    Message:               Server Bundles not deployed in workspace masdev 
    Reason:                Pending
    Status:                False
    Type:                  Ready
    Last Transition Time:  2024-01-23T22:35:44Z
    Message:               Build generated and tagged with 20240123T220921-8.7
    Reason:                Build completed
    Status:                True
    Type:                  BuildReady
    Last Transition Time:  2024-01-24T20:35:42Z
    Message:                Task: Database update failed. Deployment all - not deployed
    Reason:                Pending
    Status:                False
    Type:                  DeploymentReady
    Message:               unknown playbook failure
status: error prepare-db: Task: Database update failed
    Reason:                Failed
    Status:                False
    Type:                  DeploymentCR
    Last Transition Time:  2024-01-23T22:12:39Z
    Message:               
    Reason:                
    Status:                False
    Type:                  Failure
    Last Transition Time:  2024-01-24T22:13:05Z
    Message:               Last reconciliation succeeded
    Reason:                Successful
    Status:                False
    Type:                  Successful
    Last Transition Time:  2024-01-23T22:07:09Z
    Message:               Running reconciliation
    Reason:                Running
    Status:                True
    Type:                  Running
  Milestones:
    Activated:
      Anonymous Id:    a8889d08-63bd-4ab3-9cfa-6dfb8e058f7b
      Timestamp:       2024-01-23T22:07:52.398832+00:00
      Transaction Id:  9b801f9e-efdf-400c-a9c8-e60cc9b15e9a
  Settings:
    Aio:
      Install:  true
    Customization:
    Customization List:
    Db:
      Db Schema:  maximo
      Maxinst:
        Bypass Upgrade Version Check:  false
        db2Vargraphic:                 true
        Demodata:                      false
        Index Space:                   MAXINDEX
        Table Space:                   MAXDATA
    Deployment:
      Build Tag:    latest
      Default JMS:  false
      Ephemeral Storage:
        Limits:
          Admin Build:          100Gi
          Server Bundle Build:  100Gi
        Requests:
          Admin Build:          30Gi
          Server Bundle Build:  30Gi
      Imported Certs:
      Mode:  up
      Persistent Volumes:
      Server Bundles:
        Bundle Type:          all
        Is Default:           true
        Is Mobile Target:     true
        Is User Sync Target:  true
        Name:                 all
        Replica:              1
        Route Sub Domain:     all
      Server Timezone:        GMT
      Use Mas Smtpcfg:        true
    Languages:
      Base Lang:  EN
      Secondary Langs:
    Manual Cert Mgmt:  false
    Opentelemetry:
      Collector:
        Image Tag:  0.81.0
        Replicas:   1
      Enabled:      false
    Resources:
      Manage Admin:
        Limits:
          Cpu:     2
          Memory:  4Gi
        Requests:
          Cpu:     0.5
          Memory:  500Mi
      Server Bundles:
        Limits:
          Cpu:     6
          Memory:  10Gi
        Requests:
          Cpu:     0.5
          Memory:  2Gi
  Version:
    Reconciled:  
  Versions:
    Reconciled:  8.7.3
Events: