ORA-03896: Unable to load the sharding wallet successfully.

[nakinov]

Hi Everyone,

I've successfully deployed Oracle 21c Sharded Database using standalone Docker Containers. Everything works fine the first time and while the environment is not rebooted. The moment when I restart some of the containers or the complete VM, I am unable to perform any sharded ddl, because the wallet location somehow gets lots.

SQL> insert into my_table values (1,'test');
*
ERROR at line 1:
ORA-03896: Unable to load the sharding wallet successfully.
ORA-06512: at "SYS.DBMS_SESSION", line 155
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_POOLADMIN", line 25013
ORA-03896: Unable to load the sharding wallet successfully.
ORA-06512: at "SYS.DBMS_SESSION", line 141
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_POOLADMIN", line 24910
ORA-06512: at "GSMADMIN_INTERNAL.DBMS_GSM_POOLADMIN", line 25031
ORA-06512: at line 1

Below is the information about the wallet's location on the CATALOG, SHARD1 and SHARD2 containers.

CATALOG

#####################################################################################
SQL> select value from v$parameter where name='wallet_root';

VALUE

/opt/oracle/product/21c/dbhome_1/admin

SQL> select guid from v$pdbs where con_id = sys_context('userenv','con_id');

GUID

020CCE4FEE460AD6E0636614000A9A39

bash-4.2$ pwd
/opt/oracle/admin/CATCDB/xdb_wallet
bash-4.2$ ls -ltr
total 8
-rw-------. 1 oracle oinstall 3835 Aug 4 15:11 ewallet.p12
-rw-------. 1 oracle oinstall 3880 Aug 4 15:11 cwallet.sso

SQL> select value from v$parameter where name='wallet_root';

SHARD1

#####################################################################################

SQL> select value from v$parameter where name='wallet_root';

VALUE

/opt/oracle/oradata/dbconfig/ORCL1CDB/admin

SQL> alter session set container=ORCL1PDB;

SQL> select guid from v$pdbs where con_id = sys_context('userenv','con_id');

GUID

020CEA753FD90AEEE0636714000AF57F

bash-4.2$ pwd
/opt/oracle/oradata/dbconfig/ORCL1CDB/admin/020CEA753FD90AEEE0636714000AF57F/shard

bash-4.2$ ls -ltr
total 4
-rw-------. 1 oracle oinstall 2877 Aug 4 15:27 cwallet.sso

bash-4.2$ pwd
/opt/oracle/oradata/dbconfig/ORCL1CDB/admin/shard

bash-4.2$ ls -ltr
total 4
-rw-------. 1 oracle oinstall 2661 Aug 3 22:01 cwallet.sso

SHARD2

#####################################################################################

SQL> select value from v$parameter where name='wallet_root';

VALUE

/opt/oracle/oradata/dbconfig/ORCL2CDB/admin

SQL> alter session set container=ORCL2PDB;

SQL> select guid from v$pdbs where con_id = sys_context('userenv','con_id');

GUID

020D07702D360AF0E0636814000A24E5

bash-4.2$ pwd
/opt/oracle/oradata/dbconfig/ORCL2CDB/admin/020D07702D360AF0E0636814000A24E5/shard

bash-4.2$ ls -ltr
total 4
-rw-------. 1 oracle oinstall 2877 Aug 4 15:27 cwallet.sso

bash-4.2$ pwd
/opt/oracle/oradata/dbconfig/ORCL2CDB/admin/shard

bash-4.2$ ls -ltr
total 4
-rw-------. 1 oracle oinstall 2661 Aug 3 22:09 cwallet.sso

Can you please take a look into this?

Thank you,
Lazar

[jpverma85]

Hi Lazar,

I tried to reproduce this issue but could not in my test setup using 21c RDBMS and GSM Images. Also, I can see there are two wallets for Catalog Database:

[opc@vm-oracle-sharding deploy]$ docker exec -it catalog /bin/bash
bash-4.2$
bash-4.2$ sqlplus sys as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on Tue Aug 15 20:48:45 2023
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle. All rights reserved.

Enter password:

Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> select value from v$parameter where name='wallet_root';

VALUE

/opt/oracle/product/21c/dbhome_1/admin

SQL> select guid,con_id from v$pdbs;

GUID CON_ID

---

026CB878DD8104E0E0636614000A1776 2
026CC6A6EE580AD1E0636614000A96D9 3

SQL> show pdbs

CON_ID CON_NAME               OPEN MODE  RESTRICTED

---

 2 PDB$SEED                  READ ONLY  NO
 3 CAT1PDB              READ WRITE NO

SQL> exit
Disconnected from Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
bash-4.2$ cd /opt/oracle/product/21c/dbhome_1/admin
bash-4.2$ ls -lrt
total 0
drwxr-x---. 2 oracle oinstall 25 Aug 8 16:23 shard
drwxr-x---. 3 oracle oinstall 19 Aug 8 16:23 026CC6A6EE580AD1E0636614000A96D9
bash-4.2$ ls -rlt /opt/oracle/product/21c/dbhome_1/admin/shard/
total 4
-rw-------. 1 oracle oinstall 2549 Aug 8 16:23 cwallet.sso
bash-4.2$ ls -rlt /opt/oracle/product/21c/dbhome_1/admin/026CC6A6EE580AD1E0636614000A96D9/
total 0
drwxr-x---. 2 oracle oinstall 25 Aug 8 16:23 shard
bash-4.2$ ls -rlt /opt/oracle/product/21c/dbhome_1/admin/026CC6A6EE580AD1E0636614000A96D9/shard/
total 4
-rw-------. 1 oracle oinstall 2829 Aug 8 16:39 cwallet.sso
bash-4.2$
bash-4.2$

[saurabhuja]

I am also able to run DDL queries after restart of docker containers after system reboot - "docker start catalog gsm1 gsm2 shard1 shard2" and "docker exec -it gsm1 bash -c "/u01/app/oracle/product/21c/gsmhome_1/bin/sqlplus sharding_demo/sharding_demo@SHARDDIRECTOR1""

Is there a way we consistently reproduce it?