Error after updating rods password

[DOC-MEX]

• main
• 4-3-stable

Bug Report

I have updated the default password of the rods user. This change triggers this error: [CAT_INVALID_AUTHENTICATION: rcAuthCheck failed. . I have re-run iinit and also restarted the service but that does not correct the problem. I can see it that it the delayServer is not activated after that.

sudo su irods -c "/var/lib/irods/irodsctl status"
irodsServer :
  Process 642603
  Process 642604

The moment I change the password back to the default "rods" value, everything is back to normal. The log error messages stop and the irodsDelayServer is initialized

{"log_category":"delay_server","log_level":"info","log_message":"Initializing delay server ...","server_host":"opendata-20","server_pid":645988,"server_timestamp":"2024-04-26T12:22:16.704Z","server_type":"delay_server","server_zone":"earlhamZone"}

iRODS Version, OS and Version

4.3.1 Ubuntu 20

What did you try to do?

restart the service and running iinit

Expected behavior

Observed behavior (including steps to reproduce, if applicable)

These are the error logs when the password of rods is not the default "rods"

{"log_category":"api","log_level":"info","log_message":"rsAuthCheck: chlCheckAuth status = -826000","request_api_name":"","request_api_number":110000,"request_api_version":"d","request_client_user":"rods","request_host":"127.0.0.1","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"opendata-20","server_pid":642722,"server_timestamp":"2024-04-26T12:13:19.245Z","server_type":"agent","server_zone":"earlhamZone"}
 {"log_category":"api","log_level":"info","log_message":"Error occurred invoking auth plugin operation [CAT_INVALID_AUTHENTICATION: rcAuthCheck failed.\n\n] [ec=-826000]","request_api_name":"","request_api_number":110000,"request_api_version":"d","request_client_user":"rods","request_host":"127.0.0.1","request_proxy_user":"rods","request_release_version":"rods4.3.1","server_host":"opendata-20","server_pid":642722,"server_timestamp":"2024-04-26T12:13:19.246Z","server_type":"agent","server_zone":"earlhamZone"}
 {"log_category":"authentication","log_level":"info","log_message":"Error occurred while authenticating user [rods] [CAT_INVALID_AUTHENTICATION: failed to perform request\n\n] [ec=-826000]","server_host":"opendata-20","server_pid":642603,"server_timestamp":"2024-04-26T12:13:19.381Z","server_type":"server","server_zone":"earlhamZone"}
 {"log_category":"server","log_level":"error","log_message":"Caught exception in migrate_delay_server(): iRODS Exception:\n    file: /irods_source/lib/core/src/client_connection.cpp\n    function: void irods::experimental::client_connection::connect_and_login(const std::string &, const int, const irods::experimental::fully_qualified_username &)\n    line: 161\n    code: -178000 (AUTHENTICATION_ERROR)\n    message:\n        Client login error\nstack trace:\n--------------\n 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1\n 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1\n 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1\n 3# std::__1::__function::__func<initServerMain(RsComm*, bool, bool)::$_13, std::__1::allocator<initServerMain(RsComm*, bool, bool)::$_13>, void ()>::operator()() at rodsServer.cpp:?\n 4# irods::experimental::cron::cron_task::operator()() in /usr/sbin/irodsServer\n 5# irods::experimental::cron::cron::run() in /usr/sbin/irodsServer\n 6# void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, main::$_10> >(void*) at rodsServer.cpp:?\n 7# 0x00007FEFCEA10609 in /lib/x86_64-linux-gnu/libpthread.so.0\n 8# clone in /lib/x86_64-linux-gnu/libc.so.6\n\n","server_host":"opendata-20","server_pid":642603,"server_timestamp":"2024-04-26T12:13:19.446Z","server_type":"server","server_zone":"earlhamZone"}

I wonder if having another user with admin privileges or that this machine is federated to another could be related to the problem, but It looks as if I have to update the password in another file/place.

[korydraughn]

Some parts of the server do not respond to those types of changes. For example, the delay server.

Because of that, processes such as the delay server will continue to use old information until it is restarted.

The recommendation is ... if you change the password of the iRODS user responsible for managing the server (i.e. the user under the service account), restart the iRODS server.

That also applies to any clients using pooling and relying on rodsadmin-level accounts which have had their password changed (e.g. NFSRODS, HTTP API (if not running in 4.2 compatibility mode)).

Another way to protect against this is to create a secondary rodsadmin account for carrying out administrative tasks only. This allows you to leave the iRODS account responsible for the running of the server as is.

We'll add documentation explaining this.

[DOC-MEX]

I see, if I create a dedicated user for admin task, should I use this new user to replace rods in server_config.json, I mean, change the value of zone_user from "zone_user": "rods" to "zone_user": "<New_admin_user>"? and how can I prevent the user rods being used if cannot really change its old default password?

[korydraughn]

I see, if I create a dedicated user for admin task, should I use this new user to replace rods in server_config.json, I mean, change the value of zone_user from "zone_user": "rods" to "zone_user": "<New_admin_user>"?

No. Creating the secondary rodsadmin simply allows you to do things to the server without affecting tasks managed by the primary rodsadmin. You still shouldn't change the password of the primary rodsadmin though.

and how can I prevent the user rods being used if cannot really change its old default password?

If the goal is to change the password of the original rodsadmin, then a secondary rodsadmin account won't help. Once you change the password, you must restart the server. There's no way around that.

[DOC-MEX]

OK! Well, the only issue I am seeing is that when I change the password of the original rodsadmin and restart the server, I still get the CAT_INVALID_AUTHENTICATION error. even if I create another adming user, I don't want to leave the rods user with the default well-know password... I suppose I should have changed it when I reinstall irods on this machine, maybe that would be the actual final solution.

[trel]

That would definitely 'fix' it. But changing the password should not result in CAT_INVALID_AUTHENTICATION on restart... that suggests either a typo, or ... something else is happening. Can you confirm your main rodsadmin is NOT using PAM? Not sure what else could be going wrong.

[DOC-MEX]

I see, I will double-check and try again.
Sorry what do you mean by rods using (or not using) PAM? How can I check that?

[korydraughn]

In your irods_environment.json file, look for the property, "irods_authentication_scheme".

What value is that set to?

[trel]

The rodsadmin could be using LDAP or Active Directory to get authenticated... but if you're not sure about it - you're probably using 'native'.

PAM for the service account is not something we have under testing, so I was just mentioning it as another dark corner that could be happening here...

[DOC-MEX]

I don't see that "irods_authentication_scheme" but can see "zone_auth_scheme": "native" in the config file.

[korydraughn]

@DOC-MEX Any new leads on this? Did you fix it?

Can you list the exact sequence of steps you took (with commands) to encounter this situation? Start from the working system.

[DOC-MEX]

@korydraughn sorry, I did not explore this further, I simply reinstalled it and and set up a different password for rods from the very beginning.

[korydraughn]

No problem. Did the reinstall work as you expect?

[DOC-MEX]

yeah, that solved my issue.

[korydraughn]

I've confirmed there's an issue with changing the password of the rodsadmin user managing the server.

The messages I see aren't exactly the same, but I see CAT_INVALID_AUTHENTICATION every few seconds which means it must have something to do with the delay server migration logic. The log messages you posted also contain references to the migration logic. The server is still responsive too.

This was done using a 4.3.2 server.

We will investigate.