Web log-in broken

[djelenc]

I have an issue with web-login. We are using gitlab 16.10.1 and at some point, it has become impossible for users to to log-in via web. (If you believe this is a gitlab issue, please let me know and I'll try my luck upstream.)

Unfortunately, I cannot tell whether the issue began with this version of Gitlab or with some other prior to that.

I am sure, however, that the issue did not manifest at an upgrade, but at a later point. What I mean is that after the upgrade, I immediately checked the instance, and it worked. The issue came about only after existing login credentials expired and the web application required users to perform a fresh log in. Users cannot log in since.

When logging in, Gitlab web app always responds by saying credentials are invalid. However, we are certain the credentials are in fact valid:

• I've reset a few users' passwords using rails console, but they are still unable to log in.
• However, the users can access projects (e.g. git clone) using git over HTTPS with their usernames and passwords. In this case (the same) credentials are accepted as valid.

The issue seems to be related to the web application itself. I’m looking at the logs, but I cannot see anything useful. (I’ve checked basically every log in /var/log/gitlab/gitlab/*.log, but I see no errors.)

Some more background:

• We were using LDAP and basic Gitlab login to authenticate users, but the organization has since turned off the LDAP. (We are to switch to OIDC next, but this is not yet implemented in our gitlab instance.)
• After the organization turned off the LDAP, users were able to login via web using their Gitlab credentials without problems.
• Now, out of a sudden, this is no longer possible.
• I am running an nginx reverse-proxy in front of this docker-compose setup.

Any idea will be greatly appreciated. Below I enclose my docker-compose config.

version: '2.3'

services:
  redis:
    restart: always
    image: redis:6.2
    command:
    - --loglevel warning
    volumes:
    - redis-data:/data:Z

  postgresql:
    restart: always
    image: sameersbn/postgresql:14-20230628
    volumes:
    - postgresql-data:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm,btree_gist

  gitlab:
    restart: always
    image: sameersbn/gitlab:16.10.1
    depends_on:
    - redis
    - postgresql
    ports:
    - "22:22"
    - "127.0.0.1:10080:80"
    volumes:
    - gitlab-data:/home/git/data:Z
    healthcheck:
      test: ["CMD", "/usr/local/sbin/healthcheck"]
      interval: 5m
      timeout: 10s
      retries: 3
      start_period: 5m
    environment:
    - DEBUG=false

    - GITLAB_PAGES_ENABLED=true
    - GITLAB_PAGES_HTTPS=true
    - GITLAB_PAGES_PORT=443
    - GITLAB_PAGES_DOMAIN=...
    - GITLAB_PAGES_ACCESS_CONTROL=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=...
    - GITLAB_TIMEZONE=...

    - GITLAB_HTTPS=true
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=...redacted...
    - GITLAB_PORT=443
    - GITLAB_SSH_PORT=22
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE=...
    - GITLAB_SECRETS_SECRET_KEY_BASE=...
    - GITLAB_SECRETS_OTP_KEY_BASE=...

    - GITLAB_ROOT_PASSWORD=
    - GITLAB_ROOT_EMAIL=

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=...
    - GITLAB_EMAIL_REPLY_TO=...
    - GITLAB_INCOMING_EMAIL_ADDRESS=...

    - GITLAB_BACKUP_SCHEDULE=weekly
    - GITLAB_BACKUP_TIME=04:00

    - SMTP_ENABLED=true
    - SMTP_DOMAIN=...
    - SMTP_HOST=...
    - SMTP_PORT=...
    - SMTP_USER=
    - SMTP_PASS=
    - SMTP_STARTTLS=false
    - SMTP_AUTHENTICATION=login

    - IMAP_ENABLED=false
    - IMAP_HOST=imap.gmail.com
    - IMAP_PORT=993
    - IMAP_USER=mailer@example.com
    - IMAP_PASS=password
    - IMAP_SSL=true
    - IMAP_STARTTLS=false


    - OAUTH_ENABLED=false
    - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    - OAUTH_ALLOW_SSO=
    - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    - OAUTH_AUTO_LINK_LDAP_USER=false
    - OAUTH_AUTO_LINK_SAML_USER=false
    - OAUTH_AUTO_LINK_USER=true
    - OAUTH_EXTERNAL_PROVIDERS=

    - OAUTH_CAS3_LABEL=cas3
    - OAUTH_CAS3_SERVER=
    - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    - OAUTH_CAS3_LOGIN_URL=/cas/login
    - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    - OAUTH_GOOGLE_API_KEY=
    - OAUTH_GOOGLE_APP_SECRET=
    - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    - OAUTH_FACEBOOK_API_KEY=
    - OAUTH_FACEBOOK_APP_SECRET=

    - OAUTH_TWITTER_API_KEY=
    - OAUTH_TWITTER_APP_SECRET=

    - OAUTH_GITHUB_API_KEY=
    - OAUTH_GITHUB_APP_SECRET=
    - OAUTH_GITHUB_URL=
    - OAUTH_GITHUB_VERIFY_SSL=

    - OAUTH_GITLAB_API_KEY=
    - OAUTH_GITLAB_APP_SECRET=

    - OAUTH_BITBUCKET_API_KEY=
    - OAUTH_BITBUCKET_APP_SECRET=
    - OAUTH_BITBUCKET_URL=

    - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    - OAUTH_SAML_IDP_SSO_TARGET_URL=
    - OAUTH_SAML_ISSUER=
    - OAUTH_SAML_LABEL="Our SAML Provider"
    - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    - OAUTH_SAML_GROUPS_ATTRIBUTE=
    - OAUTH_SAML_EXTERNAL_GROUPS=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_USERNAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    - OAUTH_CROWD_SERVER_URL=
    - OAUTH_CROWD_APP_NAME=
    - OAUTH_CROWD_APP_PASSWORD=

    - OAUTH_AUTH0_CLIENT_ID=
    - OAUTH_AUTH0_CLIENT_SECRET=
    - OAUTH_AUTH0_DOMAIN=
    - OAUTH_AUTH0_SCOPE=

volumes:
  redis-data:
  postgresql-data:
  gitlab-data:

[frenchbeard]

When you say

We were using LDAP and basic Gitlab login to authenticate users, but the organization has since turned off the LDAP. (We are to switch to OIDC next, but this is not yet implemented in our gitlab instance.)

was it a mix of both regardless of user types, or differences between local privileged roles and LDAP-bound unprivileged users, or a mix of both account types for all users ?

Since LDAP sync doesn't happen all the time, but should be once a day if default configuration is kept (and checked only if authenticating a LDAP user through any authentication provider). As mentioned here : https://docs.gitlab.com/ee/administration/auth/ldap/#security, it may be linked with the ldap_blocked status which results in any auth type to fail for the user (not sure if displayed in the admin panel, though).

It may be a little far-fetched, but couldn't the following have happened:

• when both LDAP and Gitlab credentials were used, most users ended up as "LDAP" users with a DN
• when LDAP was disabled, they kept the attribute, and the associated DN, but until a password change, no check was made against the LDAP (there's the speculation on my part), and users without ability to be checked against the previous LDAP gained ldap_blocked in the process
• since this doesn't get removed by a password reset, such state would have persisted through your attempts

If not done already, a quick check for this particular attribute through the rails console may yield something.

This still wouldn't explain the HTTP auth working, but I'm not knowledgeable enough about their differences in the internals to be sure. Also, since I never tested the case where you disabled LDAP, no idea if I'm just overspeculating, hope this helps in any way.

[djelenc]

Thank you for your feedback, @frenchbeard. To answer your question.

Was it a mix of both regardless of user types, or differences between local privileged roles and LDAP-bound unprivileged users, or a mix of both account types for all users ?

There are two users that were able to login in with their local accounts and their LDAP accounts when the system worked. These users are admins of the Gitlab instance.

Then there is a bunch of normal users that only had LDAP accounts, not local ones. The instance is set in a way that the admin of Gitlab had to confirm their (LDAP) account before they could actually use the Gitlab instance. (The process was such that after the first log-in using LDAP credentials, an account was created, but it was immediately blocked. And admin could unblock the account and give the user then access.)

Also, I’ve checked the status of the two local (admin) usesrs and it is set to active, not ldap_blocked. Here are pretty-printed attributes for my user (local and admin account) with some fields redacted for privacy.

{
    "id"=>2,
    "email"=>"...",
    "encrypted_password"=>"$2a$10$...",
    "reset_password_token"=>nil,
    "reset_password_sent_at"=>nil,
    "remember_created_at"=>Fri, 26 Jan 2024 09:54:46.894155000 CET +01:00,
    "sign_in_count"=>604,
    "current_sign_in_at"=>Wed, 27 Mar 2024 15:02:01.946820000 CET +01:00,
    "last_sign_in_at"=>Wed, 27 Mar 2024 15:01:53.495105000 CET +01:00,
    "current_sign_in_ip"=>"....",
    "last_sign_in_ip"=>"....",
    "created_at"=>Tue, 11 Mar 2014 16:59:03.598136000 CET +01:00,
    "updated_at"=>Fri, 26 Apr 2024 10:08:20.440457000 CEST +02:00,
    "name"=>"David ...",
    "admin"=>true,
    "projects_limit"=>99,
    "failed_attempts"=>0,
    "locked_at"=>nil,
    "username"=>"david",
    "can_create_group"=>true,
    "can_create_team"=>true,
    "state"=>"active",
    "color_scheme_id"=>3,
    "password_expires_at"=>nil,
    "created_by_id"=>1,
    "avatar"=>"avatar.png",
    "confirmation_token"=>".....",
    "confirmed_at"=>Tue, 11 Mar 2014 16:59:03.486587000 CET +01:00,
    "confirmation_sent_at"=>Wed, 06 Feb 2019 13:41:12.348830000 CET +01:00,
    "unconfirmed_email"=>nil,
    "hide_no_ssh_key"=>false,
    "last_credential_check_at"=>Sun, 04 Feb 2024 22:54:02.095932000 CET +01:00,
    "notification_email"=>"....",
    "hide_no_password"=>false,
    "password_automatically_set"=>false,
    "encrypted_otp_secret"=>nil,
    "encrypted_otp_secret_iv"=>nil,
    "encrypted_otp_secret_salt"=>nil,
    "otp_required_for_login"=>false,
    "otp_backup_codes"=>nil,
    "public_email"=>"",
    "dashboard"=>"projects",
    "project_view"=>"files",
    "consumed_timestep"=>nil,
    "layout"=>"fixed",
    "hide_project_limit"=>false,
    "unlock_token"=>nil,
    "otp_grace_period_started_at"=>nil,
    "external"=>false,
    "incoming_email_token"=>"......",
    "require_two_factor_authentication_from_group"=>false,
    "two_factor_grace_period"=>48,
    "last_activity_on"=>Fri, 26 Apr 2024,
    "notified_of_own_activity"=>false,
    "preferred_language"=>"en",
    "theme_id"=>nil,
    "include_private_contributions"=>false,
    "feed_token"=>".......",
    "accepted_term_id"=>nil,
    "private_profile"=>false,
    "commit_email"=>nil,
    "auditor"=>false,
    "admin_email_unsubscribed_at"=>nil,
    "group_view"=>nil,
    "managing_group_id"=>nil,
    "note"=>nil,
    "roadmap_layout"=>nil,
    "static_object_token"=>nil,
    "first_name"=>nil,
    "last_name"=>nil,
    "role"=>"software_developer",
    "user_type"=>"human",
    "static_object_token_encrypted"=>nil,
    "otp_secret_expires_at"=>nil,
    "onboarding_in_progress"=>false,
    "color_mode_id"=>1,
    "otp_secret"=>nil
}