Sync upstream configuration

[kkimurak]

Draft:

This PR syncs configuration files.
Current target versions are based on gitlab-foss v15.0.0:

• gitlab-foss: v15.0.0
• gitaly: v15.0.0
• gitlab-shell: v14.3.0
• gitlab-pages: v1.58.0

This PR is based on #2569 .

TODO:

• add update process to assets/runtime/function
• parameterize
• re-check
• check compatibility for renamed parameter

[sachilles]

@kkimurak The ci pipeline failed at stage test by testing the image bootup. It failes due to connection issues:

• Hostname in DNS cache was stale, zapped
• Trying 127.0.0.1:80...
• connect to 127.0.0.1 port 80 failed: Connection refused
• Failed to connect to localhost port 80 after 0 ms: Connection refused

Is this related to numerous changes in the config.toml?

[sachilles]

Do you have an idea how to proceed or make progress? How can we work together efficiently?

When I see the amount of changes, a four-eyes review would be the best. Do you have any objections and/or suggestions for a platform for real-time exchange?

[sachilles]

Unfortunately, several GitLab critical security releases (15.0.1, 14.10.4, and 14.9.5) have been published yesterday (see https://about.gitlab.com/releases/2022/06/01/critical-security-release-gitlab-15-0-1-released/).

I guess we should maintain the official releases in parallel with this WIP PR. From your point of view, is there any reason not to at least merge the PR #2569 so that the security release can be prepared?

[kkimurak]

Sorry for late, I was bit busy.

@kkimurak The ci pipeline failed at stage test by testing the image bootup. It failes due to connection issues:

• Hostname in DNS cache was stale, zapped
• Trying 127.0.0.1:80...
• connect to 127.0.0.1 port 80 failed: Connection refused
• Failed to connect to localhost port 80 after 0 ms: Connection refused

Is this related to numerous changes in the config.toml?

I think so. Maybe upstream config for [gitlab]::url in assets/runtime/config/gitaly/config.toml is not able to be applied directly. /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket is used in upstream config but we use /home/git/gitlab/tmp/sockets/gitlab.socket when we start gitlab workhorse.

note: I just copied and pasted upstream config, then started checking git diff.

Unfortunately, several GitLab critical security releases (15.0.1, 14.10.4, and 14.9.5) have been published yesterday (see https://about.gitlab.com/releases/2022/06/01/critical-security-release-gitlab-15-0-1-released/).

I guess we should maintain the official releases in parallel with this WIP PR. From your point of view, is there any reason not to at least merge the PR #2569 so that the security release can be prepared?

Just please merge #2569 and prepare for security release. I'm not sure v15.0.0 works but at least test is passing. And I have learned from history that the releasing is much important than missing (or stale) parameters for this project. Thank you for your work.

BTW, I'm going to work on this PR this weekend (since 2022-06-03T20:00:00+0900)

[sachilles]

This sounds good. I can confirm that GitLab CE works at least for me in version 15.0.0 and 15.0.1. The only downside is that I do not use all features.

[sachilles]

I just added an additional section about the state (important notes) of the release in the release notes.

[kkimurak]

I'm back, confirmed that gitlab works on relative URL enabled/disabled. Also simple git push/pull works.

Here're list of changes for configurations.

• gitaly
  • config.toml
    • add commented-out config runtime_dir
    • fix value for commented-out config tls_listen_addr (lacking trailing " )
    • add commented-out config for git (fetch.fsckObjects=true)
    • add custom_hooks_dir : moved from gitlab-shell configuration. see deprecations page. TODO: parameterize?
    • move [gitlab] section before [[concurrency]] section
    • add section [gitlab.http-settings]
      • TODO: apply parameters
    • add many more commented-out configurations for resource limitation
• gitlab-pages
  • config
    • add listen-http, pages-root, pages-domain
    • sort based on upstream config
• gitlab-shell
  • config.yml
    • add comment for gitlab_relative_url_root (we don't use http+unix:// protocol so we have nothing to do)
    • remove deprecated http_settings::self_signed_cert : see upstream mr 602 : since v13.26.0, v14.0.0
    • add ssl_cert_dir: TODO maybe we need to set it using SSL_CERTIFICATE_PATH, but this is the path for file itself. We need directory path.
    • default log format is changed to json : see upstream MR 476 : since v13.21.0
• gitlabhq
  • database.yml: No notable update. just added commented-out load-balancing configurations
    • stale: official documentation says that DB_POOL (pool: in database.yml) is not very practical and gitlab override the number of allowed connections in the database connection-pool based on the configured number of application threads. See docs about client-side connection pool
  • gitlab.yml: too many changes
    • Merge Request -> merge request
    • add header note for those who submitting MR
    • some URLs for docs are updated. Eg. http://doc.gitlab.com/ce is changed to https://docs.gitlab.com/ee
    • some empty line are removed
    • add commented-out config gitlab:cdn_host to support Rails asset host: see upstream MR 67710 : since v14.2.0 : TODO: parameterize?
    • remove unicorn from comment, now only Puma configurations are written.
    • add gitlab:allowed_hosts to control Rails.application.config.hosts : see upstream MR 55491 : since v13.10.0 : TODO: parameterize?
    • add commented-out config gitlab:email_smtp_secret_file: see upstream MR 67802 : since v14.2.0 : TODO: parameterize?
    • add commented-out config gitlab:application_settings_cache_seconds : see upstream MR 51889 : since v13.9.0 : TODO: parameterize?
    • add commented-out config display_initial_root_password (default to false): Security commit, no MR. see upstream commit b48d80d : since v14.5.0
    • many update on incoming_email section. They are all commented out.q
    • add object_store section on top-level as default value : TODO: parameterize? (need to check compatibility with GITLAB_OBJECT_STORE_CONNECTION_PROVIDER and so on)
    • add local_store section on top-level : see upstream MR 55470 : since v13.11.0
    • add packages:dpkg_deb_path ; see upstream MR 44029 : since v13.6.0
    • rename packages:path to packages:storage_path
    • add ci_secure_files section for secure files API: see upstream MR 78227 : since v14.8.0 : TODO: parameterize?
    • add commented-out config ldap:secret_file
    • add commented-out config ldap:servers:main:retry_empty_result_with_codes
    • add commented-out config kerberos:simple_ldap_linking_allowed_realms
    • add commented-out config omniauth:saml_message_max_byte_size
    • add forti_authenticator section for one time password method : see upstream MR 45055 : since v13.5.0
    • add forti_token_cloud section
    • add encrypted_settings section and commented-out config entrypted_settings:path
    • remove gitaly:client_path : see upstream MR 61987 : since 14.0.0
    • some additional configs for backup:upload
    • remove pseudonymizer configurations : see upstream MR 86087 : since v15.0.0
    • many update on cron_jobs (also on ee_cron_jobs that we don't use) : deleted or modified schedules ware not parameterized.
    • many more updates on non-production (develop/test) settings
  • puma.rb: No notable changes for us.
    • shorter call for Gitlab::Cluster::PumaWorkerKillerInitializer.start
    • additional behavior via environment variable PUMA_WAIT_FOR_LESS_BUSY_WORKER (default to 0.001) and DISABLE_PUMA_NAKAYOSHI_FORK
    • add low-level error handling to Sentry. See upstream MR 66712
  • resque.yml: No notable changes.
    • Merge Request -> merge request
    • DB wording: slave -> replica
    • update URL for docs
  • secrets.yml: No notable changes for configuration.
    • note: minimum secret key length is increased from 30 chars to 32 chars. first introduced: v13.12.0 : see upstream MR 53602
    • note: Do we need to parameterize openid_connect_signing_key and encrypted_settings_key_base?
  • smtp_settings.rb: NO notable changes for configuration.
    • Merge Request -> merge request
    • add commented-out configurations example for those who uses encrypted smtp credentials.
      to achieve this, add local variable secrets to refer Gitlab::Email::SmtpConfig.secrets.{user_name,password}

note: http+unix protocol is used in some configuration but I kept traditional value http://localhost:8181{GITLAB_RELATIVE_URL} for simplicity.

[kkimurak]

Additional note: I have noticed that puma dies on startup so that gitlab did not come up. I have submit a separate PR #2579

[kkimurak]

Now I can continue developing thanks to #2579 is merged

[kkimurak]

@sachilles Is there a backporting request? If so, I'll split the commit to make it easier (although it will take some time, of course).

[sachilles]

@kkimurak If we will be able to merge in the near future, I'd like to avoid backporting.

[sachilles]

@kkimurak Sorry for the extremely long delay in processing this PR. Can we try to incorporate this PR next (and before the next release of GitLab)?

[kkimurak]

Hi @sachilles , I apologize for leaving the pull request for so long.
I thought a lot, but I want to split this PR into some feature-based individual PRs (and minor typo PRs). This PR contains too many changes so that make diff larger, it makes harder to review the functionality.

As usual, the new version of GitLab will be released on the 22nd. I'm sorry, but I've been busy with work this month and it may be difficult to get things done (including checking against the new release, rebase on some concurrent works e.g. #2598 #2610 ) by then. I think I will be able to work on it in early October.

[kkimurak]

I'm still busy and progress is slow, but the work is definitely nearing completion. Most of the changes were cleaned up and sorted by released tag. A little work left over from v15.4 and confirmation of new additions in v15.5 is required.
Anyway, I'm really sorry I'm a month late to do just this. I hope I can work on in a week...

[kkimurak]

I also want to split them into separate pull requests for easier review.
I haven't decided where to split it, but if you have any suggestions (and requests for access via configuration parameters that are currently largely untouched) I'd be happy to hear them.