Auto-docs: generate pages for properties from the code

[Deflaimun]

Resolves https://github.com/redpanda-data/documentation-private/issues/1969, https://github.com/redpanda-data/documentation-private/issues/853, https://github.com/redpanda-data/documentation-private/issues/1979, https://github.com/redpanda-data/documentation-private/issues/2059, https://github.com/redpanda-data/documentation-private/issues/2184, https://github.com/redpanda-data/documentation-private/issues/2273, https://github.com/redpanda-data/documentation-private/issues/2297, https://github.com/redpanda-data/documentation-private/issues/2277, https://github.com/redpanda-data/documentation-private/issues/2376, https://github.com/redpanda-data/documentation-private/issues/2375, https://github.com/redpanda-data/documentation-private/issues/2374, https://github.com/redpanda-data/documentation-private/issues/2379, https://github.com/redpanda-data/documentation-private/issues/2417, https://github.com/redpanda-data/documentation-private/issues/2298, https://github.com/redpanda-data/documentation-private/issues/2275, https://github.com/redpanda-data/documentation-private/issues/2044, https://github.com/redpanda-data/documentation-private/issues/1518, https://github.com/redpanda-data/documentation-private/issues/2146, https://github.com/redpanda-data/documentation-private/issues/2097, https://github.com/redpanda-data/documentation-private/issues/1846, https://github.com/redpanda-data/documentation-private/issues/1634, https://github.com/redpanda-data/documentation-private/issues/1432, https://github.com/redpanda-data/documentation-private/issues/1383, https://github.com/redpanda-data/documentation-private/issues/1268, https://github.com/redpanda-data/documentation-private/issues/1197, https://github.com/redpanda-data/documentation-private/issues/1029, https://github.com/redpanda-data/documentation-private/issues/647, https://github.com/redpanda-data/documentation-private/issues/2161

Review deadline: May 23rd

Previews:

https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/properties/broker-properties/
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/properties/cluster-properties/
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/properties/object-storage-properties/
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/upgrade/deprecated/

New index page
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/properties/

Modifications

This PR removes the hand-written pages:
https://docs.redpanda.com/current/reference/cluster-properties/
https://docs.redpanda.com/current/reference/tunable-properties/
https://docs.redpanda.com/current/reference/node-properties/
https://docs.redpanda.com/current/reference/node-configuration-sample/

And replaces with auto-generated pages:
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/broker-properties/
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/cluster-properties/

New auto-generated page added:
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/reference/properties/object-storage-properties/

Deprecated properties that were found on the automation are written in
https://deploy-preview-334--redpanda-docs-preview.netlify.app/current/upgrade/deprecated/

Source

Values are extracted from the code.
Properties automation project related PR #365

Source can be found https://github.com/redpanda-data/docs/pull/365/files#diff-69553b5d91035866cd1ac289a68c80ca9a35fb43ef43b9a792f7375bbfba5192

Metadata:

Total properties read 450
Total Broker properties read 51
Total Cluster properties read 289
Total Cloud properties read 81

There´s 2 properties with empty description. Percentage of error 0.44%.
There´s 41 properties with empty visibility. Percentage of error 9.11%.
There´s 27 deprecated_properties. Percentage 6.0%.

[netlify]

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	57f7550
🔍 Latest deploy log	https://app.netlify.com/sites/redpanda-docs-preview/deploys/664fa599edc0da0008dcb415
😎 Deploy Preview	https://deploy-preview-334--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Deflaimun]

ENG: Questions about visibility.
Currently we have these values:

user
tunable
None (null)

Current guidance is
rpk cluster config export will export all properties with user visibility
rpk cluster config export --all will export both user and tunable .

Questions:
1 - What happens when visibility is not set (None,null)?
2 - The current guidance shows an rpk command to apparently export cluster properties. Does the same command also export broker properties?

There's currently 41 properties with empty visibility

advertised_pandaproxy_api
aggregate_metrics
api_doc_dir
broker_tls
brokers
client_cache_max_size
client_identifier
client_keep_alive
cluster_id
consumer_heartbeat_interval
consumer_instance_timeout
consumer_rebalance_timeout
consumer_request_max_bytes
consumer_request_min_bytes
consumer_request_timeout
consumer_session_timeout
disable_metrics
disable_public_metrics
group_max_session_timeout_ms
group_min_session_timeout_ms
kafka_rpc_server_tcp_recv_buf
kafka_rpc_server_tcp_send_buf
pandaproxy_api
pandaproxy_api_tls
produce_ack_level
produce_batch_delay
produce_batch_record_count
produce_batch_size_bytes
produce_compression_type
produce_shutdown_delay
retries
retry_base_backoff
rpc_client_connections_per_peer
rpc_server_tcp_recv_buf
rpc_server_tcp_send_buf
sasl_mechanism
schema_registry_api
schema_registry_api_tls
schema_registry_replication_factor
scram_password
scram_username

[Deflaimun]

The rpk properties weren't captured by the automation. Where can we obtain them on source code? @BenPope

rpk:
  # Add optional flags to have rpk start Redpanda with specific parameters.
  # The available start flags are found in: /src/v/config/configuration.cc
  additional_start_flags:
    - "--overprovisioned"
    - "--smp=2"
    - "--memory=4G"
    - "--default-log-level=info"

  # The Kafka API configuration
  kafka_api:
    # A list of broker addresses that rpk will use
    brokers:
    - 192.168.72.34:9092
    - 192.168.72.35:9092

    # The TLS configuration to be used when interacting with the Kafka API.
    # If present, TLS is enabled. If missing or null, TLS is disabled.
    tls:
      # The path to the client certificate (PEM). Only required if client authentication is
      # enabled in the broker.
      cert_file: ~/certs/cert.pem
      # The path to the client certificate key (PEM). Only required if client authentication is
      # enabled in the broker.
      key_file: ~/certs/key.pem
      # The path to the root CA certificate (PEM).
      truststore_file: ~/certs/ca.pem

    # The SASL config, if enabled in the brokers.
    sasl:
      user: user
      password: 'pass'
      type: SCRAM-SHA-256

  # The Admin API configuration
  admin_api:
    # A list of the brokers' Admin API addresses that rpk will use.
    addresses:
    - 192.168.72.34:9644
    - 192.168.72.35:9644
    # The TLS configuration to be used when with the Admin API.
    # If present, TLS is enabled. If missing or null, TLS is disabled.
    tls:
      # The path to the client certificate (PEM). Only required if client authentication is
      # enabled in the broker.
      cert_file: ~/certs/admin-cert.pem
      # The path to the client certificate key (PEM). Only required if client authentication is
      # enabled in the broker.
      key_file: ~/certs/admin-key.pem
      # The path to the root CA certificate (PEM).
      truststore_file: ~/certs/admin-ca.pem

  # Available tuners. Set to true to enable, false to disable.

  # Setup NIC IRQs affinity, sets up NIC RPS and RFS, sets up NIC XPS, increases socket
  # listen backlog, increases the number of remembered connection requests, bans the
  # IRQ Balance service from moving distributed IRQs.
  # Default: false
  tune_network: false

  # Sets the preferred I/O scheduler for given block devices.
  # It can work using both the device name or a directory, in which the device where
  # directory is stored is optimized. Sets either 'none' or 'noop' scheduler
  # if supported.
  # Default: false
  tune_disk_scheduler: false

  # Disables IOPS merging.
  # Default: false
  tune_disk_nomerges: false

  # Distributes IRQs across cores with the method deemed the most appropriate for the
  # current device type (i.e. NVMe).
  # Default: false
  tune_disk_irq: false

  # Installs a systemd service to run fstrim weekly, or starts the default fstrim service
  # which comes with most Linux distributions.
  # Default: false
  tune_fstrim: false

  # Disables hyper-threading, sets the ACPI-cpufreq governor to 'performance'. Additionaly
  # if system reboot is allowed: disables Intel P-States, disables Intel C-States,
  # disables Turbo Boost.
  # Default: false
  tune_cpu: true

  # Increases the number of allowed asynchronous IO events.
  # Default: false
  tune_aio_events: false

  # Syncs NTP.
  # Default: false
  tune_clocksource: true

  # Tunes the kernel to prefer keeping processes in-memory instead of swapping them out.
  # Default: false
  tune_swappiness: false

  # Enables transparent hugepages (THP) to reduce TLB misses.
  # Default: false
  tune_transparent_hugepages: false

  # Enables memory locking.
  # Default: false
  enable_memory_locking: false

  # Installs a custom script to process coredumps and save them to the given directory.
  # Default: false
  tune_coredump: false

  # The directory where all coredumps are saved after they're processed.
  # Default: ''
  coredump_dir: "/var/lib/redpanda/coredump"

  # Creates a "ballast" file, so if a Redpanda broker runs out of space,
  # you can delete the ballast file to allow the broker to resume operations and then
  # delete a topic or records to reduce the space used by Redpanda.
  # Default: false
  tune_ballast_file: false

  # The path where the ballast file is created.
  # Default: "/var/lib/redpanda/data/ballast"
  ballast_file_path: "/var/lib/redpanda/data/ballast"

  # The ballast file size.
  # Default: "1GiB"
  ballast_file_size: "1GiB"

  # (Optional) The vendor, VM type, and storage device type that Redpanda runs on, in
  # the format <vendor>:<vm>:<storage>. This hints to rpk which configuration values it
  # should use for the Redpanda IO scheduler.
  # Default: ''
  well_known_io: "aws:i3.xlarge:default"

[BenPope]

The rpk properties weren't captured by the automation. Where can we obtain them on source code? @BenPope

I'm not even sure who ask, maybe those options exist under here: https://github.com/redpanda-data/redpanda/commits/dev/src/go/rpk/pkg/redpanda ?

[Deflaimun]

Properties with empty descriptions:

• cloud_storage_upload_ctrl_update_interval_ms
• compaction_ctrl_update_interval_ms

[Feediver1]

Looks great @Deflaimun !