[FEATURE] Create Longhorn CLI focusing on non CR resource operations

[innobead]

Is your feature request related to a problem? Please describe (👍 if you like this request)

Create a Longhorn CLI focusing on non-custom-resource operations, as users can operate resources via Kubectl or similar tools, K8s user experience. However, some operations like install prerequisites, preflight check before installation/upgrade, runtime periodical check, headless support bundle download, check known issue mentioned in the knowledge pages, etc. are only partially operated through UI.

We should provide a CLI to help with these headless operations for general functions, debugging, benchmarking, troubleshooting, etc.

Change https://github.com/longhorn/longhorn-preflight to longhorn/cli and implement the corresponding functions.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[shuo-wu]

We can leverage this CLI to guide users understand & config Longhorn systems on demands:

1. Use an interactive way to help config Longhorn nodes, disks, settings, backups, and so on.
2. Step by Step. In each step, we will explain what's for and list all options.

For example:

$ longhorn-cli guide

Do you need to launch v1 volumes in your clusters? v1 volumes are low-spec friendly but have limited performance.
yes/no: yes

How many CPUs you wish to reserve for the component (longhorn-instance-manger pod) that runs v1 volumes on each node?
Please input the reserved CPU percentage:

......

======

Do you need to add extra disks for the below nodes?
  1. Node 1: default disk
  2. Node 2: default disk
  3. Node 2: default disk
Please input the nodes we want to add disks: 1, 2, 3

The available disk type is:
  1. Block device (for v2 volumes)
  2. Filesystem (for v1 volumes)
Please select the disk type: 2

Please input the disk path: /var/lib/longhorn-extra

......
======

Do you have a backupstore?
  Currently, Longhorn supports:
  1. NFS. nfs://xxxxxx
  2. S3. xxx
  3. CIFS. xxx
If YES, please input the backup target: s3://xxxxx

What's the credential of the backup target?
Please input the secret name: minio-secret 
......

---

Besides, this CLI can handle some complicated operations such as recurring-job creation and configuration. Without this, users have to get a template YAML, check all fields and the corresponding options, modify it, create CRs, and bind it to some volumes. It's too trivial. An interactive CLI definitely helps.

[derekbit]

Some functionalities of the CLI

• Installation
  • Environment check ([IMPROVEMENT] Environment setup and check tooling container #6657)
  • GKE COS doc(1.7.0): support GKE Container-Optimized OS website#884
• Filesystem
  • Manual trim ([IMPROVEMENT] Allow users to issue filesystem trim via kubectl #4932)
  • Filesystem check
• Snapshot
  • Snapshot integrity check ([FEATURE] Support manual trigger of data integrity checks and fixes for replicas of a volume #5035)
• Backup and Restore
  • Restore backup without Longhorn system (https://longhorn.io/docs/1.6.0/advanced-resources/data-recovery/export-from-replica/)
  • Backup integrity check ([IMPROVEMENT] Backup and restore data Integrity check and repair #4571)
• Upgrade pre-check
  • [IMPROVEMENT] Pre-check Longhorn cluster conditions before upgrading Longhorn #8343

[c3y1huang]

Filesystem

• Filesystem check

@derekbit , could you help to create an issue for the filesystem check with some context?

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • feat: Longhorn toolbox CLI cli#14
  • feat: Longhorn toolbox CLI go-common-libs#33
  • feat: Longhorn toolbox CLI longhorn-engine#1096

• Which areas/issues this PR might have potential impacts on?
  Area troubleshoot, CLI
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at Feat(LEP): Longhorn toolbox CLI #8428

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at TBU

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at TBU

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at