[TASK][MEDIUM] Unlock advanced features of markdown in docs

[pan3793]

What's the level of this task?

MEDIUM

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

• Sphinx, and maybe some HTML and CSS skills

Background and Goals

Kyuubi uses Sphinx to build the docs, and RST and Markdown are mixed-used.

RST is a much more powerful format but easy to mess up and hard to learn, while Markdown is more popular and easy to learn.

Currently, Kyuubi uses a legacy markdown==3.4.1 for Sphinx markdown support, which lacks some advanced features, e.g. variable substitution, which is likely to be used for "version"

While the recommended powerful markdown solution is myst_parser https://www.sphinx-doc.org/en/master/usage/markdown.html, we should investigate and migrate to this one.

Implementation steps

• configure Sphinx to use myst_parser
• make sure everything in docs looks well, especially the configuration page https://kyuubi.readthedocs.io/en/master/configuration/settings.html
• converting at least one RST docs to markdown which has variable substitution

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program