Interactive use + Pydantic models for plots

[vladsavelyev]

Use Pydantic

• Plots, tables, plot data (Plot, DataTable, Dataset, and derived classes). Helps validate dumps for JavaScript internally.
• Configuration in interactive functions and command line (ClConfig). Validates run parameters.

Future improvements would include validating custom content with Pydantic.

Partially addresses #1790

Support interactive usage

Support interactive usage of MultiQC, i.e. in notebooks. Partly addresses #1051

Split the multiqc.py into submodules with better separated stages: parse logs, render plots and tables, write data to disk. That allowed to add functions like multiqc.parse_logs(), that parses more logs and appends them into the same report without rendering HTML or writing stuff to disk; multiqc.list_plots() multiqc.show_plot() without rendering all plots; multiqc.write_results() to only render and write results to disk.

Config parameters can be passed individually to any interactive method (like verbose, modules_order, etc).

Can also add custom sections into the report with multiqc.add_custom_content_section().

The write_report() function triggers HTML rendering, module ordering, adds special-case software versions and runs performance modules, and writes data and report. Though all those things are also separated internally and can be exposed more granuarly if needed.

Example notebook

Check this notebook for example usage:

https://github.com/MultiQC/example-notebook/blob/master/multiqc_example.ipynb

Performance benchmark

Tested the branch against main (with all recent performance improvements merged into both). Getting consistent speed up just from the refactoring:

main
Run took 103.14 seconds

• 5.14s: Searching files
• 54.13s: Running modules
• 20.40s: Compressing report data
  3628933760 peak memory footprint

interactive-use-2
Run took 77.98 seconds

• 5.15s: Searching files
• 29.62s: Running modules
• 18.11s: Compressing report data
  3638501952 peak memory footprint

Before performance PRs (196c9738)
Run took 115.36 seconds

• 11.50s: Searching files
• 55.16s: Running modules
• 35.53s: Compressing report data
  5196197376 peak memory footprint