New module: Hostile

[SumeetTiwari07]

• This comment contains a description of changes (with reason)

• There is example tool output for tools in the https://github.com/MultiQC/test-data repository or attached to this PR
• Code is tested and works locally (including with --strict flag)
• docs/modulename.md is created
• Everything that can be represented with a plot instead of a table is a plot
• Report sections have a description and help text (with self.add_section)
• There aren't any huge tables with > 6 columns (explain reasoning if so)
• Each table column has a different colour scale to its neighbour, which relates to the data (e.g. if high numbers are bad, they're red)
• Module does not do any significant computational work

[SumeetTiwari07]

""

[vladsavelyev]

Thanks a lot for the contribution!

Left a few comments.

Also, it's needed to add self.ignore_samples (see https://multiqc.info/docs/development/modules/#filtering-by-parsed-sample-names) and self. add_software_version (see https://multiqc.info/docs/development/modules/#saving-version-information) calls.

        ...
        for f in self.find_log_files("hostile", filehandles=True):
            self.parse_logs(f)

        self.add_software_version(...)

        self.parse_data = self.ignore_samples(self.parse_data)

        if len(self.parse_data) == 0:
            raise ModuleNoSamplesFound

        log.info(f"Found {len(self.parse_data)} reports")

        self.write_data_file(self.parse_data, "multiqc_hostile")
        ...

[vladsavelyev]

Please also create a PR in https://github.com/MultiQC/test-data with test examples.

[SumeetTiwari07]

Thank you for reviewing the code and all the suggestions.
I made the changes that were recommended and updated the code as suggested. 069e1c6,
51f61da
Also added the test-data (MultiQC/test-data#319)

[vladsavelyev]

self.parse_data is already assumed to be indexed by sample names (i.e. self.ignore_samples takes that assumption), so we can't create a different sample name here. Better move this line into parse_logs. And clean with self.clean_s_name() instead of manually calling split (the function knows about the extensions to remove)

[SumeetTiwari07]

for f_name, values in self.parse_data.items():
    database = os.path.basename(values[0]["index"])
    data[f_name] = {"Cleaned reads": values[0]["reads_out"], "Host reads": values[0]["reads_removed"]}

Removed that line.

[vladsavelyev]

database is initialized in the loop for each sample separately and this line is outside of the loop. Need to decide how to handle the situation if samples have different database

[SumeetTiwari07]

Since Hostile does not allow the screening against multiple databases in a single run, a batch of sequences will be screened against the same database. However, modified the code in which the database is joined with the categories.

        for f_name, values in self.parse_data.items():
            database = os.path.basename(values[0]["index"])
            data[f_name] = {f"Cleaned reads (DB: {database})": values[0]["reads_out"], f"Host reads (DB: {database})": values[0]["reads_removed"]}

        ## categories
        all_categories = [inner_key for outer_dict in data.values() for inner_key in outer_dict.keys()]
        cats = list(set(all_categories))
        #cats = ["Cleaned reads", "Host reads"]

[SumeetTiwari07]

Does it make sense?

[SumeetTiwari07]

Suggested change

	description=f"This plot shows the number of cleaned reads vs host-reads per sample (database index: {database}).",
	description=f"This plot shows the number of cleaned reads vs host-reads per sample (DB: Database).",

[vladsavelyev]

Can there be more than one entry in the JSON file? How that should be handled? Please add such example to the test-data repo.

[SumeetTiwari07]

No there won't be more than one entry in the JSON file.
if len(parse_data) > 0: deleted from conde.

Suggested change

	if len(parse_data) > 0:

[vladsavelyev]

Should reads_out and reads_removed sum up to reads_in? Doesn't look like in the test data :

cat test_data/data/modules/hostile/hostile.SAMPLE-A1.json
[
    {
        "version": "1.1.0",
        "aligner": "bowtie2",
        "index": "human-t2t-hla",
        "options": [],
        "fastq1_in_name": "SAMPLE-A1.fastq.gz",
        "fastq1_in_path": "SAMPLE-A1.fastq.gz",
        "fastq1_out_name": "SAMPLE-A1.clean.fastq.gz",
        "fastq1_out_path": "SAMPLE-A1.clean.fastq.gz",
        "reads_in": 241805,
        "reads_out": 234757,
        "reads_removed": 70484,
        "reads_removed_proportion": 0.2915
    }
]

Bar plots assume categories do not overlapp

[SumeetTiwari07]

Yes, it should. I will update the count in the reports.

[SumeetTiwari07]

Pull request raised to change the counts.

• 322
• 321

[vladsavelyev]

We should use real outputs from the tool in the test data. Can you run Hostile to generate some?

[vladsavelyev]

I pushed some updates, and with the test example MultiQC/test-data@2eee448, we are good to go - merging this PR now.

[SumeetTiwari07]

I pushed some updates, and with the test example MultiQC/test-data@2eee448, we are good to go - merging this PR now.

Thank you so much. I too added some more logs from a metagenomic sampling.