demultiplex_script.py

Demutliplex a MiSEQ or NextSEQ run, perform QC using FastQC and MultiQC and deliver files either to VIGASP for analysis or NIRD for archiving

Replace with relevant run id. Example : "190912_M06578_0001_000000000-CNNTP". RunID breaks down like this (date +%y%m%d/yymmdd_MACHINE-SERIAL-NUMBER_AUTOINCREASING-NUMBER-OF-RUN_000000000-FlowcellID-used-for-this-run .

Note: don't bother with enforcing ISO dates for the directory name. It is an Illumina standard and they do not care.

Software requirements

Python > v3.9
bcl2fastq ( from https://emea.support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software/downloads.html )
FastQC    ( https://www.bioinformatics.babraham.ac.uk/projects/fastqc/ )
MultiQC   ( pip3 install multiqc )

Directory structure on seqtech00

├── bin															binaries and symlinks of binaries live here
├── clarity 													exported Illumina Clarity directory
│   ├── gls_events	
│   ├── logs 													clarity logs go here
│   ├── miseq													miseq-clarity stopover directory
│   │   └── M06578												per serial number
│   │   	└── samplesheets									samplesheets for this serial number gohere
│   └── nextseq													nextseq-clarity stopover directory
│   	└── NB552450											per serial number
│   		└── samplesheets									samplesheets for this serial number gohere
├── demultiplex													demultiplexed data directory
├── for_transfer												data ready to be transfered over to NIRD or VIGASP
├── logs														all demultiplexing logs go here
├── rawdata														raw data directory, sequencers write here
│   ├── bad_runs												runs which are bad, or rejected
│   └── control_runs											water/other control runs
└── samplesheets												cummulative backups of all samplesheets
├── M06578 -> /data/clarity/miseq/M06578/samplesheets/			symlinmk to sample sheets for convinience
└── NB552450 -> /data/clarity/nextseq/NB552450/samplesheets/	symlinmk to sample sheets for convinience

Procedure

• MiSeq writes as MiSEQ- to /data/scratch; shared folder Z:\ (alias rawdata) in MiSeq
• Lab members modify an existing SampleSheet.csv file to include the new project data, then save the new file to the <RunId> folder in Z:\ and a copy within Z:\SampleSheets\ as <RunId>\SampleSheet.csv

Example:

Z:\190912_M06578_0001_000000000-CNNTP
        ├── SampleSheets.csv
Z:\SampleSheets
        ├── 190912_M06578_0001_000000000-CNNTP.csv

• Cron job runs every 30 minutes if it finds a new run, RTAComplete.txt and SampleSheet.csv files within the run new, it starts the demultiplexing script
• It can be manually started as below

/usr/bin/python3 /data/bin/demultiplex_script.py \<RunID\>

as the relevant user.