SquiggleKit

A toolkit for accessing and manipulating nanopore signal data

Full documentation: https://psy-fer.github.io/SquiggleKitDocs/

publication: SquiggleKit: A toolkit for manipulating nanopore signal data

Pre-print: SquiggleKit: A toolkit for manipulating nanopore signal data

Cleanup in progress

Coming changes

Fast5_fetcher: merge single files into multi-fast5 files

SquigglePull: python3, read from multi-fast5

SquigglePlot: python3, read from multi-fast5, image size args, arg clean-up

Segmenter: dynamic file formats and more stability

MotifSeq: Improved background modelling, custom modelling, RNA specific tools, custom alignment methods

Overview

Tool	Category	Description
Fast5_fetcher	File management	Fetches fast5 files given a filtered input list
SquigglePull	Signal extraction	Extracts event or raw signal from data files
SquigglePlot	Signal visualisation	Visualisation tool for signal data
Segmenter	Signal analysis	Finds adapter stall, and homopolymer regions
MotifSeq	Signal analysis	Finds nucleotide sequence motifs in signal, i.e.“Ctrl+F”

Requirements

Following a self imposed guideline, most things written to handle nanopore data or bioinformatics in general, will use as little 3rd party libraries as possible, aiming for only core libraries, or have all included files in the package.

In the case of fast5_fetcher.py and batch_tater.py, only core python libraries are used. So as long as Python 2.7+ is present, everything should work with no extra steps.

There is one catch. Everything is written primarily for use with Linux. Due to MacOS running on Unix, so long as the GNU tools are installed (see below), there should be minimal issues running it. Windows however may require more massaging. The Windows-Subsystem-Linux must be installed. Follow the instructions here to do this.

SquiggleKit tools were not made to be executable to allow for use with varying python environments on various operating systems. To make them executable, add #! paths, such as #!/usr/bin/env python2.7 as the first line of each of the files, then add the SquiggleKit directory to the PATH variable in ~/.bashrc, export PATH="$HOME/path/to/SquiggleKit:$PATH"

Install

git clone https://github.com/Psy-Fer/SquiggleKit.git

Use pip for python 2 and pip3 for python 3. User environments may vary.

for fast5_fetcher.py, SquigglePull.py, SquigglePlot.py segmenter.py:

• numpy
• matplotlib
• h5py
• sklearn
• ont_fast5_api
• pyslow5

pip install numpy h5py sklearn matplotlib
# do this after to get around an annoying version check bug
pip install pyslow5

for MotifSeq.py:

• all of the above
• scipy
• scrappie
• mlpy 3.5.0 (only use pip3 in python 3 - see below)

pip install scipy scrappie

Installing mlpy:

Python2

• Download the Files
• Install Instructions

Python3.6 (Python3.7 not working because of a cython/gsl issue)

pip3 install machine-learning-py

VBZ compressed fast5

If your fast5 files are compressed with ONT VBZ, then you will need to install their VBZ plugin for hdf5/h5py

https://github.com/nanoporetech/vbz_compression/releases

This will fix errors like this:

OSError: Can't read data (can't open directory: /home/jamfer/squigglekit_env/lib/hdf5/plugin)
extract_fast5_all():failed to read readID: read_6c3c3be4-e5b0-89aa-b0a0-a1c0167ce136 Traceback (most recent call last):
File "SquigglePull.py", line 211, in extract_f5_all
for col in hdf[read]['Raw/Signal'][()]:
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

Quick start

fast5_fetcher

If using MacOS, and NOT using homebrew, install it here:

homebrew installation instructions

then install gnu-tar with:

brew install gnu-tar

Building the index (not required for multi_fast5)

How the index is built depends on which file structure you are using. It will work with both tarred and un-tarred file structures. Tarred is preferred. (zip and other archive methods are being investigated)

- Raw structure (not preferred)

for file in $(pwd)/reads/*/*;do echo $file; done >> name.index

gzip name.index

- Local basecalled structure

for file in $(pwd)/reads.tar; do echo $file; tar -tf $file; done >> name.index

gzip name.index

- Parallel basecalled structure

for file in $(pwd)/fast5/*fast5.tar; do echo $file; tar -tf $file; done >> name.index

If you have multiple experiments, then cat them all together and gzip.

for file in ./*.index; do cat $file; done >> ../all.name.index

gzip all.name.index

Basic use on a local computer

using a filtered paf file as input:

python fast5_fetcher.py -p my.paf -s sequencing_summary.txt.gz -i name.index.gz -o ./fast5

SquigglePull

All raw data:

python SquigglePull.py -rv -p ~/data/test/reads/1/ -f all > data.tsv

Positional event data:

python SquigglePull.py -ev -p ./test/ -t 50,150 -f pos1 > data.tsv

SquigglePlot

Plot individual fast5 file:

python SquigglePlot.py -i ~/data/test.fast5

Plot files in path

python SquigglePlot.py -p ~/data/ --plot_colour -g

Plot first 2000 data points of each read from signal file and save at 300dpi pdf:

python SquigglePlot.py -s signals.tsv.gz --plot_colour teal -n 2000 --dpi 300 --no_show o--save test.pdf --save_path ./test/plots/

Segmenter

Identify any segments in folder and visualise each one

Use f to full screen a plot, and ctrl+w to close a plot and move to the next one.

python segmenter.py -p ./test/ -v

Stall identification

python segmenter.py -s signals.tsv.gz -ku -j 100 > signals_stall_segments.tsv

MotifSeq

Find kmer motif:

fasta format for model:

my_kmer.fa

>my_kmer_name
ATCGATCGCTATGCTAGCATTACG

find the best match to that k-mer in the signal:

python MotifSeq.py -s signals.tsv -i my_kmer.fa > signals_kmer.tsv

Limitations

k-mer length should not really be below 12nt, below this things get hairy based on modelling

The p-values and hit probabilities provided are based on loose modelling of negative background scores for a number of k-mers. It is currently only modelled on R9.4 model, not R10 or RNA.

Acknowledgements

I would like to thank the members of my lab, Shaun Carswell, Kirston Barton, Hasindu Gamaarachchi, Kai Martin, Tansel Ersavas, Brent Bevear, Jillian Hammond, and Martin Smith, from the Genomic Technologies team from the Garvan Institute for their feedback on the development of these tools.

License

The MIT License