This repository contains the notebooks used to create the analyses in Vazquez-Baeza et al. 2016. Each notebook tries to summarize a particular step and be as standalone as possible. In this document we briefly describe each of the analyses needed for the paper and when possible we group them together, so as to provide more cohesion.
-
This repository does not provide all the data files used, as the size would exceed the limits allowed in GitHub. However, we provide the main data files (metadata and OTU tables), from which the rest of the data can be generated using these commands. The rest of the sequence data can be freely accessed through Qiita (remember to log in) in study 833.
-
In several locations in the notebooks, there are cells that reference a remote address (i.e. in
sshorscpcommands), you don't need to execute these commands, the files being fetched should already be provided in this repository.
The metadata processing required two steps, one for cleanup of the data and another one to prepare the data for Qiita.
1-metadata: the metadata is cleaned up and filtered to remove samples that we didn't use in the rest of the analyses. We also calculate the dysbiosis index as defined in Gevers et. al. 2014.
1.1-metadata-for-qiita: adds the needed fields and columns to the mapping file, and creates a sample and prep template that were used to upload the data into the Qiita study.
2-alpha-diversity: this notebook includes the following alpha diversity comparisons: fat, protein, age, weight and disease state. As well as a comparison of the human-trained dysbiosis index and alpha diversity. Of note, we did this comparisons for several metrics, but only used Faith's phylogenetic diversity in the manuscript.
3-beta-diversity: this notebook includes the creation of the beta diversity plots for the dog dataset only. Biplots and statistics to assess clustering significance are also performed as part of this notebook. 3.1-beta-diversity-antibiotics.ipynb: compares the differences between samples according to their history fo antibiotic usage.
4-group-significance: this notebook tests statistical significance between the disease affected and unaffected dogs, and plots their relative abundance as a heatmap. While none of the plots shown in this notebook were used in this paper, it helped guide our analysis for the next few notebooks.
5-feature-exploration: this notebook looks at a few different ways to filter out the data so as to avoid OTUs that are not well represented througout the samples.
After realizing that the human-trained dysbiosis index didn't perform as well in dogs, we decided to use CCREPE to train a new dysbiosis index using the dog data alone. In 6-md-index-ccrepe we calculate the checkerboard scores and asscociated significance tables. These results are used in 6.1-md-index-ccrepe-visualizations, where we visualize them in a variety of ways, ultimately resolving that we should use Cytoscape to do that. The final section of this notebook shows the plots relating alpha diversity and the index.
The ROC curves and feature importance scores are created in 7-classifier and 7.1-classifier-feature-importance (respectively). Here we use R and hack_ml to create the plots and tables.
In 8-comparison we explore the combined data and perform a few comparisons that were ultimately not used in the paper. 8.1-comparison is concerned with making the data between humans and dogs as comparable as possible.
PICRUSt predictions were generated at the galaxy server. In 9-picrust we compare the combined human and dog samples, and in 9.1-picrust-nsti we use the NSTI (nearest sequenced taxon index) to assess the quality of our predictions.
In 10-minamoto-md-index notebook we use the dog-trained dysbiosis index in a different dataset, that was processed mainly in a separate supercomputer.
In 11-sequence-counts, we explore the number of sequences that were assigned to an OTU per sample. Specifically we compare the differences betwee closed and open reference protocols.
This Dockerfile should install any additional dependencies for these notebooks to work.