ngs-71-hmp-partition.ipynb

{
 "metadata": {
  "name": "hmp-partition"
 },
 "nbformat": 3,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "source": [
      "## Partitioning the HMP mock data set",
      "",
      "Partitioning breaks your data set into many smaller chunks so that they can be assembled more easily.  See the k-mer percolation paper",
      "(at http://ivory.idyll.org/blog/dec-11/kmer-percolation-posted.html) for more information.",
      "",
      "Here we are starting with the 'keep.abundfilt' files from the previous tutorial."
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "cd /hmp-mock-tutorial/tutorial-files/"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/hmp-mock-tutorial/tutorial-files",
        ""
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "input": [
      "!ls "
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "mock-pass1.kh  SRR172903.fastq.gz.keep",
        "pass1.report   SRR172903.fastq.gz.keep.below",
        ""
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "source": [
      "The *keep file is the diginormed file.  The *below file is the high abundance filtered file (previously diginormed).  We'll work with the *below file for partitioning."
     ]
    },
    {
     "cell_type": "markdown",
     "source": [
      "### Initial round of partitioning"
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "!/usr/local/src/khmer/scripts/load-graph.py -k 32 -N 4 -x 1e9 /hmp-mock-tutorial/tutorial-files/mock.part1 *.keep.below"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "",
        "PARAMETERS:",
        " - kmer size =    32 \t\t(-k)",
        " - n hashes =     4 \t\t(-N)",
        " - min hashsize = 1e+09 \t(-x)",
        "",
        "Estimated memory usage is 5e+08 bytes (n_hashes x min_hashsize / 8)",
        "--------",
        "Saving hashtable to /hmp-mock-tutorial/tutorial-files/mock.part1",
        "Loading kmers from sequences in ['SRR172903.fastq.gz.keep.below']",
        "We WILL build the tagset (for partitioning/traversal).",
        "making hashtable",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "consuming input SRR172903.fastq.gz.keep.below",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 173329",
        "... consume_fasta_and_tag 100000 3740875",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 331079",
        "... consume_fasta_and_tag 200000 6999381",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 478738",
        "... consume_fasta_and_tag 300000 9934364",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 618144",
        "... consume_fasta_and_tag 400000 12609144",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 750495",
        "... consume_fasta_and_tag 500000 15075380",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 878339",
        "... consume_fasta_and_tag 600000 17403236",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1000123",
        "... consume_fasta_and_tag 700000 19562316",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1118523",
        "... consume_fasta_and_tag 800000 21640685",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1233489",
        "... consume_fasta_and_tag 900000 23642814",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1345752",
        "... consume_fasta_and_tag 1000000 25572425",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1453713",
        "... consume_fasta_and_tag 1100000 27402321",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1555632",
        "... consume_fasta_and_tag 1200000 29083589",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1657176",
        "... consume_fasta_and_tag 1300000 30780858",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1761860",
        "... consume_fasta_and_tag 1400000 32599153",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1856788",
        "... consume_fasta_and_tag 1500000 34133509",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 1949954",
        "... consume_fasta_and_tag 1600000 35645119",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2055000",
        "... consume_fasta_and_tag 1700000 37524852",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2146880",
        "... consume_fasta_and_tag 1800000 39023433",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2246849",
        "... consume_fasta_and_tag 1900000 40794289",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2349896",
        "... consume_fasta_and_tag 2000000 42651231",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2475003",
        "... consume_fasta_and_tag 2100000 45207071",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2573725",
        "... consume_fasta_and_tag 2200000 46950070",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2662797",
        "... consume_fasta_and_tag 2300000 48418556",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2753164",
        "... consume_fasta_and_tag 2400000 49912362",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2862523",
        "... consume_fasta_and_tag 2500000 52019731",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 2958756",
        "... consume_fasta_and_tag 2600000 53709664",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3076077",
        "... consume_fasta_and_tag 2700000 56066157",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3182488",
        "... consume_fasta_and_tag 2800000 58049012",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3278612",
        "... consume_fasta_and_tag 2900000 59730543",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3374433",
        "... consume_fasta_and_tag 3000000 61400950",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3468644",
        "... consume_fasta_and_tag 3100000 63026213",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3561313",
        "... consume_fasta_and_tag 3200000 64618172",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3655723",
        "... consume_fasta_and_tag 3300000 66265133",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3762355",
        "... consume_fasta_and_tag 3400000 68247983",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3872634",
        "... consume_fasta_and_tag 3500000 70337066",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 3998420",
        "... consume_fasta_and_tag 3600000 73025420",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4104080",
        "... consume_fasta_and_tag 3700000 75018608",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4207522",
        "... consume_fasta_and_tag 3800000 76938617",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4310752",
        "... consume_fasta_and_tag 3900000 78847043",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4416038",
        "... consume_fasta_and_tag 4000000 80809411",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4520772",
        "... consume_fasta_and_tag 4100000 82758547",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4626596",
        "... consume_fasta_and_tag 4200000 84738514",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4733898",
        "... consume_fasta_and_tag 4300000 86756232",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4843303",
        "... consume_fasta_and_tag 4400000 88844438",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 4955053",
        "... consume_fasta_and_tag 4500000 91010381",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "n tags: 5074383",
        "... consume_fasta_and_tag 4600000 93393413",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving hashtable in /hmp-mock-tutorial/tutorial-files/mock.part1.ht",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving tagset in /hmp-mock-tutorial/tutorial-files/mock.part1.tagset",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "fp rate estimated to be 0.000",
        ""
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "input": [
      "!/usr/local/src/khmer/scripts/partition-graph.py --threads 8 -s 1e5 /hmp-mock-tutorial/tutorial-files/mock.part1"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "--",
        "SUBSET SIZE 100000.0",
        "N THREADS 8",
        "--",
        "loading ht /hmp-mock-tutorial/tutorial-files/mock.part1.ht",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "** Traverse all the things: stop_big_traversals is false.",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "enqueued 51 subset tasks",
        "starting 8 threads",
        "---",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 0",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 1",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 2",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 3",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 4",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 5",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 6",
        "done starting threads",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 7",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 16777984-94136181661369437: 100000 <- 81157",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 8",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 260208025003308444-409745112244459880: 100000 <- 84023",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 2",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 9",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 94136181661369437-260208025003308444: 100000 <- 86169",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 1",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 1297627398891647638-1531300037474237491: 100000 <- 85730",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 7",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 1110102924880473729-1297627398891647638: 100000 <- 84881",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 10",
        " saving: /hmp-mock-tutorial/tutorial-files/mock.part1 6",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 11",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 409745112244459880-608327443995449189: 100000 <- 85644",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 3",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 12",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 13",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 608327443995449189-875288249355193519: 100000 <- 88133",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 4",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 14",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 875288249355193519-1110102924880473729: 100000 <- 88031",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 5",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 15",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 1531300037474237491-1732095455870257259: 100000 <- 86170",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 8",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 16",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 1732095455870257259-1859652016459854531: 100000 <- 87256",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 9",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 17",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 2306524097106133802-2644798478190932419: 100000 <- 87792",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 12",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 18",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 2644798478190932419-3004065993864211518: 100000 <- 88715",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 13",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 19",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 2049402294476703608-2306524097106133802: 100000 <- 87385",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 11",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 20",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 1859652016459854531-2049402294476703608: 100000 <- 86682",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 10",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 21",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 3261806917173220934-3537254637783596083: 100000 <- 88266",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 15",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 22",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 3004065993864211518-3261806917173220934: 100000 <- 86975",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 14",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 23",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 3537254637783596083-3883423667295379143: 100000 <- 88368",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 16",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 24",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 4684904421329469439-4995705942896480352: 100000 <- 83553",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 20",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 25",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 3883423667295379143-4209207214124708075: 100000 <- 87997",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 17",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 26",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 4995705942896480352-5357141808125845843: 100000 <- 86501",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 21",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 27",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 4462585361236842383-4684904421329469439: 100000 <- 88161",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 19",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 5357141808125845843-5800708136555691008: 100000 <- 88304",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 22",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 5800708136555691008-6107043204167586587: 100000 <- 84683",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 23",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 28",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 29",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 30",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 4209207214124708075-4462585361236842383: 100000 <- 86337",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 18",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 31",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 6107043204167586587-6322961371608382971: 100000 <- 85189",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 24",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 32",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 6322961371608382971-6541038361576452014: 100000 <- 88838",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 25",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 33",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 6541038361576452014-6838615406063056811: 100000 <- 88617",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 26",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 34",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 7183086351678416748-7326493404245058367: 100000 <- 89300",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 28",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 6838615406063056811-7183086351678416748: 100000 <- 88286",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 27",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 35",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 36",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 7326493404245058367-7603665690809285299: 100000 <- 86898",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 29",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 37",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 7603665690809285299-7796014725684922511: 100000 <- 86154",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 30",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 38",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 7796014725684922511-8063127332508974054: 100000 <- 81914",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 31",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 39",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 8063127332508974054-8441389612599857478: 100000 <- 88577",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 32",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 40",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 8441389612599857478-8840307798304368111: 100000 <- 88416",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 33",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 41",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 9137411250090126794-9589039204914260018: 100000 <- 89151",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 35",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 42",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 8840307798304368111-9137411250090126794: 100000 <- 86327",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 34",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 43",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 9589039204914260018-10072697330203029710: 100000 <- 87679",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 36",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 44",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 10579730981856932046-11032294443884874459: 100000 <- 88811",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 38",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 10072697330203029710-10579730981856932046: 100000 <- 88110",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 37",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 45",
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 46",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 11032294443884874459-11054958194172862715: 100000 <- 90162",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 39",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 47",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 11054958194172862715-11433623482846595706: 100000 <- 84114",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 40",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 48",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 11433623482846595706-11928722690215839611: 100000 <- 86212",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 41",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 49",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 11928722690215839611-12298410884273713855: 100000 <- 81395",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 42",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "starting: /hmp-mock-tutorial/tutorial-files/mock.part1 50",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 12298410884273713855-12518855533948649386: 100000 <- 80133",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 43",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 12518855533948649386-12797728323970822374: 100000 <- 81437",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 44",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 12797728323970822374-13323821898357289006: 100000 <- 83259",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 45",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 13323821898357289006-13689482642694583194: 100000 <- 79767",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 46",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 13689482642694583194-14292954440285978686: 100000 <- 86515",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 47",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 14292954440285978686-15473098948992473882: 100000 <- 87307",
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 48",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...subset-part 15473098948992473882-16635994639072819950: 100000 <- 86801",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 49",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving: /hmp-mock-tutorial/tutorial-files/mock.part1 50",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "exiting",
        "---",
        "done making subsets! see /hmp-mock-tutorial/tutorial-files/mock.part1.subset.*.pmap",
        ""
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "input": [
      "ls"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "mock.part1.ht              mock.part1.subset.34.pmap",
        "mock.part1.info            mock.part1.subset.35.pmap",
        "mock.part1.subset.0.pmap   mock.part1.subset.36.pmap",
        "mock.part1.subset.10.pmap  mock.part1.subset.37.pmap",
        "mock.part1.subset.11.pmap  mock.part1.subset.38.pmap",
        "mock.part1.subset.12.pmap  mock.part1.subset.39.pmap",
        "mock.part1.subset.13.pmap  mock.part1.subset.3.pmap",
        "mock.part1.subset.14.pmap  mock.part1.subset.40.pmap",
        "mock.part1.subset.15.pmap  mock.part1.subset.41.pmap",
        "mock.part1.subset.16.pmap  mock.part1.subset.42.pmap",
        "mock.part1.subset.17.pmap  mock.part1.subset.43.pmap",
        "mock.part1.subset.18.pmap  mock.part1.subset.44.pmap",
        "mock.part1.subset.19.pmap  mock.part1.subset.45.pmap",
        "mock.part1.subset.1.pmap   mock.part1.subset.46.pmap",
        "mock.part1.subset.20.pmap  mock.part1.subset.47.pmap",
        "mock.part1.subset.21.pmap  mock.part1.subset.48.pmap",
        "mock.part1.subset.22.pmap  mock.part1.subset.49.pmap",
        "mock.part1.subset.23.pmap  mock.part1.subset.4.pmap",
        "mock.part1.subset.24.pmap  mock.part1.subset.50.pmap",
        "mock.part1.subset.25.pmap  mock.part1.subset.5.pmap",
        "mock.part1.subset.26.pmap  mock.part1.subset.6.pmap",
        "mock.part1.subset.27.pmap  mock.part1.subset.7.pmap",
        "mock.part1.subset.28.pmap  mock.part1.subset.8.pmap",
        "mock.part1.subset.29.pmap  mock.part1.subset.9.pmap",
        "mock.part1.subset.2.pmap   mock.part1.tagset",
        "mock.part1.subset.30.pmap  mock-pass1.kh",
        "mock.part1.subset.31.pmap  pass1.report",
        "mock.part1.subset.32.pmap  SRR172903.fastq.gz.keep",
        "mock.part1.subset.33.pmap  SRR172903.fastq.gz.keep.below",
        ""
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "input": [
      "!/usr/local/src/khmer/scripts/merge-partitions.py mock.part1"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "loading 51 pmap files (first one: mock.part1.subset.0.pmap)",
        "merging mock.part1.subset.0.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.2.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.1.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.7.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.6.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.3.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.4.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.5.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.8.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.9.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.12.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.13.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.11.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.10.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.15.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.14.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.16.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.20.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.17.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.21.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.19.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.22.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.23.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.18.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.24.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.25.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.26.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.28.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.27.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.29.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.30.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.31.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.32.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.33.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.35.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.34.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.36.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.38.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.37.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.39.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.40.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.41.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.42.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.43.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.44.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.45.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.46.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.47.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.48.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.49.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "merging mock.part1.subset.50.pmap",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "saving merged to mock.part1.pmap.merged",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "removing pmap files",
        ""
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "input": [
      "!/usr/local/src/khmer/scripts/annotate-partitions.py mock.part1 *.keep.below"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "loading partition map from: mock.part1.pmap.merged",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "outputting partitions for SRR172903.fastq.gz.keep.below",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 100000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 200000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 300000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 400000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 500000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 600000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 700000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 800000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 900000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1000000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1100000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1200000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1300000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1400000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1500000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1600000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1700000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1800000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 1900000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2000000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2100000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2200000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2300000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2400000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2500000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2600000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2700000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2800000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 2900000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3000000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3100000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3200000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3300000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3400000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3500000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3600000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3700000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3800000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 3900000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4000000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4100000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4200000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4300000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4400000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4500000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... output_partitions 4600000 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "output 477730 partitions for SRR172903.fastq.gz.keep.below",
        "partitions are in SRR172903.fastq.gz.keep.below.part",
        ""
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "input": [
      "!/usr/local/src/khmer/scripts/extract-partitions.py mock.part1 *keep.below.part"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "---",
        "reading partitioned files: ['SRR172903.fastq.gz.keep.below.part']",
        "outputting to files named \"mock.part1.groupN.fa\"",
        "min reads to keep a partition: 5",
        "max size of a group file: 1000000",
        "partition size distribution will go to mock.part1.dist",
        "---",
        "... 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 1900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 2900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 3900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... 4500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2 groups",
        "...x2 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 1900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 2900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 3900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "...x2 4500000",
        ""
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "source": [
      "### Done!",
      "",
      "You can see now that you have some grouped files which you can assemble in parallel.  This was actually a very small dataset, so you have only 2 of these files.  However, for larger metagenomes, you could have millions of partitions and thus want to create hundreds of these grouped files.  ",
      "",
      "Now take the '\\*.group\\*.fa' files and proceed to assembly.  If you're looking for more resources on how we do this, check out these tutorials.  ",
      "",
      "http://ged.msu.edu/angus/metag-assembly-2011/index.html",
      "http://ged.msu.edu/angus/nih-hmp-2012/index.html",
      "",
      ""
     ]
    }
   ]
  }
 ]
}