How to create APT tar balls for the GMS from inside The Genome Institute

This page documents the creation of tarballs which are used during the sGMS install process. This will be useful whenever a debian package dependency needs to be added/removed/modified for the sGMS.

The apt-repos for TGI are on http://repo.gsc.wustl.edu, to make changes to the packages that are stored here, TGI uses a git repo called genome-snapshot-deps.

• To get a version of genome-snapshot-deps,

 git clone https://github.com/genome/genome-snapshot-deps

• Once you have cloned this repo, to change the packages dependencies for precise go into the 'precise/' folder. This has a bunch of files which indicate the dependencies for installing the sGMS on precise. These are of the form genome-snapshot-deps-*.depends. So, for example, to add a perl package, modify the genome-snapshot-deps-perl.depends file. To add a python package, modify the genome-snapshot-deps-python.depends file and so on.

• After modifying the .depends file, commit the changes using git commit

• Next, the change log has to be manually updated for precise (this has to be done manually for changes to the precise packages, I think for lucid this is done automatically). The change-log is of this format,

  genome-snapshot-deps (2014.03.10-1~Ubuntu~precise) precise; urgency=low
 
       * [2e1d398] add circos
  
  -- Avinash Ramu <aramu@genome.wustl.edu>  Thu, 10 Mar 2014 12:36:30 -0500
   

The first and last lines need to have the date the change is made in them. If there is more than one change on a given day use something like 2014.03.10-1, 2014.03.10-2, 2014.03.10-3 to indicate them in the first line, this has to be a unique entry for every change made otherwise there is no way to distinguish this from another entry with the same entry. '[2e1d398]' - this is the commit ID of the commit where circos was added to the '.depends' file.

• After updating the changelog, commit the changes using git commit

• The next step is to upload these changes to the repo.gsc.wustl.edu site. This is mostly automated and is done by running make debian DISTRO=precise this creates a bunch of files that for example look like

 genome-snapshot-deps-apps-external_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-apps-tgi_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-perl_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-python_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-refcov_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-trans-abyss_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-ur_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps-workflow_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps_2014.03.10-1~Ubuntu~precise.dsc
 genome-snapshot-deps_2014.03.10-1~Ubuntu~precise.tar.gz
 genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_all.deb
 genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes
 genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.precise-genome-development.upload

The genome-snapshot-deps_2014.03.10-1Ubuntuprecise_amd64.changes (replace with your version here) file should be signed automatically by the last step in the make debian DISTRO=precise process with your GPG key unless there is some problem. Note: if you do a make debian DISTRO=precise the GPG key of the person who made the last entry in the precise/changelog file is required to sign the 'changes' file. So you can only sign the changes file if you made the last entry in the changelog(I need to double check this with someone in APIPE, but it sounds logical.) So make sure your GPG keys are setup correctly before attempting this process, there are a bunch of tutorials within the TGI and on the inter webs which tell you how to accomplish this.

• After the make step, try

 debsign ../genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes

(replace with your changes file), if things went well to this point, this file should already be signed with your ID and this step is not necessary.

• The next step is to upload these changes to repo.gsc.wustl.edu which is done by

 dput precise-genome-development genome-snapshot-deps_2014.03.10-1~Ubuntu~precise_amd64.changes

Note : replace "genome-snapshot-deps_2014.03.10-1Ubuntuprecise_amd64.changes" with your changes file.

• Now that we have uploaded these changes to repo.gsc.wustl.edu, we need to make a tarball with the updated packages. Note: the sGMS installs the debian packages from the tarball(which it downloads from an FTP site https://xfer.genome.wustl.edu/gxfer1/project/gms/) instead of downloading the packages directly from 'repo.gsc.wustl.edu'. The reasons for this are unknown but might be to mitigate load on that server.

This process is accomplished with the help of another Makefile inside the 'gms' repo. To get this repo do,

 git clone https://github.com/genome/gms.git 
 cd gms/setup/stage
 make

This will create a mirror of the repo.gsc.wustl.edu and make a tar ball with all the packages in it. The tar ball has a timestamp on it which is the current date unless specified manually. This tar ball is stored in the gms/setup/archive files. The make step also automatically copies the tar ball over to the FTP staging site which is, /gscmnt/sata102/info/ftp-staging/pub/software/gms/testdata/GMS1/setup/archive-files, currently the end of the Make process complains about setting timestamps on the files in the staging directory, this might be a permissions related issue.

• After the make step, double check to make sure that the updated tar ball and a 'mirror*.config' file have been copied over to the staging directory.

• Now this tar-ball can be used in the sGMS install process by updating the 'APT_DUMP_VERSION=', line in the gms Makefile to the latest version of the APT DUMP. So for example APT_DUMP_VERSION=2013.03.10 will attempt to install using apt-mirror-min-ubuntu-12.04-2014.03.10.tgz.