wiki:BigCompute

Version 66 (modified by Barbera van Schaik, 13 years ago) (diff)

--

Port applications to Dutch Life Science Grid

People

  • AMC: Antoine van Kampen, Barbera van Schaik, Silvia D Olabarriaga, Mark Santcroos
  • Sara/BiGGrid: Tom Visser
  • UMCG: Morris Swertz, Freerk van Dijk

Description Software is going to be implemented as workflow components. The workflows will run on the Dutch life science grid.

Implemented workflow components at AMC

This list of workflow components are already available. We can expand it with Pindel and (parts of) the GATK pipeline.

  • Splitting of fastq files
  • Building a BWA index on the genome sequence (base space and color space)
  • BWA for shotgun reads (base space and color space) It is possible to do parameter sweeps. Output is in bam format
  • Merge bam results
  • Samtools pileup
  • Varscan (pileup to snp, indel and cns)
  • Bam2coverage creates a UCSC wiggle file to display the genome coverage (per 50kbp)
  • Coverage-per-base determines the coverage for every base in the genome and it summarizes the results (coverage versus frequency)
  • Annovar (works for hg18, working on other assemblies). This is a pipeline to annotate variants (gene, dbsnp, hapmap, 1000g, conservation, etc)
  • FastqC

Implemented components of the Groningen pipeline

Template (grid component)

Alignment, realignment, recalibration, stats

  • pe0--fastqc.ftl (FastqToFastQC, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/quality/Workflow/FastqToFastQC.gwendia)
  • pe00-bwa-align-pair1.ftl (BwaIllumina, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/BwaIllumina.gwendia)
  • pe01-bwa-align-pair2.ftl (BwaIllumina, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/BwaIllumina.gwendia)
  • pe02-bwa-sampe.ftl (BwaIllumina, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/BwaIllumina.gwendia)
  • pe03-sam-to-bam.ftl (BwaIllumina, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/BwaIllumina.gwendia)
  • pe04a-HsMetrics.ftl (CalculateHsMetrics, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/CalculateHsMetrics.gwendia)
  • pe04b-picardQC.ftl (PicardQC, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/PicardQC.gwendia)
  • pe04-sam-sort.ftl (SamSort, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/SamSort.gwendia)
  • pe05-mark-duplicates.ftl (MarkDuplicates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/MarkDuplicates.gwendia)
  • pe06-realign.ftl (ReAlign, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/ReAlign.gwendia)
  • pe07-fixmates.ftl (FixMates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/FixMates.gwendia)
  • pe08-covariates-before.ftl (GatkCovariates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/GatkCovariates.gwendia)
  • pe09-recalibrate.ftl (GatkRecalibrate, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/GatkRecalibrate.gwendia)
  • pe10-sam-sort.ftl (SamSort, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/SamSort.gwendia)
  • pe11-covariates-after.ftl (GatkCovariates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/GatkCovariates.gwendia)
  • pe12-analyze-covariates.ftl (GatkAnalyzeCovariates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/GatkAnalyzeCovariates.gwendia)

Merge bam per sample and perform SNP and indel calling

  • vc00a-unified-genotyper.ftl to do
  • vc00b-variant-filtration.ftl to do
  • vc00c-variant-eval.ftl to do
  • vc00d-picardMetrics.ftl (PicardQC, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/PicardQC.gwendia)
  • vc00-merge.ftl to do
  • vc00.merge.ftl to do
  • vc01-coverage.ftl to do
  • vc01.unified_genotyper.ftl to do
  • vc02.picardQC.ftl (PicardQC, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/PicardQC.gwendia)
  • vc02-realigner-target-creator.ftl to do
  • vc03.coverage.ftl to do
  • vc03-realign.ftl (ReAlign, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/ReAlign.gwendia)
  • vc04-fixmates.ftl (FixMates, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/FixMates.gwendia)
  • vc05-indel-genotyper-v2.ftl to do
  • vc06-filter-indels.ftl to do
  • vc07-unified-genotyper.ftl to do
  • vc08-make-indel-mask.ftl to do
  • vc09-variant-filtration.ftl to do
  • vc10-variant-eval.ftl to do
  • vc11-name-sort-bam.ftl (SamSort, lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF/gvnl/Workflows/SamSort.gwendia)
  • vc12-pindel.ftl to do

Data access rights

To ensure that the most limited group of people has access to the data we have created a subgroup "gvnl" within the "vlemed" Virtual Organisation (VO). For people to become part of this group, it is required that they have a Grid certificate and that they are part of the "vlemed" VO. On the following page there is information on how to get a certificate, how to get into the "vlemed" VO: http://www.bioinformaticslaboratory.nl/twiki/bin/view/EBioScience/EBioInfra#Access

For more information about data access see http://www.bioinformaticslaboratory.nl/twiki/bin/view/EBioScience/DataManagement

Things to address

  • Available disk space on the grid storage elements / worker nodes

Data location on grid

Data

The data is located on the storage element at Sara and only readable and writable for the vlemed/gvnl group. This screencast demonstrates how to access the data from the Vbrowser: http://www.youtube.com/watch?v=FicwWGAbubQ

Storage location (resource): srm.grid.sara.nl

Path: /pnfs/grid.sara.nl/data/vlemed/gvnl

Workflows and databases

These directories are open to all members of the vlemed VO

  • Workflows: lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF
  • Databases: lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_DB

The directories that contain the workflows have the following structure:

Directory Description
bin dependent binaries like bwa, samtools
GasW component description, describes which executable has to run on grid and it specifies the in and output files/parameters
Java dependent jar files, e.g. the GATK jar
parameterFiles text files that contain exactly one line with the parameters that you would like to provide to bwa or another component
Scufl old workflow description files, can be ignored
shFiles these files are executed on the grid and described by the GASW descriptor, the gvnl shFiles are based on the Groningen templates
Workflows Workflow descriptions, clicking on them will start the Moteur plugin. Input files/parameters can be specified in the fields. If you click on "Run" the jobs are submitted to the grid

The shell files can (in most cases) run on any linux cluster. In that case you need to place the shell file and the dependent executable(s) in one directory. At the start of each shell file is an example on how to run them.

Workflow execution

On mini pilot. Split lines-per-file: 8,000,000. Start: 18-12-2010 16:40

Lane split BWA merge Comments Elapsed time (s)
A4a_L4_HUModqRBUDIBAPE done 22740
A4a_L6_HUModqRADDIAAPE done 37750
A4a_L7_HUModqRBVDIBAPE failed
A4b_L3_HUModqRAFDIBAPE done 32256
A4b_L6_HUModqRBTDIBAPE failed
R2A _L1_HUModqRADDIBAPE failed
R2A _L1_HUModqRAFDIBAPE done 35819
R2A _L5_HUModqRAEDIAAPE failed
R2B _L3_HUModqRBTDIAAPE done 23754
R2B _L4_HUModqRBUDIAAPE failed pair 1 not in correct gzip format?
R2B _L6_HUModqRBTDIBAPE failed
R2C _L2_HUModqRBUDIBAPE done 40763
R2C _L2_HUModqRBVDIBAPE failed
R2C _L7_HUModqRBVDIAAPE done 39374
Unknown_L6_HUModqRBUDIAAPE done 30307

Note: The Merge component is red (marked as failed) in the finished workflows. The bam files where produced successfully.

BWA alignment on mini pilot without splitting the data

  • 19-12-2010 14:00 done - elapsed time 325886 s

PicardQC on finished bam files

Coverage-per-base on finished bam files

  • 20-12-2010 12:40 done - elapsed time 89634 s

Mark-duplicates on finished bam files

  • 20-12-2010 12:47 done - elapsed time 10614 s

Re-align on finished bam files

  • 20-12-2010 17:48 done - elapsed time 9165 s

Gvnl pipeline on remaining samples (split, bwa, merge, coverage-per-base, picard-qc, mark-duplicates, re-align) Start: 20-12-2010 18:50

Lane Progress Elapsed time
A4a_L7_HUModqRBVDIBAPE done 208542
A4b_L6_HUModqRBTDIBAPE done 74476
R2A _L1_HUModqRADDIBAPE done 72636
R2A _L5_HUModqRAEDIAAPE done 55242
R2B _L4_HUModqRBUDIAAPE failed
R2B _L6_HUModqRBTDIBAPE done 227859
R2C _L2_HUModqRBVDIBAPE done 44730
  • Note1: Workflow stops after merge-bam. Working on it.
  • Note2: The Dutch grid seems very busy at the moment. That could explain the longer execution times.

Gvnl pipeline on theoretical reads

  • 22-12-2010 19:14 done - elapsed time 38587 s
  • Workflow does continue after the merge-bam now
  • To do: implement recalibration step, and a check on all bam files of the split input before the merge step

Alternatives

Clusters

  • Groningen
    • Description here about code template and automatic PBS script generation. Job submission/monitoring
  • Leiden
  • Huygens
  • Lisa
  • Philips
  • DAS

Grid

Attachments (3)

Download all attachments as: .zip