Initial port of ref-isoforms to Nextflow

.gitignore

@@ -3,3 +3,7 @@ nextflow
 template-workflow
 .*.swp
 .*.swo
+.DS_STORE
+output/**
+.idea/**
+**/__pycache__

.gitlab-ci.yml

@@ -8,4 +8,5 @@ variables:
     # The workflow should define `--out_dir`, the CI template sets this.
     # Only common file inputs and option values need to be given here
     # (not things such as -profile)
-    NF_WORKFLOW_OPTS: "--fastq test_data"
+    NF_WORKFLOW_OPTS: "--fastq test_data/fastq \
+      --ref_genome test_data/SIRV_150601a.fasta --ref_annotation test_data/SIRV_isofroms.gtf"

README.md

@@ -1,22 +1,26 @@
-# Workflow template
+# Pipeline for annotating genomes using long read transcriptomics
 
 This repository contains a [nextflow](https://www.nextflow.io/) workflow
-template that can be used as the basis for creating new workflows.
-
-> This workflow is not intended to be used by end users.
+for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads.
 
+## Overview
+* cDNA or direct RNA reads are optionally preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) for trimming and orientation. <br>
+* Reads are then mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2) <br>
+* Transcripts are assembled by[stringtie](http://ccb.jhu.edu/software/stringtie) in long read mode (with or without a guide reference annotation) to generate the GFF annotation.
+* The annotation generated by the pipeline is compared to the reference annotation (if supplied) using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml) 
+* An html report is generated, which contains various summary statistics and plots of the data.
 
 ## Quickstart
-
 The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and 
 software resources, as such nextflow will need to be installed before attempting
 to run the workflow.
 
 The workflow can currently be run using either
-[Docker](https://www.docker.com/products/docker-desktop) or
+[Docker](https://www.docker.com/products/docker-desktop),
+[Singularity](https://sylabs.io/singularity/) or
 [conda](https://docs.conda.io/en/latest/miniconda.html) to provide isolation of
-the required software. Both methods are automated out-of-the-box provided
-either docker of conda is installed.
+the required software. Each method is automated out-of-the-box provided
+either docker, singularity or conda is installed.
 
 It is not required to clone or download the git repository in order to run the workflow.
 For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex).
@@ -26,21 +30,47 @@ For more information on running EPI2ME Labs workflows [visit out website](https:
 To obtain the workflow, having installed `nextflow`, users can run:
 
 ```
-nextflow run epi2me-labs/wf-template --help
+nextflow run epi2me-labs/wf-isoforms --help
 ```
 
 to see the options for the workflow.
 
+**Workflow inputs**
+- Directory containing cDNA/direct RNA reads (or path to single file) in fastq/fastq.gz format
+- Reference genome in fast format
+- Optional reference annotation in GFF2/3 format 
+
+**Example workflow run**
+```
+# To run a small and quick example using synthetic data
+nextflow run wf-isoforms/ --fastq test_data/fastq  --ref_genome genome.fasta --ref_annotation reference.gff
+--out_dir outdir/ -profile conda
+```
+
+```
+# To evaluate the workflow on a larger Drosophila dataset
+chmod u+x ./run_evaluation_dmel.sh outdir
+```
+
 **Workflow outputs**
 
-The primary outputs of the workflow include:
-
-* a simple text file providing a summary of sequencing reads,
-* an HTML report document detailing the primary findings of the workflow.
-
+* wf-isoforms-report.html
+  - Summary and plots of reads, alignments and the transcript assembly and annotation
+* str_merged.gff
+  - The stringtie-generated transcript annotations
+* str_merged.stats
+  - gffcomare file with statistics regarding the accuracy of the assembled query transcripts in relation to the reference annotation 
+* str_merged.annotated.gtf
+  - A gffcomapre output file with extra columns relating to comparison with the reference annotation
+* str_transcriptome.fas
+  - A transcriptome made from the query reads 
+* merged_transcriptome.fas
+  -A transcriptome made from the combined query reads and reference annotation
 
+  
 ## Useful links
 
 * [nextflow](https://www.nextflow.io/)
 * [docker](https://www.docker.com/products/docker-desktop)
+* [Singularity](https://sylabs.io/singularity/)
 * [conda](https://docs.conda.io/en/latest/miniconda.html)

bin/check_sample_sheet.py

@@ -16,21 +16,21 @@ def main():
     try:
         samples = pd.read_csv(args.sample_sheet, sep=None)
         if 'alias' in samples.columns:
-            if 'sample_name' in samples.columns:
+            if 'sample_id' in samples.columns:
                 sys.stderr.write(
                     "Warning: sample sheet contains both 'alias' and "
-                    'sample_name, using the former.')
-            samples['sample_name'] = samples['alias']
+                    'sample_id, using the former.')
+            samples['sample_id'] = samples['alias']
         if 'barcode' not in samples.columns \
-                or 'sample_name' not in samples.columns:
+                or 'sample_id' not in samples.columns:
             raise IOError()
     except Exception:
         raise IOError(
             "Could not parse sample sheet, it must contain two columns "
-            "named 'barcode' and 'sample_name' or 'alias'.")
+            "named 'barcode' and 'sample_id' or 'alias'.")
     # check duplicates
     dup_bc = samples['barcode'].duplicated()
-    dup_sample = samples['sample_name'].duplicated()
+    dup_sample = samples['sample_id'].duplicated()
     if any(dup_bc) or any(dup_sample):
         raise IOError(
             "Sample sheet contains duplicate values.")

bin/generate_pychopper_stats.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+"""Generate CSV of pychopper stats."""
+
+# -*- coding: utf-8 -*-
+
+import argparse
+import os
+import sys
+
+import pandas as pd
+
+
+def parse_args(argv=sys.argv[1:]):
+    """Parse arguments."""
+    description = """Script to run the isoform workflow """
+    parser = argparse.ArgumentParser(description=description)
+    parser.add_argument("--data", required=True, help="")
+    parser.add_argument("--output_dir", required=True, help="")
+    return parser.parse_args(argv)
+
+
+def generate_pychopper_stats(tsv, output):
+    """Make CSV of pychopper stats."""
+    classified_path = os.path.join(output, "pychopper_stats.csv")
+    df = pd.read_csv(tsv, sep="\t", index_col="Name")
+    classified = df.loc[df["Category"] == "Classification"]\
+        .copy().reset_index().rename(columns={'Name': 'Classification'})
+    classified["Percentage"] = \
+        100 * classified["Value"] / classified["Value"].sum()
+    tuning = df.loc[df["Category"] == "AutotuneSample"]\
+        .copy().reset_index().rename(columns={'Name': 'Filter'})
+    tuning.to_csv(classified_path)
+
+
+def main(args):
+    """Run entry point."""
+    assert os.path.isfile(args.data)
+    assert os.path.isdir(args.output_dir)
+    generate_pychopper_stats(tsv=args.data, output=args.output_dir)
+
+
+if __name__ == '__main__':
+    main(args=parse_args())

bin/generate_tracking_summary.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+"""Generate per-transcript class sumarrarry files from gffcompare."""
+
+import argparse
+import os
+import sys
+
+import pandas as pd
+
+
+def parse_args(argv=sys.argv[1:]):
+    """Parse arguments."""
+    description = """Script to run the isoform workflow """
+    parser = argparse.ArgumentParser(description=description)
+    parser.add_argument("--tracking", required=True, help="")
+    parser.add_argument("--output_dir", required=True, help="")
+    parser.add_argument("--annotation", required=False, default=None, help="")
+    return parser.parse_args(argv)
+
+
+def generate_tracking_summary(tracking_file, output_dir, annotations=None):
+    """Write per transcript class gffcompare tracking files."""
+    # results/gffcompare/gff_comparestringtie.tracking
+    write_empty_tsvs = False
+    tracking_headings = [
+        "query_transfrag_id", "query_locus_id", "ref_gene_id",
+        "class", "details"]
+    nice_names = {
+        '=': 'complete', 'c': 'contained', 'k': 'containment',
+        'm': 'retained', 'n': 'retained (partial)', 'j': 'multi',
+        'e': 'single', 'o': 'overlap', 's': 'opposite',
+        'x': 'exonic', 'i': 'intron', 'y': 'contains', 'p': 'runon',
+        'r': 'repeat', 'u': 'unknown'}
+
+    if os.path.exists(annotations):
+        tracking = pd.read_csv(
+            tracking_file, sep="\t", names=tracking_headings[1:],
+            index_col=0)
+
+        d = pd.DataFrame(tracking['class'].value_counts()) \
+            .reset_index().rename(columns={'index': 'class', 'class': 'count'})
+        d['description'] = [nice_names[x] for x in d['class']]
+
+        # write a separate table for each class
+        for class_code, table in tracking.groupby('class'):
+            if not write_empty_tsvs and table.empty:
+                print("Skipping: No transcripts found for: {}".format(
+                    class_code))
+                continue
+
+            path = tracking_file + ".{}.tsv".format(class_code)
+            table.to_csv(path)
+    else:
+        print("Skipping classification summary as no annotation provided.")
+
+
+def main(args):
+    """Run entry point."""
+    assert os.path.isfile(args.tracking)
+    assert os.path.isdir(args.output_dir)
+    if args.annotation:
+        os.path.isfile(args.annotation)
+    generate_tracking_summary(args.tracking, output_dir=args.output_dir,
+                              annotations=args.annotation)
+
+
+if __name__ == '__main__':
+    main(parse_args(sys.argv[1:]))

bin/ping.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+"""Send workflow ping."""
+
+import argparse
+import json
+import uuid
+
+from epi2melabs import ping
+
+
+def get_uuid(val):
+    """Construct UUID from string."""
+    return uuid.UUID(str(val))
+
+
+def main():
+    """Run the entry point."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--hostname", required=True, default=None,
+        help="ping some meta")
+    parser.add_argument(
+        "--opsys", required=True, default=None,
+        help="ping some meta")
+    parser.add_argument(
+        "--session", default=None,
+        help="ping some meta")
+    parser.add_argument(
+        "--message", required=True, default=None,
+        help="message to include in the ping")
+    parser.add_argument(
+        "--meta", default=None,
+        help="JSON file of metadata to be included in the ping")
+    parser.add_argument(
+        "--revision", default='unknown',
+        help="git branch/tag of the executed workflow")
+    parser.add_argument(
+        "--commit", default='unknown',
+        help="git commit of the executed workflow")
+    parser.add_argument(
+        "--disable", action='store_true',
+        help="Run the script but don't send the ping")
+    args = parser.parse_args()
+
+    meta = None
+    if args.meta:
+        with open(args.meta, "r") as json_file:
+            meta = json.load(json_file)
+
+    if not args.disable:
+        ping.Pingu(
+            get_uuid(args.session),
+            hostname=args.hostname,
+            opsys=args.opsys
+        ).send_workflow_ping(
+            workflow='wf-isoforms',
+            message=args.message,
+            revision=args.revision,
+            commit=args.commit,
+            meta=meta
+        )
+
+
+if __name__ == "__main__":
+    main()

bin/plot_aln_stats.py

@@ -0,0 +1,186 @@
+#!/usr/bin/env python
+"""Plot a seqkit alignment stats file."""
+
+import argparse
+import warnings
+
+import matplotlib
+from matplotlib.backends.backend_pdf import PdfPages
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import six
+
+matplotlib.use('Agg')
+
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore")
+    import seaborn as sns
+
+warnings.resetwarnings()
+_ = sns
+
+# Parse command line arguments:
+parser = argparse.ArgumentParser(
+    description="""Plot a seqkit alignment stats file.""")
+parser.add_argument(
+    '-r', metavar='report_pdf', type=str, help="Report PDF (stats.pdf).",
+    default="stats.pdf")
+parser.add_argument(
+    'input', metavar='input_tsv', type=str, help="Input TSV.")
+
+
+class Report:
+    """Class for plotting utilities on the top of matplotlib.
+
+    Plots are saved in the specified file through the PDF backend.
+    """
+
+    def __init__(self, pdf):
+        """
+        Init Report with a matplotlib PdfPages instance.
+
+        :param self: object.
+        :param pdf: Output pdf.
+        :returns: The report object.
+        :rtype: Report
+        """
+        self.pdf = pdf
+        self.plt = plt
+        self.pages = PdfPages(pdf)
+
+    def _set_properties_and_close(self, fig, title, xlab, ylab):
+        """Set title, axis labels and close the figure.
+
+        :param self: object.
+        :param fig: The current figure.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :returns: None
+        :rtype: object
+        """
+        plt.xlabel(xlab)
+        plt.ylabel(ylab)
+        plt.title(title)
+        self.pages.savefig(fig)
+        plt.close(fig)
+
+    def plot_boxplots(self, data_map, title="", xlab="", ylab="",
+                      xticks_rotation=0, xticks_fontsize=5):
+        """Plot multiple pairs of data arrays.
+
+        :param self: object.
+        :param data_map:
+            A dictionary with labels as keys and lists as data values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param xticks_rotation: Rotation value for x tick labels.
+        :param xticks_fontsize: Fontsize for x tick labels.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+        plt.boxplot(list(data_map.values()))
+        plt.xticks(np.arange(len(data_map)) + 1, data_map.keys(),
+                   rotation=xticks_rotation, fontsize=xticks_fontsize)
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def plot_bars_simple(self, data_map, title="", xlab="", ylab="", alpha=0.6,
+                         xticks_rotation=0, auto_limit=False):
+        """Plot simple bar chart from input dictionary.
+
+        :param self: object.
+        :param data_map: A dictionary with labels as keys and data as values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param alpha: Alpha value.
+        :param xticks_rotation: Rotation value for x tick labels.
+        :param auto_limit: Set y axis limits automatically.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+
+        labels = list(data_map.keys())
+        data = list(data_map.values())
+        positions = np.arange(len(labels))
+        plt.bar(positions, data, align='center', alpha=alpha)
+        plt.xticks(positions, labels, rotation=xticks_rotation)
+
+        if auto_limit:
+            low, high = min(data), max(data)
+            plt.ylim([(low - 0.5 * (high - low)), (high + 0.5 * (high - low))])
+
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def plot_histograms(self, data_map, title="", xlab="", ylab="", bins=50,
+                        alpha=0.7, legend_loc='best', legend=True,
+                        vlines=None):
+        """Plot histograms of multiple data arrays.
+
+        :param self: object.
+        :param data_map:
+            A dictionary with labels as keys and data arrays as values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param bins: Number of bins.
+        :param alpha: Transparency value for histograms.
+        :param legend_loc: Location of legend.
+        :param legend: Plot legend if True.
+        :param vlines:
+            Dictionary with labels and positions of vertical lines to draw.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+
+        for label, data in six.iteritems(data_map):
+            if len(data) > 0:
+                plt.hist(data, bins=bins, label=label, alpha=alpha)
+        if vlines is not None:
+            for label, pos in six.iteritems(vlines):
+                plt.axvline(x=pos, label=label)
+        if legend:
+            plt.legend(loc=legend_loc)
+
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def close(self):
+        """Close PDF backend.
+
+        Do not forget to call this at the end of your
+        script or your output will be damaged!
+
+        :param self: object
+        :returns: None
+        :rtype: object
+        """
+        self.pages.close()
+
+
+if __name__ == '__main__':
+    args = parser.parse_args()
+
+    plotter = Report(args.r)
+
+    # Plot overview panel:
+    stats = pd.read_csv(args.input, sep="\t")
+    perc = stats[["PrimAlnPerc", "MultimapPerc"]].copy()
+    num = stats[["PrimAln", "SecAln", "SupAln", "Unmapped", "TotalReads",
+                "TotalRecords"]].copy()
+
+    perc.plot(kind='bar')
+    plt.title("Percent primary and multimapping reads")
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    num.plot(kind='bar')
+    plt.title("Number of alignment records")
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    plotter.close()

bin/plot_gffcmp_stats.py

@@ -0,0 +1,471 @@
+#!/usr/bin/env python
+
+"""Plot a gffcompare stats file."""
+
+import argparse
+from collections import OrderedDict
+import warnings
+
+import matplotlib
+from matplotlib.backends.backend_pdf import PdfPages
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import six
+
+
+matplotlib.use('Agg')
+
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore")
+    import seaborn as sns
+
+warnings.resetwarnings()
+_ = sns
+
+# Parse command line arguments:
+parser = argparse.ArgumentParser(
+    description="""Plot a gffcompare stats file.""")
+parser.add_argument(
+    '-r', metavar='report_pdf', type=str,
+    help="Report PDF (plot_gffcmp_stats.pdf).",
+    default="plot_gffcmp_stats.pdf")
+parser.add_argument(
+    '-t', metavar='tracking_tsv', type=str,
+    help="Tracking file produced by gffcompare.", default=None)
+parser.add_argument(
+    'input', metavar='input_txt', type=str,
+    help="Input gffcompare stats file.")
+
+
+class Report:
+    """Matplotlib plotting utilities."""
+
+    def __init__(self, pdf):
+        """Init class with PdfPahges instance.
+
+        Plots are saved in the specified file through the PDF backend.
+
+        :param self: object.
+        :param pdf: Output pdf.
+        :returns: The report object.
+        :rtype: Report
+
+        """
+        self.pdf = pdf
+        self.plt = plt
+        self.pages = PdfPages(pdf)
+
+    def _set_properties_and_close(self, fig, title, xlab, ylab):
+        """Set title, axis labels and close the figure.
+
+        :param self: object.
+        :param fig: The current figure.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :returns: None
+        :rtype: object
+        """
+        plt.xlabel(xlab)
+        plt.ylabel(ylab)
+        plt.title(title)
+        self.pages.savefig(fig)
+        plt.close(fig)
+
+    def plot_boxplots(self, data_map, title="", xlab="", ylab="",
+                      xticks_rotation=0, xticks_fontsize=5):
+        """Plot multiple pairs of data arrays.
+
+        :param self: object.
+        :param data_map: A dictionary with labels as keys and lists as data
+            values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param xticks_rotation: Rotation value for x tick labels.
+        :param xticks_fontsize: Fontsize for x tick labels.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+        plt.boxplot(list(data_map.values()))
+        plt.xticks(np.arange(len(data_map)) + 1, data_map.keys(),
+                   rotation=xticks_rotation, fontsize=xticks_fontsize)
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def plot_bars_simple(self, data_map, title="", xlab="", ylab="", alpha=0.6,
+                         xticks_rotation=0, auto_limit=False):
+        """Plot simple bar chart from input dictionary.
+
+        :param self: object.
+        :param data_map: A dictionary with labels as keys and data as values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param alpha: Alpha value.
+        :param xticks_rotation: Rotation value for x tick labels.
+        :param auto_limit: Set y axis limits automatically.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+
+        labels = list(data_map.keys())
+        data = list(data_map.values())
+        positions = np.arange(len(labels))
+        plt.bar(positions, data, align='center', alpha=alpha)
+        plt.xticks(positions, labels, rotation=xticks_rotation)
+
+        if auto_limit:
+            low, high = min(data), max(data)
+            plt.ylim([(low - 0.5 * (high - low)), (high + 0.5 * (high - low))])
+
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def plot_histograms(self, data_map, title="", xlab="", ylab="", bins=50,
+                        alpha=0.7, legend_loc='best', legend=True,
+                        vlines=None):
+        """Plot histograms of multiple data arrays.
+
+        :param self: object.
+        :param data_map: A dictionary with labels as keys and data arrays
+            as values.
+        :param title: Figure title.
+        :param xlab: X axis label.
+        :param ylab: Y axis label.
+        :param bins: Number of bins.
+        :param alpha: Transparency value for histograms.
+        :param legend_loc: Location of legend.
+        :param legend: Plot legend if True.
+        :param vlines: Dictionary with labels and positions of vertical lines
+            to draw.
+        :returns: None
+        :rtype: object
+        """
+        fig = plt.figure()
+
+        for label, data in six.iteritems(data_map):
+            if len(data) > 0:
+                plt.hist(data, bins=bins, label=label, alpha=alpha)
+        if vlines is not None:
+            for label, pos in six.iteritems(vlines):
+                plt.axvline(x=pos, label=label)
+        if legend:
+            plt.legend(loc=legend_loc)
+
+        self._set_properties_and_close(fig, title, xlab, ylab)
+
+    def close(self):
+        """Close PDF backend.
+
+        Do not forget to call this at the end of your
+        script or your output will be damaged!
+
+        :param self: object
+        :returns: None
+        :rtype: object
+        """
+        self.pages.close()
+
+
+def _parse_stat_line(sl):
+    """Parse a stats line."""
+    res = {}
+    tmp = sl.split(':')[1]
+    tmp = tmp.split('|')
+    res['sensitivity'] = float(tmp[0].strip())
+    res['precision'] = float(tmp[1].strip())
+    return res
+
+
+def _parse_matching_line(line):
+    """Parse a metching line."""
+    tmp = line.split(':')[1].strip()
+    return int(tmp)
+
+
+def _parse_mn_line(line):
+    """Parse a miss or novel line."""
+    res = {}
+    tmp = line.split(':')[1].strip()
+    tmp = tmp.split('/')
+    res['value'] = int(tmp[0])
+    tmp = tmp[1].split('(')
+    res['value_total'] = int(tmp[0].strip())
+    res['percent'] = float(tmp[1].split('%)')[0])
+    return res
+
+
+def _parse_total_line(line):
+    """Parse a total line."""
+    res = {}
+    tmp = line.split(':')[1].strip()
+    tmp = tmp.split('in')
+    res['transcripts'] = int(tmp[0].strip())
+    tmp = tmp[1].split('loci')
+    res['loci'] = int(tmp[0].strip())
+    tmp = int(tmp[1].split('(')[1].split(' ')[0])
+    res['me_transcripts'] = tmp
+    return res
+
+
+def parse_gffcmp_stats(txt):
+    """Parse a gffcompare stats file.
+
+    :param txt: Path to the gffcompare stats file.
+    :returns: Return as tuple of dataframes containing:
+        perfromance statistics, match statistics, miss statistics,
+        novel statistics, total statistics.
+    :rtype: tuple
+    """
+    sensitivity = []
+    precision = []
+    level = []
+
+    matching = OrderedDict()
+
+    missed_level = []
+    missed = []
+    missed_total = []
+    missed_percent = []
+
+    novel_level = []
+    novel = []
+    novel_total = []
+    novel_percent = []
+
+    total_target = []
+    total_loci = []
+    total_transcripts = []
+    total_multiexonic = []
+
+    fh = open(txt, 'r')
+    for line in fh:
+        line = line.strip()
+        if len(line) == 0:
+            continue
+        # Parse totals:
+        if line.startswith('#     Query mRNAs'):
+            total_target.append('Query')
+            r = _parse_total_line(line)
+            total_loci.append(r['loci'])
+            total_transcripts.append(r['transcripts'])
+            total_multiexonic.append(r['me_transcripts'])
+
+        if line.startswith('# Reference mRNAs '):
+            total_target.append('Reference')
+            r = _parse_total_line(line)
+            total_loci.append(r['loci'])
+            total_transcripts.append(r['transcripts'])
+            total_multiexonic.append(r['me_transcripts'])
+
+        # Parse basic statistics:
+        if line.startswith('Base level'):
+            st = _parse_stat_line(line)
+            level.append('Base')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Exon level'):
+            st = _parse_stat_line(line)
+            level.append('Exon')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Intron level'):
+            st = _parse_stat_line(line)
+            level.append('Intron')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Intron chain level'):
+            st = _parse_stat_line(line)
+            level.append('Intron chain')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Transcript level'):
+            st = _parse_stat_line(line)
+            level.append('Transcript')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Locus level'):
+            st = _parse_stat_line(line)
+            level.append('Locus')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+
+        # Parse match statistics:
+        if line.startswith('Matching intron chains'):
+            m = _parse_matching_line(line)
+            matching['Intron chains'] = [m]
+        if line.startswith('Matching transcripts'):
+            m = _parse_matching_line(line)
+            matching['Transcripts'] = [m]
+        if line.startswith('Matching loci'):
+            m = _parse_matching_line(line)
+            matching['Loci'] = [m]
+
+        # Parse missing statistics:
+        if line.startswith('Missed exons'):
+            missed_level.append('Exons')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+        if line.startswith('Missed introns'):
+            missed_level.append('Introns')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+        if line.startswith('Missed loci'):
+            missed_level.append('Loci')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+
+        # Parse novel statistics:
+        if line.startswith('Novel exons'):
+            novel_level.append('Exons')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+        if line.startswith('Novel introns'):
+            novel_level.append('Introns')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+        if line.startswith('Novel loci'):
+            novel_level.append('Loci')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+
+    fh.close()
+
+    df_stats = pd.DataFrame(OrderedDict(
+        [('Sensitivity', sensitivity), ('Precision', precision)]), index=level)
+    df_match = pd.DataFrame(matching, index=['Matching'])
+
+    df_miss = pd.DataFrame(
+        OrderedDict(
+                [('Total', missed_total),
+                 ('Missed', missed),
+                 ('Percent missed', missed_percent)]), index=missed_level)
+
+    df_novel = pd.DataFrame(
+        OrderedDict(
+                [('Total', novel_total),
+                 ('Novel', novel),
+                 ('Percent novel', novel_percent)]), index=novel_level)
+
+    df_total = pd.DataFrame(OrderedDict(
+        [('Loci', total_loci), ('Transcripts', total_transcripts),
+         ('Multiexonic', total_multiexonic)]), index=total_target)
+
+    return df_stats, df_match, df_miss, df_novel, df_total
+
+
+if __name__ == '__main__':
+    args = parser.parse_args()
+
+    stats, match, miss, novel, total = parse_gffcmp_stats(args.input)
+    tracking = pd.read_csv(args.t, sep="\t", header=None, usecols=[0, 3],
+                           names=['Count', 'Overlaps'])
+
+    tracking = tracking.groupby("Overlaps").count().reset_index()
+    tracking = tracking.sort_values("Overlaps")
+
+    plotter = Report(args.r)
+
+    # Plot overview panel:
+
+    plt.figure(1)
+    plt.subplot(2, 2, 1)
+    total.plot(ax=plt.gca(), kind='barh', sharex=False, title='Totals')
+    plt.tight_layout()
+
+    plt.subplot(2, 2, 2)
+    stats.plot(ax=plt.gca(), kind='barh', legend=True, sharex=False,
+               title='Performance').legend(loc='best')
+    plt.tight_layout()
+
+    plt.subplot(2, 2, 3)
+
+    miss.copy().drop(
+        'Percent missed', axis=1).plot(
+            ax=plt.gca(), kind='barh',
+            legend=True, sharex=False,
+            title='Missed')
+
+    plt.tight_layout()
+
+    plt.subplot(2, 2, 4)
+
+    novel.copy().drop(
+        'Percent novel', axis=1).plot(
+            ax=plt.gca(), kind='barh', legend=True, sharex=False,
+            title='Novel')
+
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    # Plot individual panels:
+
+    total.plot(kind='barh', subplots=True, legend=False, sharex=False)
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    stats.plot(kind='barh', subplots=True, legend=False, sharex=False)
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    match.plot(kind='barh', subplots=True, legend=False)
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    miss.plot(kind='barh', subplots=True, legend=False, sharex=False)
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    novel.plot(kind='barh', subplots=True, legend=False, sharex=False)
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    def fix_names(s):
+        """Map trancript classification codes."""
+        names = {
+            '=': 'ExactMatch:=',
+            'c': 'Contained:c',
+            'k': 'ReverseContained:k',
+            'm': 'RetainedIntron:m',
+            'n': 'PartRetainedIntron:n',
+            'j': 'PartialMatch:j',
+            'e': 'TransFragMatch:e',
+            's': 'OppositeMatch:s',
+            'o': 'OtherSameStrand:o',
+            'x': 'ExonicOpposite:o',
+            'y': 'RefInIntrons:y',
+            'p': 'PolymeraseRunon:p',
+            'r': 'Repeat:r',
+            'u': 'Intergenic:u',
+            'i': 'FullyIntronic:i',
+            }
+        return names[s]
+
+    # Plot overlaps panel:
+    tracking.Overlaps = tracking.Overlaps.apply(fix_names)
+    tracking = tracking.set_index("Overlaps")
+    tracking.plot(kind='bar', title="Overlaps detected by gffcompare",
+                  colormap='Paired')
+    plt.tight_layout()
+    plotter.pages.savefig()
+    tracking["Percent"] = tracking.Count * 100 / tracking.Count.sum()
+    tracking[["Percent"]].plot(
+        kind='bar', title="Overlaps detected by gffcompare", colormap='Paired')
+    plt.tight_layout()
+    plotter.pages.savefig()
+
+    plotter.close()

bin/report.py

@@ -2,10 +2,512 @@
 """Create workflow report."""
 
 import argparse
+from collections import OrderedDict
 
+from aplanat import bars, lines
 from aplanat.components import fastcat
 from aplanat.components import simple as scomponents
 from aplanat.report import WFReport
+from aplanat.util import Colors
+from bokeh.layouts import gridplot
+from bokeh.models import ColumnDataSource
+from bokeh.palettes import Category10_10
+from bokeh.plotting import figure
+from bokeh.transform import dodge
+import numpy as np
+import pandas as pd
+
+
+def simple_hbar(df, y, right, title="", color=Colors.cerulean,
+                fig_kwargs={}, plot_kwargs={}):
+    """Create a simple barplot.
+
+    :param groups: the grouping variable (the x-axis values).
+    :param values: the data for bars are drawn (the y-axis values).
+    :param kwargs: kwargs for bokeh figure.
+
+    Move to planat when it's working?
+    """
+    defaults = {
+        'output_backend': 'webgl',
+        'plot_height': 300, 'plot_width': 600}
+    defaults.update(fig_kwargs)
+
+    p = figure(y_range=df[y], height=250, title=title,
+               toolbar_location=None, tools="")
+
+    plot_kwargs.update({'height': 0.2})
+    p.hbar(y=df[y], right=df[right], **plot_kwargs)
+
+    return p
+
+
+def _parse_stat_line(sl):
+    """Parse a stats line."""
+    res = {}
+    tmp = sl.split(':')[1]
+    tmp = tmp.split('|')
+    res['sensitivity'] = float(tmp[0].strip())
+    res['precision'] = float(tmp[1].strip())
+    return res
+
+
+def _parse_matching_line(line):
+    """Parse a metching line."""
+    tmp = line.split(':')[1].strip()
+    return int(tmp)
+
+
+def _parse_mn_line(line):
+    """Parse a miss or novel line."""
+    res = {}
+    tmp = line.split(':')[1].strip()
+    tmp = tmp.split('/')
+    res['value'] = int(tmp[0])
+    tmp = tmp[1].split('(')
+    res['value_total'] = int(tmp[0].strip())
+    res['percent'] = float(tmp[1].split('%)')[0])
+    return res
+
+
+def _parse_total_line(line):
+    """Parse a total line."""
+    res = {}
+    tmp = line.split(':')[1].strip()
+    tmp = tmp.split('in')
+    res['transcripts'] = int(tmp[0].strip())
+    tmp = tmp[1].split('loci')
+    res['loci'] = int(tmp[0].strip())
+    tmp = int(tmp[1].split('(')[1].split(' ')[0])
+    res['me_transcripts'] = tmp
+    return res
+
+
+def parse_gffcmp_stats(txt):
+    """Parse a gffcompare stats file.
+
+    :param txt: Path to the gffcompare stats file.
+    :returns: Return as tuple of dataframes containing:
+        perfromance statistics, match statistics, miss statistics,
+        novel statistics, total statistics.
+    :rtype: tuple
+    """
+    sensitivity = []
+    precision = []
+    level = []
+
+    matching = OrderedDict()
+
+    missed_level = []
+    missed = []
+    missed_total = []
+    missed_percent = []
+
+    novel_level = []
+    novel = []
+    novel_total = []
+    novel_percent = []
+
+    total_target = []
+    total_loci = []
+    total_transcripts = []
+    total_multiexonic = []
+
+    fh = open(txt, 'r')
+    for line in fh:
+        line = line.strip()
+        if len(line) == 0:
+            continue
+        # Parse totals:
+        if line.startswith('#     Query mRNAs'):
+            total_target.append('Query')
+            r = _parse_total_line(line)
+            total_loci.append(r['loci'])
+            total_transcripts.append(r['transcripts'])
+            total_multiexonic.append(r['me_transcripts'])
+
+        if line.startswith('# Reference mRNAs '):
+            total_target.append('Reference')
+            r = _parse_total_line(line)
+            total_loci.append(r['loci'])
+            total_transcripts.append(r['transcripts'])
+            total_multiexonic.append(r['me_transcripts'])
+
+        # Parse basic statistics:
+        if line.startswith('Base level'):
+            st = _parse_stat_line(line)
+            level.append('Base')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Exon level'):
+            st = _parse_stat_line(line)
+            level.append('Exon')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Intron level'):
+            st = _parse_stat_line(line)
+            level.append('Intron')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Intron chain level'):
+            st = _parse_stat_line(line)
+            level.append('Intron chain')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Transcript level'):
+            st = _parse_stat_line(line)
+            level.append('Transcript')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+        if line.startswith('Locus level'):
+            st = _parse_stat_line(line)
+            level.append('Locus')
+            sensitivity.append(st['sensitivity'])
+            precision.append(st['precision'])
+
+        # Parse match statistics:
+        if line.startswith('Matching intron chains'):
+            m = _parse_matching_line(line)
+            matching['Intron chains'] = [m]
+        if line.startswith('Matching transcripts'):
+            m = _parse_matching_line(line)
+            matching['Transcripts'] = [m]
+        if line.startswith('Matching loci'):
+            m = _parse_matching_line(line)
+            matching['Loci'] = [m]
+
+        # Parse missing statistics:
+        if line.startswith('Missed exons'):
+            missed_level.append('Exons')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+        if line.startswith('Missed introns'):
+            missed_level.append('Introns')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+        if line.startswith('Missed loci'):
+            missed_level.append('Loci')
+            r = _parse_mn_line(line)
+            missed.append(r['value'])
+            missed_total.append(r['value_total'])
+            missed_percent.append(r['percent'])
+
+        # Parse novel statistics:
+        if line.startswith('Novel exons'):
+            novel_level.append('Exons')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+        if line.startswith('Novel introns'):
+            novel_level.append('Introns')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+        if line.startswith('Novel loci'):
+            novel_level.append('Loci')
+            r = _parse_mn_line(line)
+            novel.append(r['value'])
+            novel_total.append(r['value_total'])
+            novel_percent.append(r['percent'])
+
+    fh.close()
+
+    df_stats = pd.DataFrame(OrderedDict(
+        [('Sensitivity', sensitivity), ('Precision', precision)]), index=level)
+    df_match = pd.DataFrame(matching, index=['Matching'])
+
+    df_miss = pd.DataFrame(
+        OrderedDict(
+                [('Total', missed_total),
+                 ('Missed', missed),
+                 ('Percent missed', missed_percent)]), index=missed_level)
+
+    df_novel = pd.DataFrame(
+        OrderedDict(
+                [('Total', novel_total),
+                 ('Novel', novel),
+                 ('Percent novel', novel_percent)]), index=novel_level)
+
+    df_total = pd.DataFrame(OrderedDict(
+        [('Loci', total_loci), ('Transcripts', total_transcripts),
+         ('Multiexonic', total_multiexonic)]), index=total_target)
+
+    return df_stats, df_match, df_miss, df_novel, df_total
+
+
+def grouped_bar(df, title=""):
+    """Create grouped bar plot from pandas dataframe.
+
+    :param pandas.DataFrame
+        Index:
+            str: the x group labels - groups cluserted using these
+        Columns:
+           numeric: sub-groups of data - each sub group has same colour
+
+    :returns bokaoh.plotting.figure instance
+    """
+    min_ = 0
+    max_ = df.to_numpy().max()
+    max_ = max_ + (max_ * 0.3)  # Add some padding at top of plot for legends
+    yrange = int(min_), int(max_)
+
+    df['x_groups'] = df.index
+    df = df.reset_index(drop=True)
+    source = ColumnDataSource(data=df)
+
+    p = figure(x_range=df['x_groups'], y_range=yrange, height=250, title=title,
+               toolbar_location=None, tools="")
+    i = 0
+    # Use the dodge method to plot groups of bars
+    # https://docs.bokeh.org/en/latest/docs/user_guide/categorical.html
+    dodge_range = (-0.25, 0.25)
+    current_dodge = dodge_range[0]
+    dodge_increment = abs(dodge_range[0] - dodge_range[1]) \
+        / (len(df.columns) - 1)
+
+    for col in df.columns:
+
+        num_colors = df.shape[1] - 1
+        colors = list(zip(*[[Category10_10[x]] * (len(df.columns) - 1)
+                      for x in range(num_colors)]))
+        colors = [item for sublist in colors for item in sublist]
+
+        if col == 'x_groups':
+            continue
+        color = colors[i]
+        i += 1
+
+        width = df.size / 60
+        p.vbar(x=dodge('x_groups', current_dodge, range=p.x_range), top=col,
+               width=width, source=source, color=color, legend_label=col)
+        current_dodge += dodge_increment
+
+    p.x_range.range_padding = 0.1
+    p.xgrid.grid_line_color = None
+    p.legend.location = "top_left"
+    p.legend.orientation = "horizontal"
+    return p
+
+
+def workflow_plots(report, df_aln_stats_file,
+                   gff_cmp_stats_file, gff_cmp_tracking_file):
+    """Create various sections and plots in a WfReport.
+
+    :param report: aplanat WFReport
+    :param df_aln_stats_file: alignment stats. Output of `seqkit bam -s`
+    :param gff_cmp_stats_file: gffcompare stats file
+    :param gff_cmp_tracking_file: gffcompare tracking file
+    :return: None
+    """
+    df_aln_stats = pd.read_csv(df_aln_stats_file, sep='\t')
+    df_aln_stats = df_aln_stats.select_dtypes([np.number]).dropna(axis=1)
+
+    section = report.add_section()
+    section.markdown('''
+    ### Read mapping summary
+
+    Output of [seqkit](https://bioinf.shenwei.me/seqkit/) bam -s''')
+
+    section.table(df_aln_stats)
+
+    # Percentage primary and secondary mapping
+    df_perc = df_aln_stats[['PrimAlnPerc', 'MultimapPerc']]
+    bar_perc = bars.simple_bar(
+        df_perc.columns.values, df_perc.iloc[0].values,
+        title='% primary and multimapping reads', colors=Colors.cerulean)
+
+    # Counts of read mapping class
+    df_counts = df_aln_stats.drop(columns=['PrimAlnPerc', 'MultimapPerc'])
+    bar_counts = bars.simple_bar(
+        df_counts.columns.values, df_counts.iloc[0].values,
+        title='Number of alignment records', colors=Colors.cerulean
+    )
+
+    grid = gridplot([bar_perc, bar_counts], ncols=2,
+                    plot_width=400, plot_height=400)
+    section.plot(grid)
+
+    # If gffcompare has not been run, finish report here
+    if not gff_cmp_stats_file or not gff_cmp_tracking_file:
+        return
+
+    stats, _, miss, novel, total = \
+        parse_gffcmp_stats(gff_cmp_stats_file)
+
+    # Plot overview panel:
+    section = report.add_section()
+    section.markdown('''
+    ### Annotation summary
+
+    The following plots summarize some of the output from
+    [gffcompare](https://ccb.jhu.edu/software/stringtie/gffcompare.shtml)
+
+    * **Totals**:
+    Comparison of the number of stringtie-generated
+    transcripts, multiexonic transcripts and
+    loci (I'm not exactly sure what defines this class at the moment) between
+    reference
+
+    * **Performance**:
+    How accurate are the query transcript annotations with respect to the
+    reference at various levels.
+
+    * **Missed**:
+    Features present in the reference, but absent in the query
+
+    * **Novel**:
+    Features present in the query transcripts, but absent in the reference
+    ''')
+
+    bar_totals = grouped_bar(total, title="Totals")
+    bar_performance = grouped_bar(stats, title="Performance")
+    bar_missed = grouped_bar(miss, title="Missed")
+    bar_novel = grouped_bar(novel, title="Novel")
+
+    grid = gridplot([bar_totals, bar_performance, bar_missed, bar_novel],
+                    ncols=2, plot_width=400, plot_height=400)
+    section.plot(grid)
+
+    def fix_names(s):
+        """Map trancript classification codes."""
+        names = {
+            '=': 'ExactMatch:=',
+            'c': 'Contained:c',
+            'k': 'ReverseContained:k',
+            'm': 'RetainedIntron:m',
+            'n': 'PartRetainedIntron:n',
+            'j': 'PartialMatch:j',
+            'e': 'TransFragMatch:e',
+            's': 'OppositeMatch:s',
+            'o': 'OtherSameStrand:o',
+            'x': 'ExonicOpposite:o',
+            'y': 'RefInIntrons:y',
+            'p': 'PolymeraseRunon:p',
+            'r': 'Repeat:r',
+            'u': 'Intergenic:u',
+            'i': 'FullyIntronic:i',
+            }
+        return names[s]
+
+    # Plot overlaps panel:
+    section = report.add_section()
+    section.markdown('''
+    ## Query transfrag class assignments
+
+    The classes that are assinged by
+    [gffcompare](https://ccb.jhu.edu/software/stringtie/gffcompare.shtml),
+    which describe the relationship between query transfrag and the most
+    similar reference transcript.
+
+    [This diagram](https://ccb.jhu.edu/software/stringtie/
+    gffcompare_codes.png) illustrates the different classes.
+        ''')
+
+    tracking = pd.read_csv(gff_cmp_tracking_file, sep="\t", header=None,
+                           usecols=[0, 3], names=['Count', 'Overlaps'])
+
+    tracking = tracking.groupby("Overlaps").count().reset_index()
+    tracking = tracking.sort_values("Overlaps")
+    tracking.Overlaps = tracking.Overlaps.apply(fix_names)
+    tracking["Percent"] = tracking.Count * 100 / tracking.Count.sum()
+
+    tracking_bar = simple_hbar(
+        tracking, 'Overlaps', 'Count', title="totals")
+    tracking_bar_perc = simple_hbar(
+        tracking, 'Overlaps', 'Percent', title='percent'
+    )
+
+    grid = gridplot([tracking_bar, tracking_bar_perc], ncols=2,
+                    plot_width=400, plot_height=400)
+    section.plot(grid)
+
+
+def pychopper_plots(report, df):
+    """Make plots from pychopper.cdna_classifier.py.
+
+    :param report: aplanat WFReport
+    :param df: result DataFrame
+    """
+    section = report.add_section()
+    section.markdown('''
+    ### pychopper summary statisitcs
+
+    The following plots summarize the output of [cdna_classifier.py]
+    (https://github.com/nanoporetech/pychopper)
+
+    * **Classification of output reads**:
+        * Primers_found: Reads with primers found in correct orientation at
+        both ends.
+        * Rescue: Reads 'rescued' from fused reads
+        * Unusable: Read with missing or incorrect primer orientation
+    * **Strand of oriented reads**:
+        * Strand of read relative to the mRNA
+    * **Strand of rescued read**:
+        * Strand of read that were rescued from fused reads
+    * **Number of primer alignment hits in unclassified reads**:
+        * Note: Need to look into what this means
+    * **Number of primer alignment hits in rescued reads**:
+        * Note: Need to look into what this means
+    * **Number of usable segments per rescued read**:
+        * Number of usable segments (primer-flanked, correctly oriented
+        regions) per fused read.
+    * **Usable bases as a function of cutoff**:
+        * The cutoff value supplied to the primer alignment tool.
+        Note: What are usabel bases in this conext
+    * ** Log10 length distribution of trimmed away sequences**:
+        * todo
+    ''')
+
+    def g(df, index, title):
+        df_ = df[df.index == index]
+        groups = df_.Name.values
+        bar_ = bars.simple_bar(
+            groups, df_['Value'].values,
+            title=title, colors=Colors.cerulean)
+        return bar_
+
+    df1 = df.set_index('Name', drop=True)
+    df1 = df1.T[['Primers_found', 'Rescue', 'Unusable']]
+    bar_class = bars.simple_bar(
+            df1.columns.values, df1.iloc[0].values,
+            title='Classification of output reads', colors=Colors.cerulean)
+
+    plots = [
+        bar_class,
+        g(df, 'Strand', 'Strand of oriented read'),
+        g(df, 'RescueStrand', 'Strand of rescued reads'),
+        g(df, 'UnclassHitNr', 'Number of hits in unclassified reads'),
+        g(df, 'RescueHitNr', 'Number of hits in rescued reads'),
+        g(df, 'RescueSegmentNr', 'Number of usable segments per rescued read')
+        ]
+
+    q = round(df.loc['Parameter', 'Value'], 4)
+    df_at = df[df.index == 'AutotuneSample'].astype('float')
+
+    # Add vertical line at x=q
+    ymin, ymax = df_at['Value'].min(), df_at['Value'].max()
+    plots.append(lines.line([df_at['Name'].values.tolist(), [q, q]],
+                            [df_at['Value'].values.tolist(), [ymin, ymax]],
+                            title=("Usable bases as function of cutoff(q).Best"
+                                   " q={}").format(q), colors=['blue', 'red']
+                            ))
+    df_unusable = df[df.index == 'Unusable'].astype('float')
+
+    plots.append(lines.line([np.log10(1 + df_unusable['Name'])],
+                            [df_unusable['Value']],
+                            title=("Log10 length distribution of trimmed away"
+                                   " sequences.")
+                            ))
+
+    grid = gridplot(plots, ncols=2,
+                    plot_width=400, plot_height=400)
+    section.plot(grid)
 
 
 def main():
@@ -25,14 +527,40 @@ def main():
     parser.add_argument(
         "--commit", default='unknown',
         help="git commit of the executed workflow")
+
+    parser.add_argument(
+        "--alignment_stats", required=True,
+        help="TSV summary file of alignment statistics")
+
+    parser.add_argument(
+        "--gffcompare_tracking", required=False, default=None,
+        help="TSV summary file of alignment statistics")
+    parser.add_argument(
+        "--gffcompare_stats", required=False, default=None,
+        help="TSV summary file of alignment statistics")
+
+    parser.add_argument(
+        "--pychop_report", required=True,
+        help="TSV summary file of pychopper statistics")
     args = parser.parse_args()
 
     report = WFReport(
-        "Workflow Template Sequencing report", "wf-template",
+        "Workflow for assembling transcript isoforms", "wf-isoforms",
         revision=args.revision, commit=args.commit)
 
+    # Add reads summary section
     report.add_section(
         section=fastcat.full_report(args.summaries))
+
+    # workflow-specific plotting
+    workflow_plots(report, args.alignment_stats,  args.gffcompare_stats,
+                   args.gffcompare_tracking)
+
+    if args.pychop_report:
+        df_chop_stats = pd.read_csv(args.pychop_report, sep='\t', index_col=0)
+        pychopper_plots(report, df_chop_stats)
+
+    # Arguments and software versions
     report.add_section(
         section=scomponents.version_table(args.versions))
     report.add_section(

bin/run_fastq_qc.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python
+"""Get fastq QC reports."""
+
+# -*- coding: utf-8 -*-
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from pysam import FastxFile
+
+
+def parse_args(argv=sys.argv[1:]):
+    """Parse args."""
+    description = """Script to run the isoform workflow """
+    parser = argparse.ArgumentParser(description=description)
+    parser.add_argument("--fastq", required=True, help="")
+    parser.add_argument("--output_dir", required=True, help="")
+    return parser.parse_args(argv)
+
+
+def run_fastq_qc(fastq_path, output):
+    """Write QC info to files."""
+    qualities = list()
+    mean_qualities = list()
+    lengths = list()
+
+    with FastxFile(fastq_path) as fq:
+        for rec in fq:
+            # ONT calculation for "mean Q score"
+            quals = np.fromiter(
+                (ord(x) - 33 for x in rec.quality),
+                dtype=int, count=len(rec.quality))
+            mean_p = np.mean(np.power(10, quals / -10))
+            mean_qualities.append(-10 * np.log10(mean_p))
+            # all qualities
+            qualities.extend(quals)
+            lengths.append(len(quals))
+
+    with open(os.path.join(output, "base_qual.txt"), 'w') as f:
+        f.write("\n".join((str(q) for q in qualities)))
+
+    with open(os.path.join(output, "read_qual.txt"), 'w') as f:
+        f.write("\n".join((str(q) for q in mean_qualities)))
+
+    with open(os.path.join(output, "lengths.txt"), 'w') as f:
+        f.write("\n".join((str(_l) for _l in lengths)))
+
+
+def main(args):
+    """Run entry point."""
+    assert os.path.isfile(args.fastq)
+    assert os.path.isdir(args.output_dir)
+    run_fastq_qc(fastq_path=args.fastq, output=args.output_dir)
+
+
+if __name__ == '__main__':
+    main(args=parse_args())

data/OPTIONAL_FILE_1

@@ -0,0 +1,11 @@
+# Nothing to see here. A sentinel file to replace real data.
+# e.g.:
+#
+# input:
+#     file some_data
+#     file extra_data
+# script:
+# def extra = extra_data.name != 'OPTIONAL_FILE' ? "--extra-data $opt" : ''
+# """
+# command ${some_data} ${extra}
+# """

denovo.nf

@@ -0,0 +1,205 @@
+import sys
+import time
+import os
+from os import path
+import pandas as pd
+from glob import glob
+import re
+from threading import Lock
+from itertools import zip_longest
+from collections import OrderedDict, namedtuple
+
+if not workflow.overwrite_configfiles:
+    configfile: "config.yml"
+
+WORKDIR = path.abspath(path.join(config["workdir_top"], config["pipeline"]))
+workdir: WORKDIR
+SNAKEDIR = path.dirname(workflow.snakefile)
+
+include: "snakelib/utils.snake"
+
+in_fastq = config["reads_fastq"]
+if not path.isabs(in_fastq):
+	in_fastq = path.join(SNAKEDIR, in_fastq)
+
+
+class Node:
+    def __init__(self, Id, File, Left, Right, Parent, Level):
+        self.Id = Id
+        self.File = File
+        self.Left = Left
+        self.Right = Right
+        self.Parent = Parent
+        self.Level = Level
+        self.Done = False
+        self.RightSide = False
+    def __repr__(self):
+        return "Node:{} Level: {} File: {} Done: {} Left: {} Right: {} Parent: {}".format(self.Id, self.Level, self.File, self.Done, self.Left.Id if self.Left is not None else None, self.Right.Id if self.Right is not None else None, self.Parent.Id if self.Parent is not None else None)
+
+def grouper(n, iterable, fillvalue=None):
+    "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
+    args = [iter(iterable)] * n
+    return zip_longest(fillvalue=fillvalue, *args)
+
+def build_job_tree(batch_dir):
+    batches = glob("{}/isONbatch_*.cer".format(batch_dir))
+    batch_ids = [int(re.search('/isONbatch_(.*)\.cer$', x).group(1)) for x in batches]
+    global LEVELS
+    global JOB_TREE
+    LEVELS = OrderedDict()
+    LEVELS[0] = []
+    for Id, bf in sorted(zip(batch_ids, batches), key=lambda x: x[0]):
+        n = Node(Id, "clusters/isONcluster_{}.cer".format(Id), None, None, None, 0)
+        n.Done = True
+        JOB_TREE[Id] = n
+        LEVELS[0].append(n)
+    level = 0
+    max_id = LEVELS[0][-1].Id
+    while len(LEVELS[level]) != 1:
+        next_level = level + 1
+        LEVELS[next_level] = []
+        for l, r in grouper(2, LEVELS[level]):
+            if r is None:
+                LEVELS[level].pop()
+                l.Level += 1
+                LEVELS[next_level].append(l)
+                continue
+            max_id += 1
+            new_batch = "clusters/isONcluster_{}.cer".format(max_id)
+            new_node = Node(max_id, new_batch, l, r, None, next_level)
+            l.Parent = new_node
+            r.Parent =  new_node
+            r.RightSide = True
+            LEVELS[next_level].append(new_node)
+            JOB_TREE[max_id] = new_node
+        level = next_level
+    global ROOT
+    ROOT = JOB_TREE[len(JOB_TREE)-1].Id
+    JOB_TREE[ROOT].RightSide = True
+    print("Merge clustering job tree nodes:",file=sys.stderr)
+    for n in JOB_TREE.values():
+        print("\t{}".format(n),file=sys.stderr)
+
+
+def generate_rules(levels, snk):
+    init_template = """
+rule cluster_job_%d:
+    input:
+        left = "sorted/batches/isONbatch_%d.cer",
+    output: "clusters/isONcluster_%d.cer"
+    shell: "isONclust2 cluster -x %s -v -Q -l %s -o %s %s; sync"
+
+    """
+    template = """
+rule cluster_job_%d:
+    input:
+        left = "clusters/isONcluster_%d.cer",
+        right = "clusters/isONcluster_%d.cer",
+    output: "clusters/isONcluster_%d.cer"
+    shell: "isONclust2 cluster -x %s -v -Q -l %s -r %s -o %s %s; sync"
+
+    """
+
+    link_template="""
+rule link_root:
+    input: "clusters/isONcluster_%d.cer",
+    output: "clusters/isONcluster_ROOT.cer"
+    shell: "ln -s `realpath {input}` {output}"
+
+    """
+
+    fh = open(snk, "w")
+    for nr, l in levels.items():
+        for n in l:
+            purge = "-z" if n.RightSide else ""
+            if nr == 0 or n.Left is None or n.Right is None:
+                jr = init_template % (n.Id, n.Id, n.Id, config["cls_mode"], "{input.left}", "{output}", purge)
+                fh.write(jr)
+            else:
+                jr = template % (n.Id, n.Left.Id, n.Right.Id, n.Id, config["cls_mode"], "{input.left}", "{input.right}", "{output}", purge)
+                fh.write(jr)
+    global ROOT
+    fh.write(link_template % ROOT)
+    fh.write("\nROOT = %d" % ROOT)
+    fh.flush()
+    fh.close()
+
+def count_fastq_bases(fname, size=128000000):
+    fh = open(fname, "r")
+    count = 0
+    while True:
+        b = fh.read(size)
+        if not b:
+            break
+        count += b.count("A")
+        count += b.count("T")
+        count += b.count("G")
+        count += b.count("C")
+        count += b.count("U")
+    fh.close()
+    return count
+
+def preprocess_reads(fq):
+    pc_opts = config["pychopper_opts"]
+    concat = config["concatenate"]
+    thr = config["cores"]
+
+    out_fq = "processed_reads/full_length_reads.fq"
+    if os.path.isdir("processed_reads"):
+        return out_fq
+
+    shell("mkdir -p processed_reads")
+    if concat:
+        print("Concatenating reads under directory: " + fq)
+        shell("find %s  -regextype posix-extended -regex '.*\.(fastq|fq)$' -exec cat {{}} \\; > processed_reads/input_reads.fq" % fq)
+    else:
+        shell("ln -s `realpath %s` processed_reads/input_reads.fq" % fq)
+
+    if config["run_pychopper"]:
+        print("Running pychopper of fastq file: processed_reads/input_reads.fq")
+        shell("(cd processed_reads; cdna_classifier.py -t %d %s input_reads.fq full_length_reads.fq)" % (thr, pc_opts))
+    else:
+        shell("ln -s `realpath processed_reads/input_reads.fq` processed_reads/full_length_reads.fq")
+
+    return out_fq
+
+
+ROOT = None
+DYNAMIC_RULES="job_rules.snk"
+if ((not os.path.isfile(os.path.join(WORKDIR,"sorted","sorted_reads.fastq"))) or (not os.path.isfile(os.path.join(SNAKEDIR, DYNAMIC_RULES)))):
+    print("Preprocessing read in fastq file:", in_fastq)
+    proc_fastq = preprocess_reads(in_fastq)
+    print("Counting records in input fastq:", proc_fastq)
+    nr_bases = count_fastq_bases(proc_fastq)
+    print("Bases in input: {} megabases".format(int(nr_bases/10**6)))
+    if config['batch_size'] < 0:
+        config['batch_size'] = int(nr_bases/1000/config["cores"])
+
+    print("Batch size is: {}".format(config['batch_size']))
+
+    init_cls_options = """ --batch-size {} --kmer-size {} --window-size {} --min-shared {} --min-qual {}\
+                         --mapped-threshold {} --aligned-threshold {} --min-fraction {} --min-prob-no-hits {} -M {} -P {} -g {} -c {} -F {} """
+    init_cls_options = init_cls_options.format(config["batch_size"], config["kmer_size"], config["window_size"], config["min_shared"], config["min_qual"], \
+                        config["mapped_threshold"], config["aligned_threshold"], config["min_fraction"], config["min_prob_no_hits"], config["batch_max_seq"], config["consensus_period"],
+    config["consensus_minimum"], config["consensus_maximum"], config["min_left_cls"])
+
+    shell("""
+            rm -fr clusters sorted
+            mkdir -p sorted; isONclust2 sort {} -v -o sorted {};
+            mkdir -p clusters;
+        """.format(init_cls_options, proc_fastq))
+    JOB_TREE = OrderedDict()
+    LEVELS = None
+
+    build_job_tree("sorted/batches")
+    generate_rules(LEVELS, "{}/job_rules.snk".format(SNAKEDIR))
+
+include: DYNAMIC_RULES
+
+
+
+rule all:
+    input: rules.link_root.output
+    output: directory("final_clusters")
+    shell:
+        """ isONclust2 dump -v -i sorted/sorted_reads_idx.cer -o final_clusters {input}; sync """

environment.yaml

@@ -1,4 +1,4 @@
-name: epi2melabs-nf-template-workflow
+name: epi2melabs-wf-isoforms
 channels:
     - epi2melabs
     - bioconda
@@ -7,5 +7,21 @@ channels:
 dependencies:
     - python==3.8.*
     - aplanat >=0.5.0
+    - epi2melabs
+    - minimap2
+    - samtools
+    - bedtools
+    - pychopper
+    - pandas
+    - seaborn
+    - requests
+    - gffread
+    - seqkit
+    - csvtk
+    - stringtie==2.1.1
+    - gffcompare
+    - curl
     - pysam
+    - spoa==3.4.0
     - fastcat
+    - isonclust2

lib/fastqingress.nf

@@ -1,6 +1,22 @@
+process handleSingleFile {
+    label "isoforms"
+    cpus 1
+    input:
+        file reads
+    output:
+        path "$reads.simpleName"
+    script:
+        def name = reads.simpleName
+        def reads_dir = 'reads_dir'
+    """
+    mkdir $name
+    mv $reads $name
+    """
+}
+
 
 process checkSampleSheet {
-    label "artic"
+    label "isoforms"
     cpus 1
     input:
         file "sample_sheet.txt"
@@ -13,20 +29,19 @@ process checkSampleSheet {
 
 
 /**
- * Load a sample sheet into a Nextflow channel to map barcodes
- * to sample names.
+ * Take an input file and sample name to return a channel with
+ * a single named sample.
  *
- * @param samples CSV file according to MinKNOW sample sheet specification
- * @return A Nextflow Channel of tuples (barcode, sample name)
- */ 
-def check_sample_sheet(samples)
+ *
+ * @param input_file Single fastq file
+ * @param sample_name Name to give the sample
+ * @return Channel of tuples (path, sample_id, type)
+ */
+def handle_single_file(input_file, sample_name)
 {
-    println("Checking sample sheet.")
-    sample_sheet = Channel.fromPath(samples, checkIfExists: true)
-    sample_sheet = checkSampleSheet(sample_sheet)
-        .splitCsv(header: true)
-        .map { row -> tuple(row.barcode, row.sample_name) }
-    return sample_sheet
+    singleFile = Channel.fromPath(input_file)
+    sample = handleSingleFile(singleFile)
+    return sample.map { it -> tuple(it, sample_name === null ? it.simpleName : sample_name, 'test_sample') }
 }
 
 
@@ -38,6 +53,7 @@ def check_sample_sheet(samples)
  * @param maxdepth maximum depth to traverse
  * @return list of files.
  */ 
+
 def find_fastq(pattern, maxdepth)
 {
     files = []
@@ -86,91 +102,233 @@ def sanitize_fastq(input_folder, staging)
 
 
 /**
- * Resolves input folder containing barcode subdirectories
- * or a flat set of fastq data to a Nextflow Channel. Removes barcode
- * directories with no fastq files.
+ * Take an input directory return the barcode and non barcode 
+ * sub directories contained within.
  *
- * @param input_folder Top level input folder to locate fastq data
- * @param sample_sheet List of tuples mapping barcode to sample name
- *     or a simple string for non-multiplexed data.
- * @return Channel of tuples (path, sample_name)
+ *
+ * @param input_directory Top level input folder to locate sub directories
+ * @return A list containing sublists of barcode and non_barcode sub directories
  */
-def resolve_barcode_structure(input_folder, sample_sheet)
+def get_subdirectories(input_directory)
 {
-    println("Checking input directory structure.")
-    barcode_dirs = file("$input_folder/barcode*", type: 'dir', maxdepth: 1)
-    not_barcoded = find_fastq(file(input_folder), 1)
-    samples = null
-    if (barcode_dirs) {
-        println(" - Found barcode directories")
-        // remove empty barcode_dirs
-        valid_barcode_dirs = []
-        invalid_barcode_dirs = []
-        for (d in barcode_dirs) {
-            if(!find_fastq(d, 1)) {
-                invalid_barcode_dirs << d
-            } else {
-                valid_barcode_dirs << d
-            }
-        }
-        if (invalid_barcode_dirs.size() > 0) {
-            println(" - Some barcode directories did not contain .fastq(.gz) files:")
-            for (d in invalid_barcode_dirs) {
-                println("   - ${d}")
-            }
-        }
-        // link sample names to barcode through sample sheet
-        if (!sample_sheet) {
-            sample_sheet = Channel
-                .fromPath(valid_barcode_dirs)
-                .filter(~/.*barcode[0-9]{1,3}$/)  // up to 192
-                .map { path -> tuple(path.baseName, path.baseName) }
-        }
-        samples = Channel
-            .fromPath(valid_barcode_dirs)
-            .filter(~/.*barcode[0-9]{1,3}$/)  // up to 192
-            .map { path -> tuple(path.baseName, path) }
-            .join(sample_sheet)
-            .map { barcode, path, sample -> tuple(path, sample) }
-    } else if (not_barcoded) {
-        println(" - Found fastq files, assuming single sample")
-        sample = (sample_sheet == null) ? "unknown" : sample_sheet
-        samples = Channel
-            .fromPath(input_folder, type: 'dir', maxDepth:1)
-            .map { path -> tuple(path, sample) }
-    }
-    return samples
+    barcode_dirs = file("$input_directory/barcode*", type: 'dir', maxdepth: 1)
+    all_dirs = file("$input_directory/*", type: 'dir', maxdepth: 1)
+    non_barcoded = ( all_dirs + barcode_dirs ) - all_dirs.intersect(barcode_dirs)
+    return [barcode_dirs, non_barcoded]
 }
 
 
 /**
- * Take an input directory and sample sheet to return a channel of
- * named samples.
+ * Load a sample sheet into a Nextflow channel to map barcodes
+ * to sample names.
  *
- * @param input_folder Top level input folder to locate fastq data
- * @param sample_sheet List of tuples mapping barcode to sample name
- *     or a simple string for non-multiplexed data.
- * @return Channel of tuples (path, sample_name)
- */
-def fastq_ingress(input_folder, output_folder, samples, sanitize)
+ * @param samples CSV file according to MinKNOW sample sheet specification
+ * @return A Nextflow Channel of tuples (barcode, sample name, sample type)
+ */ 
+def get_sample_sheet(sample_sheet)
 {
-    // EPI2ME harness 
-    if (sanitize) {
-        staging = file(output_folder).resolve("staging")
-        input_folder = sanitize_fastq(file(input_folder), staging)
-    }
-    // check sample sheet
-    sample_sheet = null
-    if (samples) {
-        sample_sheet = check_sample_sheet(samples)
-    }
-    // resolve whether we have demultiplexed data or single sample
-    data = resolve_barcode_structure(input_folder, sample_sheet)
-    // return error if data empty after processing
-    if (data == null) {
-        println("")
-        println("Error: `--fastq` Unable to find FASTQ files or BARCODE folders in the provided --fastq path")
+    println("Checking sample sheet.")
+    sample_sheet = file(sample_sheet);
+    is_file = sample_sheet.isFile()
+
+    if (!is_file) {
+        println('Error: `--samples` is not a file.')
         exit 1
     }
-    return data
+
+    return checkSampleSheet(sample_sheet)
+        .splitCsv(header: true)
+        .map { row -> tuple(
+            row.barcode, 
+            row.sample_id, 
+            row.type ? row.type : 'test_sample') 
+        }
+}
+
+
+/**
+ * Take a list of input directories and return directories which are 
+ * valid, i.e. contains only .fastq(.gz) files.
+ *
+ *
+ * @param input_dirs List of barcoded directories (barcodeXX,mydir...)
+ * @return List of valid directories
+ */
+def get_valid_directories(input_dirs)
+{
+    valid_dirs = []
+    no_fastq_dirs = []
+    invalid_files_dirs = []
+    for (d in input_dirs) {
+        valid = true
+        fastq = find_fastq(d, 1)
+        all_files = file(d.resolve("*"), type: 'file', maxdepth: 1)
+        non_fastq = ( all_files + fastq ) - all_files.intersect(fastq)
+
+        if (non_fastq) {
+            valid = false
+            invalid_files_dirs << d
+        }
+        if (!fastq) {
+            valid = false
+            no_fastq_dirs << d
+        }
+        if (valid) {
+            valid_dirs << d
+        }
+    }
+    if (valid_dirs.size() == 0) {
+        error_message = "Error: None of the directories given contain .fastq(.gz) files."
+        println(error_message)
+        exit 1
+    }
+    if (no_fastq_dirs.size() > 0) {
+        println("Warning: Excluding directories not containing .fastq(.gz) files:")
+        for (d in no_fastq_dirs) {
+            println("   - ${d}")
+        }
+    }
+    if (invalid_files_dirs.size() > 0) {
+        println("Warning: Excluding directories containing non .fastq(.gz) files:")
+        for (d in invalid_files_dirs) {
+            println("   - ${d}")
+        }
+    }
+    return valid_dirs
+}
+
+
+/**
+ * Take an input directory and sample name to return a channel
+ * with a single named sample.
+ *
+ *
+ * @param input_directory Directory of fastq files
+ * @param sample_name Name to give the sample
+ * @return Channel of tuples (path, sample_id, type)
+ */
+def handle_flat_dir(input_directory, sample_name) 
+{
+    valid_dirs= get_valid_directories([ file(input_directory) ])
+    return Channel.fromPath(valid_dirs)
+        .map { it -> tuple(it, sample_name === null ? it.baseName : sample_name, 'test_sample') }
+}
+
+
+/**
+ * Take a list of barcode directories and a sample sheet to return 
+ * a channel of named samples.
+ *
+ *
+ * @param barcoded_dirs List of barcoded directories (barcodeXX,...)
+ * @param sample_sheet List of tuples mapping barcode to sample name
+ *     or a simple string for non-multiplexed data.
+ * @return Channel of tuples (path, sample_id, type)
+ */
+def handle_barcoded_dirs(barcoded_dirs, sample_sheet)
+{
+    valid_dirs = get_valid_directories(barcoded_dirs)
+    // link sample names to barcode through sample sheet
+    if (!sample_sheet) {
+        sample_sheet = Channel
+            .fromPath(valid_dirs)
+            .filter(~/.*barcode[0-9]{1,3}$/)  // up to 192
+            .map { path -> tuple(path.baseName, path.baseName, 'test_sample') }
+    }
+    return Channel
+        .fromPath(valid_dirs)
+        .filter(~/.*barcode[0-9]{1,3}$/)  // up to 192
+        .map { path -> tuple(path.baseName, path) }
+        .join(sample_sheet)
+        .map { barcode, path, sample, type -> tuple(path, sample, type) }
+}
+
+
+/**
+ * Take a list of non-barcode directories to return a channel 
+ * of named samples. Samples are named by directory baseName.
+ *
+ *
+ * @param non_barcoded_dirs List of directories (mydir,...)
+ * @return Channel of tuples (path, sample_id, type)
+ */
+def handle_non_barcoded_dirs(non_barcoded_dirs)
+{
+    valid_dirs = get_valid_directories(non_barcoded_dirs)
+    return Channel.fromPath(valid_dirs)
+        .map { path -> tuple(path, path.baseName, 'test_sample') }
+}
+
+
+/**
+ * Take an input (file or directory) and return a channel of
+ * named samples.
+ *
+ *
+ * @param input Top level input file or folder to locate fastq data.
+ * @param sample string to name single sample data.
+ * @param sample_sheet Path to sample sheet CSV file.
+ * @return Channel of tuples (path, sample_id, type)
+ */
+def fastq_ingress(input, output_folder, sample, sample_sheet, sanitize)
+{
+    println("Checking fastq input.")
+    input = file(input);
+
+    // Handle file input
+    if (input.isFile()) {
+        // Assume sample is a string at this point
+        println('Single file input detected.')
+        if (sample_sheet) {
+            println('Warning: `--sample_sheet` given but single file input found. Ignoring.')
+        }
+        return handle_single_file(input, sample)
+    }
+
+    // Handle directory input
+    if (input.isDirectory()) {
+        // EPI2ME harness 
+        if (sanitize) {
+            staging = file(output_folder).resolve("staging")
+            input = sanitize_fastq(file(input), staging)
+        }
+
+        // Get barcoded and non barcoded subdirectories
+        (barcoded, non_barcoded) = get_subdirectories(input)
+
+        // Case 03: If no subdirectories, handle the single dir
+        if (!barcoded && !non_barcoded) {
+            println("Single directory input detected.")
+            if (sample_sheet) {
+                println('Warning: `--sample_sheet` given but single non-barcode directory found. Ignoring.')
+            }
+            return handle_flat_dir(input, sample)
+        }
+
+        if (sample) {
+            println('Warning: `--sample` given but multiple directories found, ignoring.')
+        }
+
+        // Case 01, 02, 04: Handle barcoded and non_barcoded dirs
+        // Handle barcoded folders
+        barcoded_samples = Channel.empty()
+        if (barcoded) {
+            println("Barcoded directories detected.")
+            if (sample_sheet) {
+                sample_sheet = get_sample_sheet(sample_sheet)
+            }
+            barcoded_samples = handle_barcoded_dirs(barcoded, sample_sheet)
+        }
+
+        non_barcoded_samples = Channel.empty()
+        if (non_barcoded) {
+            println("Non barcoded directories detected.")
+            if (!barcoded && sample_sheet) {
+                println('Warning: `--sample_sheet` given but no barcode directories found.')
+            }
+            non_barcoded_samples = handle_non_barcoded_dirs(non_barcoded)
+        }
+
+        return barcoded_samples.mix(non_barcoded_samples)
+    }
 }

lib/ping.nf

@@ -0,0 +1,36 @@
+
+process pingMessage {
+    label "isoforms"
+    cpus 1
+    input:
+        val message
+        path json
+    script:
+        hostname = InetAddress.getLocalHost().getHostName()
+        opsys = System.properties['os.name'].toLowerCase()
+        disable = params.disable_ping ? '--disable' : ''
+        meta = json.name != 'OPTIONAL_FILE' ? "--meta $json": ''
+    """
+    ping.py \
+        --hostname $hostname \
+        --opsys "$opsys" \
+        --session $workflow.sessionId \
+        --message $message \
+        $meta $disable
+    """
+}
+
+
+// send a start message
+workflow start_ping {
+    main:
+       pingMessage("Started", Channel.fromPath("$projectDir/data/OPTIONAL_FILE"))
+}
+
+// send an end message
+workflow end_ping {
+    take:
+        json
+    main:
+        pingMessage("Finished", json)
+}

main.nf

@@ -11,21 +11,23 @@
 //        as an entry point when using this workflow in isolation.
 
 import groovy.json.JsonBuilder
+import java.util.ArrayList;
 nextflow.enable.dsl = 2
 
 include { fastq_ingress } from './lib/fastqingress' 
+include { start_ping; end_ping } from './lib/ping'
 
 
 process summariseReads {
     // concatenate fastq and fastq.gz in a dir
 
-    label "pysam"
+    label "isoforms"
     cpus 1
     input:
-        tuple path(directory), val(sample_name)
+        tuple path(directory), val(sample_name), val(type)
     output:
-        path "${sample_name}.stats"
-    shell:
+        tuple val(sample_name), path('*'), emit: summary
+    script:
     """
     fastcat -s ${sample_name} -r ${sample_name}.stats -x ${directory} > /dev/null
     """
@@ -33,19 +35,33 @@ process summariseReads {
 
 
 process getVersions {
-    label "pysam"
+    label "isoforms"
     cpus 1
     output:
         path "versions.txt"
     script:
     """
     python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
+    python -c "import aplanat; print(f'aplanat,{aplanat.__version__}')" >> versions.txt
+    python -c "import pandas; print(f'pandas,{pandas.__version__}')" >> versions.txt
+    python -c "import seaborn; print(f'seaborn,{seaborn.__version__}')" >> versions.txt
     fastcat --version | sed 's/^/fastcat,/' >> versions.txt
+    minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
+    samtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
+    bedtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
+    python -c "import pychopper; print(f'pychopper,{pychopper.__version__}')" >> versions.txt
+    gffread --version | sed 's/^/gffread,/' >> versions.txt
+    seqkit version | head -n 1 | sed 's/ /,/' >> versions.txt
+    csvtk version | head -n 1 | sed 's/ /,/' >> versions.txt
+    stringtie --version | sed 's/^/stringtie,/' >> versions.txt
+    gffcompare --version | head -n 1 | sed 's/ /,/' >> versions.txt
+    spoa --version | sed 's/^/spoa,/' >> versions.txt
     """
 }
 
+
 process getParams {
-    label "pysam"
+    label "isoforms"
     cpus 1
     output:
         path "params.json"
@@ -57,19 +73,282 @@ process getParams {
     """
 }
 
-process makeReport {
-    label "pysam"
+process preprocess_reads {
+    /* 
+    Concatenate reads from a sample directory.
+    Optionally classify, trim, and orient cDNA reads using cdna_classifier from pychopper
+    */
+ 
+    label "isoforms"
+    cpus params.threads
+
     input:
-        path "seqs.txt"
-        path "versions/*"
-        path "params.json"
+        tuple path(directory), val(sample_id), val(type) 
     output:
-        path "wf-template-*.html"
+         tuple val(sample_id), path("full_length_reads.fq"), emit: full_len_reads
+         tuple val(sample_id), path('cdna_classifier_report.tsv'),  emit: report
+//          val "${sample_id}", emit: sample_id
+//          path 'cdna_classifier_report.tsv', optional: true, emit: cdna_class_report
+         // Not sure if this is an antipattern, but it's function is to publish all the files to the publishDir dir
+    script:
+        """
+        fastcat -s ${sample_id} -r ${sample_id}.stats -x ${directory} > input_reads.fq
+    
+        if [[ ${params.use_pychopper} == true ]];
+        then
+            cdna_classifier.py -t ${params.threads} ${params.pychopper_opts} input_reads.fq full_length_reads.fq
+            generate_pychopper_stats.py --data cdna_classifier_report.tsv --output .
+        else
+            ln -s `realpath input_reads.fq` full_length_reads.fq
+        fi
+        """
+}
+
+process generate_fq_stats {
+    label "isoforms"
+
+    input:
+        tuple(sample_id), path(fastq), val(unused)
+
+    output:
+        path "*"
     script:
-        // report naming
-        report_name = "wf-template-" + params.report_name + '.html'
     """
-    report.py $report_name --versions versions seqs.txt --params params.json
+        run_fastq_qc.py --fastq ${fastq} --output .
+    """
+}
+
+process build_minimap_index{
+    /*
+    Build minimap index from reference genome
+    */
+    label "isoforms"
+    cpus params.threads
+
+    input:
+        file reference
+    output:
+        path "genome_index.mmi", emit: index
+    script:
+    """
+        minimap2 -t ${params.threads} ${params.minimap_index_opts} -I 1000G -d "genome_index.mmi" ${reference}
+    """
+}
+
+
+process map_reads{
+    /*
+    Map reads to reference using minimap2.
+    Filter reads by mapping quality.
+    Filter reads where length of poly(A) > max_poly_run at either ends of the read (defined by poly_context)
+    */
+    label "isoforms"
+    cpus params.threads
+
+    input:
+       file index
+       file reference
+       tuple val(sample_id), file(fastq_reads)
+
+    output:
+       tuple val(sample_id), path("reads_aln_sorted.bam"), emit: bam
+       tuple val(sample_id), path("read_aln_stats.tsv"), emit: stats
+    script:
+        def ab = "reads_aln_sorted.bam"
+        def af = "internal_priming_fail.tsv"
+        def fs = "context_internal_priming_fail_start.fasta"
+        def fe = "context_internal_priming_fail_end.fasta"
+        def fasta_reads = "reads.fa"
+        def ContextFilter = """AlnContext: { Ref: "${reference}", LeftShift: -${params.poly_context}, RightShift: ${params.poly_context},
+           RegexEnd: "[Aa]{${params.max_poly_run},}",
+           Stranded: True,Invert: True, Tsv: "internal_priming_fail.tsv"} """
+       """
+       seqkit fq2fa ${fastq_reads} -o ${fasta_reads};
+       minimap2 -t ${params.threads} -ax splice ${params.minimap2_opts} ${index} ${fasta_reads}\
+       | samtools view -q ${params.minimum_mapping_quality} -F 2304 -Sb -\
+       | seqkit bam -j ${params.threads} -x -T '${ContextFilter}' -\
+       | samtools sort -@ ${params.threads} -o ${ab} -;
+
+       ((seqkit bam -s -j ${params.threads} ${ab} 2>&1)  | tee read_aln_stats.tsv ) || true
+
+       if [[ -s ${af} ]];
+       then
+           tail -n +2 ${af} | awk '{{print ">" \$1 "\\n" \$4 }}' - > ${fs}
+           tail -n +2 ${af} | awk '{{print ">" \$1 "\\n" \$6 }}' - > ${fe}
+       fi
+       """
+}
+
+process plot_aln_stats{
+    /*
+    Create a pdf of alignemnt statistis. 
+    */
+    label 'isoforms'
+
+    input:
+        tuple val(sample_id), path (aln_stats)
+    output:
+        path "*"
+    script:
+    """
+    plot_aln_stats.py ${aln_stats} -r read_aln_stats.pdf
+    """
+}
+
+process split_bam{
+    /*
+    Partition BAM file into loci or bundles with `params.bundle_min_reads` minimum size
+    If no splitting required, just create single symbolic link to a single bundle.
+
+    Output tuples containing `sample_id` so bundles can be combined later in th pipeline.
+    */
+
+    label 'isoforms'
+    cpus params.threads
+
+    input:
+        tuple val(sample_id), path(bam)
+    output:
+        tuple val(sample_id), path('*.bam'), emit: bundles
+    script:
+    if (params["bundle_min_reads"] != false)
+        """
+        seqkit bam -j ${params.threads} -N ${params.bundle_min_reads} ${bam} -o  bam_bundles/
+        mv bam_bundles/* .
+        """
+    else
+        """
+        mkdir -p ./${sample_id}_bam_bundles
+        ln -s ${bam} ${sample_id}_bam_bundles/000000000_ALL:0:1_bundle.bam
+        """
+}
+
+
+process stringtie{
+    /*
+    Takes in aligned reads in bam format that may be a chunk of a larger alignment file.
+    $G_FLAG specifies whether or not to use reference annotation as a guide in transcript assembly. 
+
+    Output gff annotation files in a tuple with `sample_id` for combining into samples late rin the pipeline.
+    */
+    label 'isoforms'
+    cpus params.threads
+
+    input:
+        tuple val(sample_id), path(bam)
+        file ref_annotation
+    output:
+        tuple val(sample_id), path('*.gff'), emit: gff_bundles
+    script:
+        def out_filename = bam.name.replaceFirst(~/\.[^\.]+$/, '') + '.gff'
+        def label = "STR.${bam.name.split('_')[0].toInteger()}."
+        def G_FLAG = ref_annotation.name.startsWith('OPTIONAL_FILE') ? '' : "-G ${ref_annotation}"
+    """
+    stringtie --rf ${G_FLAG} -L -v -p ${params.threads} ${params.stringtie_opts} -o  ${out_filename} \
+    ${bam} 2>/dev/null
+    """
+}
+
+
+process merge_gff_bundles{
+    /*
+    Merge gff bundles into a single gff file.
+    */
+    label 'isoforms'
+
+    input:
+        tuple val(sample_id), path (gff_bundle)
+    output:
+        tuple val(sample_id), path('*.gff'), emit: gff
+    script:
+    def merged_gff = "str_merged_${sample_id}.gff"
+    """
+    echo '#gff-version 2' >> $merged_gff;
+    echo '#pipeline-nanopore-isoforms: stringtie' >> $merged_gff;
+
+    for fn in ${gff_bundle};
+    do
+        grep -v '#' \$fn >> $merged_gff
+
+    done
+    """
+}
+
+process run_gff_compare{
+    /*
+    Compare query and reference annotations.
+    */
+
+    label 'isoforms'
+
+    input:
+       tuple val(sample_id), path(query_annotation)
+       path ref_annotation
+    output:
+        tuple val(sample_id), path('str_merged.annotated.gtf'), emit: merged_annotated
+        tuple val(sample_id), path('str_merged.stats'), emit: stats
+        tuple val(sample_id), path('str_merged.tracking'), emit: tracking
+    script:
+        """
+        echo "Doing comparison of reference annotation: ${ref_annotation} and the current annotation"
+        gffcompare -o str_merged -r ${ref_annotation} ${params.gffcompare_opts} ${query_annotation}
+        generate_tracking_summary.py --tracking str_merged.tracking --output_dir . --annotation ${ref_annotation}
+
+        if [[ ${params.plot_gffcmp_stats} == true ]];
+        then
+            plot_gffcmp_stats.py -r str_gffcmp_report.pdf -t str_merged.tracking str_merged.stats;
+        fi
+        """
+}
+
+
+process run_gffread{
+        /*
+        Write out a transctiptome file based on the gff annotations.
+        */
+        label 'isoforms'
+
+        input:
+            tuple val(sample_id), path(gff_merged), path(merged_ann_gff)
+            path reference_seq
+        output:
+            tuple val(sample_id), path('*.fas'), emit: transcriptome
+        script:
+        def str_transcriptome = "${sample_id}_str_transcriptome.fas"
+        def merged_transcriptome = "${sample_id}_merged_transcriptome.fas"
+        """
+
+        gffread -g ${reference_seq} -w ${str_transcriptome} ${gff_merged}
+        if [ -f ${merged_ann_gff} ]
+        then
+            gffread -F -g ${reference_seq} -w ${merged_transcriptome} ${merged_ann_gff}
+        else
+            touch ${merged_transcriptome}
+        fi
+        """
+}
+
+process makeReport {
+
+    label "isoforms"
+
+    input:
+        path versions
+        path params
+        tuple val(sample_id), path(seqs), path(aln_stats), path(gffcompare_tracking), path(gffcompare_stats),
+                path(cdna_class_report)
+    output:
+        tuple val(sample_id), path("wf-isoforms-*.html"), emit: report
+    script:
+        def report_name = "wf-isoforms-${sample_id}_report.html"
+        def opt_gff_track = gffcompare_tracking.name.startsWith('OPTIONAL_FILE') ? '' : "--gffcompare_tracking ${gffcompare_tracking}"
+        def opt_gff_stats = gffcompare_stats.name.startsWith('OPTIONAL_FILE') ? '' : "--gffcompare_stats ${gffcompare_stats}"
+    """
+    report.py ${report_name} --versions ${versions} ${seqs} --params params.json \
+    --alignment_stats ${aln_stats} \
+    ${opt_gff_track} \
+    ${opt_gff_stats} \
+    --pychop_report ${cdna_class_report}
     """
 }
 
@@ -79,12 +358,13 @@ process makeReport {
 // decoupling the publish from the process steps.
 process output {
     // publish inputs to output directory
-    label "pysam"
-    publishDir "${params.out_dir}", mode: 'copy', pattern: "*"
+    label "isoforms"
+    publishDir "${params.results_dir}/${sample_id}", mode: 'copy', pattern: "*"
+
     input:
-        path fname
+        tuple val(sample_id), file(fname)
     output:
-        path fname
+        file fname
     """
     echo "Writing output files"
     """
@@ -95,22 +375,131 @@ process output {
 workflow pipeline {
     take:
         reads
+        ref_genome
+        ref_annotation
     main:
-        summary = summariseReads(reads)
+
+        map_sample_ids_cls = {it -> 
+        /* Harmonize tuples
+        output:
+            tuple val(sample_id), path('*.gff')
+        When there are multiple paths, will emit:
+            [sample_id, [path, path ..]]
+        when there's a single path, this:
+            [sample_id, path]
+        This closure makes both cases:
+            [[sample_id, path][sample_id, path]].
+        */
+            if (it[1].getClass() != java.util.ArrayList){
+                // If only one path, `it` will be [sample_id, path]
+                return [it]
+            }
+            l = [];
+            for (x in it[1]){
+                l.add(tuple(it[0], x))
+            }
+            return l
+        }
+
+        summariseReads(reads)
+        sample_ids = summariseReads.out.summary.collect({it -> it[0]})
         software_versions = getVersions()
         workflow_params = getParams()
-        report = makeReport(summary, software_versions.collect(), workflow_params)
+        preprocess_reads(reads)
+//         generate_fq_stats(preprocess_reads.out.sample)  skip for now
+
+        build_minimap_index(ref_genome)
+        map_reads(build_minimap_index.out.index, ref_genome, preprocess_reads.out.full_len_reads)
+//         plot_aln_stats(map_reads.out.bam)
+        split_bam(map_reads.out.bam)
+        stringtie(split_bam.out.bundles.flatMap(map_sample_ids_cls), ref_annotation)
+        merge_gff_bundles(stringtie.out.gff_bundles.groupTuple())
+
+        use_ref_ann = !ref_annotation.name.startsWith('OPTIONAL_FILE')
+
+        if (use_ref_ann){
+            run_gff_compare(merge_gff_bundles.out.gff, ref_annotation)
+            run_gffread(merge_gff_bundles.out.gff.join(run_gff_compare.out.merged_annotated),
+                        ref_genome)
+            gff_tracking = run_gff_compare.out.tracking
+            gff_stats = run_gff_compare.out.stats
+        }else{
+            // Create dummy file paths to satisfy required path inputs of processes
+            // Add to tuple with sample_id to guide to correct sample
+            gff_tracking = sample_ids.combine(Channel.fromPath("$projectDir/data/OPTIONAL_FILE")).view()
+            gff_stats = sample_ids.combine(Channel.fromPath("$projectDir/data/OPTIONAL_FILE_1"))
+        }
+
+        makeReport(
+                    software_versions,
+                    workflow_params,
+                    summariseReads.out.summary
+                    .join(map_reads.out.stats)
+                    .join(gff_tracking)
+                    .join(gff_stats)
+                    .join(preprocess_reads.out.report)
+                    )
+        if (use_ref_ann){
+            results = preprocess_reads.out.report
+                        .concat(merge_gff_bundles.out.gff,
+                        run_gff_compare.out.stats,
+                        run_gff_compare.out.merged_annotated,
+                        run_gffread.out.transcriptome,
+                        makeReport.out.report
+                        )
+        }else{  // Write a minimal report if no reference annotation is given
+            results = preprocess_reads.out.report
+                        .concat(merge_gff_bundles.out.gff,
+                                makeReport.out.report
+                                )
+            }
     emit:
-        summary.concat(report)
+        results
+        telemetry = workflow_params
 }
 
 // entrypoint workflow
 WorkflowMain.initialise(workflow, params, log)
 workflow {
 
-    samples = fastq_ingress(
-        params.fastq, params.out_dir, params.samples, params.sanitize_fastq)
+    start_ping()
+    params.results_dir = "${params.out_dir}/output"
 
-    results = pipeline(samples)
-    output(results)
+    fastq = file(params.fastq, type: "file")
+
+     if (!fastq.exists()) {
+        println("--fastq: File doesn't exist, check path.")
+        exit 1
+    }
+
+    if (params.ref_genome){
+        ref_genome = file(params.ref_genome, type: "file")
+        if (!ref_genome.exists()) {
+            println("--reference: File doesn't exist, check path.")
+            exit 1
+        }
+    }
+
+    ref_annotation = null
+    if (params.ref_annotation){
+        ref_annotation = file(params.ref_annotation, type: "file")
+        if (!ref_annotation.exists()) {
+            println("--annotation: File doesn't exist, check path.")
+            exit 1
+        }
+    }else{
+        ref_annotation = file("$projectDir/data/OPTIONAL_FILE")
+    }
+
+    reads = fastq_ingress(
+        params.fastq, params.out_dir, params.sample, params.sample_sheet, params.sanitize_fastq
+    )
+
+    pipeline(reads, ref_genome, ref_annotation)
+
+    output(
+        pipeline.out.results
+        )
+
+    end_ping(pipeline.out.telemetry)
 }

nextflow.config

@@ -13,10 +13,17 @@
 params {
     help = false
     fastq = null
-    out_dir = "output"
-    samples = null
+    ref_genome = null
+    ref_annotation = null
+    // Process cDNA reads using pychopper, turn off for direct RNA:
+    use_pychopper = true
+    threads = 4
+
+    out_dir = null
+    sample = null
+    sample_sheet = null
     sanitize_fastq = false
-    wfversion = "v0.0.7"
+    wfversion = "v0.0.1"
     aws_image_prefix = null
     aws_queue = null
     report_name = "report"
@@ -25,13 +32,48 @@ params {
     validate_params = true
     show_hidden_params = false
     schema_ignore_params = 'show_hidden_params,validate_params,monochrome_logs,aws_queue,aws_image_prefix,wfversion'
+
+
+    // Options passed to pychopper:
+    pychopper_opts = ""
+
+    // Extra option passed to minimap2 when generating index
+    minimap_index_opts = "-k14"
+
+    // Extra options passed to minimap2
+    minimap2_opts = "-uf"
+    // Add this for SIRV data:
+    // "--splice-flank=no"
+
+    // Minmum mapping quality
+    minimum_mapping_quality = 40
+
+    // Internal priming filter context size:
+    poly_context = 24
+
+    // Maximum allowed poly(A) length in the genome near the 3' end of mapping: 
+    max_poly_run = 8
+
+    // Minimium number of reads in BAM bundles:
+    bundle_min_reads = 50000
+
+    // Options passed to stringtie:
+    stringtie_opts = " --conservative "
+
+    // Options passed to gffcompare:
+    gffcompare_opts = " -R "
+
+    // Plot gffcompare results:
+    plot_gffcmp_stats = true
+
+    disable_ping = false
 }
 
 manifest {
-    name            = 'epi2me-labs/wf-template'
+    name            = 'epi2me-labs/wf-isoforms'
     author          = 'Oxford Nanopore Technologies'
-    homePage        = 'https://github.com/epi2me-labs/wf-template'
-    description     = 'Template workflow'
+    homePage        = 'https://github.com/epi2me-labs/wf-isoforms'
+    description     = 'RNA/cDNA isoform analysis workflow'
     mainScript      = 'main.nf'
     nextflowVersion = '>=20.10.0'
     //version       = 'v0.0.7'  // TODO: do switch to this?
@@ -44,6 +86,16 @@ executor {
     }
 }
 
+// used by default for "standard" (docker) and singularity profiles,
+// other profiles may override.
+process {
+    withLabel:isoforms {
+        container = "ontresearch/wf-isoforms:${params.wfversion}"
+    }
+    shell = ['/bin/bash', '-euo', 'pipefail']
+}
+
+
 profiles {
     // the "standard" profile is used implicitely by nextflow
     // if no other profile is given on the CLI
@@ -54,22 +106,22 @@ profiles {
             //    also adds host user to the within-container group
             runOptions = "--user \$(id -u):\$(id -g) --group-add 100"
         }
-        process {
-            withLabel:pysam {
-                container = "ontresearch/wf-template:${params.wfversion}"
-            }
-            shell = ['/bin/bash', '-euo', 'pipefail']
+    }
+
+    // using singularity instead of docker
+    singularity {
+        singularity {
+            enabled = true
+            autoMounts = true
         }
     }
-    
-    // profile using conda environments rather than docker
-    // containers
+
+
+    // profile using conda environments
     conda {
-        docker {
-            enabled = false
-        }
+        docker.enabled = false
         process {
-            withLabel:pysam {
+            withLabel:isoforms {
                 conda = "${projectDir}/environment.yaml"
             }
             shell = ['/bin/bash', '-euo', 'pipefail']
@@ -80,19 +132,24 @@ profiles {
         }
     }
 
+    // Using AWS batch.
+    // May need to set aws.region and aws.batch.cliPath
     awsbatch {
         process {
             executor = 'awsbatch'
             queue = "${params.aws_queue}"
             memory = '8G'
-            withLabel:pysam {
-                container = "${params.aws_image_prefix}-wf-template:${params.wfversion}"
+            withLabel:isoforms {
+                container = "${params.aws_image_prefix}-wf-isoforms:${params.wfversion}"
             }
             shell = ['/bin/bash', '-euo', 'pipefail']
         }
     }
-    aws.region = 'eu-west-1'
-    aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'
+
+    // local profile for simplified development testing
+    local {
+        process.executor = 'local'
+    }
 }
 
 

nextflow_schema.json

@@ -20,18 +20,95 @@
                     "type": "string",
                     "description": "Directory containing fastq input files. May contain fastq files directly or directories name barcodeXX relating to independent samples."
                 },
-                "samples": {
+                "sample": {
                     "type": "string",
-                    "description": "CSV file with columns named `barcode` and `sample_name`. (or simply a sample name for non-multiplexed data)"
+                    "description": "A sample name for non-multiplexed data. Permissible if passing a file or directory of .fastq(.gz)."
+                },
+                "sample_sheet": {
+                    "type": "string",
+                    "description": "CSV file with columns named `barcode`, `sample_name` and `type`. Permissible if passing a directory containing barcodeXX sub-directories."
                 },
                 "sanitize_fastq": {
                     "type": "boolean",
                     "description": "Use additional heuristics to identify barcodes from file paths.",
                     "help_text": "Enabling this option will group together files into samples by the presence of strings of the form `barcodeXXX` present in filenames, rather than simply files grouped into directories (as output by MinKNOW and the Guppy basecaller)."
+                },
+                "plot_gffcmp_stats": {
+                    "type": "boolean",
+                    "description": "Create a pdf of plots from showing gffcompare results"
+                },
+                "gffcompare_opts": {
+                    "type": "string",
+                    "description": "Extra options for gffcompare -r",
+                    "default": " -R "
+                },
+                "ref_genome": {
+                    "type": "string",
+                    "description": "Path to reference genome sequence [.fa/.fq/.fa.gz/fq.gz]"
+                },
+                "ref_annotation": {
+                    "type": "string",
+                    "description": "A reference annotation of gff format"
+                },
+                "use_pychopper": {
+                    "type": "boolean",
+                    "description": "Use pychopper to preprcess reads",
+                    "default": true
+                },
+                "pychopper_opts": {
+                    "type": "string",
+                    "description": "Extra pychopper opts"
+                },
+                "threads": {
+                    "type": "integer",
+                    "default": 8
+                },
+                "minimap_index_opts": {
+                    "type": "string",
+                    "description": "minimap2 extra indexing options.",
+                    "default": "-k14"
+                },
+                "minimap2_opts": {
+                    "type": "string",
+                    "description": "minimap2 extra mapping options.",
+                    "default": "-uf"
+                },
+                "minimum_mapping_quality": {
+                    "type": "integer",
+                    "description": "filter aligned reads by MAPQ quality.",
+                    "default": 40
+                },
+                "poly_context": {
+                    "type": "integer",
+                    "description": "Region size at end of reads to apply poly(A) filter.",
+                    "default": 24
+                },
+                "max_poly_run": {
+                    "type": "integer",
+                    "description": "Max poly(A) region allowed with poly_context-sized end regions.",
+                    "default": 8
+                },
+                "bundle_min_reads": {
+                    "type": "integer",
+                    "description": "Minimum size of bam bundle for parallel processing."
+                },
+                "use_guide_annotation": {
+                    "type": "boolean",
+                    "description": "Use reference annotation in stringtie transcript assembly.",
+                    "default": "true"
+                },
+                "stringtie_opts": {
+                    "type": "string",
+                    "description": "Extra options for stringtie transcript assembly.",
+                    "default": " --conservative "
+                },
+                "disable_ping": {
+                "type": "boolean"
                 }
             },
             "required": [
-                "fastq"
+                "fastq",
+                "ref_genome"
             ]
         },
         "meta_data": {
@@ -99,4 +176,4 @@
             "type": "boolean"
         }
     }
-}
+}

run_evaluation_dmel.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Usage: ./run_evaluation_dmel.sh pathto/outputdir
+
+# See the isONcorrect paper https://www.nature.com/articles/s41467-020-20340-8 where this dataset is described
+
+OUTDIR=$1;
+
+FASTQ_URL="http://ftp.sra.ebi.ac.uk/vol1/fastq/ERR358/005/ERR3588905/ERR3588905_1.fastq.gz"
+REF_URL="http://ftp.ensembl.org/pub/release-99/fasta/drosophila_melanogaster/dna/Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa.gz"
+GFF_URL="http://ftp.ensembl.org/pub/release-99/gff3/drosophila_melanogaster/Drosophila_melanogaster.BDGP6.28.99.gff3.gz"
+
+RESULTS_DIR="$OUTDIR/results"
+DATA_DIR="$OUTDIR/data"
+READS_DIR="$DATA_DIR/reads"
+FASTQ="$READS_DIR/ERR3588905_1.fastq"
+REF="$DATA_DIR/Drosophila_melanogaster.BDGP6.28.dna.toplevel.fa"
+GFF="$DATA_DIR/Drosophila_melanogaster.BDGP6.28.99.gff3"
+
+echo $READS_DIR;
+
+rm -fr $OUT_DIR/results
+mkdir -p $OUTDIR/data
+
+if [ ! -f $REF ];
+then (cd $DATA_DIR; curl -L -C - -O $REF_URL); gzip -d ${REF}.gz
+fi
+
+if [ ! -f $GFF ]
+then
+	(cd $DATA_DIR; curl -L -C - -O $GFF_URL); gzip -d ${GFF}.gz
+fi
+
+if [ ! -f $FASTQ ];
+then (cd $READS_DIR; curl -L -C - -O $FASTQ_URL); gzip -d ${FASTQ}.gz
+fi
+
+nextflow run ../wf-isoforms --fastq $READS_DIR  \
+--reference_genome $REF --annotation $GFF -profile conda --out_dir $OUTDIR \
+-w $OUTDIR/workspace -resume

test_data/SIRV_150601a.fasta.fai

@@ -0,0 +1,7 @@
+SIRV1	12643	7	80	81
+SIRV2	6911	12816	80	81
+SIRV3	10943	19821	80	81
+SIRV4	16122	30908	80	81
+SIRV5	14606	47239	80	81
+SIRV6	12837	62035	80	81
+SIRV7	148957	75040	80	81

test_data/SIRV_isofroms.gtf

@@ -0,0 +1,354 @@
+SIRV1	LexogenSIRVData	exon	1001	1484	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_0"; 
+SIRV1	LexogenSIRVData	exon	6338	6473	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_1"; 
+SIRV1	LexogenSIRVData	exon	6561	6813	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_2"; 
+SIRV1	LexogenSIRVData	exon	7553	7814	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_3"; 
+SIRV1	LexogenSIRVData	exon	10283	10366	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_4"; 
+SIRV1	LexogenSIRVData	exon	10445	10786	.	-	0	gene_id "SIRV1"; transcript_id "SIRV101"; exon_assignment "SIRV101_5"; 
+SIRV1	LexogenSIRVData	exon	1007	1484	.	-	0	gene_id "SIRV1"; transcript_id "SIRV102"; exon_assignment "SIRV102_0"; 
+SIRV1	LexogenSIRVData	exon	6338	6813	.	-	0	gene_id "SIRV1"; transcript_id "SIRV102"; exon_assignment "SIRV102_1"; 
+SIRV1	LexogenSIRVData	exon	7553	7814	.	-	0	gene_id "SIRV1"; transcript_id "SIRV102"; exon_assignment "SIRV102_2"; 
+SIRV1	LexogenSIRVData	exon	10283	10366	.	-	0	gene_id "SIRV1"; transcript_id "SIRV102"; exon_assignment "SIRV102_3"; 
+SIRV1	LexogenSIRVData	exon	1001	1484	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_0"; 
+SIRV1	LexogenSIRVData	exon	6338	6473	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_1"; 
+SIRV1	LexogenSIRVData	exon	6561	6813	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_2"; 
+SIRV1	LexogenSIRVData	exon	7553	7814	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_3"; 
+SIRV1	LexogenSIRVData	exon	10283	10366	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_4"; 
+SIRV1	LexogenSIRVData	exon	10648	10791	.	-	0	gene_id "SIRV1"; transcript_id "SIRV103"; exon_assignment "SIRV103_5"; 
+SIRV1	LexogenSIRVData	exon	6450	6473	.	-	0	gene_id "SIRV1"; transcript_id "SIRV105"; exon_assignment "SIRV105_0"; 
+SIRV1	LexogenSIRVData	exon	6561	6813	.	-	0	gene_id "SIRV1"; transcript_id "SIRV105"; exon_assignment "SIRV105_1"; 
+SIRV1	LexogenSIRVData	exon	7553	7814	.	-	0	gene_id "SIRV1"; transcript_id "SIRV105"; exon_assignment "SIRV105_2"; 
+SIRV1	LexogenSIRVData	exon	10283	10366	.	-	0	gene_id "SIRV1"; transcript_id "SIRV105"; exon_assignment "SIRV105_3"; 
+SIRV1	LexogenSIRVData	exon	10594	10640	.	-	0	gene_id "SIRV1"; transcript_id "SIRV105"; exon_assignment "SIRV105_4"; 
+SIRV1	LexogenSIRVData	exon	1001	1484	.	-	0	gene_id "SIRV1"; transcript_id "SIRV106"; exon_assignment "SIRV106_0"; 
+SIRV1	LexogenSIRVData	exon	7553	7808	.	-	0	gene_id "SIRV1"; transcript_id "SIRV106"; exon_assignment "SIRV106_1"; 
+SIRV1	LexogenSIRVData	exon	10554	10786	.	-	0	gene_id "SIRV1"; transcript_id "SIRV106"; exon_assignment "SIRV106_2"; 
+SIRV1	LexogenSIRVData	exon	10648	10791	.	-	0	gene_id "SIRV1"; transcript_id "SIRV107"; exon_assignment "SIRV107_0"; 
+SIRV1	LexogenSIRVData	exon	10883	11242	.	-	0	gene_id "SIRV1"; transcript_id "SIRV107"; exon_assignment "SIRV107_1"; 
+SIRV1	LexogenSIRVData	exon	11404	11643	.	-	0	gene_id "SIRV1"; transcript_id "SIRV107"; exon_assignment "SIRV107_2"; 
+SIRV1	LexogenSIRVData	exon	10712	10791	.	+	0	gene_id "SIRV1"; transcript_id "SIRV109"; exon_assignment "SIRV109_0"; 
+SIRV1	LexogenSIRVData	exon	10883	11057	.	+	0	gene_id "SIRV1"; transcript_id "SIRV109"; exon_assignment "SIRV109_1"; 
+SIRV1	LexogenSIRVData	exon	11435	11643	.	+	0	gene_id "SIRV1"; transcript_id "SIRV109"; exon_assignment "SIRV109_2"; 
+SIRV2	LexogenSIRVData	exon	1001	1661	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_0"; 
+SIRV2	LexogenSIRVData	exon	1742	1853	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_1"; 
+SIRV2	LexogenSIRVData	exon	1974	2064	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_2"; 
+SIRV2	LexogenSIRVData	exon	2675	2802	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_3"; 
+SIRV2	LexogenSIRVData	exon	2882	3010	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_4"; 
+SIRV2	LexogenSIRVData	exon	3106	3374	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_5"; 
+SIRV2	LexogenSIRVData	exon	3666	3825	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_6"; 
+SIRV2	LexogenSIRVData	exon	3967	4094	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_7"; 
+SIRV2	LexogenSIRVData	exon	4339	4479	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_8"; 
+SIRV2	LexogenSIRVData	exon	4688	4800	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_9"; 
+SIRV2	LexogenSIRVData	exon	5789	5907	.	-	0	gene_id "SIRV2"; transcript_id "SIRV201"; exon_assignment "SIRV201_10"; 
+SIRV2	LexogenSIRVData	exon	1036	1661	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_0"; 
+SIRV2	LexogenSIRVData	exon	1742	1853	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_1"; 
+SIRV2	LexogenSIRVData	exon	1974	2064	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_2"; 
+SIRV2	LexogenSIRVData	exon	2675	2802	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_3"; 
+SIRV2	LexogenSIRVData	exon	2882	3010	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_4"; 
+SIRV2	LexogenSIRVData	exon	3106	3325	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_5"; 
+SIRV2	LexogenSIRVData	exon	3666	3825	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_6"; 
+SIRV2	LexogenSIRVData	exon	3967	4094	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_7"; 
+SIRV2	LexogenSIRVData	exon	4339	4479	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_8"; 
+SIRV2	LexogenSIRVData	exon	4688	4800	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_9"; 
+SIRV2	LexogenSIRVData	exon	5789	5911	.	-	0	gene_id "SIRV2"; transcript_id "SIRV202"; exon_assignment "SIRV202_10"; 
+SIRV2	LexogenSIRVData	exon	3666	3825	.	-	0	gene_id "SIRV2"; transcript_id "SIRV203"; exon_assignment "SIRV203_0"; 
+SIRV2	LexogenSIRVData	exon	3967	4094	.	-	0	gene_id "SIRV2"; transcript_id "SIRV203"; exon_assignment "SIRV203_1"; 
+SIRV2	LexogenSIRVData	exon	4339	4479	.	-	0	gene_id "SIRV2"; transcript_id "SIRV203"; exon_assignment "SIRV203_2"; 
+SIRV2	LexogenSIRVData	exon	4688	4800	.	-	0	gene_id "SIRV2"; transcript_id "SIRV203"; exon_assignment "SIRV203_3"; 
+SIRV2	LexogenSIRVData	exon	5752	5895	.	-	0	gene_id "SIRV2"; transcript_id "SIRV203"; exon_assignment "SIRV203_4"; 
+SIRV2	LexogenSIRVData	exon	3644	3825	.	-	0	gene_id "SIRV2"; transcript_id "SIRV204"; exon_assignment "SIRV204_0"; 
+SIRV2	LexogenSIRVData	exon	3967	4479	.	-	0	gene_id "SIRV2"; transcript_id "SIRV204"; exon_assignment "SIRV204_1"; 
+SIRV2	LexogenSIRVData	exon	4688	4732	.	-	0	gene_id "SIRV2"; transcript_id "SIRV204"; exon_assignment "SIRV204_2"; 
+SIRV2	LexogenSIRVData	exon	1109	1631	.	+	0	gene_id "SIRV2"; transcript_id "SIRV205"; exon_assignment "SIRV205_0"; 
+SIRV2	LexogenSIRVData	exon	4034	4457	.	+	0	gene_id "SIRV2"; transcript_id "SIRV206"; exon_assignment "SIRV206_0"; 
+SIRV3	LexogenSIRVData	exon	1945	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV301"; exon_assignment "SIRV301_0"; 
+SIRV3	LexogenSIRVData	exon	4569	4779	.	+	0	gene_id "SIRV3"; transcript_id "SIRV301"; exon_assignment "SIRV301_1"; 
+SIRV3	LexogenSIRVData	exon	6058	7988	.	+	0	gene_id "SIRV3"; transcript_id "SIRV301"; exon_assignment "SIRV301_2"; 
+SIRV3	LexogenSIRVData	exon	8128	8207	.	+	0	gene_id "SIRV3"; transcript_id "SIRV301"; exon_assignment "SIRV301_3"; 
+SIRV3	LexogenSIRVData	exon	8756	8939	.	+	0	gene_id "SIRV3"; transcript_id "SIRV301"; exon_assignment "SIRV301_4"; 
+SIRV3	LexogenSIRVData	exon	1964	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV302"; exon_assignment "SIRV302_0"; 
+SIRV3	LexogenSIRVData	exon	6058	7822	.	+	0	gene_id "SIRV3"; transcript_id "SIRV302"; exon_assignment "SIRV302_1"; 
+SIRV3	LexogenSIRVData	exon	1964	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV303"; exon_assignment "SIRV303_0"; 
+SIRV3	LexogenSIRVData	exon	4569	4779	.	+	0	gene_id "SIRV3"; transcript_id "SIRV303"; exon_assignment "SIRV303_1"; 
+SIRV3	LexogenSIRVData	exon	6058	7822	.	+	0	gene_id "SIRV3"; transcript_id "SIRV303"; exon_assignment "SIRV303_2"; 
+SIRV3	LexogenSIRVData	exon	1964	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_0"; 
+SIRV3	LexogenSIRVData	exon	4004	4080	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_1"; 
+SIRV3	LexogenSIRVData	exon	4569	4779	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_2"; 
+SIRV3	LexogenSIRVData	exon	6058	6333	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_3"; 
+SIRV3	LexogenSIRVData	exon	7271	7366	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_4"; 
+SIRV3	LexogenSIRVData	exon	7873	7988	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_5"; 
+SIRV3	LexogenSIRVData	exon	8125	8207	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_6"; 
+SIRV3	LexogenSIRVData	exon	8756	8937	.	+	0	gene_id "SIRV3"; transcript_id "SIRV304"; exon_assignment "SIRV304_7"; 
+SIRV3	LexogenSIRVData	exon	4004	4080	.	+	0	gene_id "SIRV3"; transcript_id "SIRV305"; exon_assignment "SIRV305_0"; 
+SIRV3	LexogenSIRVData	exon	4569	4779	.	+	0	gene_id "SIRV3"; transcript_id "SIRV305"; exon_assignment "SIRV305_1"; 
+SIRV3	LexogenSIRVData	exon	6571	6718	.	+	0	gene_id "SIRV3"; transcript_id "SIRV305"; exon_assignment "SIRV305_2"; 
+SIRV3	LexogenSIRVData	exon	1945	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV306"; exon_assignment "SIRV306_0"; 
+SIRV3	LexogenSIRVData	exon	4004	4080	.	+	0	gene_id "SIRV3"; transcript_id "SIRV306"; exon_assignment "SIRV306_1"; 
+SIRV3	LexogenSIRVData	exon	6058	8292	.	+	0	gene_id "SIRV3"; transcript_id "SIRV306"; exon_assignment "SIRV306_2"; 
+SIRV3	LexogenSIRVData	exon	1964	2005	.	+	0	gene_id "SIRV3"; transcript_id "SIRV307"; exon_assignment "SIRV307_0"; 
+SIRV3	LexogenSIRVData	exon	4004	4080	.	+	0	gene_id "SIRV3"; transcript_id "SIRV307"; exon_assignment "SIRV307_1"; 
+SIRV3	LexogenSIRVData	exon	4575	4774	.	+	0	gene_id "SIRV3"; transcript_id "SIRV307"; exon_assignment "SIRV307_2"; 
+SIRV3	LexogenSIRVData	exon	6058	6333	.	+	0	gene_id "SIRV3"; transcript_id "SIRV307"; exon_assignment "SIRV307_3"; 
+SIRV3	LexogenSIRVData	exon	8756	8939	.	+	0	gene_id "SIRV3"; transcript_id "SIRV307"; exon_assignment "SIRV307_4"; 
+SIRV3	LexogenSIRVData	exon	1001	1167	.	-	0	gene_id "SIRV3"; transcript_id "SIRV308"; exon_assignment "SIRV308_0"; 
+SIRV3	LexogenSIRVData	exon	1533	1764	.	-	0	gene_id "SIRV3"; transcript_id "SIRV308"; exon_assignment "SIRV308_1"; 
+SIRV3	LexogenSIRVData	exon	1903	1982	.	-	0	gene_id "SIRV3"; transcript_id "SIRV308"; exon_assignment "SIRV308_2"; 
+SIRV3	LexogenSIRVData	exon	8798	8975	.	-	0	gene_id "SIRV3"; transcript_id "SIRV309"; exon_assignment "SIRV309_0"; 
+SIRV3	LexogenSIRVData	exon	9190	9298	.	-	0	gene_id "SIRV3"; transcript_id "SIRV309"; exon_assignment "SIRV309_1"; 
+SIRV3	LexogenSIRVData	exon	9435	9943	.	-	0	gene_id "SIRV3"; transcript_id "SIRV309"; exon_assignment "SIRV309_2"; 
+SIRV3	LexogenSIRVData	exon	8760	8966	.	-	0	gene_id "SIRV3"; transcript_id "SIRV310"; exon_assignment "SIRV310_0"; 
+SIRV3	LexogenSIRVData	exon	9190	9324	.	-	0	gene_id "SIRV3"; transcript_id "SIRV310"; exon_assignment "SIRV310_1"; 
+SIRV3	LexogenSIRVData	exon	9668	9914	.	-	0	gene_id "SIRV3"; transcript_id "SIRV310"; exon_assignment "SIRV310_2"; 
+SIRV3	LexogenSIRVData	exon	4602	4762	.	-	0	gene_id "SIRV3"; transcript_id "SIRV311"; exon_assignment "SIRV311_0"; 
+SIRV4	LexogenSIRVData	exon	8323	8372	.	-	0	gene_id "SIRV4"; transcript_id "SIRV403"; exon_assignment "SIRV403_0"; 
+SIRV4	LexogenSIRVData	exon	8630	8990	.	-	0	gene_id "SIRV4"; transcript_id "SIRV403"; exon_assignment "SIRV403_1"; 
+SIRV4	LexogenSIRVData	exon	13673	13828	.	-	0	gene_id "SIRV4"; transcript_id "SIRV403"; exon_assignment "SIRV403_2"; 
+SIRV4	LexogenSIRVData	exon	15020	15122	.	-	0	gene_id "SIRV4"; transcript_id "SIRV403"; exon_assignment "SIRV403_3"; 
+SIRV4	LexogenSIRVData	exon	8323	8372	.	-	0	gene_id "SIRV4"; transcript_id "SIRV404"; exon_assignment "SIRV404_0"; 
+SIRV4	LexogenSIRVData	exon	8630	8990	.	-	0	gene_id "SIRV4"; transcript_id "SIRV404"; exon_assignment "SIRV404_1"; 
+SIRV4	LexogenSIRVData	exon	13673	13822	.	-	0	gene_id "SIRV4"; transcript_id "SIRV404"; exon_assignment "SIRV404_2"; 
+SIRV4	LexogenSIRVData	exon	14593	14623	.	-	0	gene_id "SIRV4"; transcript_id "SIRV404"; exon_assignment "SIRV404_3"; 
+SIRV4	LexogenSIRVData	exon	8630	8990	.	-	0	gene_id "SIRV4"; transcript_id "SIRV405"; exon_assignment "SIRV405_0"; 
+SIRV4	LexogenSIRVData	exon	13673	13937	.	-	0	gene_id "SIRV4"; transcript_id "SIRV405"; exon_assignment "SIRV405_1"; 
+SIRV4	LexogenSIRVData	exon	3638	4103	.	-	0	gene_id "SIRV4"; transcript_id "SIRV406"; exon_assignment "SIRV406_0"; 
+SIRV4	LexogenSIRVData	exon	5008	5158	.	-	0	gene_id "SIRV4"; transcript_id "SIRV406"; exon_assignment "SIRV406_1"; 
+SIRV4	LexogenSIRVData	exon	8324	8372	.	-	0	gene_id "SIRV4"; transcript_id "SIRV408"; exon_assignment "SIRV408_0"; 
+SIRV4	LexogenSIRVData	exon	8630	8747	.	-	0	gene_id "SIRV4"; transcript_id "SIRV408"; exon_assignment "SIRV408_1"; 
+SIRV4	LexogenSIRVData	exon	8847	8990	.	-	0	gene_id "SIRV4"; transcript_id "SIRV408"; exon_assignment "SIRV408_2"; 
+SIRV4	LexogenSIRVData	exon	13673	13828	.	-	0	gene_id "SIRV4"; transcript_id "SIRV408"; exon_assignment "SIRV408_3"; 
+SIRV4	LexogenSIRVData	exon	15020	15122	.	-	0	gene_id "SIRV4"; transcript_id "SIRV408"; exon_assignment "SIRV408_4"; 
+SIRV4	LexogenSIRVData	exon	1001	1346	.	+	0	gene_id "SIRV4"; transcript_id "SIRV409"; exon_assignment "SIRV409_0"; 
+SIRV4	LexogenSIRVData	exon	1679	1885	.	+	0	gene_id "SIRV4"; transcript_id "SIRV409"; exon_assignment "SIRV409_1"; 
+SIRV4	LexogenSIRVData	exon	2390	3403	.	+	0	gene_id "SIRV4"; transcript_id "SIRV409"; exon_assignment "SIRV409_2"; 
+SIRV4	LexogenSIRVData	exon	1456	1885	.	+	0	gene_id "SIRV4"; transcript_id "SIRV410"; exon_assignment "SIRV410_0"; 
+SIRV4	LexogenSIRVData	exon	2252	2771	.	+	0	gene_id "SIRV4"; transcript_id "SIRV410"; exon_assignment "SIRV410_1"; 
+SIRV5	LexogenSIRVData	exon	1057	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2315	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_2"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_3"; 
+SIRV5	LexogenSIRVData	exon	3484	3643	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_4"; 
+SIRV5	LexogenSIRVData	exon	5381	5450	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_5"; 
+SIRV5	LexogenSIRVData	exon	5544	5626	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_6"; 
+SIRV5	LexogenSIRVData	exon	6112	6169	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_7"; 
+SIRV5	LexogenSIRVData	exon	6328	6452	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_8"; 
+SIRV5	LexogenSIRVData	exon	6659	6722	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_9"; 
+SIRV5	LexogenSIRVData	exon	6827	6957	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_10"; 
+SIRV5	LexogenSIRVData	exon	7145	7307	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_11"; 
+SIRV5	LexogenSIRVData	exon	7682	7762	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_12"; 
+SIRV5	LexogenSIRVData	exon	7871	8016	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_13"; 
+SIRV5	LexogenSIRVData	exon	8278	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_14"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_15"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV501"; exon_assignment "SIRV501_16"; 
+SIRV5	LexogenSIRVData	exon	1020	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2156	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_2"; 
+SIRV5	LexogenSIRVData	exon	2271	2488	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_3"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_4"; 
+SIRV5	LexogenSIRVData	exon	3484	3643	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_5"; 
+SIRV5	LexogenSIRVData	exon	5381	5450	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_6"; 
+SIRV5	LexogenSIRVData	exon	5544	5626	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_7"; 
+SIRV5	LexogenSIRVData	exon	6112	6169	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_8"; 
+SIRV5	LexogenSIRVData	exon	6328	6452	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_9"; 
+SIRV5	LexogenSIRVData	exon	6659	6722	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_10"; 
+SIRV5	LexogenSIRVData	exon	6827	6957	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_11"; 
+SIRV5	LexogenSIRVData	exon	7145	7307	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_12"; 
+SIRV5	LexogenSIRVData	exon	7682	7762	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_13"; 
+SIRV5	LexogenSIRVData	exon	7871	8016	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_14"; 
+SIRV5	LexogenSIRVData	exon	8278	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_15"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_16"; 
+SIRV5	LexogenSIRVData	exon	10859	10989	.	+	0	gene_id "SIRV5"; transcript_id "SIRV502"; exon_assignment "SIRV502_17"; 
+SIRV5	LexogenSIRVData	exon	8202	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV503"; exon_assignment "SIRV503_0"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV503"; exon_assignment "SIRV503_1"; 
+SIRV5	LexogenSIRVData	exon	11134	11142	.	+	0	gene_id "SIRV5"; transcript_id "SIRV503"; exon_assignment "SIRV503_2"; 
+SIRV5	LexogenSIRVData	exon	11134	13606	.	+	0	gene_id "SIRV5"; transcript_id "SIRV504"; exon_assignment "SIRV504_0"; 
+SIRV5	LexogenSIRVData	exon	1001	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2156	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_2"; 
+SIRV5	LexogenSIRVData	exon	2271	2315	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_3"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_4"; 
+SIRV5	LexogenSIRVData	exon	3484	3643	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_5"; 
+SIRV5	LexogenSIRVData	exon	5381	5450	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_6"; 
+SIRV5	LexogenSIRVData	exon	5544	5626	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_7"; 
+SIRV5	LexogenSIRVData	exon	6112	6169	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_8"; 
+SIRV5	LexogenSIRVData	exon	6328	6452	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_9"; 
+SIRV5	LexogenSIRVData	exon	6827	6957	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_10"; 
+SIRV5	LexogenSIRVData	exon	7145	7307	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_11"; 
+SIRV5	LexogenSIRVData	exon	7682	7762	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_12"; 
+SIRV5	LexogenSIRVData	exon	7871	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_13"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_14"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV505"; exon_assignment "SIRV505_15"; 
+SIRV5	LexogenSIRVData	exon	1009	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV506"; exon_assignment "SIRV506_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2398	.	+	0	gene_id "SIRV5"; transcript_id "SIRV506"; exon_assignment "SIRV506_1"; 
+SIRV5	LexogenSIRVData	exon	1028	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_0"; 
+SIRV5	LexogenSIRVData	exon	1926	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2156	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_2"; 
+SIRV5	LexogenSIRVData	exon	2271	2315	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_3"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_4"; 
+SIRV5	LexogenSIRVData	exon	3484	3598	.	+	0	gene_id "SIRV5"; transcript_id "SIRV507"; exon_assignment "SIRV507_5"; 
+SIRV5	LexogenSIRVData	exon	1009	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2156	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_2"; 
+SIRV5	LexogenSIRVData	exon	2271	2315	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_3"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_4"; 
+SIRV5	LexogenSIRVData	exon	3484	3643	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_5"; 
+SIRV5	LexogenSIRVData	exon	5381	5450	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_6"; 
+SIRV5	LexogenSIRVData	exon	5544	5626	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_7"; 
+SIRV5	LexogenSIRVData	exon	6112	6169	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_8"; 
+SIRV5	LexogenSIRVData	exon	6328	6452	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_9"; 
+SIRV5	LexogenSIRVData	exon	6659	6722	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_10"; 
+SIRV5	LexogenSIRVData	exon	6827	6957	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_11"; 
+SIRV5	LexogenSIRVData	exon	7145	7307	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_12"; 
+SIRV5	LexogenSIRVData	exon	7682	7762	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_13"; 
+SIRV5	LexogenSIRVData	exon	7871	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_14"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_15"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV508"; exon_assignment "SIRV508_16"; 
+SIRV5	LexogenSIRVData	exon	8316	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV509"; exon_assignment "SIRV509_0"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV509"; exon_assignment "SIRV509_1"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV509"; exon_assignment "SIRV509_2"; 
+SIRV5	LexogenSIRVData	exon	11312	11866	.	+	0	gene_id "SIRV5"; transcript_id "SIRV509"; exon_assignment "SIRV509_3"; 
+SIRV5	LexogenSIRVData	exon	1029	1149	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2033	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_1"; 
+SIRV5	LexogenSIRVData	exon	2120	2156	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_2"; 
+SIRV5	LexogenSIRVData	exon	2271	2315	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_3"; 
+SIRV5	LexogenSIRVData	exon	3299	3404	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_4"; 
+SIRV5	LexogenSIRVData	exon	3484	3643	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_5"; 
+SIRV5	LexogenSIRVData	exon	5381	5450	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_6"; 
+SIRV5	LexogenSIRVData	exon	5544	5626	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_7"; 
+SIRV5	LexogenSIRVData	exon	6112	6169	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_8"; 
+SIRV5	LexogenSIRVData	exon	6328	6452	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_9"; 
+SIRV5	LexogenSIRVData	exon	6827	6957	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_10"; 
+SIRV5	LexogenSIRVData	exon	7145	7307	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_11"; 
+SIRV5	LexogenSIRVData	exon	7682	7762	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_12"; 
+SIRV5	LexogenSIRVData	exon	7871	8016	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_13"; 
+SIRV5	LexogenSIRVData	exon	8278	8381	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_14"; 
+SIRV5	LexogenSIRVData	exon	8455	8585	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_15"; 
+SIRV5	LexogenSIRVData	exon	10859	10991	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_16"; 
+SIRV5	LexogenSIRVData	exon	11134	11867	.	+	0	gene_id "SIRV5"; transcript_id "SIRV510"; exon_assignment "SIRV510_17"; 
+SIRV5	LexogenSIRVData	exon	1009	1143	.	+	0	gene_id "SIRV5"; transcript_id "SIRV511"; exon_assignment "SIRV511_0"; 
+SIRV5	LexogenSIRVData	exon	1988	2398	.	+	0	gene_id "SIRV5"; transcript_id "SIRV511"; exon_assignment "SIRV511_1"; 
+SIRV5	LexogenSIRVData	exon	2178	2406	.	-	0	gene_id "SIRV5"; transcript_id "SIRV512"; exon_assignment "SIRV512_0"; 
+SIRV6	LexogenSIRVData	exon	1001	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_2"; 
+SIRV6	LexogenSIRVData	exon	2471	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_3"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_4"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_5"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_6"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_7"; 
+SIRV6	LexogenSIRVData	exon	11206	11826	.	+	0	gene_id "SIRV6"; transcript_id "SIRV601"; exon_assignment "SIRV601_8"; 
+SIRV6	LexogenSIRVData	exon	1125	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_2"; 
+SIRV6	LexogenSIRVData	exon	2781	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_3"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_4"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_5"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_6"; 
+SIRV6	LexogenSIRVData	exon	11206	11279	.	+	0	gene_id "SIRV6"; transcript_id "SIRV602"; exon_assignment "SIRV602_7"; 
+SIRV6	LexogenSIRVData	exon	9000	10968	.	+	0	gene_id "SIRV6"; transcript_id "SIRV603"; exon_assignment "SIRV603_0"; 
+SIRV6	LexogenSIRVData	exon	1088	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_2"; 
+SIRV6	LexogenSIRVData	exon	1846	2026	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_3"; 
+SIRV6	LexogenSIRVData	exon	2471	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_4"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_5"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_6"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_7"; 
+SIRV6	LexogenSIRVData	exon	11035	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_8"; 
+SIRV6	LexogenSIRVData	exon	11206	11837	.	+	0	gene_id "SIRV6"; transcript_id "SIRV604"; exon_assignment "SIRV604_9"; 
+SIRV6	LexogenSIRVData	exon	1131	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_2"; 
+SIRV6	LexogenSIRVData	exon	1846	2026	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_3"; 
+SIRV6	LexogenSIRVData	exon	2471	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_4"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_5"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_6"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_7"; 
+SIRV6	LexogenSIRVData	exon	11032	11331	.	+	0	gene_id "SIRV6"; transcript_id "SIRV605"; exon_assignment "SIRV605_8"; 
+SIRV6	LexogenSIRVData	exon	2286	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV606"; exon_assignment "SIRV606_0"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV606"; exon_assignment "SIRV606_1"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV606"; exon_assignment "SIRV606_2"; 
+SIRV6	LexogenSIRVData	exon	10725	10788	.	+	0	gene_id "SIRV6"; transcript_id "SIRV606"; exon_assignment "SIRV606_3"; 
+SIRV6	LexogenSIRVData	exon	1131	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV607"; exon_assignment "SIRV607_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV607"; exon_assignment "SIRV607_1"; 
+SIRV6	LexogenSIRVData	exon	1846	2026	.	+	0	gene_id "SIRV6"; transcript_id "SIRV607"; exon_assignment "SIRV607_2"; 
+SIRV6	LexogenSIRVData	exon	2471	2540	.	+	0	gene_id "SIRV6"; transcript_id "SIRV607"; exon_assignment "SIRV607_3"; 
+SIRV6	LexogenSIRVData	exon	3024	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV608"; exon_assignment "SIRV608_0"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV608"; exon_assignment "SIRV608_1"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV608"; exon_assignment "SIRV608_2"; 
+SIRV6	LexogenSIRVData	exon	11206	11270	.	+	0	gene_id "SIRV6"; transcript_id "SIRV608"; exon_assignment "SIRV608_3"; 
+SIRV6	LexogenSIRVData	exon	1138	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV609"; exon_assignment "SIRV609_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV609"; exon_assignment "SIRV609_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV609"; exon_assignment "SIRV609_2"; 
+SIRV6	LexogenSIRVData	exon	1846	2120	.	+	0	gene_id "SIRV6"; transcript_id "SIRV609"; exon_assignment "SIRV609_3"; 
+SIRV6	LexogenSIRVData	exon	2473	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV610"; exon_assignment "SIRV610_0"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV610"; exon_assignment "SIRV610_1"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV610"; exon_assignment "SIRV610_2"; 
+SIRV6	LexogenSIRVData	exon	10725	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV610"; exon_assignment "SIRV610_3"; 
+SIRV6	LexogenSIRVData	exon	11206	11690	.	+	0	gene_id "SIRV6"; transcript_id "SIRV610"; exon_assignment "SIRV610_4"; 
+SIRV6	LexogenSIRVData	exon	1304	1381	.	+	0	gene_id "SIRV6"; transcript_id "SIRV611"; exon_assignment "SIRV611_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV611"; exon_assignment "SIRV611_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1950	.	+	0	gene_id "SIRV6"; transcript_id "SIRV611"; exon_assignment "SIRV611_2"; 
+SIRV6	LexogenSIRVData	exon	1088	1186	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_0"; 
+SIRV6	LexogenSIRVData	exon	1469	1534	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_1"; 
+SIRV6	LexogenSIRVData	exon	1641	1735	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_2"; 
+SIRV6	LexogenSIRVData	exon	1846	2026	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_3"; 
+SIRV6	LexogenSIRVData	exon	2471	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_4"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_5"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_6"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_7"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_8"; 
+SIRV6	LexogenSIRVData	exon	11206	11825	.	+	0	gene_id "SIRV6"; transcript_id "SIRV612"; exon_assignment "SIRV612_9"; 
+SIRV6	LexogenSIRVData	exon	3106	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_0"; 
+SIRV6	LexogenSIRVData	exon	7105	7448	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_1"; 
+SIRV6	LexogenSIRVData	exon	7806	7923	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_2"; 
+SIRV6	LexogenSIRVData	exon	10725	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_3"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_4"; 
+SIRV6	LexogenSIRVData	exon	11206	11824	.	+	0	gene_id "SIRV6"; transcript_id "SIRV613"; exon_assignment "SIRV613_5"; 
+SIRV6	LexogenSIRVData	exon	2517	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV614"; exon_assignment "SIRV614_0"; 
+SIRV6	LexogenSIRVData	exon	2741	2828	.	+	0	gene_id "SIRV6"; transcript_id "SIRV614"; exon_assignment "SIRV614_1"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV614"; exon_assignment "SIRV614_2"; 
+SIRV6	LexogenSIRVData	exon	7806	7923	.	+	0	gene_id "SIRV6"; transcript_id "SIRV614"; exon_assignment "SIRV614_3"; 
+SIRV6	LexogenSIRVData	exon	10725	10815	.	+	0	gene_id "SIRV6"; transcript_id "SIRV614"; exon_assignment "SIRV614_4"; 
+SIRV6	LexogenSIRVData	exon	10238	10818	.	+	0	gene_id "SIRV6"; transcript_id "SIRV615"; exon_assignment "SIRV615_0"; 
+SIRV6	LexogenSIRVData	exon	11032	11108	.	+	0	gene_id "SIRV6"; transcript_id "SIRV615"; exon_assignment "SIRV615_1"; 
+SIRV6	LexogenSIRVData	exon	11206	11330	.	+	0	gene_id "SIRV6"; transcript_id "SIRV615"; exon_assignment "SIRV615_2"; 
+SIRV6	LexogenSIRVData	exon	2286	2620	.	+	0	gene_id "SIRV6"; transcript_id "SIRV616"; exon_assignment "SIRV616_0"; 
+SIRV6	LexogenSIRVData	exon	2741	2814	.	+	0	gene_id "SIRV6"; transcript_id "SIRV616"; exon_assignment "SIRV616_1"; 
+SIRV6	LexogenSIRVData	exon	3107	3164	.	+	0	gene_id "SIRV6"; transcript_id "SIRV616"; exon_assignment "SIRV616_2"; 
+SIRV6	LexogenSIRVData	exon	10725	10788	.	+	0	gene_id "SIRV6"; transcript_id "SIRV616"; exon_assignment "SIRV616_3"; 
+SIRV6	LexogenSIRVData	exon	1545	1820	.	-	0	gene_id "SIRV6"; transcript_id "SIRV617"; exon_assignment "SIRV617_0"; 
+SIRV6	LexogenSIRVData	exon	2359	2547	.	-	0	gene_id "SIRV6"; transcript_id "SIRV618"; exon_assignment "SIRV618_0"; 
+SIRV7	LexogenSIRVData	exon	1004	2675	.	-	0	gene_id "SIRV7"; transcript_id "SIRV701"; exon_assignment "SIRV701_0"; 
+SIRV7	LexogenSIRVData	exon	2994	3111	.	-	0	gene_id "SIRV7"; transcript_id "SIRV701"; exon_assignment "SIRV701_1"; 
+SIRV7	LexogenSIRVData	exon	43029	43077	.	-	0	gene_id "SIRV7"; transcript_id "SIRV701"; exon_assignment "SIRV701_2"; 
+SIRV7	LexogenSIRVData	exon	114681	114988	.	-	0	gene_id "SIRV7"; transcript_id "SIRV701"; exon_assignment "SIRV701_3"; 
+SIRV7	LexogenSIRVData	exon	147609	147923	.	-	0	gene_id "SIRV7"; transcript_id "SIRV701"; exon_assignment "SIRV701_4"; 
+SIRV7	LexogenSIRVData	exon	1001	2675	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_0"; 
+SIRV7	LexogenSIRVData	exon	2994	3111	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_1"; 
+SIRV7	LexogenSIRVData	exon	4096	4179	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_2"; 
+SIRV7	LexogenSIRVData	exon	4726	4810	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_3"; 
+SIRV7	LexogenSIRVData	exon	43029	43077	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_4"; 
+SIRV7	LexogenSIRVData	exon	114681	114916	.	-	0	gene_id "SIRV7"; transcript_id "SIRV702"; exon_assignment "SIRV702_5"; 
+SIRV7	LexogenSIRVData	exon	1001	2675	.	-	0	gene_id "SIRV7"; transcript_id "SIRV703"; exon_assignment "SIRV703_0"; 
+SIRV7	LexogenSIRVData	exon	2994	3111	.	-	0	gene_id "SIRV7"; transcript_id "SIRV703"; exon_assignment "SIRV703_1"; 
+SIRV7	LexogenSIRVData	exon	3810	3896	.	-	0	gene_id "SIRV7"; transcript_id "SIRV703"; exon_assignment "SIRV703_2"; 
+SIRV7	LexogenSIRVData	exon	114681	114988	.	-	0	gene_id "SIRV7"; transcript_id "SIRV703"; exon_assignment "SIRV703_3"; 
+SIRV7	LexogenSIRVData	exon	147609	147918	.	-	0	gene_id "SIRV7"; transcript_id "SIRV703"; exon_assignment "SIRV703_4"; 
+SIRV7	LexogenSIRVData	exon	55850	56097	.	-	0	gene_id "SIRV7"; transcript_id "SIRV704"; exon_assignment "SIRV704_0"; 
+SIRV7	LexogenSIRVData	exon	78842	78963	.	-	0	gene_id "SIRV7"; transcript_id "SIRV704"; exon_assignment "SIRV704_1"; 
+SIRV7	LexogenSIRVData	exon	114681	114738	.	-	0	gene_id "SIRV7"; transcript_id "SIRV704"; exon_assignment "SIRV704_2"; 
+SIRV7	LexogenSIRVData	exon	1006	2675	.	-	0	gene_id "SIRV7"; transcript_id "SIRV705"; exon_assignment "SIRV705_0"; 
+SIRV7	LexogenSIRVData	exon	2994	3111	.	-	0	gene_id "SIRV7"; transcript_id "SIRV705"; exon_assignment "SIRV705_1"; 
+SIRV7	LexogenSIRVData	exon	43029	43077	.	-	0	gene_id "SIRV7"; transcript_id "SIRV705"; exon_assignment "SIRV705_2"; 
+SIRV7	LexogenSIRVData	exon	114681	114988	.	-	0	gene_id "SIRV7"; transcript_id "SIRV705"; exon_assignment "SIRV705_3"; 
+SIRV7	LexogenSIRVData	exon	147609	147925	.	-	0	gene_id "SIRV7"; transcript_id "SIRV705"; exon_assignment "SIRV705_4"; 
+SIRV7	LexogenSIRVData	exon	56032	56097	.	-	0	gene_id "SIRV7"; transcript_id "SIRV706"; exon_assignment "SIRV706_0"; 
+SIRV7	LexogenSIRVData	exon	70884	70987	.	-	0	gene_id "SIRV7"; transcript_id "SIRV706"; exon_assignment "SIRV706_1"; 
+SIRV7	LexogenSIRVData	exon	78842	78963	.	-	0	gene_id "SIRV7"; transcript_id "SIRV706"; exon_assignment "SIRV706_2"; 
+SIRV7	LexogenSIRVData	exon	114681	114988	.	-	0	gene_id "SIRV7"; transcript_id "SIRV706"; exon_assignment "SIRV706_3"; 
+SIRV7	LexogenSIRVData	exon	147609	147957	.	-	0	gene_id "SIRV7"; transcript_id "SIRV706"; exon_assignment "SIRV706_4"; 
+SIRV7	LexogenSIRVData	exon	56038	56097	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_0"; 
+SIRV7	LexogenSIRVData	exon	70884	70987	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_1"; 
+SIRV7	LexogenSIRVData	exon	78842	78908	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_2"; 
+SIRV7	LexogenSIRVData	exon	78929	78963	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_3"; 
+SIRV7	LexogenSIRVData	exon	114687	114960	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_4"; 
+SIRV7	LexogenSIRVData	exon	147609	147957	.	-	0	gene_id "SIRV7"; transcript_id "SIRV708"; exon_assignment "SIRV708_5"; 

test_data/sample_sheet

@@ -0,0 +1,3 @@
+barcode,sample_id,alias,type
+barcode01,SRR12480552,SRR12480552,test_sample1
+barcode02,SRR12447502,SRR12447502,test_sample2

tests.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+
+# A couple of simple tests with different combinations of CLI options
+
+if [[ "$#" -ne 1 ]]; then
+    echo "Please supply path to out_dir"
+    exit 1
+fi
+
+SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+cd $SCRIPT_DIR;
+
+singledir="test_data/fastq"
+multisampledir="test_data/demultiplexed_fastq"
+
+echo $singledir
+echo $multisampledir
+results=()
+
+# Test1 single sample
+OUTPUT=$1/test1;
+nextflow run . --fastq $singledir --ref_genome test_data/SIRV_150601a.fasta \
+--ref_annotation test_data/SIRV_isofroms.gtf -profile conda --out_dir ${OUTPUT} -w ${OUTPUT}/workspace -resume;
+
+results+=("test1: $?")
+
+# Test2 multiple samples demultiplexed
+OUTPUT=$1/test2;
+nextflow run . --fastq $multisampledir --ref_genome test_data/SIRV_150601a.fasta \
+--ref_annotation test_data/SIRV_isofroms.gtf -profile conda --out_dir ${OUTPUT} -w ${OUTPUT}/workspace \
+--sample_sheet test_data/sample_sheet -resume;
+
+results+=("test2: $?")
+
+# Test3 single sample. No reference annotation
+OUTPUT=$1/test3;
+nextflow run . --fastq $singledir --ref_genome test_data/SIRV_150601a.fasta \
+ -profile conda --out_dir ${OUTPUT} -w ${OUTPUT}/workspace -resume;
+
+results+=("test3: $?")
+
+# Test4 single sample. force split_bam to make multiple alignment bundles
+OUTPUT=$1/test4;
+nextflow run . --fastq $singledir  --ref_genome test_data/SIRV_150601a.fasta \
+--ref_annotation test_data/SIRV_isofroms.gtf -profile conda --out_dir ${OUTPUT} -w ${OUTPUT}/workspace \
+--bundle_min_reads 5 -resume;
+
+results+=("test4: $?")
+
+echo "Exit status codes for each test"
+for value in "${results[@]}"; do
+     echo "${value}"
+done