kfukuta/wf-transcriptomes-v202
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'docs' into 'dev'

Browse Source 
New docs format

See merge request epi2melabs/workflow-containers/wf-isoforms!50
This commit is contained in:
Neil Horner 2022-04-08 16:33:50 +00:00
commit 00255c86f7
7 changed files with 304 additions and 184 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,26 @@
repos:
- repo: local
hooks:
- id: docs_schema
name: docs_schema
entry: parse_docs -p docs -e .md -s intro links -oj nextflow_schema.json
language: python
always_run: true
pass_filenames: false
additional_dependencies:
- epi2melabs
- id: docs_readme
name: docs_readme
entry: parse_docs -p docs -e .md -s header intro quickstart links -ot README.md
language: python
always_run: true
pass_filenames: false
additional_dependencies:
- epi2melabs
- repo: https://gitlab.com/pycqa/flake8
rev: 3.7.9
hooks:
- id: flake8
additional_dependencies:
- flake8-import-order==0.18.1
entry: flake8 bin --import-order-style google --statistics

84
README.md
View File

@ -1,25 +1,29 @@
# wf-isoforms # wf-isoforms
This repository contains a [nextflow](https://www.nextflow.io/) workflow This repository contains a [nextflow](https://www.nextflow.io/) workflow
for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads. <br> for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads.
It has been adapted from two existing Snakemake pipelines: It has been adapted from two existing Snakemake pipelines:
* https://github.com/nanoporetech/pipeline-nanopore-ref-isoforms * https://github.com/nanoporetech/pipeline-nanopore-ref-isoforms
* https://github.com/nanoporetech/pipeline-nanopore-denovo-isoforms * https://github.com/nanoporetech/pipeline-nanopore-denovo-isoforms
---
## Overview
* cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper)
for identification of full-length reads, as well as trimming and orientation correction (This step is omited for
direct RNA reads)
Reference-based approach ## Introduction
* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2) <br>
This workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA)
Oxford Nanopore reads.
### Preprocesing
cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper)
for the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for
direct RNA reads).
### Reference-aided approach
* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2)
* Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie) * Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie)
in long read mode (with or without a guide reference annotation) to generate the GFF annotation. in long read mode (with or without a guide reference annotation) to generate the GFF annotation.
* The annotation generated by the pipeline is compared to the reference annotation * The annotation generated by the pipeline is compared to the reference annotation.
using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml) using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml)
de novo-based approach (experimental!) ### de novo-based approach (experimental!)
* Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2) * Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2)
* If a reference genome is supplied, cluster quality metrics are determined by comparing * If a reference genome is supplied, cluster quality metrics are determined by comparing
with clusters generated from a minimap2 alignment. with clusters generated from a minimap2 alignment.
@ -27,14 +31,13 @@ de novo-based approach (experimental!)
* Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene. * Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene.
* Full-length reads are then mapped to these polished CDS. * Full-length reads are then mapped to these polished CDS.
* Transcripts are assembled by stringtie as for the reference-based approach. * Transcripts are assembled by stringtie as for the reference-based approach.
* Note: This is currently not supported with direct RNA reads. * __Note__: This approach is currently not supported with direct RNA reads.
For both approaches:
* An html report is generated, which contains various summary statistics and plots of the data.
---
## Quickstart
### Workflow inputs
- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples
(in fastq/fastq.gz format)
- Reference genome in fasta format (required for reference-based assembly).
- Optional reference annotation in GFF2/3 format.## Quickstart
The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and
software resources, as such nextflow will need to be installed before attempting software resources, as such nextflow will need to be installed before attempting
@ -51,7 +54,7 @@ It is not required to clone or download the git repository in order to run the w
For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex). For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex).
### Workflow options **Workflow options**
To obtain the workflow, having installed `nextflow`, users can run: To obtain the workflow, having installed `nextflow`, users can run:
@ -61,14 +64,11 @@ nextflow run epi2me-labs/wf-isoforms --help
to see the options for the workflow. to see the options for the workflow.
**Workflow inputs**
- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples
(in fastq/fastq.gz format)
- Reference genome in fasta format (required for reference-based assembly)
- Optional reference annotation in GFF2/3 format
**Example execution of a workflow for reference-based transcript assembly** **Example execution of a workflow for reference-based transcript assembly**
This uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with
This example uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with
--minimap2_opts '-uf --splice-flank=no' --minimap2_opts '-uf --splice-flank=no'
``` ```
OUTPUT=~/output; OUTPUT=~/output;
@ -88,20 +88,15 @@ nextflow run . --fastq test_data/fastq --denovo --ref_genome test_data/SIRV_1506
--sample sample_id -resume --sample sample_id -resume
``` ```
A full list of options can be seen in nextflow_schema.json. Below are some commonly used ones. A full list of options can be seen in nextflow_schema.json. Below are some commonly used ones.
They can be set in the config like this: <br>
&nbsp;&nbsp; opt = 50<br>
or at the command line:<br>
&nbsp;&nbsp; --opt 50
- Threshold for including isoforms into interactive table `transcript_table_cov_thresh = 50` - Threshold for including isoforms into interactive table `transcript_table_cov_thresh = 50`
- Run the denovo pipeline `denovo = true` (default false) - Run the denovo pipeline `denovo = true` (default false)
- To run the workflow with direct RNA reads `--direct_rna` (skips the pychopper step). - To run the workflow with direct RNA reads `--direct_rna` (skips the pychopper step).
Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts` Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts`, for example:
<br>
For example:
- When using the SIRV synthetic test data - When using the SIRV synthetic test data
- `minimap2_opts = '-uf --splice-flank=no'` - `minimap2_opts = '-uf --splice-flank=no'`
- pychopper needs to know which cDNA synthesis kit used - pychopper needs to know which cDNA synthesis kit used
@ -110,30 +105,8 @@ For example:
- pychopper can use one of two available backends for identifying primers in the raw reads - pychopper can use one of two available backends for identifying primers in the raw reads
- nhmmscan `pychopper opts = '-m phmm'` - nhmmscan `pychopper opts = '-m phmm'`
- edlib `pychopper opts = '-m edlib'` - edlib `pychopper opts = '-m edlib'`
<br>Note: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan.
---
## Workflow outputs
* wf-isoforms-report.html
- Summary and plots of reads, alignments and the transcript assembly and annotation
- One file per run with results from each sample merged
Note transcript isoform table is currently only available for the reference-guilded assembly.
Note: If using a large dataset with 10s of thousands of predicted isoforms, to speed up searching, the table can be limited
by read coverage, which can be set with `transcript_table_cov_thresh = 50` (default 50x)
#### Output files
Each sample will also have it's own directory containing the following (dependent on assembly approach and input options)
* {sampleid}_gffcompare
* Directory containing all output from gffcompare
* transcripts_{sample_id}.gff
* Annotation of query transcripts
* {sample_id}_transcriptome.fas
- A transcriptome derived from the query reads
* {sample_id}_merged_transcriptome.fas
- A transcriptome derived from the query reads + reference annotation
__Note__: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan.
## Useful links ## Useful links
@ -145,4 +118,3 @@ Each sample will also have it's own directory containing the following (dependen
* [spoa](https://github.com/rvaser/spoa) * [spoa](https://github.com/rvaser/spoa)
* [inONclust](https://github.com/ksahlin/isONclust) * [inONclust](https://github.com/ksahlin/isONclust)
* [isONclust2](https://github.com/nanoporetech/isONclust2) * [isONclust2](https://github.com/nanoporetech/isONclust2)

8
docs/header.md Normal file
View File

@ -0,0 +1,8 @@
# wf-isoforms
This repository contains a [nextflow](https://www.nextflow.io/) workflow
for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads.
It has been adapted from two existing Snakemake pipelines:
* https://github.com/nanoporetech/pipeline-nanopore-ref-isoforms
* https://github.com/nanoporetech/pipeline-nanopore-denovo-isoforms

32
docs/intro.md Normal file
View File

@ -0,0 +1,32 @@
## Introduction
This workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA)
Oxford Nanopore reads.
### Preprocesing
cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper)
for the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for
direct RNA reads).
### Reference-aided approach
* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2)
* Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie)
in long read mode (with or without a guide reference annotation) to generate the GFF annotation.
* The annotation generated by the pipeline is compared to the reference annotation.
using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml)
### de novo-based approach (experimental!)
* Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2)
* If a reference genome is supplied, cluster quality metrics are determined by comparing
with clusters generated from a minimap2 alignment.
* A consensus sequence for each cluster is generated using [spoa](https://github.com/rvaser/spoa)
* Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene.
* Full-length reads are then mapped to these polished CDS.
* Transcripts are assembled by stringtie as for the reference-based approach.
* __Note__: This approach is currently not supported with direct RNA reads.
### Workflow inputs
- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples
(in fastq/fastq.gz format)
- Reference genome in fasta format (required for reference-based assembly).
- Optional reference annotation in GFF2/3 format.

10
docs/links.md Normal file
View File

@ -0,0 +1,10 @@
## Useful links
* [nextflow](https://www.nextflow.io/)
* [docker](https://www.docker.com/products/docker-desktop)
* [Singularity](https://sylabs.io/singularity/)
* [conda](https://docs.conda.io/en/latest/miniconda.html)
* [racon](https://github.com/isovic/racon)
* [spoa](https://github.com/rvaser/spoa)
* [inONclust](https://github.com/ksahlin/isONclust)
* [isONclust2](https://github.com/nanoporetech/isONclust2)

71
docs/quickstart.md Normal file
View File

@ -0,0 +1,71 @@
## Quickstart
The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and
software resources, as such nextflow will need to be installed before attempting
to run the workflow.
The workflow can currently be run using either
[Docker](https://www.docker.com/products/docker-desktop),
[Singularity](https://sylabs.io/singularity/) or
[conda](https://docs.conda.io/en/latest/miniconda.html) to provide isolation of
the required software. Each method is automated out-of-the-box provided
either docker, singularity or conda is installed.
It is not required to clone or download the git repository in order to run the workflow.
For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex).
**Workflow options**
To obtain the workflow, having installed `nextflow`, users can run:
```
nextflow run epi2me-labs/wf-isoforms --help
```
to see the options for the workflow.
**Example execution of a workflow for reference-based transcript assembly**
This example uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with
--minimap2_opts '-uf --splice-flank=no'
```
OUTPUT=~/output;
nextflow run wf-isoforms/ --fastq test_data/fastq --ref_genome test_data/SIRV_150601a.fasta --ref_annotation test_data/SIRV_isofroms.gtf
--minimap2_opts '-uf --splice-flank=no' --out_dir outdir -w workspace_dir -profile conda -resume
```
```
# To evaluate the workflow on a larger Drosophila dataset
./evaluation/run_evaluation_dmel.sh outdir
```
**Example workflow for denovo transcript assembly**
```
OUTPUT=~/output
nextflow run . --fastq test_data/fastq --denovo --ref_genome test_data/SIRV_150601a.fasta -profile local --out_dir ${OUTPUT} -w ${OUTPUT}/workspace \
--sample sample_id -resume
```
A full list of options can be seen in nextflow_schema.json. Below are some commonly used ones.
- Threshold for including isoforms into interactive table `transcript_table_cov_thresh = 50`
- Run the denovo pipeline `denovo = true` (default false)
- To run the workflow with direct RNA reads `--direct_rna` (skips the pychopper step).
Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts`, for example:
- When using the SIRV synthetic test data
- `minimap2_opts = '-uf --splice-flank=no'`
- pychopper needs to know which cDNA synthesis kit used
- SQK-PCS109: use `pychopper_opts = '-k PCS109'` (default)
- SQK-PCS110: use `pychopper_opts = '-k PCS110'`
- pychopper can use one of two available backends for identifying primers in the raw reads
- nhmmscan `pychopper opts = '-m phmm'`
- edlib `pychopper opts = '-m edlib'`
__Note__: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan.

253
nextflow_schema.json
View File

@ -53,9 +53,9 @@
] ]
}, },
"global_options": { "global_options": {
"title": "Global options", "title": "Global options",
"type": "object", "type": "object",
"description": "Options for both sub-workflows", "description": "Options for both sub-workflows",
"properties": { "properties": {
"threads": { "threads": {
"type": "integer", "type": "integer",
@ -92,128 +92,127 @@
} }
} }
}, },
"reference_wf_options": { "reference_wf_options": {
"title": "Options for reference-based workflow", "title": "Options for reference-based workflow",
"type": "object", "type": "object",
"description": "Parameters that are used solely for the referenc-guided workflow", "description": "Parameters that are used solely for the referenc-guided workflow",
"properties": { "properties": {
"plot_gffcmp_stats": { "plot_gffcmp_stats": {
"type": "boolean", "type": "boolean",
"description": "Create a pdf of plots from showing gffcompare results" "description": "Create a pdf of plots from showing gffcompare results"
}, },
"gffcompare_opts": { "gffcompare_opts": {
"type": "string", "type": "string",
"description": "Extra options for gffcompare -r", "description": "Extra options for gffcompare -r",
"default": " -R " "default": " -R "
}, },
"minimap_index_opts": { "minimap_index_opts": {
"type": "string", "type": "string",
"description": "minimap2 extra indexing options.", "description": "minimap2 extra indexing options.",
"default": "-k14" "default": "-k14"
}, },
"minimap2_opts": { "minimap2_opts": {
"type": "string", "type": "string",
"description": "minimap2 extra mapping options.", "description": "minimap2 extra mapping options.",
"default": "-uf" "default": "-uf"
}, },
"minimum_mapping_quality": { "minimum_mapping_quality": {
"type": "integer", "type": "integer",
"description": "filter aligned reads by MAPQ quality.", "description": "filter aligned reads by MAPQ quality.",
"default": 40 "default": 40
}, },
"poly_context": { "poly_context": {
"type": "integer", "type": "integer",
"description": "Region size at end of reads to apply poly(A) filter.", "description": "Region size at end of reads to apply poly(A) filter.",
"default": 24 "default": 24
}, },
"max_poly_run": { "max_poly_run": {
"type": "integer", "type": "integer",
"description": "Max poly(A) region allowed with poly_context-sized end regions.", "description": "Max poly(A) region allowed with poly_context-sized end regions.",
"default": 8 "default": 8
} }
}
},
"denovo_wf_options": {
"title": "Options for de novo-based workflow",
"type": "object",
"description": "Parameters that are used solely for the de novo workflow",
"properties": {
"batch_size": {
"type": "integer",
"description": "Maximum sequences per input batch (-1 means no limit)",
"default": -1
},
"batch_max_seq": {
"type": "integer",
"description": "Maximum sequences per input batch (-1 means no limit)",
"default": -1
},
"cls_mode": {
"type": "string",
"description": "Clustering mode",
"default": "sahlin"
},
"kmer_size": {
"type": "integer",
"description": "Kmer size",
"default": 11
},
"window_size": {
"type": "integer",
"description": "Window size",
"default": 15
},
"min_left_cls": {
"type": "integer",
"description": "Minimum cluser size in the left batch",
"default": 2
},
"consensus_period": {
"type": "integer",
"description": "Consensus period (-1 means no consensus)",
"default": 500
},
"consensus_minimum": {
"type": "integer",
"description": "Minimum consensus sample size:",
"default": 50
},
"consensus_maximum": {
"type": "integer",
"description": "Maximum consensus sample size",
"default": -150
},
"min_shared": {
"type": "integer",
"description": "Minimum number of minimizers shared between read and cluster",
"default": 5
},
"min_qual": {
"description": "Minimum average quality value",
"type": "number",
"default": 7.0
},
"mapped_threshold": {
"description": "Minimum mapped fraction of read to be included in cluster",
"type": "number",
"default": 0.65
},
"aligned_threshold": {
"tpye": "number",
"description": "Minimum aligned fraction of read to be included in cluster",
"default": 0.2
},
"min_fraction": {
"type": "number",
"description": "Minimum fraction of minimizers shared compared to best hit, in order to continue mapping",
"default": 0.8
},
"min_prob_no_hits": {
"type": "number",
"description": "Minimum probability for i consecutive minimizers to be different between read and representative",
"default": 0.2
} }
},
"denovo_wf_options": {
"title": "Options for de novo-based workflow",
"type": "object",
"description": "Parameters that are used solely for the de novo workflow",
"properties": {
"batch_size": {
"type": "integer",
"description": "Maximum sequences per input batch (-1 means no limit)",
"default": -1
},
"batch_max_seq": {
"type": "integer",
"description": "Maximum sequences per input batch (-1 means no limit)",
"default": -1
},
"cls_mode": {
"type": "string",
"description": "Clustering mode",
"default": "sahlin"
},
"kmer_size": {
"type": "integer",
"description": "Kmer size",
"default": 11
},
"window_size": {
"type": "integer",
"description": "Window size",
"default": 15
},
"min_left_cls": {
"type": "integer",
"description": "Minimum cluser size in the left batch",
"default": 2
},
"consensus_period": {
"type": "integer",
"description": "Consensus period (-1 means no consensus)",
"default": 500
},
"consensus_minimum": {
"type": "integer",
"description": "Minimum consensus sample size:",
"default": 50
},
"consensus_maximum": {
"type": "integer",
"description": "Maximum consensus sample size",
"default": -150
},
"min_shared": {
"type": "integer",
"description": "Minimum number of minimizers shared between read and cluster",
"default": 5
},
"min_qual": {
"description": "Minimum average quality value",
"type": "number",
"default": 7.0
},
"mapped_threshold": {
"description": "Minimum mapped fraction of read to be included in cluster",
"type": "number",
"default": 0.65
},
"aligned_threshold": {
"tpye": "number",
"description": "Minimum aligned fraction of read to be included in cluster",
"default": 0.2
},
"min_fraction": {
"type": "number",
"description": "Minimum fraction of minimizers shared compared to best hit, in order to continue mapping",
"default": 0.8
},
"min_prob_no_hits" : {
"type": "number",
"description": "Minimum probability for i consecutive minimizers to be different between read and representative",
"default": 0.2
}
} }
}, },
"meta_data": { "meta_data": {
@ -231,7 +230,6 @@
"type": "boolean", "type": "boolean",
"default": false, "default": false,
"description": "Enable to prevent sending a workflow ping." "description": "Enable to prevent sending a workflow ping."
} }
} }
}, },
@ -251,7 +249,6 @@
} }
} }
}, },
"allOf": [ "allOf": [
{ {
"$ref": "#/definitions/basic_input_output_options" "$ref": "#/definitions/basic_input_output_options"
@ -296,5 +293,9 @@
"show_hidden_params": { "show_hidden_params": {
"type": "boolean" "type": "boolean"
} }
},
"docs": {
"intro": "## Introduction\n\nThis workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA) \nOxford Nanopore reads.\n\n### Preprocesing\ncDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) \nfor the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for \n direct RNA reads).\n\n### Reference-aided approach\n* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2)\n* Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie) \nin long read mode (with or without a guide reference annotation) to generate the GFF annotation.\n* The annotation generated by the pipeline is compared to the reference annotation. \nusing [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml)\n\n### de novo-based approach (experimental!)\n* Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2)\n * If a reference genome is supplied, cluster quality metrics are determined by comparing \n with clusters generated from a minimap2 alignment.\n* A consensus sequence for each cluster is generated using [spoa](https://github.com/rvaser/spoa)\n* Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene.\n* Full-length reads are then mapped to these polished CDS.\n* Transcripts are assembled by stringtie as for the reference-based approach.\n* __Note__: This approach is currently not supported with direct RNA reads.\n\n### Workflow inputs\n- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples\n (in fastq/fastq.gz format)\n- Reference genome in fasta format (required for reference-based assembly).\n- Optional reference annotation in GFF2/3 format.",
"links": "## Useful links\n\n* [nextflow](https://www.nextflow.io/)\n* [docker](https://www.docker.com/products/docker-desktop)\n* [Singularity](https://sylabs.io/singularity/)\n* [conda](https://docs.conda.io/en/latest/miniconda.html)\n* [racon](https://github.com/isovic/racon)\n* [spoa](https://github.com/rvaser/spoa)\n* [inONclust](https://github.com/ksahlin/isONclust)\n* [isONclust2](https://github.com/nanoporetech/isONclust2)"
} }
} }