diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..ecd70f9 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,26 @@ +repos: + - repo: local + hooks: + - id: docs_schema + name: docs_schema + entry: parse_docs -p docs -e .md -s intro links -oj nextflow_schema.json + language: python + always_run: true + pass_filenames: false + additional_dependencies: + - epi2melabs + - id: docs_readme + name: docs_readme + entry: parse_docs -p docs -e .md -s header intro quickstart links -ot README.md + language: python + always_run: true + pass_filenames: false + additional_dependencies: + - epi2melabs + - repo: https://gitlab.com/pycqa/flake8 + rev: 3.7.9 + hooks: + - id: flake8 + additional_dependencies: + - flake8-import-order==0.18.1 + entry: flake8 bin --import-order-style google --statistics \ No newline at end of file diff --git a/README.md b/README.md index b6eff18..301afb6 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,29 @@ # wf-isoforms - This repository contains a [nextflow](https://www.nextflow.io/) workflow -for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads.
+for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads. It has been adapted from two existing Snakemake pipelines: * https://github.com/nanoporetech/pipeline-nanopore-ref-isoforms * https://github.com/nanoporetech/pipeline-nanopore-denovo-isoforms ---- -## Overview -* cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) -for identification of full-length reads, as well as trimming and orientation correction (This step is omited for - direct RNA reads) -Reference-based approach -* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2)
+## Introduction + +This workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA) +Oxford Nanopore reads. + +### Preprocesing +cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) +for the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for + direct RNA reads). + +### Reference-aided approach +* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2) * Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie) in long read mode (with or without a guide reference annotation) to generate the GFF annotation. -* The annotation generated by the pipeline is compared to the reference annotation +* The annotation generated by the pipeline is compared to the reference annotation. using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml) -de novo-based approach (experimental!) +### de novo-based approach (experimental!) * Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2) * If a reference genome is supplied, cluster quality metrics are determined by comparing with clusters generated from a minimap2 alignment. @@ -27,14 +31,13 @@ de novo-based approach (experimental!) * Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene. * Full-length reads are then mapped to these polished CDS. * Transcripts are assembled by stringtie as for the reference-based approach. -* Note: This is currently not supported with direct RNA reads. - - -For both approaches: -* An html report is generated, which contains various summary statistics and plots of the data. ---- -## Quickstart +* __Note__: This approach is currently not supported with direct RNA reads. +### Workflow inputs +- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples + (in fastq/fastq.gz format) +- Reference genome in fasta format (required for reference-based assembly). +- Optional reference annotation in GFF2/3 format.## Quickstart The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and software resources, as such nextflow will need to be installed before attempting @@ -51,7 +54,7 @@ It is not required to clone or download the git repository in order to run the w For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex). -### Workflow options +**Workflow options** To obtain the workflow, having installed `nextflow`, users can run: @@ -61,14 +64,11 @@ nextflow run epi2me-labs/wf-isoforms --help to see the options for the workflow. -**Workflow inputs** -- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples - (in fastq/fastq.gz format) -- Reference genome in fasta format (required for reference-based assembly) -- Optional reference annotation in GFF2/3 format + **Example execution of a workflow for reference-based transcript assembly** -This uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with + +This example uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with --minimap2_opts '-uf --splice-flank=no' ``` OUTPUT=~/output; @@ -88,20 +88,15 @@ nextflow run . --fastq test_data/fastq --denovo --ref_genome test_data/SIRV_1506 --sample sample_id -resume ``` A full list of options can be seen in nextflow_schema.json. Below are some commonly used ones. -They can be set in the config like this:
-   opt = 50
-or at the command line:
-   --opt 50 - Threshold for including isoforms into interactive table `transcript_table_cov_thresh = 50` - Run the denovo pipeline `denovo = true` (default false) - To run the workflow with direct RNA reads `--direct_rna` (skips the pychopper step). -Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts` -
+Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts`, for example: + -For example: - When using the SIRV synthetic test data - `minimap2_opts = '-uf --splice-flank=no'` - pychopper needs to know which cDNA synthesis kit used @@ -110,31 +105,9 @@ For example: - pychopper can use one of two available backends for identifying primers in the raw reads - nhmmscan `pychopper opts = '-m phmm'` - edlib `pychopper opts = '-m edlib'` -
Note: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan. - ---- -## Workflow outputs -* wf-isoforms-report.html - - Summary and plots of reads, alignments and the transcript assembly and annotation - - One file per run with results from each sample merged +__Note__: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan. -Note transcript isoform table is currently only available for the reference-guilded assembly. -Note: If using a large dataset with 10s of thousands of predicted isoforms, to speed up searching, the table can be limited -by read coverage, which can be set with `transcript_table_cov_thresh = 50` (default 50x) - -#### Output files -Each sample will also have it's own directory containing the following (dependent on assembly approach and input options) -* {sampleid}_gffcompare - * Directory containing all output from gffcompare -* transcripts_{sample_id}.gff - * Annotation of query transcripts -* {sample_id}_transcriptome.fas - - A transcriptome derived from the query reads -* {sample_id}_merged_transcriptome.fas - - A transcriptome derived from the query reads + reference annotation - - ## Useful links * [nextflow](https://www.nextflow.io/) @@ -144,5 +117,4 @@ Each sample will also have it's own directory containing the following (dependen * [racon](https://github.com/isovic/racon) * [spoa](https://github.com/rvaser/spoa) * [inONclust](https://github.com/ksahlin/isONclust) -* [isONclust2](https://github.com/nanoporetech/isONclust2) - +* [isONclust2](https://github.com/nanoporetech/isONclust2) \ No newline at end of file diff --git a/docs/header.md b/docs/header.md new file mode 100644 index 0000000..2affd15 --- /dev/null +++ b/docs/header.md @@ -0,0 +1,8 @@ +# wf-isoforms + +This repository contains a [nextflow](https://www.nextflow.io/) workflow +for assembly and annotation of transcripts from Oxford Nanopore cDNA or direct RNA reads. +It has been adapted from two existing Snakemake pipelines: +* https://github.com/nanoporetech/pipeline-nanopore-ref-isoforms +* https://github.com/nanoporetech/pipeline-nanopore-denovo-isoforms + diff --git a/docs/intro.md b/docs/intro.md new file mode 100644 index 0000000..9a8935b --- /dev/null +++ b/docs/intro.md @@ -0,0 +1,32 @@ +## Introduction + +This workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA) +Oxford Nanopore reads. + +### Preprocesing +cDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) +for the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for + direct RNA reads). + +### Reference-aided approach +* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2) +* Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie) +in long read mode (with or without a guide reference annotation) to generate the GFF annotation. +* The annotation generated by the pipeline is compared to the reference annotation. +using [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml) + +### de novo-based approach (experimental!) +* Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2) + * If a reference genome is supplied, cluster quality metrics are determined by comparing + with clusters generated from a minimap2 alignment. +* A consensus sequence for each cluster is generated using [spoa](https://github.com/rvaser/spoa) +* Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene. +* Full-length reads are then mapped to these polished CDS. +* Transcripts are assembled by stringtie as for the reference-based approach. +* __Note__: This approach is currently not supported with direct RNA reads. + +### Workflow inputs +- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples + (in fastq/fastq.gz format) +- Reference genome in fasta format (required for reference-based assembly). +- Optional reference annotation in GFF2/3 format. \ No newline at end of file diff --git a/docs/links.md b/docs/links.md new file mode 100644 index 0000000..8dd5236 --- /dev/null +++ b/docs/links.md @@ -0,0 +1,10 @@ +## Useful links + +* [nextflow](https://www.nextflow.io/) +* [docker](https://www.docker.com/products/docker-desktop) +* [Singularity](https://sylabs.io/singularity/) +* [conda](https://docs.conda.io/en/latest/miniconda.html) +* [racon](https://github.com/isovic/racon) +* [spoa](https://github.com/rvaser/spoa) +* [inONclust](https://github.com/ksahlin/isONclust) +* [isONclust2](https://github.com/nanoporetech/isONclust2) \ No newline at end of file diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..ff1bff0 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,71 @@ +## Quickstart + +The workflow uses [nextflow](https://www.nextflow.io/) to manage compute and +software resources, as such nextflow will need to be installed before attempting +to run the workflow. + +The workflow can currently be run using either +[Docker](https://www.docker.com/products/docker-desktop), +[Singularity](https://sylabs.io/singularity/) or +[conda](https://docs.conda.io/en/latest/miniconda.html) to provide isolation of +the required software. Each method is automated out-of-the-box provided +either docker, singularity or conda is installed. + +It is not required to clone or download the git repository in order to run the workflow. +For more information on running EPI2ME Labs workflows [visit out website](https://labs.epi2me.io/wfindex). + + +**Workflow options** + +To obtain the workflow, having installed `nextflow`, users can run: + +``` +nextflow run epi2me-labs/wf-isoforms --help +``` + +to see the options for the workflow. + + + +**Example execution of a workflow for reference-based transcript assembly** + +This example uses a synthetic SIRV dataset, so we need to tell minimap2 about the non-canonical splice junctions with +--minimap2_opts '-uf --splice-flank=no' +``` +OUTPUT=~/output; +nextflow run wf-isoforms/ --fastq test_data/fastq --ref_genome test_data/SIRV_150601a.fasta --ref_annotation test_data/SIRV_isofroms.gtf +--minimap2_opts '-uf --splice-flank=no' --out_dir outdir -w workspace_dir -profile conda -resume +``` + +``` +# To evaluate the workflow on a larger Drosophila dataset +./evaluation/run_evaluation_dmel.sh outdir +``` + +**Example workflow for denovo transcript assembly** +``` +OUTPUT=~/output +nextflow run . --fastq test_data/fastq --denovo --ref_genome test_data/SIRV_150601a.fasta -profile local --out_dir ${OUTPUT} -w ${OUTPUT}/workspace \ +--sample sample_id -resume +``` +A full list of options can be seen in nextflow_schema.json. Below are some commonly used ones. + +- Threshold for including isoforms into interactive table `transcript_table_cov_thresh = 50` +- Run the denovo pipeline `denovo = true` (default false) +- To run the workflow with direct RNA reads `--direct_rna` (skips the pychopper step). + + +Pychopper and minimap2 can take options via `minimap2_opts` and `pychopper_opts`, for example: + + +- When using the SIRV synthetic test data + - `minimap2_opts = '-uf --splice-flank=no'` +- pychopper needs to know which cDNA synthesis kit used + - SQK-PCS109: use `pychopper_opts = '-k PCS109'` (default) + - SQK-PCS110: use `pychopper_opts = '-k PCS110'` +- pychopper can use one of two available backends for identifying primers in the raw reads + - nhmmscan `pychopper opts = '-m phmm'` + - edlib `pychopper opts = '-m edlib'` + +__Note__: edlib is set by default in the config as it's quite a lot faster. However it may be less sensitive than nhmmscan. + diff --git a/nextflow_schema.json b/nextflow_schema.json index 5e31e34..7e194eb 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -53,9 +53,9 @@ ] }, "global_options": { - "title": "Global options", - "type": "object", - "description": "Options for both sub-workflows", + "title": "Global options", + "type": "object", + "description": "Options for both sub-workflows", "properties": { "threads": { "type": "integer", @@ -92,128 +92,127 @@ } } }, - "reference_wf_options": { - "title": "Options for reference-based workflow", - "type": "object", - "description": "Parameters that are used solely for the referenc-guided workflow", - "properties": { - "plot_gffcmp_stats": { - "type": "boolean", - "description": "Create a pdf of plots from showing gffcompare results" - }, - "gffcompare_opts": { - "type": "string", - "description": "Extra options for gffcompare -r", - "default": " -R " - }, - "minimap_index_opts": { - "type": "string", - "description": "minimap2 extra indexing options.", - "default": "-k14" - }, - "minimap2_opts": { - "type": "string", - "description": "minimap2 extra mapping options.", - "default": "-uf" - }, - "minimum_mapping_quality": { - "type": "integer", - "description": "filter aligned reads by MAPQ quality.", - "default": 40 - }, - "poly_context": { - "type": "integer", - "description": "Region size at end of reads to apply poly(A) filter.", - "default": 24 - }, - "max_poly_run": { - "type": "integer", - "description": "Max poly(A) region allowed with poly_context-sized end regions.", - "default": 8 - } - + "reference_wf_options": { + "title": "Options for reference-based workflow", + "type": "object", + "description": "Parameters that are used solely for the referenc-guided workflow", + "properties": { + "plot_gffcmp_stats": { + "type": "boolean", + "description": "Create a pdf of plots from showing gffcompare results" + }, + "gffcompare_opts": { + "type": "string", + "description": "Extra options for gffcompare -r", + "default": " -R " + }, + "minimap_index_opts": { + "type": "string", + "description": "minimap2 extra indexing options.", + "default": "-k14" + }, + "minimap2_opts": { + "type": "string", + "description": "minimap2 extra mapping options.", + "default": "-uf" + }, + "minimum_mapping_quality": { + "type": "integer", + "description": "filter aligned reads by MAPQ quality.", + "default": 40 + }, + "poly_context": { + "type": "integer", + "description": "Region size at end of reads to apply poly(A) filter.", + "default": 24 + }, + "max_poly_run": { + "type": "integer", + "description": "Max poly(A) region allowed with poly_context-sized end regions.", + "default": 8 + } + } + }, + "denovo_wf_options": { + "title": "Options for de novo-based workflow", + "type": "object", + "description": "Parameters that are used solely for the de novo workflow", + "properties": { + "batch_size": { + "type": "integer", + "description": "Maximum sequences per input batch (-1 means no limit)", + "default": -1 + }, + "batch_max_seq": { + "type": "integer", + "description": "Maximum sequences per input batch (-1 means no limit)", + "default": -1 + }, + "cls_mode": { + "type": "string", + "description": "Clustering mode", + "default": "sahlin" + }, + "kmer_size": { + "type": "integer", + "description": "Kmer size", + "default": 11 + }, + "window_size": { + "type": "integer", + "description": "Window size", + "default": 15 + }, + "min_left_cls": { + "type": "integer", + "description": "Minimum cluser size in the left batch", + "default": 2 + }, + "consensus_period": { + "type": "integer", + "description": "Consensus period (-1 means no consensus)", + "default": 500 + }, + "consensus_minimum": { + "type": "integer", + "description": "Minimum consensus sample size:", + "default": 50 + }, + "consensus_maximum": { + "type": "integer", + "description": "Maximum consensus sample size", + "default": -150 + }, + "min_shared": { + "type": "integer", + "description": "Minimum number of minimizers shared between read and cluster", + "default": 5 + }, + "min_qual": { + "description": "Minimum average quality value", + "type": "number", + "default": 7.0 + }, + "mapped_threshold": { + "description": "Minimum mapped fraction of read to be included in cluster", + "type": "number", + "default": 0.65 + }, + "aligned_threshold": { + "tpye": "number", + "description": "Minimum aligned fraction of read to be included in cluster", + "default": 0.2 + }, + "min_fraction": { + "type": "number", + "description": "Minimum fraction of minimizers shared compared to best hit, in order to continue mapping", + "default": 0.8 + }, + "min_prob_no_hits": { + "type": "number", + "description": "Minimum probability for i consecutive minimizers to be different between read and representative", + "default": 0.2 } - }, - "denovo_wf_options": { - "title": "Options for de novo-based workflow", - "type": "object", - "description": "Parameters that are used solely for the de novo workflow", - "properties": { - "batch_size": { - "type": "integer", - "description": "Maximum sequences per input batch (-1 means no limit)", - "default": -1 - }, - "batch_max_seq": { - "type": "integer", - "description": "Maximum sequences per input batch (-1 means no limit)", - "default": -1 - }, - "cls_mode": { - "type": "string", - "description": "Clustering mode", - "default": "sahlin" - }, - "kmer_size": { - "type": "integer", - "description": "Kmer size", - "default": 11 - }, - "window_size": { - "type": "integer", - "description": "Window size", - "default": 15 - }, - "min_left_cls": { - "type": "integer", - "description": "Minimum cluser size in the left batch", - "default": 2 - }, - "consensus_period": { - "type": "integer", - "description": "Consensus period (-1 means no consensus)", - "default": 500 - }, - "consensus_minimum": { - "type": "integer", - "description": "Minimum consensus sample size:", - "default": 50 - }, - "consensus_maximum": { - "type": "integer", - "description": "Maximum consensus sample size", - "default": -150 - }, - "min_shared": { - "type": "integer", - "description": "Minimum number of minimizers shared between read and cluster", - "default": 5 - }, - "min_qual": { - "description": "Minimum average quality value", - "type": "number", - "default": 7.0 - }, - "mapped_threshold": { - "description": "Minimum mapped fraction of read to be included in cluster", - "type": "number", - "default": 0.65 - }, - "aligned_threshold": { - "tpye": "number", - "description": "Minimum aligned fraction of read to be included in cluster", - "default": 0.2 - }, - "min_fraction": { - "type": "number", - "description": "Minimum fraction of minimizers shared compared to best hit, in order to continue mapping", - "default": 0.8 - }, - "min_prob_no_hits" : { - "type": "number", - "description": "Minimum probability for i consecutive minimizers to be different between read and representative", - "default": 0.2 - } } }, "meta_data": { @@ -231,7 +230,6 @@ "type": "boolean", "default": false, "description": "Enable to prevent sending a workflow ping." - } } }, @@ -251,7 +249,6 @@ } } }, - "allOf": [ { "$ref": "#/definitions/basic_input_output_options" @@ -296,5 +293,9 @@ "show_hidden_params": { "type": "boolean" } + }, + "docs": { + "intro": "## Introduction\n\nThis workflow identifies RNA isoforms using either cDNA or direct RNA (dRNA) \nOxford Nanopore reads.\n\n### Preprocesing\ncDNA reads are initially preprocessed by [pychopper](https://github.com/nanoporetech/pychopper) \nfor the identification of full-length reads, as well as trimming and orientation correction (This step is omitted for \n direct RNA reads).\n\n### Reference-aided approach\n* Full length reads are mapped to a supplied reference genome using [minimap2](https://github.com/lh3/minimap2)\n* Transcripts are assembled by [stringtie](http://ccb.jhu.edu/software/stringtie) \nin long read mode (with or without a guide reference annotation) to generate the GFF annotation.\n* The annotation generated by the pipeline is compared to the reference annotation. \nusing [gffcompare](http://ccb.jhu.edu/software/stringtie/gffcompare.shtml)\n\n### de novo-based approach (experimental!)\n* Sequence clusters are generated using [isONclust2](https://github.com/nanoporetech/isONclust2)\n * If a reference genome is supplied, cluster quality metrics are determined by comparing \n with clusters generated from a minimap2 alignment.\n* A consensus sequence for each cluster is generated using [spoa](https://github.com/rvaser/spoa)\n* Three rounds of polishing using racon and minimap2 to give a final polished CDS for each gene.\n* Full-length reads are then mapped to these polished CDS.\n* Transcripts are assembled by stringtie as for the reference-based approach.\n* __Note__: This approach is currently not supported with direct RNA reads.\n\n### Workflow inputs\n- Directory containing cDNA/direct RNA reads. Or a directory containing subdirectories each with reads from different samples\n (in fastq/fastq.gz format)\n- Reference genome in fasta format (required for reference-based assembly).\n- Optional reference annotation in GFF2/3 format.", + "links": "## Useful links\n\n* [nextflow](https://www.nextflow.io/)\n* [docker](https://www.docker.com/products/docker-desktop)\n* [Singularity](https://sylabs.io/singularity/)\n* [conda](https://docs.conda.io/en/latest/miniconda.html)\n* [racon](https://github.com/isovic/racon)\n* [spoa](https://github.com/rvaser/spoa)\n* [inONclust](https://github.com/ksahlin/isONclust)\n* [isONclust2](https://github.com/nanoporetech/isONclust2)" } -} +} \ No newline at end of file