From 922c3b39326542ea4473dba6493e9dd2e50e2fa5 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Sun, 16 Aug 2026 19:49:06 -0400 Subject: [PATCH 1/3] Manual is mostly moved to github pages --- README.md | 554 ++++++------------------------------------------------ 1 file changed, 56 insertions(+), 498 deletions(-) diff --git a/README.md b/README.md index 715abbb..182694f 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,7 @@ +

+ +

+ # PoolSeqFlow **A Nextflow pipeline for allele frequency analysis from pooled Illumina sequencing data** @@ -6,7 +10,7 @@ [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19245611.svg)](https://doi.org/10.5281/zenodo.19245611) - +### ๐Ÿ“– [Read the documentation โ†’](https://ozankiratli.github.io/PoolSeqFlow/) > **Platform note:** PoolSeqFlow is developed and tested on **Linux and macOS**. Windows is not supported โ€” the resume logic relies on symbolic links and Unix-style paths that are not compatible with native Windows filesystems. @@ -14,540 +18,101 @@ ## Overview -PoolSeqFlow is a reproducible, end-to-end Nextflow pipeline designed for allele frequency estimation from **pooled Illumina short-read sequencing (Pool-seq)** data. It automates quality control, adapter trimming with intelligent clipping, reference alignment, BAM post-processing, variant calling, and VCF-to-frequency table conversion โ€” with optional variant annotation. - -The pipeline is designed for evolutionary and population genetics studies where large pools of individuals are sequenced together, and where accurate allele frequency estimates are the primary output. +PoolSeqFlow takes raw FASTQ files and a reference genome and gives back allele frequency tables. It automates quality control, adapter trimming with composition-aware clipping, alignment, BAM post-processing, variant calling and VCF-to-frequency conversion, with optional annotation. ---- - -## Pipeline Overview +Pool-seq sequences many individuals together, so the unit of analysis is not a genotype but a **frequency** โ€” and that difference runs through every stage. Multiallelic sites are preserved rather than collapsed, VCFs are re-encoded so the most-read allele is the reference, and the minimum credible frequency is derived from your pool size and ploidy instead of a fixed cutoff. See [When to use PoolSeqFlow](https://ozankiratli.github.io/PoolSeqFlow/concepts/) for what the design assumes about your data. ``` Raw FASTQ reads โ”‚ โ–ผ -[Step 0] Verify Environment & folder structure +[Step 0] Verify environment, parameters and folder structure โ”‚ โ–ผ -[Step 1] Build Reference Dictionaries (BWA, SAMtools, SnpEff) +[Step 1] Build reference dictionaries (BWA, SAMtools, SnpEff) โ”‚ โ–ผ -[Step 2] Quality Control & Trimming (FastQC โ†’ Trim Galore โ†’ smart Cutadapt clipping) +[Step 2] QC & trimming (FastQC โ†’ Trim Galore โ†’ composition-aware clipping) โ”‚ โ–ผ [Step 3] Alignment (BWA-MEM) โ”‚ โ–ผ -[Step 4] BAM Cleanup (name-sort โ†’ fixmate โ†’ coord-sort โ†’ markdup โ†’ addRG โ†’ filter โ†’ index) +[Step 4] BAM cleanup (name-sort โ†’ fixmate โ†’ coord-sort โ†’ markdup โ†’ addRG โ†’ filter โ†’ index) โ”‚ โ–ผ -[Step 5] Alignment & Coverage Reports (BAMtools, SAMtools) +[Step 5] Alignment & coverage reports (BAMtools, SAMtools) โ”‚ โ–ผ -[Step 6] Variant Calling (BCFtools) +[Step 6] Variant calling (BCFtools mpileup + call) โ”‚ - โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ–ผ โ–ผ -[Step 7] VCF โ†’ Allele Frequency Tables [Step 8] Variant Annotation (SnpEff) [optional] - (major-allele normalisation โ†’ min-frequency filter โ†’ depth/quality filter โ†’ SNP/INDEL split) + โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ–ผ โ–ผ +[Step 7] VCF โ†’ allele frequency tables [Step 8] Annotation (SnpEff, optional) ``` --- -## Features - -- **Intelligent trimming**: FastQC report parsing drives automatic Cutadapt clipping thresholds to maximise read quality while minimising data loss. -- **Pool-seq-aware frequency calling**: Allele frequency tables are produced with pool-size- and ploidy-aware minimum frequency filtering (see [Step 7](#step-7-vcf--allele-frequency-tables)). -- **Major-allele normalisation**: VCF files are re-encoded so the major allele is always the reference, enabling consistent downstream comparisons. -- **Multiallelic site support**: The pipeline is designed to handle multiallelic sites throughout the variant calling and frequency conversion steps, preserving complex variation that would be lost under biallelic-only assumptions. -- **Smart resume with permanent storage**: The pipeline uses symbolic links into a permanent output directory so large intermediate files are never duplicated and completed steps are automatically skipped on re-runs โ€” replacing Nextflow's built-in caching entirely, so there is no `-resume` flag to remember (see [Resume Logic](#resume-logic)). -- **Benchmark-driven core allocation**: Every tool's thread count is derived from a single `threads` setting rather than dividing cores evenly. Each tool is quantised to the point where its published scaling flattens out, and Trim Galore is costed on its real footprint โ€” `--cores N` runs N+4 threads โ€” so a task never quietly asks for more than the budget you gave it (see [Resource configuration](#resource-configuration)). -- **Modular design**: Each step is an independent Nextflow DSL2 module โ€” easy to modify, extend, or rerun in isolation. -- **Reproducible environments**: All dependencies are managed via a single conda environment. -- **Optional annotation**: Variant annotation via SnpEff can be toggled on/off. - ---- - -## Requirements +## Quick start -- **Linux or macOS** (symbolic link support required โ€” Windows is not supported) -- [Conda](https://docs.conda.io/en/latest/miniconda.html) or Miniconda -- Git (optional, for cloning) - -All bioinformatics tools (Nextflow, FastQC, Trim Galore, Cutadapt, BWA, SAMtools, BAMtools, BCFtools, SnpEff) are installed automatically into an isolated conda environment. - ---- - -## Quick Start - -### 1. Clone the repository +Requires Linux or macOS and [conda](https://docs.conda.io/en/miniconda.html). Every bioinformatics tool is installed for you into an isolated environment, pinned to an exact build. ```bash -git clone https://github.com/ozankiratli/PoolSeqFlow.git -cd PoolSeqFlow -chmod +x PoolSeqFlow -``` +# 1. Download the latest release +curl -LO https://github.com/ozankiratli/PoolSeqFlow/releases/latest/download/PoolSeqFlow.tar.gz +tar -xzf PoolSeqFlow.tar.gz +cd PoolSeqFlow-*/ -### 2. Configure the pipeline - -`parameters.config` holds your own paths and settings, so it is **not tracked in git** โ€” a fresh clone ships `parameters.config.template` instead. Create your copy first: - -```bash +# 2. Create your configuration cp parameters.config.template parameters.config -``` - -This is deliberately not done for you: the pipeline will not start without the file, so the settings get read rather than inherited. Now edit `parameters.config` to point to your data: - -```groovy -params { - mainDir = "/path/to/working/directory" // where the pipeline runs (e.g. compute node scratch) - projectDir = "/path/to/permanent/storage" // where outputs are permanently stored (can be a different filesystem) - dataSource = 'Data' // subdirectory of projectDir containing the FASTQs - readPattern = "*_R{1,2}.fq.gz" // glob matching your paired FASTQ files - referenceFile = 'reference.fasta.gz' // reference genome (gzipped FASTA) - gffFile = 'reference.gff.gz' // annotation (gzipped GFF) - poolSize = 50 // number of individuals in pool - diploidy = 2 // ploidy of your organism - annotate = true // run SnpEff annotation (Step 8) -} -``` - -> **Configure through `parameters.config` only.** PoolSeqFlow does not accept command-line parameter overrides, and `./PoolSeqFlow` deliberately rejects any argument beyond a single subcommand. Keeping every setting in the file means a run is fully described by something you can version, diff, and share โ€” and it matches the direction Nextflow is taking on configuration handling. -> -> It also avoids a silent failure mode. Nextflow delivers `--param` values as **strings**, so `--annotate false` sets `annotate` to the string `"false"` โ€” which Groovy evaluates as *true*, leaving annotation switched on with no warning. Written in `parameters.config`, `annotate = false` is a real boolean and behaves as expected. - -`mainDir` and `projectDir` can be the same path if you have a single storage location. They are separated to support environments where compute nodes and permanent storage are on different filesystems โ€” a common constraint in HPC setups. +# then edit it: mainDir, projectDir, readPattern, referenceFile, poolSize, diploidy -Edit `RGTags.csv` to add read group metadata for each sample (see [RG Tag Configuration](#rg-tag-configuration) below). - -### 3. Install the environment - -```bash +# 3. Build and verify the environment ./PoolSeqFlow install -``` - -### 4. Run the pipeline -```bash +# 4. Run โ€” this is also the resume command ./PoolSeqFlow run ``` -Run this again at any point to resume. Every step checks whether its outputs already exist in `projectDir` and skips itself if they do, so an interrupted run picks up where it left off without any extra flag. To force a full re-run, use `./PoolSeqFlow reset` first (see [Resume Logic](#resume-logic)). +Your project directory needs a `Data/` folder of FASTQs, a gzipped reference, and an `RGTags.csv`. Copy `RGTags.csv.template` and fill it in โ€” it is not only metadata, it decides which FASTQ pairs count as one sample and the order your result columns come out in. -### 5. Additional commands +Full walkthrough: [Install](https://ozankiratli.github.io/PoolSeqFlow/getting-started/install/) and [Quick Start](https://ozankiratli.github.io/PoolSeqFlow/getting-started/quick-start/). -| Command | Description | -|---|---| -| `./PoolSeqFlow migrate_config` | Carry an older `parameters.config` onto the current template (see [Upgrading](#upgrading-from-an-earlier-release)) | -| `./PoolSeqFlow clean` | Clean Nextflow work directories | -| `./PoolSeqFlow reset` | Remove all progress and start fresh | -| `./PoolSeqFlow uninstall` | Remove the conda environment | +> **Configure through `parameters.config` only.** PoolSeqFlow does not accept command-line parameter overrides, and `./PoolSeqFlow` deliberately rejects any argument beyond a single subcommand. A run is therefore fully described by a file you can version, diff and publish. It also avoids a silent failure: Nextflow delivers `--param` values as **strings**, so `--annotate false` sets the string `"false"`, which Groovy evaluates as *true*. --- -## Upgrading from an earlier release - -`parameters.config` belongs to you and is never touched by an update, so after pulling a new version it can be missing parameters the newer code expects. **Nothing detects this.** Step 0 verifies successfully, and a later step then fails with a bare: - -``` -.command.sh: line 17: null: command not found -``` - -An absent parameter interpolates as the literal string `null`, which is why the error names no parameter and points at a generated script. - -There is a helper for this: - -```bash -./PoolSeqFlow migrate_config -``` - -It backs your file up, rebuilds it from the current template, carries across every setting -whose parameter still exists, and reports what it kept, what is new, what this release now -computes for itself, and what it dropped. - -**Treat the result as a starting point, not an answer.** Migration can only recognise a -parameter that still exists *and still means the same thing*. A parameter whose behaviour -changed while its value still looks like an ordinary number or string will be carried -across and be silently wrong. Always compare afterwards: - -```bash -diff parameters.config parameters.config.template -``` - -Every release adds parameters, so rebuilding by hand is often the safer choice โ€” and it is -the only way to be certain you have actually looked at the new ones: - -```bash -cp parameters.config parameters.config.bak # keep your settings -cp parameters.config.template parameters.config # start from the current schema -diff parameters.config.bak parameters.config # see what changed, then re-apply yours -``` - -Back the file up *before* pulling, too โ€” it is no longer tracked in git, so an update that removes it upstream can take your copy with it. - -The Nextflow 26 / Trim Galore 2.x release is the one to watch for: it adds `params.software.unzip`, `params.trim_galore.autodetect` and the whole `params.cores` block, none of which exist in an older file. It also renames trimmed-read outputs and derives the SnpEff database name from the GFF filename, so previously completed trimming and annotation steps are redone on the first run after upgrading. - ---- - -## Resume Logic - -PoolSeqFlow implements a custom resume strategy designed for large Pool-seq datasets where intermediate files (BAMs, VCFs) can be tens to hundreds of gigabytes, and where compute nodes and permanent storage are often on separate filesystems. - -The pipeline separates two concepts: - -- **`mainDir`** โ€” the working directory where the pipeline executes (e.g. a compute node's scratch space or a fast local disk). -- **`projectDir`** โ€” the permanent storage location where all outputs are written and kept (e.g. a network-attached archive, a group storage volume, or a different mount point entirely). - -Rather than relying solely on Nextflow's built-in caching โ€” which stores copies of outputs inside `work/` and can consume significant additional disk space โ€” PoolSeqFlow writes outputs directly to `projectDir` and places **symbolic links** in `mainDir` pointing back to those permanent files. This means: - -- **No file duplication**: large BAM and VCF files exist in exactly one place on disk, in `projectDir`. -- **No data movement**: you can run the pipeline on any node that can reach your permanent storage via a symlink, without copying files between filesystems. -- **Automatic step-skipping**: on *every* `./PoolSeqFlow run`, the pipeline checks for the existence of permanent output files in `projectDir`. Any step whose outputs are already present is skipped entirely, regardless of whether the Nextflow `work/` cache is still intact. -- **Resilience across sessions**: the resume logic is filesystem-based, so it survives cluster job timeouts, system reboots, and `work/` directory cleanups that would otherwise invalidate Nextflow's native cache. - -### No `-resume` flag - -This strategy **replaces** Nextflow's `-resume` rather than supplementing it, so the wrapper never passes that flag: - -- `cleanup = true` in `nextflow.config` deletes the task working directories under `work/` once a run completes โ€” only empty hash-prefix folders are left behind. Since `-resume` replays task outputs *from* those directories, there is nothing to reuse. -- Several steps delete their own inputs once the next stage has consumed them (for example, the trimmed reads are removed after clipping). That leaves the upstream task's recorded outputs dangling, which invalidates Nextflow's cache entry anyway. - -`./PoolSeqFlow run` is therefore both "start" and "resume". `./PoolSeqFlow resume` still works as a deprecated alias and prints a notice. To start genuinely from scratch, run `./PoolSeqFlow reset` first โ€” that clears `work/`, the Nextflow metadata, and the `Output/`, `Logs/`, `Reports/` and `Reference/` folders in `projectDir`. - -One consequence worth knowing on HPC: because step-skipping happens *inside* each task rather than before it, a re-run still submits every process to the scheduler. Those jobs exit almost immediately (they test for a file, create a symlink, and copy two log files), but they are real submissions โ€” expect roughly one short job per process per sample. - -`mainDir` and `projectDir` can point to the same path if you have a single unified storage location โ€” the separation is there to gracefully handle the storage constraints common in HPC environments, not to impose them. - -> **Requires Linux or macOS.** Symbolic links behave correctly on both. Windows โ€” including WSL with certain filesystem configurations โ€” is not supported. - ---- - -## Directory Structure - -### Repository - -``` -PoolSeqFlow/ -โ”œโ”€โ”€ bin/ -โ”‚ โ”œโ”€โ”€ createDepthFile.sh # Generate per-site depth files -โ”‚ โ”œโ”€โ”€ depth2freq.awk # Convert depth to allele frequency -โ”‚ โ””โ”€โ”€ MajorAlleleToRef.py # Re-encode VCF with major allele as REF -โ”œโ”€โ”€ install/ -โ”‚ โ”œโ”€โ”€ environment.yml # Conda environment specification -โ”‚ โ””โ”€โ”€ check_install.sh # Dependency verification (./PoolSeqFlow check) -โ”œโ”€โ”€ scripts/ -โ”‚ โ”œโ”€โ”€ 0_verify_environment.nf -โ”‚ โ”œโ”€โ”€ 1_build_dictionaries.nf -โ”‚ โ”œโ”€โ”€ 2_trim_reads.nf -โ”‚ โ”œโ”€โ”€ 3_align.nf -โ”‚ โ”œโ”€โ”€ 4_clean.nf -โ”‚ โ”œโ”€โ”€ 5_reports.nf -โ”‚ โ”œโ”€โ”€ 6_variant_call.nf -โ”‚ โ”œโ”€โ”€ 7_vcf2freq.nf -โ”‚ โ””โ”€โ”€ 8_annotate_variants.nf -โ”œโ”€โ”€ nextflow.config -โ”œโ”€โ”€ parameters.config -โ”œโ”€โ”€ parameters.config.template -โ”œโ”€โ”€ poolseqflow.nf -โ”œโ”€โ”€ RGTags.csv.template -โ””โ”€โ”€ README.md -``` - -### Required project directory layout - -``` -/path/to/project/ โ† mainDir in parameters.config -โ”œโ”€โ”€ Data/ -โ”‚ โ”œโ”€โ”€ Sample1_R1.fastq.gz -โ”‚ โ”œโ”€โ”€ Sample1_R2.fastq.gz -โ”‚ โ””โ”€โ”€ ... -โ”œโ”€โ”€ RGTags.csv -โ”œโ”€โ”€ Ref.fa.gz -โ””โ”€โ”€ Ref.gff.gz -``` - -### Output structure - -``` -/path/to/project/ -โ”œโ”€โ”€ Logs/ -โ”œโ”€โ”€ Reference/ -โ”‚ โ”œโ”€โ”€ Ref.fasta -โ”‚ โ”œโ”€โ”€ Ref.fasta.{amb,ann,bwt,fai,pac,sa} -โ”‚ โ””โ”€โ”€ snpEff/ -โ””โ”€โ”€ Output/ - โ”œโ”€โ”€ Trimmed/ # Trimmed FASTQ files - โ”œโ”€โ”€ Unpaired/ # Discarded unpaired reads - โ”œโ”€โ”€ Aligned/ # Raw BAM files - โ”œโ”€โ”€ Ready/ # Cleaned, indexed BAM files - โ”œโ”€โ”€ VCF/ # Variant calls (raw + annotated) - โ”œโ”€โ”€ Frequencies/ # Allele frequency tables - โ””โ”€โ”€ Reports/ # QC and alignment reports -``` - ---- - -## Configuration - -### Resource configuration - -Two values in `parameters.config` size an entire run: - -```groovy -threads = 8 // cores a single task may use -memory = '24 GB' // memory ceiling for a single task -``` +## Commands -Every tool's thread count is derived from `threads` โ€” do not set the per-tool counts by -hand: - -| `threads` | Trim Galore `--cores` | actual threads | BWA `-t` | cutadapt | FastQC `-t` | SAMtools `-@` | Java GC | -|---|---|---|---|---|---|---|---| -| 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | -| 4 | 1 | 1 | 4 | 4 | 2 | 1 | 2 | -| 8 | 4 | 8 | 8 | 8 | 2 | 1 | 2 | -| 12+ | 8 | 12 | 8 | 8 | 2 | 1 | 2 | - -Two details explain the shape of that table. Trim Galore's `--cores N` actually runs **N+4** threads (N workers, 2 decompressors, a batcher and a writer), so the ladder picks the largest N whose full footprint still fits โ€” which is why 4 cores yields `--cores 1` rather than `--cores 4`. And SAMtools' `-@` counts *additional* threads, so `0` means one core and `1` means two. - -#### How the numbers reach the tools - -Each process declares what it needs with the `cpus` directive and passes that same number to its tool as `task.cpus`, so there is exactly one value per task and nothing can drift: - -```groovy -process Align { - cpus { params.cores.bwa } - script: - """ - bwa mem -t ${task.cpus} ... - """ -} -``` - -| Process | reserves | at `threads = 8` | -|---|---|---| -| `TrimReads` | `params.cores.trimTotal` | 8 | -| `ClipReads` | `params.cores.cutadapt` | 8 | -| `Align` | `params.cores.bwa` | 8 | -| `SortCleanBam` | `params.cores.samtools + 1` | 2 | -| `BuildSnpEffDb`, `AnnotateVariants` | `params.cores.javaGc` | 2 | -| every other step | *(single-threaded)* | 1 | - -This matters for more than bookkeeping: **Nextflow decides how many tasks to run at once by comparing `cpus` against the resources available**, so an under-declared task leads to oversubscription. Overriding `cpus` in a profile automatically changes what the tool is told, because both come from `task.cpus`. - -`TrimReads` is the one place the number is not passed through unchanged. Its reservation is Trim Galore's *footprint*, since `--cores N` really runs N+4 threads, so the script maps back to the worker count: - -```groovy -cpus { params.cores.trimTotal } // 8 at threads = 8 -trim_cores = task.cpus > 4 ? task.cpus - 4 : 1 // -> --cores 4 -``` - -Reserving the worker count instead would understate the task by four threads. The guard covers `--cores 1`, which bypasses the worker pool and is genuinely single-threaded. - -#### `threads` must fit the machine - -Because tasks now reserve what they really use, a request larger than the available cores fails immediately rather than quietly oversubscribing: - -``` -Process requirement exceeds available CPUs -- req: 12; avail: 8 -``` - -Set `threads` to the cores you actually have โ€” on HPC, the size of one node. Note the consequence on a small machine: at `threads = 8`, a single `TrimReads` task reserves all eight, so samples are trimmed one at a time instead of three at once. That is slower in wall-clock than the old behaviour, which ran three concurrently at 12 threads each on 8 cores; it is also the only version that respects the machine. - -#### `resourceLimits` is a ceiling, not an allocation - -`nextflow.config` caps requests using the same two parameters: - -```groovy -process { - resourceLimits = [ memory: params.memory, cpus: params.threads ] -} -``` - -If a task requests more than this, Nextflow reduces the request before submitting it โ€” which prevents a job that no node can satisfy from queueing forever. It does **not** reserve anything and does **not** limit concurrency on its own; that is what `cpus` does. Set `threads` and `memory` to match the node you are running on. - -### SAMtools filter flags - -Default flags in `parameters.config`: - -```groovy -params { - samFlags.filter = "0xF0C" // Remove: unmapped, mate-unmapped, secondary, QC-fail, duplicate, supplementary - samFlags.required = "0x2" // Require: properly paired -} -``` - -| Flag | Value | Effect | -|---|---|---| -| `0x004` | 4 | Exclude unmapped reads | -| `0x008` | 8 | Exclude reads with unmapped mate | -| `0x100` | 256 | Exclude secondary alignments | -| `0x200` | 512 | Exclude reads failing QC | -| `0x400` | 1024 | Exclude PCR/optical duplicates | -| `0x800` | 2048 | Exclude supplementary alignments | - ---- - -## RG Tag Configuration - -Create `RGTags.csv` in your project directory. The `ID` field must match the sample prefix in your FASTQ filenames. - -```csv -ID,SM,LB,DS,FO,PL,PU -Sample1T1Rep1,Sample1T1,Lib1,Pop1_T1_Rep1,FASTQ,ILLUMINA,Unit1 -Sample1T1Rep2,Sample1T1,Lib1,Pop1_T1_Rep2,FASTQ,ILLUMINA,Unit1 -Sample1T2Rep1,Sample1T2,Lib1,Pop1_T2_Rep1,FASTQ,ILLUMINA,Unit1 -Sample1T2Rep2,Sample1T2,Lib1,Pop1_T2_Rep2,FASTQ,ILLUMINA,Unit1 -Sample2T1Rep1,Sample2T1,Lib1,Pop2_T1_Rep1,FASTQ,ILLUMINA,Unit1 -Sample2T1Rep2,Sample2T1,Lib1,Pop2_T1_Rep2,FASTQ,ILLUMINA,Unit1 -Sample2T2Rep1,Sample2T2,Lib1,Pop2_T2_Rep1,FASTQ,ILLUMINA,Unit1 -Sample2T2Rep2,Sample2T2,Lib1,Pop2_T2_Rep2,FASTQ,ILLUMINA,Unit1 -``` - -*This csv will create a vcf file with 4 samples, Sample1T1, Sample1T2, Sample2T1, Sample2T2. (See Below)* - -| Tag | Required | Description | -|---|---|---| -| `ID` | **Yes** | Unique identifier; must match FASTQ filename prefix | -| `SM` | No | Sample / population name | -| `LB` | No | Library identifier | -| `DS` | No | Description | -| `FO` | No | Flow order (typically `FASTQ`) | -| `PL` | No | Platform (e.g., `ILLUMINA`) | -| `PU` | No | Platform unit | -| `CN` | No | Sequencing centre | -| `DT` | No | Run date (ISO8601, e.g., `2024-03-07`) | - -**Every `ID` must appear exactly once.** A row is looked up by `ID` and only the first match is read, so a repeated `ID` would silently discard the later rows and give that sample the wrong tags. Step 0 refuses to run and lists the offending values. - -Editing this file in Excel on Windows saves it with CRLF line endings, which would otherwise put a stray carriage return in the last tag of every row. Step 0 detects that, rewrites the file with Unix line endings, and reports `RGTAGS LINE ENDING CHECK: FIXED`. File permissions and ownership are preserved; if the file cannot be rewritten, the run stops and tells you the command to run. - -### `SM` decides what counts as a sample - -`ID` identifies each FASTQ pair, but **`SM` determines the samples in your variant calls.** BCFtools names VCF columns after `SM`, and any read groups sharing a value are pooled into a single column. The example above does exactly this: eight FASTQ pairs carry four distinct `SM` values, so each pair of replicates is combined and the VCF has four columns. - -| `SM` values in RGTags.csv | Resulting VCF columns | +| Command | Description | |---|---| -| `Sample1`, `Sample2`, `Sample3` | `Sample1` `Sample2` `Sample3` | -| `Population1`, `Population1`, `Sample3` | `Population1` `Sample3` | - -**Give every pool its own `SM`** when you want them analysed separately. This is what most runs want, and it is the safe default. - -**Share an `SM` deliberately** when several FASTQ pairs are really the same biological pool: - -- **One pool sequenced more than once** โ€” split across lanes or runs to reach the depth Pool-seq needs. Each run arrives as its own FASTQ pair, but they describe one set of individuals, and the allele frequencies are only correct once the reads are combined. -- **Technical replicates** of the same library that you want treated as one observation rather than compared with each other. - -Because merging happens at variant calling, it changes the numbers: read depths add together and each frequency is computed across the pooled reads. Leaving one pool split across two `SM` values instead gives you two under-powered estimates of the same thing โ€” which is easy to do by accident, since the FASTQ files look like two ordinary samples. - -### Row order decides column order - -**The order of the rows in `RGTags.csv` is the order of the sample columns** in the VCF and in the frequency tables. Put the rows in whatever order you want to read your results in โ€” treatment before control, or by time point โ€” and the output follows. - -```csv -ID,SM,DS,FO,PL,PU -Sample3,Sample3,Sample3,FASTQ,ILLUMINA,Unit1 # -> first column -Sample1,Sample1,Sample1,FASTQ,ILLUMINA,Unit1 # -> second column -Sample2,Sample2,Sample2,FASTQ,ILLUMINA,Unit1 # -> third column -``` - -When several rows share an `SM`, the merged column appears where the **first** of those rows sits. - -Reordering rows only moves columns; it never changes a value. Nothing else about the file is positional. - -### Editing `RGTags.csv` after a run - -Completed steps are skipped by looking for their output files, not by checking what produced them. So once this file has been consumed, editing it does **not** update anything that already exists โ€” the tags are inside the BAMs, and the column order is inside the VCF. - -Step 0 therefore records the file the first time it is used and compares against that record on every later run. **If it has changed, the run stops before any work happens** and the report tells you which outputs are now stale: - -| What you changed | What it invalidates | Delete and rerun | -|---|---|---| -| A tag value (`SM`, `DS`, โ€ฆ) | The BAMs, and everything called from them | `Output/Ready/`, `Output/VCF/`, `Output/Frequencies/` | -| Row order only | The VCF sample column order | `Output/VCF/`, `Output/Frequencies/` | - -Deleting the affected outputs is what clears the check โ€” the edit becomes the new baseline on the next run. Or discard everything and start over with `./PoolSeqFlow reset`. - -The record lives in `.poolseqflow_rgtags` in your project directory. Line endings and trailing whitespace are ignored when comparing; row order is not. - ---- - -## Step-by-step Description - -### Step 0: Verify Environment - -Checks that all required files and software dependencies are present before the run begins. Produces `Reports/0_verify_environment.txt`. - -It also refuses to continue when something an existing result was built from has since changed โ€” the analysis parameters in `parameters.config`, or `RGTags.csv` (see [Editing `RGTags.csv` after a run](#editing-rgtagscsv-after-a-run)). Because completed steps are skipped by looking for output files, continuing would otherwise mix old and new results in one output folder. - -### Step 1: Build Reference Dictionaries - -Creates index files for BWA, SAMtools (`.fai`), and SnpEff. Output is written to `Reference/`. - -### Step 2: Trim & QC - -1. Runs **FastQC** on raw reads. -2. Runs **Trim Galore** to remove adapters (using sequences from `parameters.config`). -3. Parses FastQC HTML reports to compute the per-cycle A/T and G/C imbalance. -4. Automatically determines the number of bases to hard-clip with **Cutadapt** to bring base-composition ratios within configured thresholds. - -### Step 3: Align - -Aligns trimmed paired reads to the reference genome using **BWA-MEM**. Output: per-sample BAM files in `Aligned/`. - -### Step 4: Clean BAM Files - -Post-processing pipeline: - -1. Name-sort (`samtools sort -n`) -2. Fix mate information (`samtools fixmate -m`) -3. Coordinate-sort (`samtools sort`) -4. Mark and remove duplicates (`samtools markdup`) -5. Add read group tags (`samtools addreplacerg`) -6. Filter alignments (`samtools view -F 0xF0C -f 0x2`) -7. Index (`samtools index`) - -Cleaned BAMs are written to `Ready/`. - -### Step 5: Generate Reports - -Produces alignment statistics (`bamtools stats`) and coverage summaries (`samtools coverage`) for each sample. Written to `Reports/`. - -### Step 6: Variant Calling - -Calls SNPs and indels with **BCFtools mpileup + call**. The resulting multi-sample VCF contains `AD` (allelic depth) and `DP` (total depth) FORMAT fields. Output: `VCF/`. - -### Step 7: VCF โ†’ Allele Frequency Tables - -1. Re-encode the VCF with the major allele as REF using `MajorAlleleToRef.py`. -2. Update `DP` from `AD` counts. -3. Apply a **ploidy- and pool-size-aware minimum frequency filter**. Variants with allele frequency below - -$$f_{\min} = \frac{1}{2 \times ploidy \times poolSize}$$ - -    are removed, as they cannot represent even a single genome copy in the pool. - -4. Apply depth and quality filters. -5. Split into SNP and INDEL VCFs. -6. Convert to tab-separated allele frequency tables written to `Frequencies/`. - -### Step 8: Annotate Variants *(optional)* +| `./PoolSeqFlow install` | Create the conda environment, then verify it | +| `./PoolSeqFlow check` | Verify an existing installation โ€” tools, helpers, config | +| `./PoolSeqFlow run` | Start โ€” or resume โ€” the pipeline | +| `./PoolSeqFlow migrate_config` | Carry an older `parameters.config` onto the current template | +| `./PoolSeqFlow clean` | Remove Nextflow work directories | +| `./PoolSeqFlow reset` | Remove all progress and start fresh (typed confirmation required) | +| `./PoolSeqFlow version` | Print the installed version | +| `./PoolSeqFlow cite` | Print how to cite this copy, and which DOI to use | +| `./PoolSeqFlow uninstall` | Remove the conda environment | -Annotates the variant VCF with **SnpEff** using the reference GFF. Enable with `params.annotate = true` in `parameters.config`. +There is no `-resume` flag. Every step checks whether its outputs already exist in permanent storage and skips itself if they do, so `run` both starts and resumes โ€” and that survives job timeouts, reboots and `work/` cleanups. [Why โ†’](https://ozankiratli.github.io/PoolSeqFlow/pipeline/resume/) --- -## Troubleshooting +## Documentation -| Problem | Solution | +| | | |---|---| -| Environment creation fails | `conda update -n base conda`, then retry `./PoolSeqFlow install` | -| Missing dependencies after install | `conda activate PoolSeqFlow` before running | -| Pipeline errors | Check `.nextflow.log` for the failing process | -| `null: command not found`, or a parameter appearing as `null` | Your `parameters.config` predates the installed version โ€” rebuild it from `parameters.config.template` (see [Upgrading](#upgrading-from-an-earlier-release)) | -| A re-run skips too many steps | Steps skip themselves when their outputs exist in `projectDir`. Delete the stale outputs, or use `./PoolSeqFlow reset` to start over | -| `-resume` appears to do nothing | Correct โ€” PoolSeqFlow does not use Nextflow's `-resume`. `./PoolSeqFlow run` already resumes (see [Resume Logic](#resume-logic)) | -| Symbolic link errors | Confirm you are on Linux or macOS, not Windows | +| [When to use it](https://ozankiratli.github.io/PoolSeqFlow/concepts/) | What pooling buys and costs, and what the pipeline assumes about your design | +| [Design decisions](https://ozankiratli.github.io/PoolSeqFlow/concepts/design-decisions/) | Why configuration is a file, why resume is filesystem-based, what each choice costs | +| [The filter chain](https://ozankiratli.github.io/PoolSeqFlow/concepts/filter-chain/) | All eight filters in order, what each removes, and how to tune them | +| [Interpreting results](https://ozankiratli.github.io/PoolSeqFlow/concepts/interpreting-results/) | The frequency table format, and the mistakes that are easy to make reading it | +| [Configuration](https://ozankiratli.github.io/PoolSeqFlow/configuration/) | Every parameter, sorted by whether it changes your results | +| [Read groups](https://ozankiratli.github.io/PoolSeqFlow/configuration/read-groups/) | `RGTags.csv`, and why `SM` decides what counts as a sample | +| [Pipeline steps](https://ozankiratli.github.io/PoolSeqFlow/pipeline/steps/) | Steps 0โ€“8 in detail | +| [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/) | Your `parameters.config` is never touched by an update โ€” read this first | +| [Troubleshooting](https://ozankiratli.github.io/PoolSeqFlow/reference/troubleshooting/) | Errors by symptom | --- @@ -559,23 +124,16 @@ Your installed copy prints its own citation, with the version filled in: ./PoolSeqFlow cite ``` -**Cite the version you actually ran, not the newest one.** Zenodo issues a separate DOI for every release, and results depend on which release produced them โ€” filters, defaults and parameter names have all changed between versions. A paper citing the current release for numbers produced by an older one is describing a method it did not use. - -- [10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) is the **all-versions** DOI. It always resolves to the newest release โ€” use it to refer to the software in general. -- For results, open that record, pick your version from the **Versions** list, and use its DOI. +**Cite the version you actually ran, not the newest one.** Zenodo issues a separate DOI for every release, and results depend on which release produced them โ€” filters, defaults and parameter names have all changed between versions. Step 0 records the versions that have run in a project in `.poolseqflow_versions`, mirrored into `Output/run_parameters.txt`. -> Kiratli, O. L. Z. (2026). *PoolSeqFlow: A Nextflow pipeline for allele frequency analysis from pooled Illumina sequencing data* \(Version *x.y.z*\) \[Computer Software\]. https://doi.org/10.5281/zenodo.19245611 +[10.5281/zenodo.19245611](https://doi.org/10.5281/zenodo.19245611) is the **all-versions** DOI: it always resolves to the newest release. Use it to refer to the software in general, and a version DOI when reporting results. [Details โ†’](https://ozankiratli.github.io/PoolSeqFlow/reference/citation/) --- ## License -This project is licensed under the [Apache 2.0 License](LICENSE). - ---- +Apache 2.0 โ€” see [LICENSE](LICENSE). The tools PoolSeqFlow invokes carry their own licenses. ## Contact -**Ozan L. Z. Kiratli** -GitHub: [@ozankiratli](https://github.com/ozankiratli) -Issues: [https://github.com/ozankiratli/PoolSeqFlow/issues](https://github.com/ozankiratli/PoolSeqFlow/issues) +**Ozan L. Z. Kiratli** ยท [@ozankiratli](https://github.com/ozankiratli) ยท [Issues](https://github.com/ozankiratli/PoolSeqFlow/issues) From f9b3437fa9ac274685c251b39a5facf9cdde7040 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Sun, 16 Aug 2026 19:51:36 -0400 Subject: [PATCH 2/3] Version check is added. --- .github/workflows/release.yml | 10 ++++++++++ PoolSeqFlow | 9 ++++++--- dev/scripts/bump-version.sh | 12 +++++++++++- docs/concepts/design-decisions.md | 1 + docs/pipeline/directories.md | 1 + docs/pipeline/steps.md | 1 + docs/reference/citation.md | 4 +++- nextflow.config | 21 +++++++++++++++++++++ scripts/0_verify_environment.nf | 27 +++++++++++++++++++++++++++ 9 files changed, 81 insertions(+), 5 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e44f024..4723c24 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -56,6 +56,16 @@ jobs: exit 1 fi + # The version lives in nextflow.config's manifest too - that is what Nextflow + # reports and what step 0 records alongside a project's outputs, so a stale + # value there would misattribute results to the wrong release. + nfversion=$(sed -n "s/^[[:space:]]*version[[:space:]]*=[[:space:]]*'\(.*\)'.*/\1/p" nextflow.config) + if [ "$nfversion" != "$declared" ]; then + echo "./PoolSeqFlow says VERSION=\"$declared\" but nextflow.config's" >&2 + echo " manifest says version = '$nfversion'" >&2 + exit 1 + fi + # On a tag, the tag has to match. On workflow_dispatch there is no tag, # and the run builds and verifies the archive without publishing. if [ "${GITHUB_REF_TYPE:-}" = "tag" ]; then diff --git a/PoolSeqFlow b/PoolSeqFlow index e506d5a..faf1ca0 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -147,6 +147,7 @@ case $COMMAND in echo " $STORAGE/Reference - indices and the SnpEff database" echo " $STORAGE/.poolseqflow_params" echo " $STORAGE/.poolseqflow_rgtags" + echo " $STORAGE/.poolseqflow_versions" [ -n "$WORKDIR" ] && echo " $WORKDIR" echo "" echo "Your raw sequencing data in $DATADIR is not touched." @@ -166,9 +167,11 @@ case $COMMAND in if [ -n "$WORKDIR" ]; then rm -rf "$WORKDIR"; fi rm -rf "$STORAGE"/Reports "$STORAGE"/Logs "$STORAGE"/Output \ "$STORAGE"/Reference "$STORAGE"/PoolSeqFlow_* - # These record the parameters and RG tags behind outputs that no longer exist; - # leaving them would make the next run fail its checks for no reason. - rm -f "$STORAGE"/.poolseqflow_params "$STORAGE"/.poolseqflow_rgtags + # These record the parameters, RG tags and pipeline versions behind outputs that + # no longer exist; leaving them would make the next run fail its checks for no + # reason, and would attribute fresh results to an earlier release. + rm -f "$STORAGE"/.poolseqflow_params "$STORAGE"/.poolseqflow_rgtags \ + "$STORAGE"/.poolseqflow_versions echo "Reset complete." ;; version) diff --git a/dev/scripts/bump-version.sh b/dev/scripts/bump-version.sh index 5af7850..f994789 100755 --- a/dev/scripts/bump-version.sh +++ b/dev/scripts/bump-version.sh @@ -70,10 +70,20 @@ awk -v entry="$ENTRY" ' sed -i -E "s|^# Version: .*|# Version: $NEW|; s|^VERSION=\".*\"|VERSION=\"$NEW\"|" "$MAIN" +# The version also lives in nextflow.config's manifest, which is what Nextflow reports and +# what step 0 records alongside a project's outputs. release.yml refuses to publish if it +# disagrees with $MAIN, so it has to move at the same time. +NFCONFIG="nextflow.config" +[ -f "$NFCONFIG" ] || { echo "ERROR: $NFCONFIG not found" >&2; exit 1; } +sed -i -E "s|^(\s*version\s*=\s*)'.*'|\1'$NEW'|" "$NFCONFIG" +grep -q "version *= *'$NEW'" "$NFCONFIG" || { + echo "ERROR: could not update the manifest version in $NFCONFIG" >&2; exit 1; } + echo "$CURRENT -> $NEW" echo " $MAIN : $(grep -c "$NEW" "$MAIN") references updated" +echo " $NFCONFIG : manifest version updated" echo " $LOG : $(printf '%s\n' "$COMMITS" | wc -l) commits since ${LAST_TAG:-start}" echo echo "Review, then:" -echo " git add $MAIN $LOG && git commit -m 'Version bump $NEW'" +echo " git add $MAIN $NFCONFIG $LOG && git commit -m 'Version bump $NEW'" echo " git tag v$NEW" diff --git a/docs/concepts/design-decisions.md b/docs/concepts/design-decisions.md index 59c8dc3..b18c12e 100644 --- a/docs/concepts/design-decisions.md +++ b/docs/concepts/design-decisions.md @@ -97,6 +97,7 @@ Two records are kept in the project directory: | Record | Covers | Written by | |---|---|---| | `.poolseqflow_params` | Analysis-affecting parameters, mirrored to a read-only `Output/run_parameters.txt` | Step 0 | +| `.poolseqflow_versions` | Every pipeline version that has run here. Recorded only โ€” a version change never stops a run | Step 0 | | `.poolseqflow_rgtags` | The consumed `RGTags.csv`, ignoring line endings and trailing whitespace | Step 0 | Path, resource and software parameters are excluded โ€” they change where and how fast the work happens, not what the answer is. Anything added in a later release counts as analysis-affecting until decided otherwise, which is the conservative direction to err in. diff --git a/docs/pipeline/directories.md b/docs/pipeline/directories.md index 7e63614..d420062 100644 --- a/docs/pipeline/directories.md +++ b/docs/pipeline/directories.md @@ -57,6 +57,7 @@ Both reference files must be gzipped. The pipeline decompresses them into `Refer /path/to/project/ โ”œโ”€โ”€ .poolseqflow_params # Analysis parameters behind these outputs โ”œโ”€โ”€ .poolseqflow_rgtags # RGTags.csv as consumed +โ”œโ”€โ”€ .poolseqflow_versions # Pipeline versions that have run here, oldest first โ”œโ”€โ”€ Logs/ # Per-step .log and .err, mirrored from every task โ”œโ”€โ”€ Reference/ โ”‚ โ”œโ”€โ”€ reference.fasta diff --git a/docs/pipeline/steps.md b/docs/pipeline/steps.md index 8aeb05d..35ae79d 100644 --- a/docs/pipeline/steps.md +++ b/docs/pipeline/steps.md @@ -39,6 +39,7 @@ The gate for everything else. It runs sixteen checks and writes `Output/Reports/ |---|---| | `.poolseqflow_params` | Analysis-affecting parameters, mirrored read-only to `Output/run_parameters.txt` | | `.poolseqflow_rgtags` | The `RGTags.csv` as consumed; line endings and trailing whitespace ignored | +| `.poolseqflow_versions` | Every pipeline version that has run in this project, with the date. **Recorded, never enforced** | Path, resource and software parameters are excluded โ€” they change where and how fast work happens, not what the answer is. diff --git a/docs/reference/citation.md b/docs/reference/citation.md index 1a52f06..5845aa7 100644 --- a/docs/reference/citation.md +++ b/docs/reference/citation.md @@ -23,7 +23,9 @@ Zenodo issues **two kinds of DOI**, and the difference matters. Results depend on which release produced them. Filters, defaults and parameter names have all changed between versions โ€” `vcffilter.minDP` went from having no effect to removing whole sites, and sample column ordering changed in 2.1.1. A paper citing the current release for numbers produced by an older one is describing a method it did not use. - Find your version with `./PoolSeqFlow version`, then open the [all-versions record](https://doi.org/10.5281/zenodo.19245611) and pick that version from the **Versions** list to get its DOI. + Find the version that produced a given set of results in that project's `Output/run_parameters.txt`, which lists every release that has run there โ€” `./PoolSeqFlow version` tells you only what is installed now, which is not the same thing once you have upgraded. Then open the [all-versions record](https://doi.org/10.5281/zenodo.19245611) and pick that version from the **Versions** list to get its DOI. + + If more than one version is listed, the outputs were not all produced by the same release: completed steps are not redone on upgrade. Say so in your methods, or `./PoolSeqFlow reset` and re-run under one version. ## Reference diff --git a/nextflow.config b/nextflow.config index 57196fb..4a94c83 100644 --- a/nextflow.config +++ b/nextflow.config @@ -1,5 +1,26 @@ includeConfig 'parameters.config' +// Pipeline identity. `version` is what Nextflow itself reports - in `nextflow info`, in +// the execution report, and as `workflow.manifest.version` - and step 0 records it in each +// project so the version behind a set of outputs can be recovered later. +// +// It is one of three places the version lives, alongside `VERSION=` and the `# Version:` +// header comment in ./PoolSeqFlow. dev/scripts/bump-version.sh updates all three, and +// .github/workflows/release.yml refuses to publish if any of them disagree. +manifest { + name = 'PoolSeqFlow' + author = 'Ozan L. Z. Kiratli' + description = 'Allele frequency analysis from pooled Illumina sequencing data' + homePage = 'https://ozankiratli.github.io/PoolSeqFlow/' + mainScript = 'poolseqflow.nf' + license = 'Apache-2.0' + doi = '10.5281/zenodo.19245611' + version = '2.2.0' + // Enforced by Nextflow, so a system install too old for the pipeline fails at once + // with a clear message rather than partway through in a confusing one. + nextflowVersion = '>=26.04.0' +} + conda.enabled = true docker.enabled = false diff --git a/scripts/0_verify_environment.nf b/scripts/0_verify_environment.nf index da916a7..2b685b1 100644 --- a/scripts/0_verify_environment.nf +++ b/scripts/0_verify_environment.nf @@ -538,6 +538,8 @@ process CheckRunParameters { manifest = analysisParams() stored = "${params.projectDir}/.poolseqflow_params" readable = "${params.dir.outputs}/run_parameters.txt" + versions = "${params.projectDir}/.poolseqflow_versions" + release = workflow.manifest.version ?: 'unknown' dir_log = "${params.dir.logs}/0_verify_environment/s7_CheckRunParameters" """ REPORTFILE="verify_environment.txt" @@ -580,6 +582,27 @@ CURRENT_PARAMS STATUS="FAIL" fi + # Which release produced these outputs. Recorded, never enforced: most releases do + # not change results, so a version change must not invalidate outputs the way a + # parameter change does. But "cite the version you ran" is unanswerable if nothing + # writes it down, and a project that several versions have touched is worth knowing + # about - the file is append-only for exactly that reason. + if [ ! -f "${versions}" ]; then + mkdir -p "\$(dirname "${versions}")" + printf '%s\\t%s\\n' "${release}" "\$(date -u '+%Y-%m-%d')" > "${versions}" + log_message "PIPELINE VERSION: ${release} - first run in this project" + elif [ "\$(tail -n 1 "${versions}" | cut -f1)" = "${release}" ]; then + log_message "PIPELINE VERSION: ${release}" + else + PREVIOUS=\$(tail -n 1 "${versions}" | cut -f1) + printf '%s\\t%s\\n' "${release}" "\$(date -u '+%Y-%m-%d')" >> "${versions}" + log_message "PIPELINE VERSION: ${release} - earlier runs here used \$PREVIOUS" + log_message "PIPELINE VERSION: Outputs already present were produced by the earlier" + log_message "PIPELINE VERSION: release, and completed steps are not redone. Cite the" + log_message "PIPELINE VERSION: version that produced the results you report." + log_message "PIPELINE VERSION: Full history: ${versions}" + fi + # Publish a readable copy next to the results. ${stored} stays the file the check # compares against; this one exists so the settings behind a set of outputs can be # read without picking through parameters.config. Read-only, because editing it @@ -591,6 +614,10 @@ CURRENT_PARAMS echo "# PoolSeqFlow analysis parameters for the outputs in ${params.dir.outputs}" echo "# Generated \$(date -u '+%Y-%m-%d %H:%M:%S UTC') - read-only; edit parameters.config instead." echo "#" + echo "# Pipeline version(s) that have run in this project, oldest first." + echo "# More than one means these outputs were not all produced by the same release." + sed 's/^/# /' "${versions}" + echo "#" cat current_params.txt } > "${readable}" chmod 444 "${readable}" From 85fbe74e446acf58a17641bba6ef1ef11393a7d4 Mon Sep 17 00:00:00 2001 From: Ozan Kiratli Date: Sun, 16 Aug 2026 20:09:54 -0400 Subject: [PATCH 3/3] Merge checks added --- .github/workflows/ci.yml | 96 ++++++++++++++++++ .github/workflows/release.yml | 63 +----------- .../MajorAlleleToRef.cpython-311.pyc | Bin 0 -> 8391 bytes dev/scripts/verify-archive.sh | 73 +++++++++++++ 4 files changed, 173 insertions(+), 59 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 bin/__pycache__/MajorAlleleToRef.cpython-311.pyc create mode 100755 dev/scripts/verify-archive.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..115712f --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,96 @@ +name: Checks + +# The gate on merging into main, and an early warning on dev. +# +# Everything here is static: nothing runs the pipeline or needs sequencing data. +# It catches the class of breakage that would otherwise only surface at tag time +# (release.yml) or on someone else's machine - a syntax error in a helper, a +# version that drifted between the three places it lives, a template that no +# longer parses, an archive missing a file a run needs. +# +# It deliberately does NOT run on push to main: by then it is too late to gate +# anything, and main is only written by merges that already passed here. + +on: + pull_request: + branches: [main] + push: + branches: [dev] + workflow_dispatch: + +permissions: + contents: read + +env: + # Pinned to the version install/environment.yml ships, so CI checks the same + # Nextflow a user will actually run. + NXF_VER: 26.04.6 + +concurrency: + group: checks-${{ github.ref }} + cancel-in-progress: true + +jobs: + checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + with: + # verify-archive.sh builds from a git ref, so it needs real history. + fetch-depth: 0 + + - name: The version agrees with itself + run: | + set -euo pipefail + declared=$(sed -n 's/^VERSION="\(.*\)"$/\1/p' PoolSeqFlow) + header=$(sed -n 's/^# Version: \(.*\)$/\1/p' PoolSeqFlow) + manifest=$(sed -n "s/^[[:space:]]*version[[:space:]]*=[[:space:]]*'\(.*\)'.*/\1/p" nextflow.config) + echo " ./PoolSeqFlow VERSION= : $declared" + echo " ./PoolSeqFlow header : $header" + echo " nextflow.config : $manifest" + [ -n "$declared" ] || { echo "no VERSION= in ./PoolSeqFlow" >&2; exit 1; } + [ "$header" = "$declared" ] && [ "$manifest" = "$declared" ] || { + echo "the three version strings disagree" >&2; exit 1; } + + - name: Shell scripts parse + run: | + set -euo pipefail + for s in PoolSeqFlow bin/*.sh install/*.sh dev/scripts/*.sh; do + bash -n "$s" && echo " ok $s" + done + + - name: Python and awk helpers parse + run: | + set -euo pipefail + python3 -m py_compile bin/MajorAlleleToRef.py && echo " ok bin/MajorAlleleToRef.py" + # Running it on empty input executes BEGIN and nothing else, which is + # enough to reject a syntax error without needing a VCF. + awk -f bin/depth2freq.awk /dev/null && echo " ok bin/depth2freq.awk" + + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: "17" + + - name: Install Nextflow + run: | + curl -s https://get.nextflow.io | bash + sudo mv nextflow /usr/local/bin/ + nextflow -version + + - name: Pipeline lints without errors + run: nextflow lint poolseqflow.nf scripts/*.nf + + # The failure this catches is the one users hit as `null: command not + # found` partway through a run - a parameter the template no longer + # resolves. Cheap to check, invisible otherwise until it bites. + - name: The config template parses + run: | + set -euo pipefail + cp parameters.config.template parameters.config + nextflow config -flat > /dev/null + echo " ok parameters.config.template resolves" + rm -f parameters.config + + - name: The release archive is a working copy + run: dev/scripts/verify-archive.sh diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4723c24..9e22e24 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -89,13 +89,14 @@ jobs: echo "name=PoolSeqFlow-$declared" >> "$GITHUB_OUTPUT" echo "Building PoolSeqFlow-$declared" - - name: Build the archive + # Same script ci.yml runs on every pull request, so what is published here + # has already been checked in review rather than only at tag time. + - name: Build and verify the archive env: NAME: ${{ steps.version.outputs.name }} run: | set -euo pipefail - mkdir -p dist - git archive --format=tar.gz --prefix="${NAME}/" -o "dist/${NAME}.tar.gz" HEAD + dev/scripts/verify-archive.sh HEAD dist # The same archive under a stable name. GitHub's # /releases/latest/download/ only resolves a filename it can @@ -107,62 +108,6 @@ jobs: ( cd dist && sha256sum ./*.tar.gz > SHA256SUMS ) ls -lh dist/ - - name: Verify the archive is a working copy of the pipeline - env: - NAME: ${{ steps.version.outputs.name }} - VERSION: ${{ steps.version.outputs.version }} - run: | - set -euo pipefail - tmp=$(mktemp -d) - tar -xzf "dist/${NAME}.tar.gz" -C "$tmp" - root="$tmp/${NAME}" - - fail() { echo "ARCHIVE CHECK FAILED: $*" >&2; exit 1; } - - # Everything a run needs. - for f in PoolSeqFlow poolseqflow.nf nextflow.config \ - parameters.config.template RGTags.csv.template \ - install/environment.yml install/check_install.sh \ - LICENSE README.md CHANGELOG.md; do - [ -f "$root/$f" ] || fail "missing from archive: $f" - done - - # All nine step modules, not however many happened to be committed. - steps_found=$(find "$root/scripts" -name '*.nf' | wc -l) - [ "$steps_found" -eq 9 ] || fail "expected 9 scripts/*.nf, found $steps_found" - - # The helpers the process scripts call by bare name via nextflow.config's PATH. - for f in atomic_mv.sh config_migrate.sh createDepthFile.sh \ - depth2freq.awk filterFalsePositives.sh MajorAlleleToRef.py; do - [ -f "$root/bin/$f" ] || fail "missing from archive: bin/$f" - done - - # Repository furniture must NOT ship. This is the other half of - # .gitattributes: if an export-ignore is dropped, this catches it. - for f in docs .github mkdocs.yml .gitignore .gitattributes dev Project; do - [ ! -e "$root/$f" ] || fail "should have been export-ignored: $f" - done - - # The executable bit is the whole reason for git archive over tar. - [ -x "$root/PoolSeqFlow" ] || fail "./PoolSeqFlow is not executable" - for s in "$root"/bin/*; do - [ -x "$s" ] || fail "$(basename "$s") is not executable" - done - - # Syntax-check every shell script that ships. - bash -n "$root/PoolSeqFlow" || fail "./PoolSeqFlow has a syntax error" - for s in "$root"/bin/*.sh "$root"/install/*.sh; do - bash -n "$s" || fail "$(basename "$s") has a syntax error" - done - - # The wrapper cannot simply be run here: it calls `conda shell.bash hook` - # at the top under `set -e`, and the runner has no conda. Check the - # version it would report instead. - grep -q "^VERSION=\"${VERSION}\"$" "$root/PoolSeqFlow" \ - || fail "extracted wrapper does not report ${VERSION}" - - echo "Archive verified: $(du -h "dist/${NAME}.tar.gz" | cut -f1)" - - name: Publish the release if: github.ref_type == 'tag' uses: softprops/action-gh-release@v3 diff --git a/bin/__pycache__/MajorAlleleToRef.cpython-311.pyc b/bin/__pycache__/MajorAlleleToRef.cpython-311.pyc new file mode 100644 index 0000000000000000000000000000000000000000..fd15c1d4c368dfdbb307605ad946cb26c27a4369 GIT binary patch literal 8391 zcmb_AZEO=)wlf~v8QXE1luLzo70)74OeG zclO+>(*cpG!fI&#LWhH8U$TGHb-0#Vn)H3ETWxT1H)+jJ%B&TzhkQfBAH*Y zfQMm%QHzOVw>p6;;$%(H$|b2^g8%^9prscp5lTi)F*0U~nQu1TY>ZiAr7>%)ELKh% zuVaFB11cq2Dm3ZHFSfx9)2bQf*UqqLXY^xQ{$e)4Mi)^T$6-v;#7}hv%2C+WO($>3 zZw}+FMg8(Zf7Yp8O*@J^$8fszI{DbDJ^7o>v(3eOpv_r5)y+1AT(gZE$YK>|t7wo> zxJd;bViobmcvHN2jG)V|le5ihOxUtoPvNJj&mm zm{riv+z`q(IHygk=Cg z_yVh9hjlR)+6`vdwX{Px*daSz{hC?T>&!C58$j1|4xnlZsP7-?6T?)(Pc;qj11)cq;WjSd;6tTBY5qP(Rzy1FQ)4 zPn!6e0#6#vtJBWWj)Z!_u2L|3Vxa3_WKc)7$*KaiLXY0G{yh}xwN$ku@BAPSNN(HM>Hf|DX(f!u$fUaK)!@=*}`p5*{8~wCW#<7sag>_+^ zep+{xIEPK+ICc)aH}duCVbjDgwh{g?=5C1o)=WV;$%I*&VW~5H1C%!qU;+$B@nI_H zz06SjC58%yIi4D0nJbe_h!0$&SjJ0d5!LrBGsdt?$j5Y=ReHVbIJZabA?j1UG0&Jk zz)$p*(?hRt*DKuQ%f|sAWCQ#w z;s5P6XzpU;n1EfVh~e4@M0*mqb+wH_muW~4&pr}_h7yY5_ zN4*nawpSI-Uo(BoegQNw`SZ7Zl?DGb= z{m{uxp@2Hq2+a&8R-ew)Wb0=MatWX0T^L@%#mOXjD8pI)kyuQbS-#TxPCm_MJa39uPhwvM2VF7%t$m zdb)^%%c|FTS|&ien~xHRkaJd-gqdp##GxRsM|s_Vc{*)m0O!6Fq(Z3jWRcrukpw{v z3|u7O=iLSt`9KyKF-7kU@SH-xSt?~*nB^JT;}6k(AH#8o1&W%P$w-ey9_vS_Lpx04 z4nqT?6KnWPUG$0vo*Y#)WJf7H>s*vm7%0pBxqQ?H?`_9vEMG1Jm!uKe;-u`Is zP?#DxcJgTNsqMY})R>cguG#Dm!57(k@PeKKq^hqd-R|9_WW!6#T0n*K}lvlkI^LWA_v zHIS+E&B=jO*DnY2fXt02I=6Se!p&~Gq7O%ykYeU|FUxb&e*ThTm}dPv19qBc{Sg)s zgzKbQWyQdHL*tA>1Yk-D%R~ZRAEW3e!v2s#1icZ3fH}}i1{EC(Xx!vQ#Sme^??e9@ zr|6lheqJ$vZD+13CMGl)WLPgOW%Nd16SRA?Lgs=R#ZnZqC`Nw>#WYF@0M(9lEX)SI zD4+qm#3{yc9$j-q4~7WHWuXS&B{m#Xh)5Wmi=Xx=`YV&(fKmb!c!q@`Pz}Q{LrN*h z*bGi7K}JJ;6=J4QgrShAqdF;z+RMH-q5&OIjC4d@-UJgf)WDCa1_&mpVV|PT%Px>{ zdbaF60%w;sR%S#l(`S*#QoR9-ZVQXP=~PW&Dd=&?-cjFE{!K0FrB$5Ydtcd6AcelQ zx!-{*f2Woi?4LHwTo%8SAS6?RY-(7+dJL^e;}d=A-xpj``+m87zY1DxOV>8Ut?3GT z+^9A$s_Is-U8dHTsEPNd%PW8O@y(ADoK)_T%UvrNVRgK))!emyYE96R%_ZAhqRo}I z)xNMh?v8vqa{K(9^Uv*VOZK*uUb1({_Kx^K`oxD~=}`Q@+&kcthm$QSeQJE3U-)ow zdc}yf%_|bnFvBs8P9eCnO z1*NS$^41=yWuM%#PwX2lRHsLBdSs_3J_Oro+?MKUv-9dS7bqliT|ws&Dc1v*B;f{`u@* z&z$?~$OX|eA)UD_pSdg@4ai3W(t&_f8I&u7@qXa6+L5T2?5(oBH9oLhX}{B*a3@Do z-O179==>?Ea*tfOC*J>}yg@8)NmFh2tzTJFK8e~PQ#*tX=@mO3Rlc7UIpGp4)uECj{wBTy4g?Mo5- z=!w&BgaTr69LDorcd-V#QthBzI~X5KtG4@@32eCJJSjU*#)n?iHN=lBJGUf1lAPVL zvwH<2t?#PMZPOe+*PYH8PcWMEA3-Cz24sNQsH~?@R7J+1)pH zI58zvZOI^{-JQ9yt^T=f^O9|I@}2v;zuKKTEV*{cu3eICw`|)jsuh^FR{iYg&7+By zU${Sar)~B*F432)Nx7c3KWv}hol)5Zdhy`1Q{wQ*3ZZK)dts}8Y0$Mcyu@_%&VQih zMUx94^(_FgwgSZ3x`G;YxN;eXYrJBC%dcK{wRwqb!9P$_fUVa9OR^q#73@u^sncFd zlG_P}>~6)eDk3#hGSp-E%iE>U|4Wbl7-sqgBcT5cfup`&?N{lK)s}oyk3;K&iTMIa z_)I$f3`FAE6#PW~lZVD1>$D$1G58!_KT%E-Jc|3Wh1NedXj6Uo#xsqYnI%79CP|xU zGi`ZXdJhcyBNV5oMO`aD%0ba{RSFv~PE8B=M4kgJ*GVL;HAPE{bMpCABZf`nQ@gaa znC{Opy6j6rOC1uHMG11iM&HXWnlE0AfE#d>Eezm;d;E#C}c4bR3?c$lMQ&{k>u)hk;imQ@mkk*2f1^T-% z9>o{g(t@$uW5fn>spN}JT3B^X>v3@oR)`TF5t+46ZyKM*GU+vU)lhW*$?r*zX;NeB z=ENZ3VWAn$2SF{lO5r|YTF@cC;&PJ99vwT$+EUN36 zikdn)rWp2Om}5{}9A+60&rUMYy5lI4r6O#2oQ2E;d<|z}Jy;x8FnWDx8!WOU(M<SiD7Q@^z3h0z%K~R*kW7U5 zJH#JHLslvE(zFNOsxbCl;6gF0m=aiUUddmmzp$szeUR02ULmdkjV~_ z>`-5Hz>8yISqD^!?3Bq)k?dU28w|Ua$yT(zdHUDk{La#r@>@NywrSULX@$6Hr&QW4 zmv)P}xC;|;J=vB zVf4MD>J*w1@&*}?CzJ9(NmNzuAVnJt(~|*aKRXRDr;+VcP9$g|FBH>8u$(jC72CfgxsJ|5g L)7Jr5{b>I`Qb4}J literal 0 HcmV?d00001 diff --git a/dev/scripts/verify-archive.sh b/dev/scripts/verify-archive.sh new file mode 100755 index 0000000..0ba111e --- /dev/null +++ b/dev/scripts/verify-archive.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# +# Build the release archive from a git ref and check it is a working copy of the +# pipeline. Used by both .github/workflows/ci.yml (on every pull request, so a +# problem surfaces in review) and release.yml (before publishing). +# +# Usage: dev/scripts/verify-archive.sh [ref] [outdir] +# defaults: HEAD dist +# +# Prints the path of the archive it built on success. +# +# Kept here rather than inline in the workflows so the two cannot drift, and so +# it can be run locally before pushing. dev/ carries export-ignore, so this file +# never ships to a user. + +set -euo pipefail + +REF="${1:-HEAD}" +OUT="${2:-dist}" + +fail() { echo "ARCHIVE CHECK FAILED: $*" >&2; exit 1; } + +version=$(git show "${REF}:PoolSeqFlow" | sed -n 's/^VERSION="\(.*\)"$/\1/p') +[ -n "$version" ] || fail "no VERSION= line in ./PoolSeqFlow at ${REF}" + +name="PoolSeqFlow-${version}" +mkdir -p "$OUT" +git archive --format=tar.gz --prefix="${name}/" -o "${OUT}/${name}.tar.gz" "$REF" + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +tar -xzf "${OUT}/${name}.tar.gz" -C "$tmp" +root="${tmp}/${name}" + +# Everything a run needs. +for f in PoolSeqFlow poolseqflow.nf nextflow.config \ + parameters.config.template RGTags.csv.template \ + install/environment.yml install/check_install.sh \ + LICENSE README.md CHANGELOG.md; do + [ -f "$root/$f" ] || fail "missing from archive: $f" +done + +# All nine step modules, not however many happened to be committed. +steps_found=$(find "$root/scripts" -name '*.nf' | wc -l) +[ "$steps_found" -eq 9 ] || fail "expected 9 scripts/*.nf, found $steps_found" + +# The helpers the process scripts call by bare name via nextflow.config's PATH. +for f in atomic_mv.sh config_migrate.sh createDepthFile.sh \ + depth2freq.awk filterFalsePositives.sh MajorAlleleToRef.py; do + [ -f "$root/bin/$f" ] || fail "missing from archive: bin/$f" +done + +# Repository furniture must NOT ship. This is the other half of .gitattributes: +# if an export-ignore is dropped, this catches it. +for f in docs .github mkdocs.yml .gitignore .gitattributes dev Project; do + [ ! -e "$root/$f" ] || fail "should have been export-ignored: $f" +done + +# The executable bit is the whole reason for git archive over tar. +[ -x "$root/PoolSeqFlow" ] || fail "./PoolSeqFlow is not executable" +for s in "$root"/bin/*; do + [ -x "$s" ] || fail "$(basename "$s") is not executable" +done + +# The version the extracted copy would report, in all three places it lives. +grep -q "^VERSION=\"${version}\"$" "$root/PoolSeqFlow" \ + || fail "extracted wrapper does not report ${version}" +grep -q "^# Version: ${version}$" "$root/PoolSeqFlow" \ + || fail "extracted wrapper's header comment does not say ${version}" +grep -q "version *= *'${version}'" "$root/nextflow.config" \ + || fail "extracted nextflow.config manifest does not say ${version}" + +echo "${OUT}/${name}.tar.gz"