Skip to content

Nextflow Integration

While kreview provides a robust, fully-featured CLI orchestrator, executing large-scale fragmentomics evaluation directly on local Posix systems (e.g. your laptop) can become a bottleneck when navigating tens of thousands of network parquet files.

For enterprise environments, kreview natively ships a standardized nf-core DSL2 Nextflow pipeline wrapper. This ensures kreview can securely attach to the execution tail of High-Performance Computing (HPC) workflows (like the MSK krewlyzer fragmentomics caller).

Architecture Overview

All Nextflow pipeline logic resides within the nextflow/ directory:

  • nextflow/main.nf — Entrypoint (routes --workflow eval or --workflow label)
  • nextflow/nextflow.config — Execution profiles, resource definitions, defaults
  • nextflow/workflows/kreview_eval.nf — Pipeline DAG (monolithic or multistage)
  • nextflow/workflows/kreview_label.nf — Standalone label-only workflow
  • nextflow/modules/local/kreview/ — Individual process modules:
    • label.nf — ctDNA labeling (runs once, shared across extractors)
    • extract.nf — Per-evaluator feature extraction (accepts --labels)
    • select_single.nf — Per-evaluator feature scoring + mRMR/hybrid-union selection
    • eval_cpu_single.nf — Per-evaluator CPU model evaluation (LR, RF, XGB)
    • eval_gpu_single.nf — Per-evaluator GPU model evaluation (TabPFN, TabPFN-FT, TabICL, TabICL-FT) — all models in one process
    • ablate_cpu_single.nf — Nested CV feature group ablation (CPU models, v0.0.20+)
    • ablate_gpu_single.nf — Nested CV feature group ablation (GPU models, v0.0.20+)
    • merge_ablation.nf — Merge CPU + GPU ablation → best_subset.json (v0.0.20+)
    • fuse.nf — Super-matrix construction (all evaluators merged)
    • scoreboard.nf — Cross-evaluator scoreboard aggregation (v0.0.15)
    • multimodal_prep.nf — Stacking matrix + raw-feature selection (v0.0.18+; GrootCV default since #96)
    • multimodal_single.nf — Per-model stacking CV, CPU + GPU variants (v0.0.18+)
    • multimodal_ablation.nf — Leave-one-evaluator-out ablation (v0.0.18+)
    • multimodal_merge.nf — Final results aggregation (v0.0.18+)
    • report.nf — the single self-contained report page + manifest (#79; the separate report_multimodal.nf was folded into it)

The pipeline is a single decomposed DAG with per-evaluator parallelism. (The former monolithic mode and its params.pipeline_mode switch were removed in v0.0.29 — it was a second, unmaintained implementation of the same pipeline.)

graph LR
    classDef step fill:#8b5cf6,stroke:#5b21b6,color:#fff;
    classDef opt fill:#f59e0b,stroke:#b45309,color:#fff;
    A["Label (1 job)"]:::step --> B["Extract ×N"]:::step
    B --> C["Select ×N"]:::step
    C --> ABL_CPU["Ablate CPU ×N"]:::opt
    C --> ABL_GPU["Ablate GPU ×N"]:::opt
    ABL_CPU --> ABL_MERGE["Merge Ablation ×N"]:::opt
    ABL_GPU --> ABL_MERGE
    ABL_MERGE --> D["Eval CPU ×N"]:::step
    ABL_MERGE --> E["Eval GPU ×N"]:::step
    C --> D
    C --> E
    C --> F["Fuse (1 job)"]:::step
    D --> S["Scoreboard"]:::step
    E --> S
    D --> MM_PREP["Multimodal Prep"]:::step
    E --> MM_PREP
    F --> MM_PREP
    MM_PREP --> MM_SINGLE["Multimodal Single ×M"]:::step
    MM_SINGLE --> MM_ABLATION["Multimodal Ablation"]:::step
    MM_ABLATION --> MM_MERGE["Multimodal Merge"]:::step
    S --> H["Report"]:::step
    D --> H
    E --> H
    MM_MERGE --> H
Use mouse to pan and zoom

Decomposed Multimodal Pipeline (v0.0.18+)

The multimodal pipeline is decomposed into 4 sequential stages: prep → single ×M → ablation → merge. The single stage is parallelized across models (CPU and GPU variants). (The legacy monolithic process and module were removed with the one-implementation cleanup; kreview eval multimodal run remains available for local, single-process runs.)

Feature Group Ablation (v0.0.20+, optional)

When params.run_ablation = true, the ABLATE stages (amber nodes) run between SELECT and EVAL. They use inner cross-validation (sensitivity_at_100spec_healthy) to identify the best feature group subset per model, producing a best_subset.json consumed by EVAL. When disabled (default), EVAL runs directly after SELECT.

For a detailed architecture overview with notebook-to-module mappings, see the Pipeline Architecture developer guide.

The workflow transparently wraps the kreview Typer CLI, binding the computation natively to the ghcr.io/msk-access/kreview:latest container.

Output Structure

In multistage mode, all process outputs are published to params.outdir via publishDir with mode: copy:

outdir/
├── labels/
│   └── labels.parquet                          # 6-tier ctDNA labels + split column
├── matrices/
│   ├── raw/                                    # Per-evaluator raw feature matrices
│   │   ├── AtacOnTarget_matrix.parquet
│   │   ├── FSCOnTarget_matrix.parquet
│   │   └── ...
│   ├── selected/                               # After mRMR/hybrid selection
│   │   ├── AtacOnTarget_matrix.parquet
│   │   ├── AtacOnTarget_eval_stats.parquet
│   │   ├── AtacOnTarget_selection_qc.json
│   │   └── ...
│   └── fused/
│       └── super_matrix.parquet                # All evaluators merged
├── models/
│   ├── ablation/                               # Feature group ablation (v0.0.20+)
│   │   ├── AtacOnTarget_ablation_cpu_results.json  # Per-fold best subsets (CPU)
│   │   ├── AtacOnTarget_ablation_gpu_results.json  # Per-fold best subsets (GPU)
│   │   ├── AtacOnTarget_best_subset.json           # Merged winning features
│   │   └── ...
│   ├── cpu/                                    # Per-evaluator CPU model results
│   │   ├── AtacOnTarget_model_results.json
│   │   ├── AtacOnTarget_lr_model.joblib
│   │   └── ...
│   ├── gpu/                                    # Per-evaluator GPU model results
│   │   ├── AtacOnTarget_gpu_model_results.json # All 4 GPU models in one JSON
│   │   ├── FSCOnTarget_gpu_model_results.json
│   │   └── ...
│   └── multimodal/
│       ├── stacking_matrix.parquet             # From prep
│       ├── prep_metadata.json                  # From prep
│       ├── stacking_rf_results.json            # From single (per-model)
│       ├── stacking_xgb_results.json
│       ├── stacking_tabpfn_ft_results.json
│       ├── ablation_results.json               # From ablation
│       └── multimodal_model_results.json       # From merge (final)
├── scoreboard_combined__all.parquet            # Cross-evaluator ranking (v0.0.15)
├── scoreboard_combined__all.csv
└── reports/
    ├── kreview_report.html                     # single self-contained page (#79)
    └── report_manifest.json                    # what the report covers, always written

Inspecting Parquet Files on the CLI

Use parq-cli to inspect parquet files directly from the terminal without Python: 

parq schema labels/labels.parquet    # View column names and types
parq meta   labels/labels.parquet    # View row count, compression, metadata

Pipeline Execution

To trigger the kreview evaluation over a massive cohort using your standard Nextflow runner, use the main.nf script:

nextflow run /path/to/kreview/nextflow/main.nf \
  --cancer_samplesheet /data/cancer.csv \
  --healthy_xs1_samplesheet /data/healthy1.csv \
  --healthy_xs2_samplesheet /data/healthy2.csv \
  --cbioportal_dir /data/msk_solid_heme/ \
  --krewlyzer_dir /data/krewlyzer_parquets/ \
  --run_gpu_eval true \
  --gpu_models "tabpfn,tabpfn_ft,tabicl,tabicl_ft" \
  --run_multimodal_eval true \
  --multimodal_selection grootcv \
  --multimodal_gpu_models "tabpfn_ft,tabicl_ft" \
  --run_ablation true \
  --ablation_inner_folds 3 \
  --ch_hotspot_maf /path/to/ch_hotspots.maf \
  --max_gpu_features 150 \
  --seed 42 \
  --deterministic true \
  -profile iris

GrootCV selection tunables

grootcv is the multimodal-selection default since #96 (confirmed on the v0.0.32 production run) and accepts two evidence-based knobs whose defaults were measured on the real v0.0.29 cohort: --multimodal_selection_cutoff (default 3.0 — shadow-importance divisor, higher admits more features) and --multimodal_selection_n_iter (default 10 — shadow-test CV repetitions; 0.96 selection agreement with 50 at ~6× less runtime). LightGBM threads are pinned to the process task.cpus automatically.

Targeted Nextflow Execution

Just like the vanilla CLI, you can limit the Nextflow computation to specific features! You are allowed to pass the --features or --tier parameters dynamically through Nextflow: 

nextflow run nextflow/main.nf \
  ...
  --features "AtacOnTarget,FSCOnTarget"

Supported Nextflow versions

kreview supports Nextflow v25–v26. manifest.nextflowVersion pins a hard floor of 25.04.0, so an older Nextflow fails immediately with a clear message instead of a confusing parse error. CI runs the stub test against both ends of the range (25.04.6 and 26.04.6).

The floor exists because nextflow.config uses env('VAR') to read environment variables, which Nextflow added in 24.11. Older releases also accepted try/catch and ${HOME} interpolation in config; Nextflow 24+ rejects both, which is why the config was reworked.

Smoke-testing the DAG (-profile stub)

Every process declares a stub: block that creates its declared outputs and nothing else. Combined with -stub-run, this exercises the whole DAG — config parsing, profile resolution, module includes and channel wiring — in seconds, with no data and no containers:

bash scripts/nextflow_stub_test.sh            # uses `nextflow` from PATH
bash scripts/nextflow_stub_test.sh /path/to/nextflow

The script checks every profile resolves, runs both the eval and label workflows, and asserts from execution_trace.txt that all 17 processes ran and every task reached COMPLETED. It also fails on a withName: selector naming a process that no longer exists — silent config rot that Nextflow only reports as a warning.

This is a wiring test, not a correctness test: no real computation happens. Run it after changing any .nf file, the DAG, or nextflow.config.

Profiling & Scaling (SLURM)

Because kreview accesses thousands of files aggressively using DuckDB, network filesystem socket limits (Ulimit N) behave entirely differently between a desktop Mac and a remote HPC SLURM cluster.

1. Local (Docker)

nextflow run ... -profile docker

When operating locally, nextflow.config strictly maps docker.runOptions = '-v /:/' to guarantee absolute URI paths don't break the container's volume map. It safely hardcaps --chunk-size at 50 to defend local Posix max-file limits.

2. High-Performance Computing (SLURM)

Two SLURM profiles are available:

Profile Partition Use Case
slurm Generic Default SLURM submission
iris cmobic_short (3h) MSK IRIS cluster with Singularity, auto-tuned CV and SHAP

nextflow run ... -profile iris

The iris profile invokes Singularity (via autoMounts = true), targets the cmobic_short partition, and overrides the fallback logic. The configuration aggressively sets --chunk-size 500 to maximize network ingestion speeds on hardware that naturally supports 102400 open network sockets. It also auto-tunes cv_folds=10 and shap_samples=5000 for production quality.

IRIS Compute Nodes Lack Internet

On IRIS, compute nodes cannot pull containers from GHCR. Use a local clone of the kreview repo instead of the -r GitHub remote: 

nextflow run /usersoftware/shahr2/github/kreview/nextflow/main.nf ...
Singularity images are cached to ~/.singularity_cache/ on the login node.

Averting Path-Staging Collapse

If Nextflow were allowed to behave natively, it would attempt to symlink all 14,000 Krewlyzer .parquet target output files independently into the isolated .work/ module directory. This mathematically guarantees a localized freeze or file-limit crash on practically all operating systems.

To fundamentally solve this problem:

The kreview Nextflow module (run.nf) completely bypasses standard path file orchestration for the target Krewlyzer output. It captures --krewlyzer_dir exactly as an absolute Native val String, sending it securely inside the Python container.

You can safely drop either an absolute directory path or the explicit path to a manifest.txt file directly into the --krewlyzer_dir mechanism!