Developer Guide¶

Guide for contributing to gbcms.

Setup¶

# Clone
git clone https://github.com/msk-access/gbcms.git
cd gbcms

# Virtual environment
python -m venv .venv
source .venv/bin/activate

# Install Rust (if not installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install (builds Rust extension, then installs the `dev` dependency group,
# which needs pip>=25.1)
pip install --upgrade "pip>=25.1" maturin
maturin develop --release

# Verify
gbcms --version

# Clone
git clone https://github.com/msk-access/gbcms.git
cd gbcms

# Create conda environment with build dependencies
# Note: clangdev (not clang) provides headers needed by bindgen
conda create -n gbcms-dev python=3.11 clangdev rust -c conda-forge
conda activate gbcms-dev

# Set libclang path for the Rust build
export LIBCLANG_PATH=$CONDA_PREFIX/lib

# Install maturin and build (the `dev` dependency group needs pip>=25.1)
pip install --upgrade "pip>=25.1" maturin
maturin develop --release

# Verify
gbcms --version

# Clone
git clone https://github.com/msk-access/gbcms.git
cd gbcms

# Install Rust (if not installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Virtual environment
python -m venv .venv
source .venv/bin/activate

# Install (builds Rust extension, then installs the `dev` dependency group,
# which needs pip>=25.1)
pip install --upgrade "pip>=25.1" maturin
maturin develop --release

# Verify
gbcms --version

Project Structure¶

flowchart LR
    subgraph Python["src/gbcms/"]
        CLI["cli.py"] --> Pipeline["pipeline.py"]
        CLI --> Normalize["normalize.py"]
        CLI --> Convert["convert.py"]
        Pipeline --> IO["io/"]
        Pipeline --> Models["models/"]
    end

    subgraph Rust["rust/src/"]
        Lib["lib.rs"] --> Count["counting/"]
        Lib --> Norm["normalize/"]
        Lib --> Shared["shared/"]
        Lib --> Ann["annotation/"]:::annot
        Count --> Eng["engine.rs"]
        Count --> VC["variant_checks.rs"]
        Count --> PH["pairhmm.rs"]
        Count --> WFA["wfa_router.rs"]
        Count --> RNA["rna.rs"]
        Count --> Parquet["parquet_writer.rs"]
        Norm --> LA["left_align.rs"]
        Norm --> DC["decomp.rs"]
        Shared --> Frag["fragment.rs"]
        Shared --> Stats["stats.rs"]
        Shared --> Filters["filters.rs"]
        Shared --> BAQ["baq.rs"]
        Shared --> BamU["bam_utils.rs"]
    end

    Pipeline --> Rust
    Normalize --> Rust

    classDef annot fill:#27ae60,color:#fff,stroke:#1e8449,stroke-width:2px;

gbcms dna ... 2>&1 | grep "Built.*bins"
# Example: "Built 42 genomic bins from 1247 variants (window=10000bp)"

Build Commands¶

# Development (fast)
maturin develop

# Release (optimized)
maturin develop --release

# Build wheel
maturin build --release --out dist

Regression Testing¶

Python Integration Tests (pytest)¶

The primary test suite covers CIGAR parsing, windowed indel detection, SNP/MNP/insertion/deletion/complex variant classification, shifted indel handling, and CLI validation:

pytest -v                                    # Full suite
pytest tests/test_shifted_indels.py -v      # Windowed + shifted indel-specific
pytest tests/test_accuracy.py -v            # Core allele-classification

Rust Unit Tests (cargo test)¶

Normalization engine tests — left-alignment, repeat detection, adaptive padding, homopolymer decomposition:

cd rust && cargo test

BAM Slice Regression Suite¶

For changes to counting logic (counting/engine.rs, counting/variant_checks.rs), run against the BAM slice test set (54 BAM slices × 2 MAFs — see ~/Downloads/bam_slice_v2_8_0/):

# Rebuild first
pip install -e . --no-build-isolation

# Run on indels MAF
gbcms dna \
    --variants filtered_indels_io.maf \
    --bam-list bam_list.txt \
    --fasta ref.fa \
    --format maf \
    --output-dir /tmp/regression_out/

# Diff against baseline: match rows by locus, never by position. The release gate's
# comparison is scripts/regression_panel/compare_panel.py (see Regression Panel).

Key invariants to check:

MNP Rescue Pass (--rescue-mnp)¶

Architecture: The rescue engine is implemented in Python (pipeline.py::_rescue_mnp_pass()) calling the existing Rust counting engine (count_bam_binned()). This was a deliberate choice:

• The rescue pass is an orchestration layer — it decides what to re-count, not how
• The expensive work (BAM I/O, read classification) stays in Rust
• Python has direct access to gbcms_diagnostic flags (set by _compute_diagnostics())
• Candidate filtering uses Python string matching on diagnostic strings
• The typical rescue workload is 5–50 synthetic SNPs per sample (< 1s overhead)

See Architecture → MNP Rescue Pass for the full design rationale and data flow diagram.

Pipeline integration point — rescue runs after diagnostics, before output:

count_bam_binned() → _merge_counts() → _compute_diagnostics() → _rescue_mnp_pass() → _write_output()

Debugging rescue:

# Enable rescue with debug logging to see per-variant decisions
gbcms dna --verbose --rescue-mnp --variants input.maf --bam sample:sample.bam --fasta ref.fa --format maf --output-dir out/

# Rescue log lines:
# WARNING — "MNP rescue enabled (--rescue-mnp, threshold=1.00): rescued rows report a component SNV's counts ..."  (once per run)
# INFO    — "MNP rescue for SAMPLE: rescued=2, skipped_grouped=1 (0.342s)"   (per-sample outcome summary)
# WARNING — "MNP rescue: 1:1000 GAGGG>AAGGA now reports component 1:1000(G>A) (ALT 88, REF 487); the MNP itself had ALT 1 (0 showing the whole haplotype), partial 88"
# DEBUG   — "MNP rescue: ... skipped — co-annotated group 3 owns its reads"
# DEBUG   — "MNP rescue: ... kept — 36 read(s) show the whole haplotype"
# WARNING — "Mixed MNP rescue at 1:1000 GAGGG>AAGGA — duplex reports component ..., simplex reports the MNP ..."  (gbcms merge)
# DEBUG   — "MNP rescue: ... not rescued (no_improvement) ..."  (partial evidence was not component carriers)
# WARNING — "MNP rescue: ... not rescued (ref_validation_failed) ..."  (anomaly: investigate)
# WARNING — "MNP rescue: synthetic SNV ... failed preparation (...) — reported as ref_fail"

Extending rescue: If adding a new rescue strategy (e.g. coordinate shift):

1. Add the strategy as a new code path in _rescue_mnp_pass()
2. Use a different method= value in the audit trail (e.g. method=coordinate_shift) and keep outcome= and original_* keys, built by gbcms.rescue_audit.format_rescue_audit()
3. Decide outcomes in a pure function (like _resolve_mnp_rescue()) so they are unit-testable
4. Log DEBUG per variant, INFO per-sample summary, WARNING for anomalies
5. Add end-to-end tests to test_rescue_mnp.py asserting the counting invariants on written rows

Test fixtures: Any manually instantiated MafWriter/VcfWriter (via __new__) must set rescue_mnp=False (or True) explicitly, or _gbcms_column_names() will raise AttributeError. See test_phase2_output.py for examples.

Variant-Type-Specific Investigation¶

When debugging specific variant types, use targeted BAM slices:

Code Standards¶

Python¶

CLI Validation Standards¶

New CLI options must follow this four-layer validation order:

1. Parse-time (Typer): Use Enum for constrained choices, min=/max= for numeric ranges. Typer rejects invalid values before any Python code runs.
2. Pre-model (cli.py command body): File extension checks, cross-option semantics (e.g. --preserve-barcode + non-MAF input), charset validation. Log at ERROR and raise typer.Exit(code=1).
3. Model-time (Pydantic): Business-logic constraints in models/core.py via Field(ge=..., le=...) and @field_validator.
4. No silent skips: Missing inputs must fail-fast or require an explicit opt-out (e.g. --lenient-bam). WARNING-level silent skips are unacceptable for missing required inputs.

The module-level docstring in cli.py documents this order and must be kept in sync when adding new options.

Rust¶

Git Workflow (git-flow)¶

gitGraph
    commit id: "main"
    branch develop
    commit id: "develop"
    branch feature/new-thing
    commit id: "work"
    checkout develop
    merge feature/new-thing
    branch release/X.Y.Z
    commit id: "bump"
    checkout main
    merge release/X.Y.Z tag: "X.Y.Z"
    checkout develop
    merge release/X.Y.Z
    checkout main
    branch hotfix/urgent-fix
    commit id: "critical fix"
    checkout main
    merge hotfix/urgent-fix tag: "X.Y.Z+1"
    checkout develop
    merge hotfix/urgent-fix

Pre-Commit Checklist¶

Before committing, verify all checks pass:

• ruff check src/ tests/ — Python linting
• black --check src/ tests/ — Python formatting
• mypy src/ — Type checking (no errors)
• cd rust && cargo clippy --all-targets -- -D warnings — Rust linting (strict, warnings-as-errors)
• cd rust && cargo test — Rust unit tests
• pytest -v — Python/integration tests
• Type hints complete on all new public functions
• Docstrings added (Google style for Python; /// on public Rust items)
• No dead code (removed, not commented out)
• No silent failures: missing inputs must fail-fast or require explicit opt-out

To run all Python checks in one pass:

ruff check src/ tests/ && \
black --check src/ tests/ && \
mypy src/ && \
pytest -v

To run all Rust checks:

cd rust && cargo clippy --all-targets -- -D warnings && cargo test

Environment Variables¶

Logging is controlled by CLI flags, not environment variables: --verbose enables DEBUG (Python and Rust — Rust records are forwarded through pyo3-log into Python's logging), and --trace additionally enables per-read Rust trace!() diagnostics. RUST_LOG has no effect on gbcms.

gbcms dna --verbose ...   # DEBUG-level logs
gbcms dna --trace ...     # per-read classification diagnostics (slow)

Building the docs¶

The site's tools are pyproject's docs dependency group (the dev group includes it). MkDocs stays below 2.0, which is incompatible with Material and this site's config. Material for MkDocs 9.7.2 and later still print a notice about MkDocs 2.0 (from 9.8.0, about Material's own end of life) on every mkdocs run; the docs workflow turns it off, and so can you:

NO_MKDOCS_2_WARNING=1 mkdocs build --strict

Generating a PDF¶

The docs include a combined print page (via mkdocs-print-site-plugin) that can be printed to PDF directly from Chrome — no extra tools required.

Steps:

1. Start the docs server:

   mkdocs serve
   

2. Open http://127.0.0.1:8000/gbcms/print_page/ in Google Chrome and wait ~10s for Mermaid diagrams to render.

3. Open the DevTools console (F12 → Console tab) and paste:

   document.querySelectorAll(
     '.md-sidebar,.md-header,.md-footer,.md-tabs,.md-source-file'
   ).forEach(e => e.remove());
   document.querySelectorAll('.md-content').forEach(e => {
     e.style.maxWidth = '100%'; e.style.padding = '0 1cm';
   });
   

4. Press Cmd+P → Save as PDF (A4, default margins, ✅ background graphics).

Related¶

• Testing Guide — Running tests and adding new test cases
• Release Guide — Release process and versioning
• Contributing — Contribution workflow and code standards
• Architecture — System design and module structure