Changelog¶
All notable changes to gbcms are documented here.
Full History
See GitHub Releases for complete release notes.
Changelog¶
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[6.7.0] - 2026-11-09 — a read counts what it reaches; the sequence judge, opt-in¶
Release gate — the regression panel against 6.6.0 (2026-11-09)¶
- The 6.6.0 tier on local slices, 411 runs: no run failed in either build; 48,022 changed cells, every one attributed to a merge (the reduction is exact).
- Concordance with the sign-out (matched level): 98.1% of the 13,323 DNA rows within the margin in 6.6.0, 97.3% in 6.7.0 (136 rows left the margin, 33 entered it). 94 of the 136 are now above the sign-out: 6.7.0 counts carriers by their bases and finds carriers that a count by alignment missed (read by read, 809 of the changed reads carry the ALT and 150 do not).
- 19 signed-out rows read AD 0, each flagged
ZERO_ALTandPARTIAL_DOMINANT, and each checked on its own: - 9 rows: the row is a part of a complex event that the input also lists. One molecule gives one
allele (RJ-25), so the complex row keeps the reads; the part reads AD 0 with status
MULTI_ALLELIC(operator, 2026-11-09: kept). - 5 rows: every ALT read carries the listed deletion and an unlisted change beside it that no REF
read carries. The reads carry another allele, named in
OBSERVED_ALLELE, so the given deletion reads AD 0 (D4, RJ-25; operator, 2026-11-09: kept). - 1 row: a 279 bp deletion with a change past its junction (a named limit of the junction judge, RJ-24). 1 row: the reads carry two or more extra changes (another allele).
- 3 rows: see "Known issues".
Known issues — three shapes that the default read judgment counts as 0¶
- A pure deletion with a listed change on the first base past its end, carried in cis (#417).
- The exact carriers of a 17 bp tandem duplication: partial on the rules path (#417).
- A delins with a listed SNV at its first base: neither row gets the reads (#425).
--read-judge sequencecounts all three. The rules path is not changed for them (operator, 2026-11-10): the sequence judge is the path to keep, and it becomes the default in 6.8.0 (#331).
Fixed — harness: the cBioPortal guard reads each output stream (Greptile on #426)¶
- The guard (
.claude/hooks/protect-cbioportal.py) took the>inside2>for a stdout redirect, so a command that sent only stderr into a protected folder passed while stdout printed the rows. Now every stdout redirect (>,>>,1>,&>) and every stderr redirect must end under a protected root (2>&1counts only after the last stdout redirect), and a root matches only as a whole folder.tests/test_protect_cbioportal_hook.pypins 18 shapes (made-up names).
Tests — Typer 0.29.0¶
- Two docs tests (every
gbcmsexample parses; theqc summarizeexamples parse) switch off the path-existence check where Typer 0.29 keeps it, on the parameter's declaration (0.29 removed a parameter'stype). What they assert is unchanged; gbcms itself runs on Typer 0.29.
Fixed — RNA: ASJD reads the same sibling verdict as the main counts (#359, RJ-5)¶
- At an RNA row with a GTF and a co-annotated sibling, the allele-specific junction test counted reads the main counts give to the sibling (a REF read that is ALT for the sibling where this row's REF is read; an ALT match the sibling explains as well or better). It now makes the same two sibling checks, so those reads vote in neither the REF nor the ALT junction partition. Only the
asjd_*columns can move; RD, AD, DP and the per-transcript counts are unchanged. Rules path only. Measured on the synthetic set: rows without a sibling are byte-identical (0 of 1,072); with a sibling added to every row, 12 of 2,144 rows move.
Tests — RNA pins (#344, #345, #346)¶
- Pinned decided behaviour, no count change: reads carrying another allele are in a transcript's DP and in no transcript AD/RD or ASJD partition; antisense reads carrying another allele appear in no column under enforcement, and count as partial evidence with
--no-strandednessor with no GTF; an edit inside an insertion's bases is decided by the reads, and an editing database cannot reach it. - Decided and pinned (RJ-32): a read with no junction fits every transcript it overlaps, so it counts in the DP and RD of each overlapping spliced transcript, and once in the row's counts. A single-exon transcript gets no per-transcript entry. No count changes.
Changed — a listed allele at the site is always another allele, and a read counts REF where it tells REF from the alleles it reaches (RJ-25, RJ-24; operator 2026-11-02)¶
- No floor for a listed allele, on both paths. A listed allele whose span meets the ALT's site
is another allele there whether or not any read shows it (the input is taken as correct, as
rule X took it for another unit count at a long deletion). The 1-fragment-and-5% test of
2026-10-14 is gone for listed alleles (the rules'
other_alleles, the sequence judge'sseen_apart); it stays for an allele nobody listed (3 fragments and 5%). A listed component of an MNP that no read shows apart from it is another allele too: a carrier that reads only the component's base cannot tell the MNP from it and is partial evidence (AD 14 to 10 in that shape). The rules path takes any listed allele of the ALT's length that meets the site and differs inside the window, not only one that differs nowhere past two bases beside the event. - A REF read is told only from the alleles it reaches. This reverses the REF clause decided
on 2026-10-14 (a REF read short of a REF-length allele was neither). An allele past the read's
end (or before its start) no longer contests it: a read ending inside a long event is judged by
the bases it has (RJ-9); a read that reaches the allele and cannot tell it (the base unread,
below
--min-baseq, or masked) stays neither. The sequence judge already ties by reach; the rules path now does too, for REF-length alleles listed or found, in the exact-carrier windows and in the MNP check. - An SNV row beside a listed indel or delins is judged by its bases, also under the rules (#398).
An SNV row where the same input lists an indel or delins whose span meets the row's site (the
sequence judge's own test) is counted by the sequence judge, as
--read-judge sequencewould; every other row, MNV rows included, keeps the rules. Prep decides it from the whole input, so no bin or call changes it; no column, flag or environment variable is added. The rules still count a routed-class row in two places, each with one warning per count: RNA mode, and a count given no reference FASTA. The rules' single-base check weighed no listed allele that changes the length, so a read that spelled one counted ALT. Measured at these rows (rules lane against judge lane, 2026-11-03): patient sets, wrong-ALT reads 1,230 to 0 and wrong definite calls 1,258 to 7; HG002, true ALT credited 454 to 520 and reads of the listed allele credited ALT 142 to 21. Precedent: a deletion of 50 bp or more already goes to a by-bases junction judge. A MNV-plus-insertion shape (67 true-ALT reads at two gate-set rows) would go to partial under the judge and stays on the rules (#331). - RJ-24 reaches a delins. A delins that replaces 50 REF bases or more (the changed REF bases once the shared prefix and suffix are trimmed) has a core, 4 bases in from both ends: a substitution wholly inside it is a context of REF's molecules, so a read that reaches and spells it is REF and a sibling there claims no REF read. A 60>30 delins has a core; 5>60 does not.
- Stopgap (#418). a found allele's background bases are left out of its reach only where the allele also differs at an unmasked base inside X's window; where its only differences inside the window are masked, those differences are its reach. Without it HG002 c246 INS 1-9 gets 185 true-ALT reads called REF against develop's 182, because the allele's one unmasked difference lies outside the window. It goes when #418 is fixed.
- Routing can raise AD. At a routed SNV row the judge counts a read that carries a listed allele
holding X's change as ALT (RJ-28), and a read whose clipped bases carry X. In rq2, 349 reads enter AD
at SNV rows, 254 of them off the oracle's anchor; HG002 truth at the routed rows shows no new
wrong-ALT.
--alignment-backendhas no effect at routed rows (the judge scores them). The judge's diagnostics (UNEXPLAINED_READS, MNV_SHARED, INDEL_SHARED) can appear on routed rows. Named limit (#418, as on develop): the rules path's REF side weighs only listed alleles that differ inside X's window (RJ-25 (5)(a)). - Two corrections against the text above (rq1, operator 2026-11-03). (a) The candidate test reads the window the event's own windows hold: one flank base each side of a pure indel, two of any other event. It read two for a pure indel too, so a listed delins that is X plus a substitution at the second flank base became a candidate whose window equals X's: no read could tell them apart and every carrier of a 1-base insertion in a 4-base run went to partial (AD 106 to 0 at one panel row, rules path). (b) The reach of a REF-length allele is measured on its difference from REF before the guard's masks, in every branch (a found allele's background bases at a mask do not widen it), so an allele that differs only at masked bases still contests a read that spans them: reached and not told is neither. It had been taken to contest nothing, and 91 read calls at six HG002 rows (five insertions, one deletion; 5 of them true ALT at c246 INS 1-9) went from neither to REF. The MNP check measures it the same way. The rules path only; the judge lane moves no row and no read call. Where the paths still differ at a delins row (the rules need the read to hold X's whole window, RJ-1 and RJ-2; the judge counts what the read covers) is older than this change, and closing it would be a third REF rule (#331); RJ-25 (6).
- Counts on HG002 (public, local; 27,589 rows), rules path against develop: 598 arm-rows change (87 without the routing); the routing changes no row that is not a class row; at the 1,302 class rows the rules lane equals the judge lane in every MAF field and read call. Phased truth at those rows (strict view): true ALT credited 454 to 521, reads of the listed allele credited ALT 142 to 21; scored targets (truth v2dbq20): true ALT credited 30,455 to 30,457, true REF called REF unchanged, true ALT called REF 1,339 to 1,336. Judge: 49 arm-rows change; truth classes equal. Speed (rq2, on HPC): patient wall time 1.02 times develop; HG002 CPU 1.28 times; nocl peak memory 411 to 905 MiB. Inputs that can be much slower: a long listed deletion with many SNVs inside it, and dense cohort lists (#412).
- Existing expectations changed by the decision (each with its reason in the test):
test_listed_sibling_floor_contract.py(8 cases, AD 14 to 10),test_locus_alleles_contract.py(a component never seen alone and one seen on 1 of 100 fragments, AD 14 to 10; an SNV among a delins's deleted bases never spelled, RD 24 to 20). New contract tests:tests/test_ref_reach_contract.py; a routed row intests/test_binning_invariance.py. - Supersedes the option A of the REF-tie question on the judge (#394) and settles #398 (SNV rows routed to the judge, above; the rules path's sibling scope for exact-carrier rows).
Deprecated — --rescue-homopolymer (removed in 6.8.0)¶
--rescue-homopolymeris deprecated in 6.7.0 and removed in 6.8.0.gbcms dna,gbcms rna, thePipeline,observe_molecules(rescue_homopolymer=...)and the Nextflow--rescue_homopolymeraccept it, warn once and ignore it, as--gtf-cache-dirwas in 6.6.0. The twin is never built, soWARN_HOMOPOLYMER_DECOMPis never set. It did nothing useful the diagnostics do not:OBSERVED_ALLELEandCOEXISTING_ALLELEingbcms_diagnosticname the allele the reads carry.- Measured on origin/develop
df5da55f(hp1): of 20 runs whose variant list holds a twin candidate (71 rows scored), the twin won at 1 of 20 candidate rows (5%), moving AD by +2 (17 to 19) while 17 reads moved into RD. At the SOX2-shaped rows (REF length 6, ALT length 1) it won at none, and the default diagnostic named the 1 bp deletion plus the change in 2 of 3. - The Rust binding's
rescue_homopolymerparameter,decomp.rsand the dual count stay until 6.8.0 removes them; Python no longer sends the flag.
Changed — RNA: a splice never lowers a base's quality (#292, RJ-31)¶
- Heuristic BAQ (
--apply-baq, on by default ingbcms rna) no longer lowers the five read bases each side of a splice junction (CIGARN) by 20. It still lowers by 20 the five bases each side of a read's own insertion or deletion, and still spares the counted variant's own indel. A base is judged by its own quality against--min-baseq, wherever it lies in the read. This reverses the junction penalty decided on 2026-05-04 (in effect for RNA since 4.1.0), under which a base near a junction passed the bar only where its quality was 20 above it: Q40 data passed at the default bar 20 and nothing passed at 30. GATK's SplitNCigarReads changes no quality at a splice and htslib's BAQ skips a spliced read whole, so no surveyed tool lowered a base for lying near a junction. - Counts change in RNA only, and only for reads spliced within five bases of the variant's bases, GTF or none (with a GTF, BAQ was already skipped within five bases of an annotated exon edge). Measured on 58 FORTE STAR slices at bar 20: of 163,882 SNV rows in the first five bases of exons, 136,257 move, ALT +14,269 and REF +5.70 million reads (+1.9%); clean rows keep 0 ALT, and the 219 germline hets gain the 984 ALT reads (1.9%) the penalty hid (76% at bar 30). DNA has no junction: unchanged.
- At a delins, ALT can fall at bar 30 (FORTE: 280 to 173 reads in reach, reproduced exactly on the builds before and after the change). That is right by bases: a base under the bar fits any allele, so with a junction's ten read bases masked a read could fit the ALT on the few bases left, whatever the masked bases showed. The 117 reads that lose ALT are all spliced with the event 0 to 4 bases from the junction; 92 lie on a long piece (20 or more aligned bases), not a short anchor. Under the change 91 of the 117 are REF and 18 partial evidence.
- The exon-edge exception (#106) stays; it now lifts only the penalty of a read's own indels.
- The limit (#409, RJ-3's short pieces): a piece too short to place its own bases, at a spliced read's end or between two junctions, is no longer hidden by the penalty and can read as an ALT it does not carry. An end piece is written at an annotated junction where its molecule spliced at a novel one a few bases on (synthetic set: 2.2% of the reads at that stratum, at hom-REF rows); a 6-base piece between two junctions with a misplaced base counted ALT on 4 of 4 reads (develop: depth only, two penalties stacked to Q0). Ticketed; two strict xfails pin them.
gbcms rna --apply-baq/--no-baqhelp and the--traceBAQ line no longer name splice junctions. No flag or Nextflow parameter changes.- One test expectation changed, with the decision:
test_baq_still_applies_to_an_mnp_ending_six_bases_from_the_edge, (REF, ALT) from (20, 0) to (20, 10): its ten carriers hold the MNP in their own bases, and the novel donor's junction penalty had masked both.
Fixed — RNA: an exon piece holding part of a deletion's bases is depth only (#292)¶
- A spliced read whose exon piece starts at a junction at a deletion's edge and ends inside
the deleted bases (or, mirrored, starts inside them and ends at a junction) holds part of
the deletion and no flank base of its own. The exact-carrier rule now counts it toward depth
only, never REF (RJ-17, amended 2026-10-30), with BAQ on or off. Before, only BAQ's
junction penalty hid the piece's bases: with
--no-baq, such reads counted REF. The CIGAR checks already required the whole span. DNA is unchanged (its reads have no junction).
Changed — a long deletion's ALT reads must tell it from another unit count present at the site (#297)¶
- At a deletion of 50 bp or more beside a tandem run of its unit (its deleted copy differing
from the run's, so the junction windows stop at that base), another unit count reads like
the given deletion until the run's copies end. Where such an allele is present at the site
(listed in the input, always: rule X, operator 2026-10-31, the input is taken as correct;
or, unlisted, spelled in full by its own bases on 3 or more fragments and 5% of those
telling the ALT from it, RJ-25's floor; a fragment spells it only where its bases fit it,
and fit REF worse, over 24 bases past where it and the ALT differ; only reads the aligner
places here, RJ-29's test), an ALT read now counts only where its confident bases where
the two first differ show the given count, or show it with a listed row applied (a phased
neighbour the input names, as RJ-24 takes a listed change); a read ending sooner is either allele: partial
evidence (RJ-25), not ALT. Errors where the two read alike cost both and decide nothing
(RJ-26). A site with no other count keeps its reads; REF reads and the sibling claim keep
their windows. Both paths: the rules (
junction) and, under--read-judge sequence, the found alleles as site states (the judge already held listed and aligner-written ones). - The other counts looked for are the other deletions at the breakpoint (fewer bases, or more
up to twice the given); an expansion at the breakpoint is not (a REF read inside its copy
fits it as well). The count that holds a row's ALT calls is named in
gbcms_diagnostic,OBSERVED_ALLELEorCOEXISTING_ALLELEat the deletion's anchor (BaseCounts.unit_count_allele). - A listed other unit count is a candidate allele always, on both paths, whether or not a read shows it (rule X, operator 2026-10-31): the given count's reads that end before the first difference stay partial evidence at a listed count no molecule carries. That is the cost of taking the input as correct, and the spike-in arm counts it (bar 7). A variant that decided a listed count absent when the deciding fragments show none of it was built, measured and rejected (REJ-20261031-001): it gained 141 spike-in reads but credited the other allele's reads as ALT and made false-absent rows. RJ-24 and RJ-25 amended 2026-10-31.
UNIT_COUNT_UNSPANNEDweighs the nearest unit count on each side (one that reads like the given deletion for its shift and 12 bases more), not any count past reach.- Where no read at the site can spell an unlisted count (the run longer than the longest
read), nothing is held for it, the counts stand, and
gbcms_diagnosticnames the row:UNIT_COUNT_UNSPANNED(change:bases/longest), a newBaseCounts.unit_count_unspanned(no new column). A listed count there is held (below). Prep holds the reference a read reaches past twice a long deletion's length (whole_ref), and the input's indel rows inside it (listed_indels). RJ-24 amended 2026-10-30. - The rule closed on one principle, on both paths (operator, 2026-10-31, option A throughout;
RJ-24 and RJ-25 amended): a read is ALT only where its own confident bases carry the given
ALT and tell it from every candidate (REF, every listed allele, each found allele that
passes the presence test). The final review of rule X found four places where the branch
departed from it, three on the rules path only; all are closed, each red first on the rules
path and under
--read-judge sequence: - A read is "the deletion with its listed neighbour" only where its confident bases show that row's change. One input row no read carries (a 1-bp deletion 270 bases past the junction, or an SNV) turned 6 tie reads into ALT on the rules path, (RD, AD, partial) (6, 0, 6) to (6, 6, 0); the sequence judge stayed (6, 0, 6). HG002 4:3610329 is unchanged.
- A listed other unit count is held wherever it first differs, past every read's reach too:
the reads that cannot tell are partial evidence, (6, 12, 0) to (6, 0, 12) on the rules
path, and the row is named
UNIT_COUNT_UNSPANNED. The cost is a row withalt_count0 until a read can tell (HG002: two rows on the rules path, ALT 15 to 0 and 5 to 0; the sequence judge already counted 0 there). - A read whose own bases spell an unlisted other unit count in full is never ALT, whatever the
presence floor (RJ-6): partial evidence, or depth only where the aligner doubts its
placement (RJ-29). The floor decides only the reads that cannot tell. The sequence judge
leaves such reads depth only (an allele under the floor is no state), so
partial_altdiffers between the paths on them,alt_countdoes not. - The presence test reads each read's qualities as the hold does (BAQ-adjusted, inside the
readable range), and "within a read's reach" is the first difference and the 24 bases the
spelling test reads (it was 4).
UNIT_COUNT_UNSPANNED'sbasesis larger by 20, and a count that first differs 4 to 23 bases before the longest read's end is named. - One presence test, RJ-25's (
carrier::is_found: 3 fragments and 5% of the fragments spelling the allele it is nearest), asked by the exact-carrier guard, the junction guard and the sequence judge's found alleles; the judge counted reads against 5% of all the site's reads. This changes presence for every allele the sequence judge finds, at every event (SNVs, MNVs and short indels as well), not only for #297's unit counts. On HG002 nothing moves on the rules path; under the sequence judge 101 reads on 25 rows (18 distinct) move. One tolerance, RJ-27's (carrier::one_error_forgiven), for the carrier rule and the junction judge's telling; no count changes. - Round 2 of the close (operator, 2026-10-31), two more places closed on the same principle:
a read placed from the right flank (its junction bases in a left soft clip) that spells an
unlisted other unit count is partial evidence like one placed from the left, not depth only
(only the aligner's own doubt, RJ-29, makes it depth only); and the sequence judge's
presence test divides by the fragments spelling the allele the found one is nearest (REF's
or X's, by the bases that differ over the window; the larger when as near), not by every
fragment either given allele beats it on, and a fragment whose mates disagree counts for
neither, as the junction guard's
conflict. On HG002 the right-flank change moves no read; the judge's presence test moves nothing on the rules path and, under the sequence judge, 37 read calls on 8 arm rows (6 distinct rows), 7 of them (5 distinct rows) in the counts: four rows lose a found allele because a telling fragment's mates disagree, one (a 106-bp insertion) gains one and 11 of its 79 ALT reads become partial evidence, and one read moves from depth to partial evidence. - Named limits until the sequence judge is the default (#331): under the judge a read that spells a deletion of another length at X's anchor, below the presence floor, is depth only where the rules path calls it partial; and on the rules path the carriers of the deletion that also carry an unlisted 1-bp insertion are all held, where the judge places the insertion as a separate event and counts ALT those that reach the run's end (RJ-24).
- The review of the pull request (2026-11-02, operator: fix all three, each red first). A listed
deletion at the breakpoint whose REF runs past the reference held (a 600-bp deletion beside a
60-bp one before twelve copies of its unit) was dropped on the rules path, so the given count's
short carriers stayed ALT; its bases cannot be read, so it is now held as a candidate no read can
tell, as any listed count is, and named
UNIT_COUNT_UNSPANNEDwhere it is the nearest on its side: (RD, AD, partial) (6, 6, 0) to (6, 0, 6) on the rules path, the sequence judge already (6, 0, 6). Where the reference past it would tell the counts apart (a flank that does not repeat) the rules path holds all the same and the judge counts ALT, (6, 0, 6) against (6, 6, 0): a named limit. A fragment whose short mate (under 24 bases past the first difference) tells the given count while the other mate spells the other count in full counts for neither, as two full mates do: (6, 3, 15) to (6, 15, 3) on the rules path and (6, 15, 0) under the sequence judge, three such fragments. And the alleles at the breakpoint past a read's reach keep only their length change and first difference, not their bases and rows, so memory no longer grows with the square of a long deletion beside a long simple repeat: a 24-kb deletion before a 24-kb (AC)n repeat peaked at 805 MB through the engine and now at 101 MB, a 60-kb one at 4.0 GB (a 100-kb one now 131 MB); counts and flags are the same. On HG002 (public; c246, tr, sub, nocl; both paths) no row and no read call moves for any of the three. RJ-24 and RJ-25 amended 2026-11-02.
Fixed — two follow-ups to the equal windows of an insertion of 50 bp or more (#264)¶
- Under
--read-judge sequence, a read that ties the ALT with another allele (REF beaten) must hold the ALT's equal window as well as that allele's (RJ-23). Before, the window check ran on the haplotype of the allele the call names, and a read inside the insert, or holding only its far junction, held the other allele's window (a haplotype with no long insert, which looks like REF's run) and counted partial evidence. It is depth only now; no ALT count can rise. At HG002 (public; c246, tr, sub, nocl) 367 read calls on 82 rows move from partial evidence to depth only, all at pure insertions of 50 bp or more; no row's REF count, ALT count or VAF changes (partial evidence falls on all 82 rows; DP falls by 42 reads on 25 rows: those reads lie off the anchor, and a read off the anchor was in DP only through its partial call, so as depth only they now count in no column). - The guard's masks are laid by offset from the anchor on every candidate haplotype along the columns of each path (RJ-23, RJ-27), not by the offsets the haplotypes have without the listed or found changes beside the insertion (Greptile's finding on PR #396): an insertion or deletion between the anchor and a masked base moves that base's offset on every haplotype a read carrying it lies on. The graph holds a copy of the span where a mask can lie per set of such changes that can lie together. A state's graph with seven such changes or more in that span (the cap is checked per graph) keeps the offsets without them (a named limit). At HG002 the masks are laid by the columns of up to five such changes at the rows with masks and no read call moves (nor in a sweep of 12,078 synthetic read pairs); the change is in the score of a read carrying such a change on a candidate (four points lower on X's haplotype with a third base at the masked base, now equal).
- A read lies on one copy of that span for every candidate (RJ-23). A state's best path took whichever copy masked its own mismatches, so a read ending in the insert, long before a listed deletion, took the deletion's copy for the ALT (masking a base where the read has a third base) and the other copy for a listed sibling of the ALT's length that differs at a masked offset: the ALT scored 0, the sibling -4, and the read was ALT from a deletion it cannot carry. A read now lies on the changes it carries and those it spans, the same for every candidate, and carries a change only where its bases span the change's place (a read ending on a deletion's padding base, or inside the repeat it slides over, carries nothing); a change it cannot place is a plain alternative for every candidate, with no mask moved for it, as on develop: the read is scored first on graphs with no copy of the masks' zone and again, on graphs with copies for the changes it spans only, where it spans one; the candidates whose best path took another copy are scored again. A read carrying the deletion is told the ALT from the sibling at the base its copy leaves unmasked, ALT.
- The graph sets a site lays for the changes its reads span (up to 32 per site) count against the
judge's cache budget (
GBCMS_JUDGE_BUDGET_MB), once each when kept. Past the budget no more are kept and a read's set is laid for it alone. No call changes; only what is cached. - The length the windows read on to (RJ-23, RJ-27) where the masks hide the base that tells them apart is counted in the columns of the path a read lies on, as the masks are. A deletion listed between the anchor and that base takes bases out of the haplotype of a read that carries it, so the windows hold as many fewer columns there; a REF molecule carrying it that reached the base was two columns short of a window measured without the deletion and was depth only.
- The window check at a pure insertion of 50 bp or more is the class rule for every call (RJ-23): a read is told only on the windows of the alleles it ties, so a call holds its own allele's equal window and that of every other class within the margin of it. Before, only a tie of another allele with the ALT held the ALT's window too; a tie of REF with another allele was checked on that allele's window alone, and a call made by the fit check on its haplotype's alone. A REF call the reach rule made (REF ties another allele the read never reaches, or one it writes its own indel across) holds REF's window alone, as on develop; a separate class plan decides it for every shape.
- The default rules path is unchanged.
Changed — the sequence judge reads an insertion of 50 bp or more over equal windows from its anchor (#264)¶
- Under
--read-judge sequence, a REF or ALT call at a pure insertion of 50 bp or more now holds RJ-23's equal windows from the anchor, as the rules read it: the anchor and as many bases on as REF's window holds (the run the insert slides over and the base past it). The ALT is read over the insert's first bases, never the whole insert. A read lying inside the insert, or holding only its far junction, is depth only, as a REF read starting inside the run is. Before this the judge counted reads inside a long insert ALT, giving the ALT read starts REF cannot have. - Where the guard's masks hide the base telling the two windows apart (a change past the run, which on the ALT's sequence lies past the insert, so it told REF's molecules apart and never the ALT's), both windows read on to the next base telling them apart, and a read short of it is depth only, REF and ALT alike (RJ-27's read-on). At HG002's 98 bp duplication at chr22:39653046 this makes 11 REF reads depth (RD 21 → 10, AD 6).
- Both windows are counted in bases of the called allele's sequence, so a listed change inside them
lengthens neither; the window check runs before another allele's call, so a read inside the
insert tying the ALT and another allele is depth only, not partial evidence; and the guard's
masks are laid by offset from the anchor on every candidate haplotype (REF's, the ALT's and
each other allele's), so a masked offset weighs on none of them. Where no base within reach tells the windows apart, every read is depth only and the
row is named
NO_TELLING_BASEingbcms_diagnostic(no column added). - Among the reads that hold the windows, the calls now follow the truth's allele shares on HG002 (per-read symmetry bar); the VAF of long insertions stays below 0.5 because fewer reads over the anchor come from ALT molecules than from REF ones in the BAM itself.
- Two measurement options, temporary and retired when the sequence judge becomes the default
(#331):
GBCMS_JUDGE_LONG_INS=windowkeeps the windows without the read-on, andoffrestores the judge before them. Reading a masked change on REF's haplotypes as the reference's where REF's molecules show they lack it was measured with a third option and is not adopted (it credits a read on other fragments' evidence, REJ-20261023-001); that option is removed. The rules path is unchanged; RNA does not take the sequence judge. - Decided 2026-10-31 (operator): the masks laid on both windows by offset are accepted and laid
on every candidate haplotype, so a listed allele of the ALT's length that differs from it at a
masked offset is told from it by no base there (the ALT's carriers and a read with a third base
at that offset are partial evidence of either, never ALT); the
cisoption is removed; the anchor is taken from the insertion's run, wherever the row is written; the rule stays for pure insertions of 50 bp or more (a delins with a longer side over 50 bp is a separate decision). On HG002 this moves 4 reads in 2 rows on tr and 4 in 2 rows on nocl and sub, each from partial evidence to REF (3 of the 4 on nocl and sub are true REF, none a true ALT), and no other row of the four truth arms; the merge with develop's band rule (#369) moves only reads on itsjudge farlines; the rules path is unchanged byte for byte.
Added — a macOS wheel for Apple silicon (arm64) (#237)¶
- The release builds
cp310-abi3-macosx_11_0_arm64onmacos-latestwith the new local action.github/actions/macos-wheel, and publishes it to PyPI and the release page (with its checksum and build-provenance attestation) beside the Linux wheel.pip install gbcmson an Apple silicon Mac with macOS 11 or newer, on any Python from 3.10, now installs a wheel; before, it built the source distribution (Rust, clang and htslib's C source). - The
Wheelsworkflow builds that wheel on a macOS runner on every change to the build, and installs it on Python 3.10 and 3.14 withpip install --only-binary=:all:. The suite runs there, and the counting run's output digests must equal the Linux ones. The same steps run on a Mac withscripts/check_wheel.sh. - The action then checks that the wheel needs no newer macOS than 11.0: its platform tag, and
the minimum macOS in each binary's load commands (
otool -l). The tests run onmacos-latest, so they would accept a wheel that needs a newer macOS; the Linux build checks its glibc floor the same way. - Intel Macs (x86_64) still build the source distribution: nothing in CI uses an Intel Mac, and a wheel for one would be a second entry in the same two jobs (or a cross build on the arm64 runner) if a user asks for it.
- Checked locally on macOS arm64, with the wheel of the stable-ABI change: the wheel is 3.2 MB, installs on Python 3.10 to 3.14, and the suite passes on all five.
Added — a native arm64 Docker image, joined with amd64 into one multi-arch tag (#236)¶
- The release builds the image on an amd64 runner and on GitHub's arm64 runner (no QEMU),
pushes each by digest, and a new job
docker-mergejoins the two into one tag (X.Y.Zandlatest). Pulling the tag on Apple silicon gets an arm64 image that runs natively; before, Docker Desktop ran the amd64 image under emulation. The image lock's hashes cover every platform's wheels, so both architectures install the same versions. - The
docker-testjob runs on both architectures. In each it builds the image on the runner, checks that the image is the runner's architecture, runs--versionandpip check, and runs the counting run on a synthetic data set. The new jobdocker-comparefails unless amd64 and arm64 write the same bytes. - The Nextflow
dockerprofile still forces--platform linux/amd64. Dropping it would let Apple silicon run the native image; that waits fordocker-compareto pass on real CI runs, and it changes what a pipeline run on a Mac uses. - Publishing: a manual run of the release workflow builds both images and pushes nothing. A tag push pushes two untagged images and joins them; the first tagged release is the first time the join runs.
Added — CI runs the image through Apptainer, as RHEL clusters do (#233)¶
- The
docker-testjob installs Apptainer, builds a SIF from the image it just built (docker-daemon://gbcms:test), and runs the CLI and a counting run on a synthetic data set in it:apptainer execthrough bash, on the read-only SIF, with the data folders bound in (what Nextflow'sautoMountsdoes), as the calling user. It fails unless the output digests equal those of the same run under Docker. It runs on every change that builds the image. - Found by reading how Singularity and Apptainer differ from Docker: both bind the caller's
$HOMEinto the container, and a Python imports~/.local/lib/python3.x/site-packagesbefore its own packages. Anyone on a cluster withpip install --userpackages in the same Python version would have had them replace the image's locked ones. Checked on the base image with Docker (a package in a bound HOME is imported; withPYTHONNOUSERSITE=1it is not). The image now setsPYTHONNOUSERSITE=1, and the CI run puts a package that raises on import in the caller's home first, in the folder the image's own Python reads (it asks that Python, so a Python upgrade moves the trap with it), and shows that Python reads it once the setting is undone. A regression, or a trap in the wrong folder, fails loudly. APYTHONPATHin the caller's environment still reaches the container (Apptainer passes the environment on;--cleanenvstops it): the installation page says so.
Fixed — the sequence judge no longer misses a read's path through a long listed insertion (#369)¶
- Under
--read-judge sequence, the band around a read's own placement widened only for the rows within its slack and two bases of the read. A listed insertion further on but within the read's length could hold the read's best path (a read copied from inside an expanded repeat or a duplication), and the band stood on a worse score. A listed insertion longer than 16 bases in a read's reach but past its band now sends every state of the read to the full programme, and the linear aligner drops its band for the read. Deletions do not trigger it. Measured on every arm: no patient read changes; on HG002 622 reads change, every one to the call the full programme gives (537 to depth only: through the insertion they fit each state alike); truth and both oracles unchanged. CPU +0 to +4% on HG002; on the patient arms within 1% of the build without the rule.docs/proposals/sequence-judge.md§10.
Changed — one wheel for every Python from 3.10: the extension uses the stable ABI (abi3-py310) (#235)¶
- The Linux wheel is now
cp310-abi3: one file that installs on CPython 3.10, 3.11, 3.12, 3.13 and 3.14, and on later ones. Before, the one wheel wascp311, and every other Python built the source distribution (a Rust toolchain and a C compiler). A release builds one wheel per platform, not one per Python. - Why one wheel and not five. PyO3 builds under
abi3-py310only if every Python API the bindings use is in the stable ABI; it builds, and these bindings (#[pyfunction],#[pyclass]with#[pyo3(get)]fields,VecandOptionconversions,py.detach,pyo3-log) use nothing else. The price is small. Measured on macOS arm64, Python 3.11 and 3.14, against a per-version wheel of the same commit: making 300,000Variantobjects and reading or writing their fields is within 6% (the 3.11 run +4%, the 3.14 run -2%); building the small tuples of aVariant(30,000 objects, three getters) is 15% slower (12.0 to 13.7 ms on 3.11); one counting call on 4,000 SNVs takes 496 ms at best with either wheel (the median varies by ±5% with machine load). The wheel is 4 KB larger (0.1%). - What the stable ABI cannot do: a free-threaded Python (3.13t, 3.14t) takes no
abi3wheel. gbcms is not built or tested free-threaded (gil_used = truein the module), so nothing is lost today. If a binding ever needs an API outside the stable ABI, remove the feature and build one wheel per Python. - Checks. The full suite passes on 3.10, 3.11, 3.12, 3.13 and 3.14 with the one wheel, and a
counting run on a synthetic data set writes byte-identical output on all five and with a
per-version wheel of the same commit. The
Wheelsworkflow installs the wheel in Rocky Linux 8 and 9 with Python 3.10 to 3.14. - For developers: the built file is
src/gbcms/_rs.abi3.sonow. Python tries a version-specific suffix first, so a leftoversrc/gbcms/_rs.cpython-3xx-*.sofrom an earlier build wins over it and runs the old engine. Move it to the Trash and runmaturin develop --releaseagain; a new test fails when the loaded extension is not the.abi3.so.
Changed — the Linux wheel installs on RHEL 8 (manylinux_2_28), and the extension links no OpenSSL or libcurl (#234)¶
- The release builds its Linux wheel in the
manylinux_2_28image (AlmaLinux 8, glibc 2.28), notmanylinux_2_34. The wheel now installs on RHEL 8 and newer; before,pip installon RHEL 8 built the source distribution. - Why the build image had to change: the curl that rust-htslib builds in refuses OpenSSL older
than 3.0, and RHEL 8 ships 1.1.1. The fix is not a newer OpenSSL but none: rust-htslib is
built without its default
curlfeature, which only serves remote files (http, https, s3, gcs), and gbcms opens only paths that exist.curl-sys,openssl-sysandopenssl-srcleaverust/Cargo.lock, and with them the OpenSSL license question of a wheel that carries OpenSSL. Counts are unchanged: a counting run on a synthetic data set (SNV, MNV, insertion, deletion; two paired-end BAMs; mFSD on) writes byte-identical output with and without the feature. On macOS arm64 the wheel falls from 5.77 to 3.20 MB. - New
Wheelsworkflow and local action.github/actions/linux-wheel(the recipe the release also uses): on a change to the build, it builds the wheel and installs it in Rocky Linux 8 and 9 containers withpip install --only-binary=:all:, runs the test suite there, and compares the output digests of a counting run (scripts/wheel_fixture_run.py) between Rocky 8, Rocky 9 and themanylinux_2_34wheel that earlier releases shipped. The same steps run on any RHEL machine withscripts/check_wheel.sh.checkpoint-wheels.ymlis unchanged: it builds older commits, which still link libcurl, so it keepsmanylinux_2_34.
Fixed — the docs say what the original's --filter_non_primary removes, and list the measured differences from 1.2.5¶
docs/reference/read-filters.mdimplied that the original GetBaseCountsMultiSample's single--filter_non_primaryflag covers supplementary alignments. It removes secondary alignments only (flag0x100); a read's supplementary records stay in its depth,REFandALT. The comparison now says so.docs/reference/architecture.mdhas a new list, "Known differences from GetBaseCountsMultiSample 1.2.5": one line per difference with its cause, the HG002 reads and rows it moves, and the rule that explains it. Docs only; no count changes.
Fixed — the Smith-Waterman local re-scoring's partial_alt reads the scores that made the call (#195)¶
- Where a complex row's semiglobal scores are unreliable, the
swstep re-scores with a local alignment and calls on those scores, but its nearby-evidence check (a REF call with ALT close) and its tie check still read the semiglobal ones, sopartial_altcould follow scores that did not make the call. Both now read the scores that made the call. REF, ALT and depth are unchanged. Only complex rows the exact-carrier rule cannot judge reach this step; on every arm (about 4.5 million read calls under each backend) none did, so no measured count changes.
Changed — the sequence judge holds one memory budget per thread (#304)¶
- Under
--read-judge sequence, a row's caches (the verdicts and programmes of reads alike) and the judge's work space now share one budget of 128 MB per thread. Past it a row caches nothing more and its later reads are judged afresh, with the same calls; work space past a quarter of the budget is freed at a row's end. Counts are unchanged. On HPC the deepest HG002 arm's peak memory falls from 2.8 to 1.7 GB for 1.8% more CPU; patient data types are within 2% of no limit at every budget.docs/proposals/sequence-judge.md§10.
Fixed — the sequence judge keeps a read's path steps only for the states its call can examine (#304)¶
- Under
--read-judge sequence, a read's best path, base by base, was kept for every state of the site with the read's programmes, which the row keeps for every read alike, so at sites of many alleles it grew with reads × states × read length. It is now kept only for each class's best state and state 0, the only ones a call is examined on. Counts are unchanged. The row caches are 6 to 7% smaller; with the 128 MB cache limit, HG002 peak memory falls 3 to 9%.
Fixed — the sequence judge's band no longer stands on a worse score in a tandem repeat (#369)¶
- Under
--read-judge sequence, the band around a read's own placement stood on its edge alone for any score down to −23, what an indel leaving its 16 columns costs. A read its aligner placed a repeat unit off slides along the repeat at no cost and leaves the band on a short gap, so the band could stand on a worse score than the read's best. The floor now takes the longest tandem-repeat period in the read's reach off the band's slack, and asks for a perfect score once the period reaches it. Measured on every arm, with path-align: on the patient arms 2 reads change (gate subset, both agreeing with the oracles); on HG002 666, truth +4 ALT on the tandem-repeat arm and no wrong calls (without path-align, 90% of them move to the full programme's call). CPU +2 to +12% on the patient arms, +16 to +38% on HG002.docs/proposals/sequence-judge.md§10.
Changed — the sequence judge examines a call on the alignment its score came from (#304)¶
- Under
--read-judge sequence, a read's call (the bases at X, the anchor, the deciding quality) is read off the path the graph programme traced, not a second, linear alignment. Tied placements in a run are as before; where the two differed (a band missing the graph's path, a deletion tied across repeat units) the call now rests on its score's own alignment. Measured on every arm: both oracles pass everywhere, HG002 truth +1 ALT; the judge takes 2 to 28% less CPU.docs/proposals/sequence-judge.md§10.4.
Added — the sequence judge, opt-in (--read-judge sequence, #304)¶
- A second read judgment for small-variant rows, behind a flag. With
--read-judge sequence, each admitted read is scored whole, every readable base, against the row's candidate sequences: REF, the row's ALT and the other alleles at its site (listed rows over its span, and alleles the reads themselves write recurringly and tell apart), each with every context of the other listed rows within a read's reach and of the recurring background the carrier guard finds. Costs are affine (a confident mismatch 4, a gap 6 plus its length); a base below--min-baseq, orN, costs nothing. A read counts ALT or REF only where its best candidate with that allele beats every other by one confident base; another allele winning by as much is partial evidence; anything closer is depth only. Contexts only explain a read's other differences: the judge does no linkage and no phasing, so a variant linked to the row elsewhere on the molecule never credits a read. Design and every measurement round:docs/proposals/sequence-judge.md; its rules RJ-28 and RJ-29 are indocs/reference/read-judgment.md. - Opt-in: the default is unchanged. Without the flag (
--read-judge rules) the size-specific rules count the reads, with output unchanged. The Nextflowread_judgeparameter (DNA only) defaults torules.haplotype, the judge's name during development, is taken assequencewith a warning. Making the judge the default is 6.8.0's work (#331). - Preparation and counting are told the same read judgment.
prepare_variantstakesread_judge("rules", the default, or"sequence"). It records each row's rows within a read's reach only for the sequence judge, the only one that reads them. Counting under the judge refuses rows prepared for the rules, with an error that names the first such row, so a caller never silently gets the judge without those rows.observe_moleculestakesread_judgeas well (default: its alignment config's,"rules") and passes it to both preparation and counting. - Three diagnostics, under the judge only (no new column; they change no count):
MNV_SHARED(chrom:pos:REF>ALT:n/m)at an SNV whose change a co-listed MNV holds (RJ-28: the MNV's carriers count at both rows, so one molecule can sit in both rows' counts and mFSD distributions); its indel twinINDEL_SHARED(chrom:pos:REF>ALT:n/m)at a row a listed indel or delins holds whole with listed neighbours (a delins that is two listed deletions together counts its carriers at each); andUNEXPLAINED_READS(n/m), the row's depth reads that fit none of its candidate sequences within ten confident mismatches (RJ-29: such a read is depth only where its aligner also doubts its placement, XS at least AS). - The alleles the reads spell enter as the bases they change (RJ-25, amended
2026-10-29). An allele the reads spell at the locus is a candidate written past the carrier
guard's masks and trimmed to its change, so a masked background base is never part of it.
Written with its carriers' bases over the guard's window, such an allele took from the
row every read carrying the masked change: at a 17-base duplication whose carriers all
carry a listed SNV beside it, 293 carriers per run were partial evidence instead of ALT.
An allele that is the row's ALT (or REF) with listed neighbours is the row's ALT (or REF)
however a neighbour is written: the judge reads every neighbour whose placements meet it,
so a molecule the input lists several ways (a delins and its two component deletions)
counts at each of those rows (RJ-28's indel twin, decided 2026-10-30), and
INDEL_SHAREDnames the shared molecule. The rules path is unchanged. - Named limits.
- Spliced reads are refused.
gbcms rna --read-judge sequenceexits with an error, and the engine refuses RNA mode under the judge; in a DNA run, a read whose CIGAR splices (anNop) is depth only. RNA under the judge is #333. - The Phase 3 options do not apply to a row's counts under the judge
(
--alignment-backendand the PairHMM probabilities): it has its own scorer, and it reads each row's candidate sequences from the reference FASTA. - It is slower than the rules; the speed work is #335.
GBCMS_JUDGE_*environment variables are measurement switches, not options: they serve the measurement rounds and are removed as their questions are decided. Removed as decided:SCORER, together with the enumerated scorer it selected (andFOUND_PROBEandOTHER's found rules other thantold, which only that scorer read);TRIAGE,TRIAGE_EXACTandTRIAGE_UNKNOWN, together with the k-mer triage;DELINS;CONTEXTS; andENDS. The default counts do not change.
Fixed — an unchecked sample no longer reads as a clean one (#227)¶
- The QC summary names a sample whose output could not be read. When one sample's
output could not be read, the combined block left it out and the closing line said "No row
needs a look". Now the block counts "k of n samples" and lists the sample under "Not
checked" with the reason, a sample whose counting failed is named too, and the closing line
is a
WARNING("1 of 3 samples not checked: ..."). The exit code is unchanged.gbcms summarizedoes the same for an input it refuses or cannot read: it summarizes the others, names the input, and still exits 1. A MultiQC report carries the warning. gbcms summarizerefuses a VCF that is not gbcms's. An ordinary VCF, a bare header and an empty.vcfread as clean outputs (every record a PASS row with no flags). A VCF now needs gbcms's ownGSINFO field, declared in its header or on its first record; the rest are named under "Not checked".- A MultiQC file left by an earlier run is removed when this run has nothing for it, the summary table as well as the warning, so a report never reads old counts. A skipped input that shares a name with a checked sample gets a row of its own in the warning instead of replacing the checked sample's.
- The MultiQC table is built in one pass over the cohort. It looked for the flags that fired once for every sample, so its time grew with the square of the cohort; the output is byte-identical.
Added — gbcms summarize, the merge NA flag and --qc_summary (#227)¶
gbcms summarize <outputs...>makes the QC summary of finished outputs. It reads.mafor.vcfoutputs (a run's, or a merge's) with the code the run's own summary uses, and changes nothing in them.--format text(the default) is the block the run logs, for one sample or many;jsonwrites one<sample>.qc_summary.jsonper output (every flag, zeros included, and every row that needs a look);tsvwrites one cohort table;multiqcwrites MultiQC custom content (gbcms_qc_summary_mqc.json, and agbcms_qc_warning_mqc.jsonsection only when a row needs a look). The exit code is 0 whether or not rows need a look.gbcms dnaandgbcms rnagain no option.- A merged MAF is summarized too, with a flag for NA combined cells. The summary reads
each input's flags under its prefix, and
COMBINED_NA("check the call") marks a row whosesimplex_duplex_*count, total or VAF isNA: an input count was missing or not a number, or the duplex and simplex rescues adopted different alleles. It is built from whatgbcms mergewrites; no column is added. - Nextflow:
--qc_summary(defaultfalse). After all samples finish, one task runsgbcms summarize --format multiqcand--format jsonover every output, merged MAFs included, and publishes them togbcms/qc_summary/.
Added — an end-of-run QC summary in the run log (#227)¶
- The run log now ends with a QC summary.
gbcms dnaandgbcms rnalog one block after the last sample: which QC flags fired and on how many rows, which rows need a look (a flag is "fix the input", "check the call" or informational; the first two need a look), what to do about each, the first rows to look at, the output fields to filter on, and the address of the QC Flags page. With several samples in one run the block combines them. The run's last line is aWARNINGwhen any row needs a look and anINFOline when none does; the exit code is unchanged. - It reads the written output and changes nothing. The block comes from the MAF or VCF the run wrote, read back after writing: no count, column, row or file differs (a test pins the output files byte-identical with and without it), and there is no new flag or output file. A failure to read the output is a warning, never a failed run.
- One table for the code and the docs.
QC_FLAGSingbcms.qc_summaryholds each flag's tier and one-line action; the QC Flags page repeats it under "End-of-run summary". Tests fail when a flag the page defines has no entry, when an entry names no flag on the page, and when the page's tiers, actions or example block differ from the code's.
Fixed — per-transcript count lists in a machine-independent order (#284)¶
transcript_read_countsandtranscript_fragment_countslist their transcripts in ascending ID order, on every machine. They listed them in the order the exon interval tree visited the overlapping exons, which depends on the CPU's SIMD backend (NEON, AVX or none) and on how equal intervals fall, so one build on one input wrote the same entries in another order on another machine (23 of 73,649 cells differed between an Apple Silicon run and an x86 Linux run of the 6.6.0 release panel). Counts are unchanged; only the order of entries is. The cells of two machines now compare as bytes, with no set comparison. The RNA docs state the order.
Changed — a read-rule change starts from its mechanism (#304)¶
- Fix the class, not the row. The procedure for changing how reads are judged
(
read-judgment.md, "How a change to read judgment is made") now asks for the mechanism first, stated without the row that showed the bug, with the rule derived from the principles. Its tests cover the other shapes of the same mechanism, any cutoff comes from a mechanism rather than from where a measurement splits, and a rule narrower than its mechanism says why. The fix is judged on data the bug was not found in. - Each decided read rule states its mechanism. The register has a Mechanism column, filled for RJ-1 to RJ-27, and new entries state theirs. Counts are unchanged.
Fixed — a GTF transcript ID that another chromosome or strand reuses is another transcript (#216)¶
- The GTF parser keyed transcripts by ID alone. An ID used on more than one chromosome
or strand (the same RefSeq
NM_on chrX and chrY, alternate loci) merged into one transcript: its exons from both places were sorted together, so the consecutive pairs made introns that exist nowhere and missed the real ones, and the whole transcript sat on the first chromosome seen. A spliced read over a real intron then matched none of them and was left out of the transcript's counts; the same introns fed the intron boundaries (RETENTION_DOMINANT, the exon-edge BAQ rule, RNA clip edges) and ASJD's known-junction check. A transcript is now an ID at one chromosome and strand, as the strand spans already were (#185): its exons, introns, boundaries andtranscript_read_counts/transcript_fragment_countsentries are its own. The parser warns once when a GTF reuses an ID. Exons of one ID at the same chromosome and strand still join (versions that differ only by suffix share a transcript). Counts are unchanged for a GTF that reuses no ID: Ensembl GRCh38.111 (FORTE's) has none among its 252,989 transcript IDs.
Fixed — the docs state the unreadable-insert rule as decided (#304)¶
- An insert of the ALT's length whose bases are all masked is the ALT, where no other
allele of that length is at the locus. The rule (RJ-20, amended 2026-10-15 in #278,
re-confirmed 2026-10-26) is applied by the exact-carrier rule, which judges pure indels by
their bases. Several pages still gave the superseded rule, that such a read is neither,
counted in
partial_alt: the register row (amendment at its end),allele-classification.md(Dispatch, Insertion, the wrong-length table), the glossary'shas_unreadable_insert, and the Read Filters line added in #329. - Those pages now lead with the current rule. They say that the CIGAR checks, which run
only for an insertion the exact-carrier rule cannot judge (no prepared reference), still
count such a read neither, in
partial_alt. Counts are unchanged.
Fixed — a MAF row at Start 0 is kept: an insertion is counted, any other fails visibly (#219)¶
- An insertion before a contig's first base is no longer dropped. A MAF row with
Reference_Allele-andStart_Position0 (Start_Positionis the base before an insertion, so 0 is before base 1) failed the internal model'spos >= 0check in the reader and vanished with a warning: no output row, noFAILrow. It is now resolved to the VCF spec's form at position 1, the base after the event (REFthe first base,ALTthe inserted bases then that base), and counted as the same event given as VCF is, as a deletion at Start 1 is (#122). MAF output keeps the row's input columns; VCF output andgbcms convertwrite the position-1 record with the row inMAF_START/MAF_REF/MAF_ALT. Where the first base isN(human contigs), theALTcontainsNand the row is a visibleFAILwithALT_CONTAINS_N. None in the sign-out data (vcf2maf writes such an event with sequence alleles at Start 1). - Any other row at Start 0 is a visible
FAILrow, not a dropped one. A substitution, a deletion, a delins, or an insertion written with sequence alleles atStart_Position0 names a base that does not exist. The reader dropped it with a warning (only the first five named); it is now kept asFAILwith the new reasonSTART_BEFORE_CONTIG, its input columns as written, not counted. VCF output andgbcms convertwrite it as the symbolic<NON_SEQUENCE>record at position 1 (REF the first base, reason inGSR). A negativeStart_Positionis still skipped with a warning. A VCFPOSof 0 is still an error.
Fixed — --min-baseq help and docs say what it does (#304)¶
--min-baseqmasks bases, never reads. The CLI help ("Minimum base quality") and the docs did not say what happens below the threshold, and Read Filters said an SNP or MNP read was "rejected". A base below it is read as unknown, likeN: it supports no allele and counts against none, and the read still counts toward depth, judged on its other bases. An MNP's differing positions are judged one by one (masked ones skipped), not by the block's lowest quality. The help text, the DNA, RNA, quickstart and Nextflow parameter tables, and Read Filters now say so. Counts are unchanged.
Changed — two code-quality rules the code did not meet¶
__all__in every module.report/mfsd_report.pyandrescue_audit.pyhad none, so a star import or a doc tool took every name, helpers and imports included. Each now lists its public API (TOOLTIPSandgenerate_mfsd_report; the rescue audit's two functions, asio.outputlists its functions and leaves its constants to be imported by name), and a test holds every module of the package to the rule.- No
unwrap()in production Rust. The four lines that had one are changed. The KS test of fragment sizes (mfsd.rs) sorted withpartial_cmp(..).unwrap(), which panics on a NaN size; a NaN now makes the test invalid (D = NaN, p = 1.0, as with too few fragments) and is warned about, and the sort is a total order. No size is ever NaN today (they are integers read from alignments). The other two could not panic (a branch that had just matchedSome; a loop guarded by the allele lengths) and are written without the unwrap (variant_checks.rs,left_align.rs). Counts and outputs are unchanged.
Fixed — companion file names, .vcf.bgz in normalize, the amplicon rule, Nextflow RNA defaults (#309)¶
- Companion files keep every dot of the output name. The mFSD Parquet and report
were named by swapping the main output's extension, which dropped a dotted
--suffix(S1.genotyped.maf→S1.fsd.parquet) and a dotted sample name, so two samples (S.1,S.2) wrote the same file. They are now{sample}{suffix}.fsd.parquet/.mfsd_report.html, as documented, and the observations Parquet honours--suffix({sample}{suffix}.observations.parquet). gbcms normalizereads.vcf.bgz, which the CLI already accepted: onevariant_format()ingbcms.io.inputnow reads a variant file's extension for the CLI, the pipeline and normalize.- The amplicon rule applies to programmatic callers too.
GbcmsRnaConfigwithlibrary_type="amplicon"now reads the library unstranded, and an unstranded protocol disables strand enforcement, as the CLI already did. --mfsd-reportrequires--format maf. The report reads its statistics from the MAF output; with--format vcfthe run is refused at configuration time.- A read calling no base over a row's window is depth only at a contig end. Where the exact-carrier rule cannot judge a row (its repeat runs to the contig end), the CIGAR checks' reference-coverage rules (a deletion's anchor covered with no gap; a deletion-direction delins' anchor covered by a clean match) counted a read whose bases there are all N as REF. Such a read is now neither (N class), as the exact-carrier rule counts it elsewhere. No existing test's counts change.
- Nextflow RNA tasks pass the RNA CLI defaults. The RNA module passed DNA's
PairHMM gap penalties (1e-4/0.1 over RNA's 5e-3/0.25) and
--no-baq(BAQ is on ingbcms rna). RNA now reads its own params,rna_gap_*andrna_apply_baq(on), as it already didrna_min_mapq; a test pins every module's param defaults to its CLI's. Measured on the 33 local FORTE slices (94 rows), old module arguments against the CLI defaults: with the GTF no row changes; without it one row's REF count was 5 reads (3 fragments) higher under the old arguments; AD and depth unchanged.
Fixed — user docs that were wrong or failed (#310)¶
- Install. One page states what the release builds (one cp311
manylinux_2_34x86_64 wheel and the sdist; every other platform builds from source with a Rust toolchain and a C compiler); the README links to it. Nextflow 24.04.0 or later, as the manifest enforces. - Examples. Every
gbcms dna|rna|merge|normalize|convertexample in the docs parses against the CLI (the synopses and three DNA examples lacked the required--output-dir). - Unstranded RNA is
--strandedness unstranded;--no-strandednessalone only stops the filtering. The RNA MAPQ text links to the measured default. - Output reference.
MFSD_QVAL_ALT_REF(13 mFSD INFO fields),GRonly with--rescue-mnp, ASJD fields in the VCF, the realnorm_*andrna_splice_spanningnames, nogene_strandMAF column (the GTF supplies the strand), version examples. - Nextflow.
--gtf,--publish_dir_modeand--tracedirdocumented;GBCMS_NORMALIZEis in no workflow. - Release gate. BAM Evidence Caveats lists C6 #143, C24 #195, C33 #214, C34 #216 and all of C31 #212, with the status their issues give.
- Guard tests (
tests/test_docs_contract.py): the fenced examples, the documented MAF columns and VCF INFO/FORMAT names against the writers, and the Nextflow parameters table againstnextflow.config.
Changed — a co-annotated sibling contests an exact-carrier ALT read by its bases (#300)¶
- The AD-claiming sibling guard re-tested every non-anchor ALT call against REF on the read's aligned slice of the variant's context, whenever a sibling was listed. An exact-carrier call had already weighed REF against ALT by its bases wherever the read could lie; where the aligner wrote carriers a unit off (a VNTR) that slice read as REF and every carrier was demoted. Listing a sibling zeroed the row's AD, even with no read of the sibling.
- Such a read is now placed whole on this row's ALT and each sibling's: the sibling claims it where it fits with as few confident mismatches or fewer (a tie is neither row's AD, partial evidence at both); a read placed on no allele keeps the aligned tests; other calls keep the guard as it was (operator, 2026-10-22).
- Measured against #294's final build:
- Synthetic same-site 40-bp VNTR compound hets, BWA-placed: AD 0/0 to each row's count alone (41/48).
- HG002 (public): 6 insertion rows gain 41 AD (a VNTR compound het 0/0 to 6/6). On 250 compound hets (500 rows) the summed distance of AD from a per-read oracle falls from 1127 to 831; reads counted ALT at both rows of a site from 17 to 5, and other-allele reads counted ALT from 13 to 0.
- Local arms (aggregates): IMPACT panel 11 rows, AD −267; the gate set 41 rows, AD −232; ACCESS 1 row, AD +1; WES and FORTE unchanged. Against every allele of the cluster (an independent BWA check of REF, the row's ALT and each sibling's ALT), 556 of the ALT calls removed were a sibling's molecules or ties counted at both rows; 51 of the calls gained are the row's, 17 are not; 126 changed calls stay unresolved.
Changed — reads of repeats whose telling base the guard masks, and reads placed a unit off (#294)¶
- The problem. In a repeat whose units differ at single bases, the guard can mask the only base telling a one-unit insertion from REF: a germline SNP that turns the next unit into the inserted one makes both haplotypes read alike over a short read. The rule then gave up and the CIGAR checks counted the reads. The fix decided on 2026-10-17, leaving such a mask unapplied, counted the SNP's haplotype as the insertion (HG002's 28-bp VNTR row: RD/AD 10/0 → 1/8, both phased ALT reads from the SNP's haplotype); it is rejected (REJ-20261020-001).
- The rule (RJ-27, amended; every event the exact-carrier rule judges).
- The windows read on past the masks to the next base telling the alleles apart. Prep fetches reference until, with every base the guard could mask masked, the windows reach one. A read short of it is depth only unless its own bases fit one allele alone and exactly.
- Reads placed apart from the event tell a germline change from the ALT's own
base. An ALT read written without its gap shows the ALT's base past REF's tract,
which looks like a change on the REF reads; a germline SNP can give the same base.
The fragments placed apart from the event (whose bases at the telling bases past it
fit the reference more often than the shifted ALT) decide:
- carried by three or more of them and 3%: a haplotype's change, masked at any share of the REF reads (an SNP haplotype at a minority of molecules, after a tumour's gain or loss, never counts as the insertion), unless its carriers are too one-sided in pair orientation against the locus's own balance for a haplotype's (binomial chance under 0.1%): an oxoG or FFPE artifact, not masked (an amplicon library, read in one orientation, weighs none);
- otherwise, read by 15 or more: the ALT's own base, never masked and never another allele; with fewer, masked where it is background.
- An error both readings share decides nothing. A read placing no length change whose REF and ALT readings miss exactly the same bases is read on with those bases unread, not counted for whichever allele tolerates the error at the length the aligner wrote.
- One error forgiven past the windows (operator, 2026-10-21). Read on past the windows, one confident error the other allele's reading does not share is forgiven where that reading misses three or more, the read writes that allele's length, and the error is a one-off (recurring on two fragments and 5% of those fitting the allele, it is another allele's base). Within the windows only a shared error is tolerated (2026-10-15).
- Reads placed a unit or a copy off are read whole. A read whose alignment is not
clean (another event of its own in the window, or two or more confident changes
from the reference) and that is not spliced is placed ungapped on each allele over
300 bases of reference past the change (
Variant.whole_ref, fetched by prep). The allele whose windows it reads exactly, with at most one mismatch elsewhere, is its call where every other allele misses by more. A read fitting the longer allele only from inside its event is depth only. - A call decided past the windows must hold wherever the read could lie (operator, 2026-10-22). Past the windows a read is read where the aligner wrote it; in a repeat the other allele may hold the same bases a copy over. The call stands only where the other allele, the whole read placed ungapped anywhere on it, misses more confident bases than the called allele; otherwise the read is depth only. HG002's chr21 89-bp duplication, whose cis SNP sits where the inserted copy's last base shows: 2 of its insertion reads, written through the copy, counted REF. The read is placed on the alleles as the reference spells them, no mask laid: a base the guard masks beside the event may be the other allele's own shifted onto the reads written through it (HG002's 16-bp TC insertion where the run meets an AT run lost 6 true carriers on the masked alleles). A read left depth only so still counts in the census of recurring substitutions: its bases were read (leaving two reads out moved a substitution on 5% of a FORTE row's REF fragments across that line, flipping 36 other reads).
- New diagnostic:
POSSIBLE_MINORITY_HAPLOTYPE(chrom:pos:REF>ALT:c/n). A change at the ALT's shifted base that two or more of the reads placed apart carry (over 1%), too few or too one-sided for a haplotype's, left unmasked: a minority haplotype carrying it would count ALT. Counts unchanged (BaseCounts.possible_minority). - Binning. The guard reads fragments within 300 bases of the variant, so every bin's
fetch now holds each member's guard window (
window::guard_window). - Cost. One indel with 17k reads: 0.22 s against develop's 0.21 s at 0.1% error, 0.48 s against 0.21 s under a 1% error stress.
- Measured against develop (final build), rows changed and RD/AD:
- HG002 (public): insertions 96 of 1440 rows (RD −64, AD +137); small insertions 50 of 1560 (RD −17, AD +47); small deletions 41 of 1560 (RD +2, AD +32); delins 19 of 800 (RD −1, AD +51); deletions of 50 bp or more none of 600.
- Local arms (aggregates): IMPACT panel 8 of 1170 rows (RD +2, AD +19); ACCESS and normals 6 of 1402 (RD +4); WES 9 of 686 (RD −5, AD +13); the gate set 151 of 4606 (RD +21, AD +47); FORTE RNA 1 of 94 rows (RD −1), with the GTF and without.
- The past-windows check alone (against the build before it): REF calls only, never AD. HG002: 27 REF calls left depth only, on 13 reads whose bases favour ALT, 13 that tie and 1 favouring REF (scored against the truth haplotypes). Local arms: 101, every one on a read an independent BWA check of REF against ALT calls ambiguous (83) or ALT (18).
- Known limits.
- In an imperfect repeat the windows anchor on the exact repeat's edge (#290), so a read lying inside it, or read from a flank the repeat runs on past, is still placed by the aligner where the windows decide it. Pinned as strict xfails.
- Insertion reads the aligner soft-clipped over a long duplication, or wrote through it as REF with mismatches, are depth only though their bases show the insertion (13 of 20 such reads at HG002's chr21 and chr22 rows; #290).
- A haplotype showing on fewer than three, or under 3%, of the reads placed apart is taken for errors and its reads count ALT where they read as the ALT (with 30 to 45 of them, an SNP haplotype on 5–10% of the molecules in about a third of runs; at depth, any under 3% of the molecules); such rows are named by the diagnostic.
- A position-specific error giving that base on 3% of the reads placed apart in both orientations, or a one-orientation artifact on fewer carriers than tell it from a haplotype (about ten), masks it: the ALT's reads short of the next telling base are depth only.
Fixed — PairHMM aligned the haplotype whole and the read's ends free¶
- The PairHMM fallback gave the library the read as its free-ended sequence: the read's
ends went unaligned at no cost and every haplotype base it did not reach was charged
as a gap, the reverse of the semiglobal alignment the
swbackend uses. An insertion read whose window held only part of the insert (a read ending at the anchor, clipped a few bases in, or a long insertion written without its gap) scored as REF (LLR -20 to -59); a read ending inside a repeat run leaned REF. The read is now aligned whole and the haplotype's ends are free: such reads are neither or ALT as their bases show, and calls on reads holding every base of both alleles are unchanged (red-first unit tests on each shape). - Measured: no read reached the PairHMM, or the WFA triage before it, on the IMPACT panel, ACCESS and normals, WES, the gate set, any HG002 arm (develop and the #294 builds) or the FORTE RNA slices (develop, with and without the GTF), so no count there changes.
- An N at the deleted base of a REF-length read leans REF by its length (operator, 2026-10-22): the regression test that held it near-neutral scored a flankless run with the old orientation; with flanks the PairHMM gave the same REF lean before this fix.
Changed — a run beside a long deletion is a neighbour where its bases rejoin the flank (#285, #296)¶
- The rule (RJ-24, amended). A run of changes past a long deletion's junction used to be another allele. It is now a phased neighbour, masked like a single change, when two things hold:
- its bases rejoin the allele's continuation after it;
- no other breakpoint gives the reads' bases exactly: neither the deletion at another start or of another length, nor the allele with one more indel.
This applies to the carriers, and to REF reads inside the deleted bases. It covers a
run inside the window as well as one running past it.
- A run that keeps mismatching after it is still another breakpoint.
- So is a run at the junction whose bases are the deleted bases' own: the deletion
shifted right.
- On the carriers alone, the diagnostic names the deletion with the run.
- Single changes are checked the same way (#296, folded in). An allele's recurring
changes in the window are taken together. Where another breakpoint gives the reads'
bases exactly, none is masked and the reads count as another allele. Before this, a
same-length deletion 2–3 bp right of the given one, or another unit count of a VNTR
whose units differ at one base, could count ALT.
- A change another input row lists is that row's allele (operator, 2026-10-19). Prep
records on each deletion of 50 bp or more the input's other substitution rows inside
the reference it holds (Variant.listed_neighbours). A carrier change one of them
lists is masked as a neighbour, never read as another breakpoint, and not named. With
the input taken as correct, HG002's 90-bp deletion reads VAF 0.11 with the deletion
alone in the input and 0.58 with its phased SNV row (truth 0.5). Its carriers there
read the same as another unit count of the repeat.
- How the guard reads.
- Each fragment is read once, its mates merged base by base.
- It is taken as an allele by its bases past the windows, read on as RJ-27 reads (12
bases, a shift test, and on through 24), or else by its four continuation bases.
- Prep now holds as many reference bases past a long deletion as it deletes.
- Measured against develop, on 587 local deletions of 50+ bp and every other arm:
- Panel: three rows change.
- The phased-DNP row: ALT 0 → 48 (sign-out 49).
- A 167-bp deletion: RD 117 → 319 and AD 293 → 284 (sign-out 352/291). Its REF
molecules carry a germline SNP on the first deleted base.
- A 67-bp deletion beside its sibling 2 bp right: AD 47 → 46.
- Gate panel's pure indels: the same three rows. Normals and ACCESS, WES, HG002's
300 deletions, and FORTE are unchanged.
- Diagnostics: at three 1,023-bp panel rows the observed allele now names the
reads' 1,086–1,118 bp deletions.
- Known limit (#297). A read of another unit count still counts ALT where the
repeat's units are identical as far as the read reaches.
Changed — a read aligned over its reading flank, confirmed from the other (#279)¶
- The rule (RJ-23, amended). Some reads are aligned straight across a variant's windows and fit the shorter allele read from their reading flank. Such a read is now also read whole from the other flank, at the longer allele's length, wherever its own bases place that flank unambiguously.
- Its bases may favour the longer allele there by two bases. That means the aligner wrote the longer allele's other copy over the reading flank: an insert, or a deleted sequence, whose end resembles the flank, aligned onto it as mismatches with the rest clipped.
- If the read holds the longer allele's window from there and reaches past it, it counts as that allele.
- If it starts inside the event, it is depth only, never the shorter allele.
- Why spanning carriers only (operator, 2026-10-16). Counting carriers that start inside the insert would give the longer allele read starts the shorter one cannot have. On HG002's long insertions that pushed inserts of 50 bp or more to a median VAF of 0.61–0.86.
- Measured against the build before it:
- HG002 insertions of 10+ bp in band: 334 → 343. Mean |VAF − 0.5| goes 0.157 → 0.153; 57 rows move nearer 0.5 and 24 farther.
- HG002 short insertions in band: 552 → 555. Deletions are unchanged.
- Gate panel's pure indels: 7 insertion rows change (AD +10, RD −29). Against the bases oracle, no ALT read is lost, and 39 REF reads move from partial evidence to depth.
- WES, normals and ACCESS, and FORTE: unchanged.
Fixed — a pure indel the exact-carrier rule cannot judge is warned¶
- A pure insertion or deletion the exact-carrier rule could not judge fell back to the CIGAR checks with nothing logged.
- Its reads now count in the fallback tally, and the variant is warned once with the reason, as a complex variant's fallback is. One example is a VNTR whose recurring masks leave no base telling the alleles apart (#294).
- Counts are unchanged.
Changed — pure insertions and deletions judged by their bases (#278)¶
- The rule (RJ-26, RJ-27). A pure insertion or deletion under 50 bp is judged by the exact-carrier rule, as complex events are: by the read's bases, not where the aligner wrote the gap; the CIGAR checks judge only what the rule cannot (no prepared reference). Its windows are its tract (where REF and ALT differ over every placement), grown through the repeat its change lengthens or shortens, with one base each side (D1). A read's own indel is placed by its bases: inside the discrimination window, of the ALT's length and spelling it, it is the ALT; another length or sequence there is another allele (partial evidence); placed only outside, a separate event. One substitution is an error, tolerated where the other allele mismatches the same base, unless it recurs inside the event (another allele) or sits in the flank, where it is a phased neighbour and the reads keep their call. Reads whose windows fit both alleles, or neither alike, are read on past them; a fit resting on too few bases is confirmed there (a read whose further bases fit the allele shifted by another length is another allele).
- Decisions (operator, 2026-10-15/16). One flank base (D1); a truncated copy of a long insert is another allele (D2: the CIGAR path's identity band was measured and not adopted); the ALT written far off with an SNV in its flank is ALT (D3); a substitution all carriers share inside the event is another allele (D4); a deletion whose bases between its flanks are all masked is ALT where its confident flanks lie the ALT's length apart (D5); a change the REF molecules carry inside a deletion's deleted bases is background on REF (D7); mates that disagree resolve by quality (D8); an unreadable insert of the ALT's length at its junction is the ALT (RJ-20, superseding C35's partial); soft-clipped bases are read (RJ-13); a pure indel grows only through repeats on the allele holding its bases.
- Why. Develop's CIGAR checks took the aligner's placement of the gap as the allele, so reads carrying the ALT written elsewhere, or another allele written where the ALT goes, were counted by where they were written. On HG002's het indels (truth VAF 0.5) against develop's CIGAR checks: insertions of 10+ bp in band (0.35-0.65) 110 -> 334 of about 560 rows, median VAF 0.150 -> 0.424; short insertions 465 -> 552 (0.429 -> 0.486); deletions 529 -> 563 (0.455 -> 0.500).
- On real data. The full local gate (411 runs) changes 1,062 dna-arm rows of the flagged strata against develop; within tolerance of a bases-only oracle (any discriminating base): sign-out 749, develop 838, this release 867 (14 rows farther from the two-flank oracle, 58 closer). The strata the gate flags moved away from the sign-out, which the oracle shows is the least concordant of the three. Matched normals: AD +3 over three rows that already carry the allele, RD -138; the 1,402 normal and ACCESS rows of the local arms AD +3, RD +171. FORTE (RNA): 11 of the 25 pure-indel rows change, all deletions, RD -230, AD +225 (with the GTF; without it RD -211, AD +226), most of the AD on a 12-bp deletion whose carriers STAR wrote as soft clips (AD 2 -> 107; 175 reads carry its junction in their bases).
- Found and fixed on the way. A tolerated error now costs both alleles alike; a
read's own indel is placed by its bases only up to its next indel; a call whose held
bases are all below
--min-baseqtakes the quality of the bases it read, so the fragment layer counts it too; a read whose bases place no change is read on to confirm the ALT; a read short of the masks' read-on is decided by its own bases where they fit one allele exactly. An adversarial review found reads of a third allele counted ALT or REF where one base past the tract was all the windows read (a C>CATT row took a +ATTC read as ALT); on its fuzz, other-allele reads counted REF fall 315 -> 52 and ALT 74 -> 6 with no correct call lost. - RNA. A read spliced at the exon edge that cuts the left flank reads from the left on its spliced reference; an unspliced read starting at that exon start, or within the five bases of overhang an aligner writes into the intron, reads the windows up to the edge, so REF and ALT molecules starting inside a run that crosses the junction count alike (an ALT-only miscount at an exon-start A run, present on develop). A read spliced just past the windows reads on along its own mRNA, not the intron: spliced reads at pure indels two to four bases before an exon's end had lost their call (REF reads neither, or partial under BAQ) where develop counted them, and with BAQ masking the bases at a junction (the RNA default, without a GTF) spliced reads beside an exon edge count as with BAQ off. The exon edges are typed: an exon end where the event starts leaves its left flank whole (#291). On the FORTE slices one row changes, with the GTF or without: RD +1, a spliced REF read the intron had left undecided.
Changed — reads judged beside the other alleles at their locus (#277)¶
- The rule (RJ-25). A read that fits the ALT counts ALT only when, for every
other allele of the ALT's length at the locus, it reads a base telling the two
apart (the ALT's base, at or above
--min-baseq); otherwise it is that allele or the ALT, neither with partial evidence. The other alleles are the co-annotated siblings that spell one and those the reads spell in full on 3 or more fragments, found by the reads' bases placed from the event's left flank, not by where the aligner wrote the gap. An allele whose every difference from the ALT is a change from the reference that the REF molecules carry too is a separate event, not another allele; an allele lacking part of the event (the germline haplotype beside a somatic change written as one MNP) is one. Where the reads show no other allele nothing changes. Complex events (the exact-carrier rule) and the MNP check's partly read reads alike; the opt-in homopolymer twin reads the same siblings. - Why. Develop credited a read that showed only the length of a complex event, or only the germline base of a merged MNP, as the ALT, even beside another allele that shows the same: on a synthetic arm, 165 such reads under stress masks, 15 where the given allele is absent at natural quality, a third of AD for a somatic MNP at 10% on a germline het. The rule credits none and loses no carrier where no other allele exists; demoting every such read (the first cut) lost carriers there and at the normals. As GATK and Octopus treat a read that cannot tell two haplotypes apart.
- On real data, against develop: 18 gate-panel rows change (AD -22, RD +197), 9
matched-normal and ACCESS rows (RD +347), 3 WES rows (RD +28) and 3 HG002 rows (RD
+9); the FORTE RNA delins is unchanged. The ALT side moves 10 panel rows, each beside
another allele the reads show (AD -43, of which 22 carriers count again under the
component and seen-share rules below); REF alike moves 185 reads (147 read the
telling base below
--min-baseq, 31 sit at two WES rows whose given REF holds a homozygous germline change); the grown event restores 766 REF reads. On every MNP and delins row of the 6.6.0 gate panel (742 and 180 rows over 235 runs, sliced locally): 39 MNP rows (RD -203, AD -49) and 13 delins (RD +230, AD -7) change, each explained by its reads. - REF alike, and components (operator, 2026-10-14). A read that fits REF counts REF only when it tells REF from every other allele of REF's length at the locus (the germline haplotype beside a merged MNP, a germline change inside a delins's deleted bases): otherwise neither. A co-annotated sibling is another allele only where the fragments spelling it in full are at least one and 5% of those spelling the allele it is nearest: an MNP's component SNV that sign-out reports beside it, never seen apart from the MNP, is part of it (20 panel carriers had been demoted for it), and at depth one error read spelling it no longer makes it another allele (a Monte Carlo at 1000x: 9 of 60 loci flipped on one error, losing about 5% of RD and AD; none now). The annotation lowers RJ-25's 3-fragment floor, not its share. As on the ALT's side, a REF-length allele whose every difference from REF is a change the ALT molecules carry too (a germline change at a base the event keeps, in cis) is a separate event, not another allele; a REF read that is ALT for a co-annotated sibling stays partial evidence (RJ-5). The older sibling claim still takes reads that fit a co-annotated sibling no read shows (#282).
- A change inside the grown event (RJ-23, amended). A REF read holding the given REF allele with a germline SNP in the repeat the event was grown through was neither, silently (RD down by a third to a half at 2 panel rows, a matched normal and a WES row). Such a change, carried by the REF molecules, is now background, masked in both windows where it sits on each allele, so REF reads and carriers carrying it keep their call. One on the ALT molecules alone stays unmasked. The same at the bases an MNP keeps inside its core (operator, 2026-10-14): a germline het there in trans made the germline haplotype's REF reads neither (RD halved, VAF 0.20 -> 0.33 in a synthetic case), and in cis carriers written with an indel at the block were neither while plain ones counted. No real row changes.
- From the adversarial review (2026-10-14). An allele the reads spell must also reach 5% of the fragments reading the same bases in full, so a recurring error at depth is no other allele; one differing from the ALT only where the guard masks is no other allele (it took every carrier); the alleles are found from the variant's own reads only (binning invariance); a molecule may differ from its allele at one base past the event (a phased SNP hid a germline allele); a molecule past a run must read a base telling its allele from the other (a shifted carrier was taken for REF at one HG002 row); every read the MNP check calls ALT follows the rule, and one set aside keeps its N. A second review: an allele the reads spell must recur on 5% of the fragments spelling the allele it is nearest (REF or the ALT), so one as common as a low-VAF ALT is found at any depth; molecules are told by all their bases (a run past the event, or a large deletion, left none placed) with up to two phased changes beside the event; a change beside an MNP's own bases is a phased neighbour, not another allele. A third review: the fragments spelling REF or the ALT are counted past the masks (a masked change on every carrier left none, so any recurring error became an allele); for an MNP the run beside it is beside it on both routes, so a carrier with a phased neighbour there counts ALT however it is aligned. A sixth review: a mask inside the event's span took any base, so a third allele's carriers counted ALT once a germline change had masked the position (it now takes the reference's base and the change's only); a change at a base the event changes, where a third allele's molecules count among the other allele's, is no separate event; and a co-annotated allele faces the separate-event test as a spelled one does.
- Read census. Told the locus's other alleles, it judges a read fitting the ALT
and another allele as neither (
FITS_OTHER). - Tests. Two
--rescue-mnptests' unrescued counts move from (REF 20, ALT 1, partial 9) to (20, 0, 10), with the operator's OK: their masked stray reads only the base it shares with the first-change allele nine fragments spell. A new test keeps a stray ALT read beside the rescue.
Changed — complex events read by one pair of equal-length windows (#253)¶
- The rule (RJ-23). A complex event (delins, an anchor-changing indel, an MNP read with an indel at the block) is read by one pair of windows of equal length, inward from the POS flank: the shorter allele's whole window (its event, grown through repeats, with two flank bases each side) against as many of the longer allele's bases, then on through the rest of the longer allele as far as the read reaches. A REF and an ALT molecule get the same number of read starts that can judge them. Before, the shorter window was padded with flank to the longer one's length, which cost reads, and events past 50 bases were read at both junctions, which let the longer allele count at either end. When both alleles pass 50 bases the windows run from the flank nearer the first difference (an event ending a long run is out of a read's reach from the other); an RNA variant whose POS flank an exon edge cuts reads from the other flank. A read judged from the right counts although it starts past POS, as the other allele's molecule starting there would.
- A change beside the event is a separate event. A change recurring within 25
bases of the event (overlapping mates count once) is masked in both windows, as
RJ-8 treats another indel outside the window, so a read carrying the given
allele counts ALT. One on the ALT reads alone (a phased neighbour, or an event
the input under-describes) is named in
gbcms_diagnosticunder theOBSERVED_ALLELE/COEXISTING_ALLELErule. Withholding those reads was measured and not adopted: on HG002 it took 5 het delins with a phased SNV in cis from VAF 0.42 to 0.09. - Placement. Windows anchor on the aligned base of the reading flank. A read
whose clip or own deletion holds that flank (an aligner using a microhomology at
the event's far end), or whose aligner wrote it over the event's other copy, is
placed from the other flank as the allele its confident bases favour: each
allele's window placed from the same aligned anchor at its own length, its bases
at or above
--min-baseq(not N) must favour one by at least one such base. A REF molecule behind a poor clip (a NovaSeq Q12 bin) shows no placement, where it could fit the ALT through its masked bases; a placement whose allele does not fit is another allele (partial when the ALT's). - Windows run past bases the alleles share. When the shorter allele's flank repeats the longer allele's next bases, both windows run through them to two bases from where the alleles differ: a REF read carrying only a substitution of the event's first base is neither allele. Unpadded windows that stopped before the alleles differ credited such reads: 235 ALT at a 17-to-1 delins in one matched normal (6.6.0, padded: 0).
- Germline changes beside the event are background. A change recurring on the REF fragments too (on 3 or more of them), a run included (a germline MNP), is masked like a germline SNP; a run on the ALT fragments alone is another allele, and no larger allele is named beside it (per-base tallies cannot spell an indel; the observed allele names it from the reads' alignments).
- Measured against 6.6.0 on every data type: the 6.6.0 gate panel's delins rows (IMPACT, ABRA2-realigned) and their matched normals, ACCESS duplex and simplex with their normals, TEMPO WES (BWA only), and HG002 germline het delins (GIAB v5.0q, truth 0.5; WGS 35x, BWA-MEM):
| 6.6.0 | 6.7.0 | |
|---|---|---|
| Panel, all delins (161 rows): within 10% of sign-out | 88/135 | 100/136 |
| Panel, span 20+ (14 rows): AD / RD; within 10% of sign-out | 1,203 / 4,117; 0/14 | 1,648 / 5,433; 8/14 |
| Panel, span under 20 (70 rows) | 9,557 / 24,053; 52/70 | 9,827 / 25,124; 60/70 |
| Panel, recurrent neighbour (49 rows) | 2,398 / 14,609; 19/49 | 3,624 / 15,590; 27/49 |
| ACCESS duplex / simplex (8 rows each): within 10% of sign-out | 4/6; 4/6 | 4/6; 4/6 |
| TEMPO WES (54 rows): within 10% of sign-out | 12/50 | 16/52 |
| HG002, span 20+: median VAF; within 0.35–0.65 | 0.464; 166/237 | 0.477; 180/236 |
| HG002, span 5–19 | 0.500; 124/149 | 0.500; 122/149 |
| Matched normals, somatic delins rows (IMPACT 105, ACCESS 14): rows with ALT | 6; 0 | 8; 0 |
The two normal rows gaining ALT gain one soft-clipped read each whose bases fit
the ALT through the window padding had cut: one of 641 judged reads, and one of
5 at a row whose other reads carry a germline substitution of the event's first
base (neither allele). Pure indels and SNVs are unchanged (195 germline het
indels: 2 count changes; 139 on WES: 2).
- Spec. Reads ending inside a long insertion that hold the REF window's length
are judged (ALT where their bases contradict REF), a carrier ending on the base
after its run is ALT, and a read one base from each allele is neither, not
partial. tests/test_carrier_window_balance.py checks that equal molecules give
AD = RD for shrinking, growing, long, both-long and run-ending shapes, in DNA and
in RNA read from the right.
Changed — deletions of 50 bp or more judged at the POS breakpoint (#254)¶
- The rule (RJ-24). A prepared pure deletion of 50 bp or more is REF or ALT by its own bases at the POS breakpoint, however the aligner wrote it (a D op, a soft clip, a split read). Both alleles are read from the left flank over one window of equal length: 4 flank bases, the microhomology, then 4 bases of the allele's own continuation (REF: the deleted bases; ALT: the right flank). Nothing past the window is read, for either allele. Short-read aligners without a realigner write most such carriers as clips: on HG002 (BWA-MEM) a CIGAR-only rule read VAF 0.000 at 287 of 300 het deletions.
- Placement. A read whose left flank is not aligned where the windows start (in
its clip, under its own deletion, or written over the deleted bases' end) is
placed from the right flank, which both alleles share, as the ALT's window sits
there; its bases at or above
--min-baseqmust favour the left flank over the deleted bases REF has before the right flank by one such base. A REF molecule starting inside the deleted span behind a poor or N clip is depth only. A decided read admitted through its clip, or placed from the right, counts toward depth in DNA. - Beside the junction. A change recurring at one window offset is masked as a separate event (as RJ-23): in the flank and microhomology, a germline SNP or MNP the REF reads carry too; an isolated change of the ALT reads' own; past the junction an isolated change on the REF reads. A run of changes is another allele, and the diagnostic names the deletion with an isolated ALT-only change as the larger allele.
- Siblings. A co-annotated sibling claims a REF read (RJ-5) only from the anchor through REF's window's last deleted base. Over the whole deleted span, as for other variants, it withdrew REF reads the windows never read there and no ALT molecule can carry: 2,667 REF reads at 10 of the panel's 60 rows (one row's VAF 0.17 against the reads' 0.17 became 0.30); now 378 at 2 rows, where the sibling sits inside REF's window. The flank bases before the anchor, which both windows read, stay outside the guard.
- Retired: the 3-base length band for deletions of 50 bp or more and its Phase 3 route; a deletion of another length is another allele at any size.
- Measured against 6.6.0:
| 6.6.0 | 6.7.0 | |
|---|---|---|
| TEMPO WES, BWA only (150 rows): AD; within 10% of sign-out | 4; 0/132 | 6,028; 40/143 |
| HG002 het deletions (280 rows): median VAF; within 0.35–0.65 | 0.000 at 287 of 300 | 0.500; 132/188 |
| Panel, ABRA2-realigned (60 rows): median VAF; within 10% of sign-out | 0.192; 50/60 | 0.188; 45/60 |
| Matched normals, somatic rows (IMPACT 60, ACCESS 2): ALT reads | 0; 0 | 0; 1 |
On the panel ABRA2 writes carriers as D ops, which 6.6.0 and the sign-out both count from the CIGAR, with REF read anywhere over the deleted span; 6.7.0 reads REF at the breakpoint, as the bases-only oracle does (40/60 within 10% of sign-out); against the reads' own VAF 6.7.0 agrees on 50/60 (6.6.0 53/60). On HG002 that oracle reads median VAF 0.514 (105/169 in the band).
Removed¶
gbcms build-gtf-cache,gbcms rna --gtf-cache-dirand the Nextflow--gtf_cache(#269). They were deprecated in 6.6.0, when they did nothing and warned; the GTF loads in a few seconds without a cache.- The command and the option are now usage errors.
- The Nextflow parameter is gone, and the pipeline no longer warns about it.
- Pipelines that still call
build-gtf-cachemust drop that step before moving to 6.7.0.
Fixed — a REF read carrying another indel inside an MNP's windows (#287)¶
- C39's exact-carrier windows counted REF for reads carrying a 24-bp insertion written right after an MNP's block: the inserted bases repeat the right flank, so the block's windows fit REF and nothing past them was read (REF 439 -> 467 at one gate row; 6.6.0 called them neither, as RJ-7 and RJ-21 require). A read whose own alignment changes the length inside the windows with an indel that does not recur as background is now judged by its bases read on to its end; where they show the indel it is another allele (neither, partial evidence). A read whose bases still fit REF stays REF, and a germline indel beside the event keeps its reads' call. Found by the local interim 6.7.0 gate; on the full gate panel 3 rows change (RD -37 in all), on the plan arms one panel, one WES and one HG002 row (RD -1 to -3), normals and ACCESS none.
Fixed — mFSD sizes an ALT fragment as the molecule carrying the ALT¶
- A fragment's size is TLEN less the deletions (plus the insertions) its reads'
CIGARs write. A deletion carrier written as a clip, or judged by its bases, has no
D op, so its fragment read longer than the molecule by the deleted bases (227
instead of 167 for 60 bp), and a fragment of a deletion over about 830 bp fell out
of mFSD. When an ALT fragment covers the variant without a read writing its length
change, the change is applied.
tests/test_mfsd_allele_size.py.
Fixed — release tooling¶
- Attribution leaves out runs whose latest attempt failed (#255).
attribute.pyprepare,checkandattributeread the builds' time records ascompare_panel.pydoes, and name each run they skip. A run that crashed partway leaves a partial MAF, which otherwise reads as changed rows. A reduced run whose latest attempt failed failscheck, andattributetreats its output as missing, whatever it holds. The 6.6.0 gate needed those MAFs set aside by hand.
[6.6.0] - 2026-10-07 — every read judged by its own bases¶
Changed — release infrastructure (#136, #137, #152)¶
- One version source per ecosystem (#136). The Python package takes its version
from
rust/Cargo.toml(maturin,dynamic = ["version"];gbcms.__version__reads the installed metadata), and the Nextflow modules and banner readmanifest.version. A release editsrust/Cargo.toml, the Nextflow manifest and the CHANGELOG instead of 11 places in two formats.scripts/release.py checkruns on every PR, and with--tagit is the release workflow's first job: nothing builds or publishes unless the tag is a bareX.Y.Zequal to Cargo.toml, its lock and the manifest, with a dated CHANGELOG section (a tag on a dev version would have published a dev wheel under the release). Ondevelopthe manifest must name a published image older than the dev package, and the package must move to the next dev version after a release. Only a tag push publishes (a manual run builds and checks), the tag reaches shell steps throughenv:, and the workflow's default token reads only. The Nextflow lint job checks every processnextflow inspectresolves runs the manifest's image;gbcms --versionand the resolved images are unchanged. - The release workflow creates the GitHub Release (#137). After PyPI and the image
are published, it creates (or updates) the Releases entry: the title and notes from
the tag's CHANGELOG section (
## [X.Y.Z] - YYYY-MM-DD — summary; cut at a section, with a link to the full file, past GitHub's body limit), the wheel and sdist attached withSHA256SUMS, and a build-provenance attestation for each artifact. - The bin window is documented as the floor it is (#152).
BIN_WINDOW(10 kb) is a minimum span that dense variants chain past, bounded by the 200-variant cap; measured on real inputs (per-sample lists max 29 kb; one input holding a cohort's variants max 47 kb) and never capped by span. The page no longer cites the retired parity suite.
Added — the regression panel, the release gate (#155)¶
- A panel chosen for coverage replaces the hand-picked 56-sample HPC matrix: every
signed-out variant is tagged with the strata that exercise distinct code paths (70,
including this release's: delins size and direction, indels beside another run,
tandem-duplication inserts, PMS2 at
--min-mapq 0, MNPs near indels, a delins overlapping its component row, ACCESS indels in homopolymers), and a greedy set cover picks the samples. Arms beyond the IMPACT/ACCESS panels: the FORTE RNA truth cohort and a cohort-MAF fillout; configurations beyond production defaults: mFSD with both Parquet outputs, the SW backend, VCF output. - The tools (
scripts/regression_panel/, no patient data): a run-list runner (resumable, SLURM-array shards, wall time and peak memory per run), a comparison (version against version, and each version against the sign-out counts by stratum), and mechanical attribution: one wheel per count-affecting merge (the new manual Checkpoint wheels workflow), run on the changed rows, names the merge where each changed cell changed (the workflow starts by hand, or from a pushedcheckpoint-wheels/X.Y.Zbranch before the release reaches the default branch). The gate and its criteria:docs/development/regression-panel.md.
Fixed — one allele given twice is not its own sibling (#246)¶
- Carriers lost at both rows. An input that gives one allele on two rows had
each row treated as the other's sibling once normalization made them one variant:
the same row given verbatim (a cohort MAF listing a recurrent indel once per
sample), or one allele written two ways (a repeat unit inserted at either end of
the repeat, as FORTE's indel probes do). A carrier the aligner wrote at the
junction counted at both rows; one written anywhere else in a repeat fit both
exactly, so the AD guard counted it at neither: +CTG at a (CTG)x11 tract had AD 7
against 62 when counted alone. Outputs before 6.6.0 undercount such rows' indel
carriers written away from the junction (an SNV base read directly was never
contested). With
--rescue-mnp, an MNP given twice was skipped as grouped. - The fix. A sibling of the row's own allele is left out of its sibling list (the guards, the Phase-3 haplotype matrix and the observed-allele scan all read that list; one entry per other allele), so each row counts every carrier, and the MNP rescue skips only a row whose group holds another allele. A row given verbatim is noted at INFO; one allele given two ways warns, naming the rows. A genuinely different allele at the site stays a sibling.
Fixed — a carrier behind a substitution is not REF; the junction ALT stays (#245)¶
- A carrier counted REF. A readable insert of the variant's length written just outside the window behind a substitution (the anchor read as another base, so the base the insert would place onto the reference does not fit) could not slide back: the read was a separate event and counted REF, though at the variant's junction it costs no more mismatches. Against develop's outputs: RC 19 reads at 2 rows (one BRCA2 allele recurring in two samples), WES 2, FORTE 2; all REF to partial, no ALT or depth changed.
- The rule (RJ-22). Before such an insert is called a separate event, every junction it can reach (through the read's contiguous aligned bases, up to 64 each way) is scored by the read's readable mismatches; a fewest-mismatch placement inside the discrimination window makes it another allele (neither, with partial evidence): never REF, and never ALT, since one read cannot tell a molecule's error from a recurring other allele. ALT is still judged only over the placements a slide reaches base by base (RJ-21). The scan-window gate admits an insert either way reaches the window. An insert with no readable base is unchanged (RJ-20).
- The strict path keeps its junction ALT when a slide could absorb a flank substitution into another insert. On RC (28 reads) each is the given ALT plus one molecule's substitution, no other allele recurring: ABRA2's placement of a singleton. Every surveyed tool credits it ALT; a contract test pins it.
Fixed — a same-length insert of other bases is judged by its bases (#243)¶
- Phase 3 overrode the bases. A read whose insertion had the variant's length and other readable bases, near the variant, went to Phase 3, whose closer haplotype let length win ALT against those bases (Smith-Waterman under both bio versions, PairHMM more often under bio 4) and could absorb another allele into REF. On the RC DNA set it was the only way a read reached Phase 3 at an insertion row: 50 of 94,530 calls.
- The rule (RJ-21). Such a read is judged by its bases, never Phase 3, and over every placement its insert can take, not only the aligner's: the insert slides a junction when the base it places onto the reference fits it (the same base, or a masked one), so a sequencing error the aligner absorbed into the flank, or a compensating mismatch, slides back. It is ALT when a placement shows the ALT by read bases (at the variant's junction, judged as the strict path judges an insert there, or the variant's haplotype in a repeat) or its bases across the window spell the ALT; otherwise another allele when a placement sits inside the discrimination window, and a separate event (RJ-8) when none does. Only the best-aligned placements count (a slide that takes in a read mismatch resolves it). An insert with no readable base of its own does not slide (RJ-20). Both backends agree. The deletion side already judged by bases.
Fixed — an insertion's ALT needs one of the read's own inserted bases read (#240)¶
- Reads whose inserted bases nobody can read counted ALT. A read carrying an
insertion of the right length whose bases are all N (fgbio masks a duplex
disagreement to N at Q2) or all below
--min-baseqwas credited ALT on length alone: the strict path found the bases unverifiable and handed the read to Phase 3, where REF pays for the gap, under both backends. Shifted placements went the same way, and the deleted-anchor and split-op reading accepted a readable flank base. On the RC DNA set that was 127 reads in 20 of 1,060 rows, mostly 1bp insertions into homopolymer runs with a duplex-masked N inside the run (the aligner writes the N as the insertion). Three rows lost 62, 15 and 11 ALT reads (430 → 368, 299 → 284, 111 → 100), the other 17 one to eight; depth is unchanged and the reads move topartial_alt. The WES loci: 21 reads in 8 of 80 rows. The FORTE sample: 8 in DNA mode (both backends) and 9 in RNA mode, all low-quality letters. RNA truth and probe samples are unchanged. - The rule (RJ-20). A read whose inserted bases are all unreadable is neither,
with partial evidence, on every path: at the junction, at another placement inside
the discrimination window, across several ops, after a deleted anchor, and as a
truncation. Written outside the window such an insert is a separate event (RJ-8).
"Readable" means not N and at or above
--min-baseq, the gate SNV bases pass. A read whose readable inserted bases match the ALT stays ALT, however many are masked. The read census holds the same rule. The post-splice path already counted such reads this way. - Community practice. Likelihood callers (GATK, bcftools, Strelka2) credit ALT on length; sequence-keyed counters do not: VarDict and freebayes drop such a read, bam-readcount and LoFreq report it as its own allele, and GetBaseCountsMultiSample counts it as neither. gbcms counts the given allele from the read's own bases.
Fixed — Fisher's exact test at any depth (#239)¶
- Strand-bias p-values were 0 at depth.
strand_bias_p_value,fragment_strand_bias_p_value(VCFSB_PVAL,FSB_PVAL) and merge's combined columns were 0, as if strongly biased, whenever the binomial C(n, forward total) passed f64's range. That needs both strands well covered: from about 1,030 reads on a strand-balanced table. Tables with one thin strand stayed correct. On the RC cfDNA rows that was 316 of 1,060 read-level p-values, 298 of them truly ≥ 0.05. The ASJD junction test (p < 0.05) read the same zeros. - The fix. The test is now the exact two-sided p of R's
fisher.test, computed in log space from the ratio of neighbouring tables, walking out from the most likely table: no factorial, nothing overflows, relative error near 10⁻¹³ at any depth, and tables too unlikely to count are never visited. - Ties follow R's rule (probability ≤ observed × (1 + 10⁻⁷)). That replaces an absolute 10⁻¹⁰ tolerance which floored strongly biased tables near 10⁻¹⁰; one printed 2.1 × 10⁻¹⁰ where the exact p is 7.0 × 10⁻⁹⁰.
- The p = 1 guards (≤ 1 ALT observation, empty or full margins) and the odds ratio are unchanged.
- The p-value stays the exact p of the raw counts. At deep coverage, read the odds ratio for the size of a bias.
- Measured on the local FORTE sample (9,536 rows).
- What moved: 397 read-level and 205 fragment-level strand-bias p-values, and 5 ASJD p-values, of which one flag clears. All counts and Parquet tables are unchanged.
- Every reported strand-bias p now equals scipy's exact value.
- statrs leaves the direct dependencies.
Changed — Rust dependencies on their current releases (#139)¶
The Rust dependencies move to their current releases, one per commit (PR #238).
No output changes. bio follows once C35 (#240) and C36 (#243) are in.
- pyo3 0.29 (from 0.27). The API migration: allow_threads is now detach,
and Variant and BaseCounts opt in to the by-value conversion that Python
passes back. The module keeps the GIL on a free-threaded interpreter, as before;
gbcms is not built or tested free-threaded.
- rust-htslib 1.0, arrow/parquet 60, and the semver-compatible updates.
(statrs went to 0.19 here, then left the dependencies with #239.)
- parquet 60 truncates column-chunk min/max statistics of strings longer than 64
bytes. The data are unchanged.
- wfa2lib-rs moves to upstream's current commit without its default features.
Those only built its benchmark binary, so clap, tracing and mimalloc leave the
build.
- bio 4.2.1 (from 3.0), held until C35 (#240) and C36 (#243) landed.
- The PairHMM: bio 4.1 fixes how it scores the bases of an extended read
insertion (rust-bio#701). That moved reads decided in Phase 3: ones whose
inserted bases are N or below the base-quality floor crossed ±2.3 (synthetic:
1.8 → 5.0), and same-length inserts of other bases leaned to ALT. Since C35 and
C36 neither reaches the PairHMM (RJ-20, RJ-21): no pure insertion is decided in
Phase 3 on the measured sets.
- Smith-Waterman: bio 4 counts a gap's first base in the open penalty
(rust-bio#660), so SW_GAP_OPEN moves from −5 to −6 and every score is
unchanged.
- Cargo.toml caps bio at ~4.2 (its minor releases have changed numerics). bio
keeps its own statrs 0.18; the new crates in the lock are unused optional ones.
- On develop with C36, bio 4 changes no output: the local FORTE sample (9,536 rows,
both backends, every Parquet table) and the cluster RC sets (144 files) are
byte-identical.
- Unchanged output:
- Local FORTE sample (RNA and DNA modes, both alignment backends, 9,536 rows
including 806 indels observed in the reads): compared after step 1, after steps
1–5, and for the final tree. No column and no Parquet table changes.
- RC sets on the cluster (28 cfDNA DNA, 33 RNA truth, 3 probe samples and 80 WES
loci; 2,212 rows): byte-identical.
Changed — dependencies, CI coverage and the image (#139)¶
Measured first (PR #231). Nothing changes at runtime.
- The dependency floors are true. At the declared floors the CLI crashed: typer
0.9.0 cannot read its list[...] options. pysam 0.21.0 has no macOS arm64 wheel
and an sdist that fails on current setuptools. Bisected one package at a time, the
minimums are typer>=0.15.4 (older typer does not cap click and breaks with
click >= 8.2) and pysam>=0.22.0; rich 13.0, pydantic 2.0 and polars 1.0 held.
All of them together pass the suite on Python 3.10 and 3.11. A new CI leg installs
the floors on every PR and checks that they are what got installed.
- click is no longer a runtime dependency. typer bundles its own copy, and gbcms
never imports click; the CLI tests do, so it moved to the test group, and the
image no longer installs it.
- CI tests the supported range. Python 3.10–3.14 all pass on the latest
releases, but CI tested only 3.11 and 3.12. PRs now run Ubuntu 3.10 (at the
floors), 3.11 (the image's) and 3.14, plus macOS 3.12, and the classifiers list
3.10–3.14. A monthly workflow (latest-deps.yml, also run by hand before a release)
runs every version on the newest releases with the semver-compatible Rust updates. On failure it opens or updates
one latest-deps issue.
- One dev-dependency list. The dev extra and a PEP 735 dev group had drifted
apart. The group lacked pytest-mock, types-pyyaml, pyyaml and mkdocs; the extra
lacked pyarrow, which the tests import. They are now the dependency groups test
and dev (dev includes test and docs, the site's tools, which
deploy-docs.yml now installs instead of its own list), kept out of the published
metadata.
pip install gbcms[dev] no longer exists. maturin develop installs dev, which
needs pip >= 25.1 (or uv); the developer docs now say so, since the old
python -m venv steps failed on Python's bundled pip. make setup installs it too.
- The image installs a hash-pinned lock. It resolved its dependencies afresh at
each build. docker/requirements.lock (linux/amd64, Python 3.11) is installed
with --require-hashes --no-deps, then pip check, and is refreshed at each
release (release guide step 4). CI now runs the image (--version, pip check).
The 6.6.0 lock takes polars 2.0, released the day before the cut: none of the
APIs it removed are used, and the suite passes on it. gbcms uses polars only in
gbcms merge, the mFSD report and the batch IO helpers, never to count.
Changed — GTF loading (#139)¶
Measured on GRCh38.111 (PR #230). Only --gtf runs are affected, and their output
is unchanged for well-formed GTFs.
- The GTF loads in 2 s instead of 9 s, plain or .gtf.gz. noodles-gtf parsed
every line before reading its feature column. A byte-level parser reads each
line's feature and chromosome first and checks only the exon lines it keeps.
Measured in gbcms rna on Ensembl 111: 1.6 s for 16 chromosomes (8.9 s before),
2.0 s for the whole genome (9.3 s), 2.4 s from .gtf.gz. GENCODE v50 loads in
4.0 s (basic) and 6.5 s (comprehensive) from .gtf.gz. Peak memory falls (407 MB
vs 470 MB for 16 chromosomes): gene_id, stored for every exon and never read,
is gone.
- It loads exactly the exons noodles did: no line differs across Ensembl 111 and
GENCODE v50 comprehensive, v50 basic and v47lift37 (26.1M lines).
- gzip and BGZF are detected from the file's content. The config already accepted
.gtf.gz, but the parser failed on it with "stream did not contain valid UTF-8".
- Malformed exon lines are rejected and warned about. The grammar is noodles-gtf's,
column by column, except in four places where noodles loaded a line wrongly:
- A whitespace run may now separate a key from its value. noodles read
transcript_id "T1" as the ID "T1", quotes included.
- Unquoted values are trimmed.
- A line with start after end is rejected; noodles kept it as a negative-length exon.
- A coordinate past 2,147,483,647 is rejected; noodles wrapped it negative.
- An exon with an empty transcript_id is rejected, as a missing one already was.
noodles merged every such exon into one transcript.
- A transcript_id that strips to nothing (".", ".1") is rejected the same way.
- An exon line that is not UTF-8 text is rejected; it used to stop the run.
- Each of these lines used to move exon_boundary_dist with a bogus boundary.
Rejected lines now get one warning with their count and the first one's line
number and reason.
- The GTF index cache is deprecated. On Ensembl it saved 1–1.4 s a sample: a hit
cost 0.9 s in the CLI (~0.3 s of it start-up) against a 1.6–2.0 s load now. It
brought a build step, a Nextflow process and a 97 MB file per chromosome set. It is
no longer used:
- --gtf-cache-dir, gbcms build-gtf-cache and the Nextflow --gtf_cache are
accepted and ignored with a warning, and are removed in 6.7.0.
- The GBCMS_BUILD_GTF_CACHE process, the cache module and its binding are gone.
- The noodles-gtf, bincode and serde dependencies are gone too. bincode 3.0 is a
compile_error! release, so there was no upgrade path.
Changed — mFSD statistics (#153, #154)¶
Measured first on ACCESS plasma labeled by the patient's buffy coat (operator
decisions 2026-09-25, refined 2026-10-05): mFSD is graded, plasma-only evidence,
never an origin call; the CH-vs-tumor prediction belongs to a separate model that
reads <sample>.fsd.parquet. Only --mfsd output changes.
- The fragment-size LLR is the mean per fragment (#153). mfsd_alt_llr and
mfsd_ref_llr (VCF MFSD_ALT_LLR, MFSD_REF_LLR) were sums, which grew about
22x with depth on real cfDNA; the mean stays comparable, with n in
mfsd_*_count. An empty fragment class writes NA for its mean size and LLR (it
wrote 0, where the docs already said NA; the N class is empty on most rows).
- The mFSD report grades the evidence (#154). Its classes are LEANS-SOMATIC
(ALT fragments significantly shorter than REF: KS q < 0.05 with the ALT ECDF
above the REF ECDF where they differ most), NO-SIZE-EVIDENCE and
INSUFFICIENT, replacing TUMOR-LIKE / CH-LIKE / AMBIGUOUS. CH-LIKE rested
on a non-significant KS, which is also the usual result for tumor variants at
these fragment counts: on 39 labeled samples it called 3 of 17 tumor variants in
CH-associated genes CH-LIKE, and about 30% of tumor variants below 50 ALT
fragments looked REF-like. Nothing leans CH now. Gene membership is a note, not
a gate (a tumor TP53 variant could never be TUMOR-LIKE), and the enrichment
1.3 gate is gone: on duplex fragments the new rule leans somatic on 22 of 97 tumor variants and 0 of 39 white-cell variants (the old rule found 16). The direction is read from the KS gap, not from the sub-nucleosomal share, which missed an ALT 31 bp shorter with no fragment under 150 bp. A significantly longer ALT is not somatic evidence. Variants below
--mfsd-report-min-altare left out of the report, as before. - The KS p-value is exact up to 10⁷ lattice cells, with Stephens' corrected asymptotic series above. The switch atn·m10,000 sent most real ALT-vs-REF pairs (348 of 512) to the uncorrected series, which overstated p 1.7–2.3x at 5 ALT fragments near p = 0.05 and up to 45x for a strong shift. The exact value compares lattice points with the observed deviation in integers (a float band could count the observed point as not reaching D near the cap), computes p directly as the share of paths leaving the band (a tiny p keeps its digits), and visits only the cells inside the band. - A variant on a contig absent from the BAM stays out of the mFSD q-values. With--mfsdits fields held 0.0, which read as a KS test with p = 0 and deflated every real variant's BH q-value; it now reads as no test (NA). -mfsd_alt_confidenceisTESTABLE/SPARSE/NONE(wasHIGH/LOW/NONE): it names how much ALT data there is (≥ 5, 1–4, 0 fragments); at 5 fragments a real size shift is detected only about 8% of the time.
Changed — merge, outputs and observability (#194, #221, #223, #224, #129, #148, #130, #131, #156, #220, #225)¶
Measured first (operator decisions 2026-09-30 for #194, 2026-10-05 for the rest);
community practice surveyed (bcftools, Picard/htsjdk, GATK, samtools/htslib,
maftools, genotype_variants, Snakemake, Nextflow, bam-readcount, LoFreq, fgbio,
DeepVariant, Strelka2). Where no tool sets a standard, gbcms now does more.
- A missing count is not a zero in gbcms merge (#194). When either flavor's
count cell is missing or not a finite number (empty, NA, nan, inf, text)
in a row it has, the combined simplex_duplex_* cell is NA, as are the
totals, VAFs and strand bias built from it, and merge warns once per column
with the number of rows (it summed them as 0, silently). A row an input lacks
still counts 0 for it. Combined VAFs are written with four decimals, as the
writers write theirs (f"{v:.4f}"), and the combined strand-bias p-values and
odds ratios as theirs (1.7045e-01, 3.0000). No real output carries such cells (0 of 180,348 count
cells measured); they come from edited files or other tools.
- Merge keeps every gbcms column per input (#223). The mFSD and RNA columns
(63 of the 89 columns the writers can emit) were taken from the first input
only, unprefixed, and every later input's were dropped: with --mfsd, which
the pipeline allows alongside merge, a merged ACCESS output showed the duplex
BAM's fragment sizes as the sample's. They are prefixed per input
(duplex_mfsd_ref_mean), the set taken from the writer, with a test that it
matches in every mode. Inputs written with --column-prefix (duplex_ as the
pipeline runs it, t_) carry their counts under that prefix and their status,
strand-bias, mFSD and RNA columns unprefixed: merge renamed nothing for them, so
the second input's status and strand bias were dropped too, and t_ counts were
never combined. Every column is now found under the writer's name.
- A row only a later input has keeps its annotations (#221): its first-input
columns (gene, sample barcode, classification...) come from the earliest
later input that has the row (matched by that input's own row, so one variant
listed for two samples keeps each sample's); they were empty. No work when
every row is in the first input, as in the pipeline.
- Merge reads its inputs' provenance (#129). The merged MAF starts with its
own #gbcms/#command lines and one #input line per input with that
input's version line (it had none). Merge warns when the inputs come from
different versions or builds (builds differ only when both name a commit), or
when only some say which version wrote them, and stops when one is a VCF-input
MAF from before 6.5.0 (vcf_pos without vcf_ref/vcf_alt, its version line
missing or older than 6.5.0; a later output can carry vcf2maf's vcf_pos) and
another is not: the
same 102k VCF records genotyped by 6.4.0 and by this version differ in 6.3% of
their rows, which merged into 12,771 half-empty rows with exit 0. The INFO line
that claimed "n/m variants have no simplex_duplex_* cells are NA; they added the two
alleles' counts, with only a warning. A row one flavor lacks is not mixed. 20
real ACCESS pairs had none; the warning names each such row.
- Builds name their commit. Provenance lines (#gbcms, VCF ##source, run
logs, gbcms --version) read gbcms v6.6.0.dev0 (9c371263); develop now
carries a .devN version, since every development build since 6.5.0 reported
6.5.0. The commit comes from git, or from GBCMS_BUILD_COMMIT (set by the
Dockerfile and the release workflow).
- VCF output of a whole-contig deletion is valid (#220, pulled in from
6.7.0). A MAF deletion at Start 1 spanning its whole contig has no reference
base before or after it; it was written with a REF padded past the contig end
with N (ACGTACN > N). It is now the symbolic <NON_SEQUENCE> record (REF
the base at POS), in counting runs and gbcms convert; the row was already FAIL
(FETCH_FAILED). None in real data.
- No partial output files (#148). Every output (MAF, VCF, the merged MAF,
convert and normalize files, both Parquet files, the mFSD report) is
written to .<name>.partial, fsynced, and renamed into place; a failed run,
including a failure while closing (a full disk), leaves nothing at the output
path and no temp file, and the observations Parquet goes into place only after
the MAF/VCF. A symlinked output keeps its link (its target is replaced), a
replaced file keeps its permission bits, and a device or FIFO (/dev/stdout)
is written in place. It left a truncated file
under the final name (htslib tools, GATK and Picard do too; only workflow
managers clean up).
- Per-BAM warnings once (#130). The --rescue-mnp re-count no longer repeats
the records-without-bases, records-without-qualities and absent --umi-tag
warnings, and those records are counted once each (the count was an upper
bound: overlapping bins fetched a record more than once).
- The run start says what the run does (#131). One INFO block lists every
resolved option (generated from the configuration, so none is left out) and
what each count- or column-changing option implies; one INFO line per BAM
gives facts from its first 20,000 records (duplicates flagged, base-quality
values, the share below --min-baseq, ALT contigs, hard-clipped primaries). It
warns only when --min-baseq removes more than 10% of sampled bases, the
header has ALT contigs, or more than 1% of primaries are hard-clipped: on 10
MSK BAMs the default removes 0.8–2.1% and none has ALT contigs or hard-clipped
primaries. Unmarked duplicates are reported, not warned: ACCESS consensus and
FORTE RNA BAMs carry none by design. The CLI's four-setting Config: line is
replaced by the block. Implications are stated against the mode's defaults (an
RNA run's are RNA's), RNA amplicon and strandedness settings included; turning
off the secondary or supplementary filter lets those alignments join fragment
evidence, never read counts. A BAM the facts cannot read is left to counting.
- The docs toolchain is pinned below MkDocs 2.0 (#225, the pin half of #138,
pulled in from 6.7.0): mkdocs>=1.6,<2 and mkdocs-material>=9.5,<10 in the
dev extras and the docs workflow (both installed unpinned), with a test; the
MkDocs 2.0 migration stays in 6.7.0.
- One page for every QC flag, each defined once (#156):
docs/reference/qc-flags.md, a table per family (status reasons, diagnostics,
rescue outcomes, ASJD, QC columns, mFSD classes, VCF record shapes) with mode,
MAF column, VCF field and what to do. Each family is a snippet section that the
pages needing the table include (normalization, architecture, RNA annotation,
output formats, mFSD report); the rest link to it. Tests fail when a flag the
code emits is missing from the page, when a flag is defined anywhere else (a
table row or bullet), when a line lists three or more flags without linking it,
or when an include names a missing section; the docs build now fails on a
missing snippet or anchor. Consolidating found copies that had drifted: the mFSD
report's class table and flowchart (thresholds 1.0 and the raw KS p-value; the
code uses 1.3/1.2, the FDR q-value and the CH-gene check), its mfsd_ch_flag
("CH-like profile"; it marks a CH gene), output formats' STRAND_DISCORDANT
(from before intronic loci got a strand), the VCF GS row (the pre-6.0
combined status; GSR was missing), and the report's own tooltips (raw p).
Changed — input and representation (#122, #123, #124, #125, #126, #147, #208, #149, #218)¶
Measured on the MSK sign-out dump (1,133,044 rows; operator decisions 2026-09-25
and 2026-10-05).
- Non-sequence MAF alleles are FAIL rows (#123). An allele that is not a base
sequence (an IUPAC code such as R, ., a stray character from a hand edit)
made the row count 0 silently; it is now a FAIL row, NON_SEQUENCE_ALLELE,
kept in MAF output. Lowercase bases are bases; - is a MAF dash allele, so a
- given as non-MAF input (the observations API) is one too. VCF output,
which cannot carry such an allele (nor an empty one), writes the symbolic
record <NON_SEQUENCE> (REF the reference base at POS, declared in the
header), in counting runs and in gbcms convert. 7 such rows in the sign-out
data.
- A REF_MISMATCH row says where the given REF sits (#218). The row stays
FAIL and uncounted; gbcms_diagnostic (VCF GD, and gbcms normalize's
new gbcms_diagnostic column) gives REF_AT_OFFSET(k) when the REF (3+
bases) matches the reference exactly within 3 bases of its position, every
such offset listed nearest first. In the sign-out data 87 of 154
REF_MISMATCH rows sit 1–3 bases off, mostly legacy ANNOVAR-annotated
indels at Start−1 whose alleles still carry the VCF anchor base. No tool
surveyed moves or explains such a row (bcftools norm --check-ref exits,
warns, excludes or fixes REF in place; maf2vcf skips it).
- VCF output of MAF input names its MAF row. Every record carries
MAF_START, MAF_REF and MAF_ALT: the row's Start and alleles as
written (a placeholder such as 0 too; percent-encoded, . when empty), so
a result can be looked up by its input, as VCF input's MAF output carries
vcf_pos, vcf_ref and vcf_alt. Also in gbcms convert.
- A MAF deletion at Start 1 is counted (#122). It has no base before it; it
is resolved to the VCF spec's position-1 form (the base after it), as gbcms's
VCF output already wrote it, and counts as the same event given as VCF does
(it was FETCH_FAILED). None in the sign-out data.
- End_Position is optional (#124). gbcms places a variant by
Start_Position and its alleles; rows without an integer End_Position were
skipped and are now read (maf2vcf converts them). gbcms merge no longer
joins on it: inputs that write it differently for one variant join into one
row (the first input's End_Position, the difference logged), and a row only
a later input has keeps that input's. Where an input lists one variant twice
with different End_Position, it joins on End_Position too, so each row
pairs with its own counterpart. The sign-out data has it, consistent
with Start and REF, on every row but one; no two of its rows differ only in
End_Position.
- VCF input's MAF output fills Tumor_Seq_Allele1 (#125) with the reference
allele: MSK's sign-out convention on every row, and maf2vcf's reading of an
empty one. Previously empty.
- One row, one allele (#126): a Tumor_Seq_Allele1 that differs from both
REF and Allele2 is not a second allele (vcf2maf's reading; cBioPortal picks
Allele1). Documented and tested; no such rows in the sign-out data.
- Engine API (#147): a decomposed or sibling_variants list shorter than
the variants is padded; a longer one raises ValueError (a short decomposed
list panicked; a long list was cut silently).
- One allele-kind rule (#208): the AD-claiming guard's pure-indel test and
the pure-indel windows use allele_kind (case-insensitive anchor; a
multi-base shared prefix is complex, as the dispatcher counts it). No prepared
sign-out row changes; it reaches only lowercase or unprepared engine input.
- is_indel in preparation is ref_len != alt_len (#149; no change).
Changed — RNA: strand at intronic loci, splices are not coverage, clips and junctions (#185, #198, #186, #173, #213)¶
Five RNA rules (operator, 2026-10-04), each measured on FORTE against the
previous build: the truth cohort (94 signed-out rows), the T9 exon-edge indel
probes (978) and the C1 splice probes (7,224 delins at exon edges and mid-exon,
where no read carries the ALT).
- Gene strand at intronic positions (#185). A position no stranded exon
covers (intronic, splice sites included) takes the strand of the transcripts
spanning it, all agreeing; where both strands' genes cover a position there is
no strand and every read counts (both genes' transcripts carry the allele).
The exon index read the first and last intron bases as exonic (an end-inclusive
interval tree built from half-open exons): fixed, for the strand and the
per-transcript counts. The unresolved-strand warning names the variants.
Truth: no row changes; T9: 95 antisense REF reads move to
rna_antisense_depth on 62 rows.
- A splice is not reference coverage (#198, RJ-17). A pure-indel read is
informative only when one aligned block between splices spans the window, REF
and ALT alike. Truth: no row changes; T9: 24,703 REF reads at splice-crossing
deletions become depth only (their bases fit both alleles). vaf (alt over
REF plus ALT) at such a deletion with carriers rises to the VAF among the
reads that show the event; no measured row's vaf moved (the T9 rows have no
carriers). A deletion written right after a read's splice (no aligned flank)
is depth only.
- Diagnostics read the counted reads (#186). OBSERVED_ALLELE and
COEXISTING_ALLELE take n/m over the reads the counts read: no antisense read
under enforcement, and the RNA mapping rule's unique mappers. Counts unchanged.
- An RNA read's clip at an exon edge is not allele evidence (#173, RJ-18).
STAR clips a junction overhang it cannot splice, so a clip reaching an exon
edge or junction end (annotated, or one the reads splice at), or ending within
five bases of one, holds the next exon's bases and is not read. Any other clip
is the read's own bases and is read as in DNA (a mid-exon MNP carrier with its
second base clipped still counts ALT). Splice probes: spurious ALT 19 → 16.
- A spliced read is read across its junctions (#213, RJ-19). The
exact-carrier windows of a read spliced near the event are built over the
reference spliced at its own junctions (the far exons read from the FASTA,
every junction the windows reach followed), not cut at the exon edge; a
junction starting inside the event splices the haplotypes at its edge, so a
delins carrier counts however the gap is written; such a read counts ALT only
when its bases also beat REF spliced at its own junction (an alternative
donor or acceptor), and is otherwise depth only. Splice probes: spurious ALT
19 → 4; REF −0.95% at probes 0–1 bp from the exon edge (reads reaching one or
two bases past the junction hold no spliced flank), −0.11% at 2–4 bp.
- All five together, against the branch point: splice probes spurious ALT
19 → 2, REF −0.39%, partial −453; T9 106 rows (REF −24,813, mostly R5); truth: 4 rows'
per-transcript columns (the exon-index fix), no count; RC DNA and WES
byte-identical (before the review follow-ups; RNA-only changes since).
- Survey: GATK splits RNA reads at N and counts a piece only when its bases
favour an allele, and runs HaplotypeCaller with -dont-use-soft-clipped-bases;
bcftools never uses spliced reads for indels; phASER, WASP and ASEReadCounter
never read clips; REDItools resolves a site's strand from the annotation
spanning it and leaves mixed strands undetermined; allele counters take every
tally over one filtered read set.
Documented — why the RNA mapping-quality default is --min-mapq 1¶
No behaviour change. The RNA default keeps reads STAR placed at two to four loci
(MAPQ 3 or 1), counted once at their primary alignment, because junction reads tie
between a gene and its processed pseudogene. Measured on FORTE: counting unique
alignments only would cost real ALT reads at genes with pseudogenes (PIK3CA E545K:
10 of 159; 12 across the truth set) and 1.6% of junction fragments at the probes;
--min-mapq 0 would add 6 ALT and 33 REF reads across the truth set. The STAR
MAPQ scale and the measurement are in docs/reference/read-filters.md, which also
corrects a note that STAR gives novel junctions low MAPQ (its MAPQ depends only on
the number of loci).
Changed — an exact-carrier ALT call needs quality-weighted evidence (#174)¶
A read's bases across a complex variant's windows are now weighed by their
quality (a base matches with 1 − e and mismatches with e/3, e its error
probability), and an ALT call needs them to favour ALT over REF by at least what
one base read at --min-baseq gives (about 2.5 log10 at the default 20). Bases
matching both alleles cancel: the evidence is the weight of the bases that
mismatch REF less those that mismatch ALT, including a long event's bases past
its junction windows.
Clearly read bases must still match the ALT, as before. A low-quality base now
counts for little instead of fitting either allele.
- Why: reads whose window was mostly low quality could be called ALT on one
clear sequencing error. Synthetic probes where no read carries the ALT found
2.0 spurious ALT reads per million in FORTE RNA, 1.8 in IMPACT, one in WES and
none in ACCESS duplex or simplex.
- Measured: on the probes, RNA spurious ALT 70 → 15–19, IMPACT 7 → 1, WES
1 → 0; on every complex DNA/WES row with ALT reads, 1–2 of 1,185 real ALT
reads lost. Two simpler rules were measured and set aside: requiring every
event base read (46 real reads lost) or at most one masked (15 lost).
- Not in this change: spliced reads whose junction an aligner placed a few
bases late at an exon edge (about 9 of the remaining RNA probe calls), and a
REF molecule with one clear error just outside the window its ALT reading is
anchored away from; 6.7.0.
Changed — reads deleting the anchor, and the ALT written across several ops, are judged by their bases (#202, #201)¶
- A read whose own deletion covers a pure indel's anchor (#202) is judged by its bases between its nearest aligned flanks: ALT when they equal the ALT (masked bases fit, at least one base read), with the aligner's placement of its gap only a tie-break; REF when they equal the REF (the reference written as the anchor deleted and re-inserted); otherwise neither, with partial evidence. Before, Phase 3 credited such a read to whichever of REF and ALT was closer, so reads holding another allele counted ALT or REF; at insertion rows they counted REF.
- The ALT written across several insertion or deletion ops (#201), such as a deletion written as two or a 1bp deletion in a run written as D2 + I1, counts ALT when the read's bases spell it. Before, it counted partial.
- Measured: develop vs this branch on RC DNA, FORTE RNA and WES, every MAF cell compared.
- The split-op rule changed no row (144 of 144 files byte-identical).
- The anchor rule changes 33 rows: ALT net 0 on RC DNA (±1 read at 6 rows), −2 on WES; REF +57 and partial +210 at the BRCA2 cluster, where reads that were a co-annotated deletion's false ALT now count REF where their bases across the row's window are REF.
- Against the read census, the changed indel rows move toward it: summed distance REF 344 → 251, ALT 40 → 44 (masked-base edge cases).
Changed — what a read contributes: adapter read-through, absent qualities, unmapped records (#176, #182, #183, #207)¶
The read-judgment spec gains RJ-10 to RJ-12, which decide what a read brings
before any rule reads it.
- A read ends at its fragment end (#176). When the insert is shorter than the
read, the bases past the mate's 5' end are adapter. They are now hard-clipped as
the read enters counting, as if the read had been trimmed, so they are neither
bases nor reach in any rule.
- Only adapter-like bases are clipped: soft-clipped, or at most two aligned past
the boundary (an aligner's chance extension into adapter), none inserted.
- A read whose bases go on aligning past the boundary, or hold an insertion
there, keeps them. TLEN is a reference distance, so it leaves out an ITD's
inserted bases and a mate's clipped 5' bases.
- Only an inward-facing pair defines a fragment; an outward one (TLEN negative
on the forward read) is left alone.
- In MSK data the first base past the boundary is A (the adapter's first base)
in 97% of clipped reads.
- Records without base qualities (QUAL *) are dropped by the read filter
and warned once per BAM (#182). Before, they voted as Q255, and the fragment
consensus margin could overflow; it now saturates.
- Unmapped records (flag 0x4) are dropped by the read filter (#183). An
unmapped mate placed at a variant counted in mq0_count, and at
--min-mapq 0 as a read (an ALT read when it carried the ALT). A read whose
mate is unmapped still counts, and mapped MAPQ-0 alignments stay countable at
--min-mapq 0 (pseudogene loci such as PMS2).
- Fixed: the complex classifier read a hard-clipped read's anchor quality
from the wrong base (#207).
- Measured: develop vs this branch on RC DNA, FORTE RNA and WES, every MAF
cell compared, at the default MAPQ and at --min-mapq 0.
- 182 rows change at the default MAPQ. RC DNA: 160 rows, ALT −57, REF −66. WES:
13 rows, REF −12. FORTE: 9 probe rows, REF −3; the truth set is unchanged.
- Every changed row is explained by reads that read through. All 57 lost ALT
reads had their ALT on a removed adapter overhang; 52 of the 54 at SNVs
showed the adapter's A.
- On SNV rows the change equals the read census's (ALT −53, REF −31). On the
57 changed pure-indel rows the engine moves toward the census: summed
distance REF 167 to 125, ALT 69 to 62.
- At --min-mapq 0: 184 rows. The two extra rows lose one REF read each to
the same clip on reads below MAPQ 20. The PMS2 row keeps its MAPQ-0 reads
(REF 5,439 at MAPQ 0 vs 5,414 at the default). No row moved by an unmapped
record.
- The FLT3 ITD a first build lost 13 ALT reads at is unchanged.
Changed — which reads count REF for a pure indel; long complex events read through the read (#200, #199)¶
Read judgment now has one spec, docs/reference/read-judgment.md:
a table of read shapes and their calls, with the decision behind each, executed
by tests/test_read_judgment_spec.py. Changes to how reads are judged are
decided against it.
- Another indel inside the discrimination window means not REF (#200). A
read carrying another insertion or deletion inside the window (not the ALT
at another placement) is neither, with partial evidence: its bases are not
REF there. This extends the sibling REF guard (#119) from co-annotated rows
to any indel. Most visible at deep slippage loci: a BRCA2 cluster where an
unannotated 1bp deletion in an A run lies inside two rows' windows, and
homopolymer runs.
- Another indel outside the window is a separate event (#200). Of any
length, it leaves the read REF where its bases across the window are REF,
with no partial evidence. Before, a 5bp-or-longer one withdrew REF in a
repeat, and kept REF with partial evidence in unique sequence.
- Long complex events read through the read (#199). The junction windows
of an event too long for one read are read on inward as far as the read
reaches. A read whose later bases contradict an allele is no longer called
that allele: for example, a substitution-only read reaching the end of the
run after a C>TA or CA>T, or a read that keeps the anchor and changes the
run length.
- Measured: develop vs this branch, on RC DNA, FORTE RNA and WES, with every
MAF cell compared.
- 112 of 144 files are byte-identical. 155 rows change, all from the
inside-window rule: REF −4,327, partial +4,324. ALT and depth are
unchanged everywhere. The long-event change moved no row.
- Checked against the read census (bases only): on the 105 changed DNA/WES
rows it counts 57,199 REF reads, develop 60,619, this branch 57,218. On
every FORTE row, the reads moved out of REF are no more than those whose
bases contradict both alleles.
- The largest moves:
- a BRCA2 cluster: the +AAG row goes 1,413 → 408 REF in one sample
(census 368), the 12bp-deletion row 1,213 → 402;
- a 1bp-deletion row in a homopolymer: 1,009 → 698;
- a FORTE T-run probe: about −1.5% REF.
Changed — code-quality sweep: logging, monitoring, dead code, duplication (#204)¶
Counts are unchanged on prepared input: 144 of 144 acceptance files (RC DNA,
FORTE RNA, WES) are byte-identical to the previous build, every MAF cell compared.
- Logging.
- Per-read WARN and DEBUG lines move to trace.
- Rows the engine can only judge degraded are warned once per counting pass,
with their count: indels and complex variants without a prepared reference
context, MNPs whose REF equals ALT, and rows with an empty allele.
- The exact-carrier fallback warning names its reason.
- Misleading texts are corrected; for example, an exact-length deletion is
no longer called "wrong-length".
- Per-read traces name their read; the Phase stats line is 1-based.
- Every decision path has a named trace: why an ALT call is kept, and each
route to the exact-carrier rule.
- Monitoring (no new columns).
- Each read's read call trace names the rule that decided it (rule=).
- The Phase stats line counts these rules per variant: reads withdrawn as
uninformative, ALT kept by its bases, exact-carrier judged or fell back,
sibling-guard exclusions and clip admissions.
- One INFO line per counting pass gives the totals.
- Prep counts what it could not fetch in full (capped or missing shift
regions, missing or short event references, missing reference contexts,
capped left-alignments) and warns once when any occurred.
- API.
- prepare_variants takes rescue_homopolymer (default off). It builds the
homopolymer twin only when the twin will be counted; the pipeline and
observe_molecules pass the flag.
- BaseCounts.singleton_alt_count and duplex_alt_count are removed. Nothing
ever wrote them; they were always 0.
- Variant.shift_region and event_ref are read-only from Python.
- A row with an empty allele counts neither and is warned once per pass; no
read can carry an empty allele. Prep fails such rows, so pipeline counts are
unchanged. Two places did see them:
- observe_molecules, which passes FAIL rows through to keep rows
positional: such a row's molecules are now OTHER (with an empty ALT they
counted REF);
- unprepared rows passed straight to the engine, which could reach the MNP
check, where len - 1 underflowed.
- Dead code and duplication.
- Unused parameters, branches and setters are removed.
- Observations and mFSD share one molecule classifier.
- Six reference-end CIGAR walks, the soft-clip rule, the scan pad and read
window, the read loops' quality, scoring and molecule-key code, and the
insertion and deletion checks' end-of-walk resolution each have one helper.
- The large-deletion band's literals are named constants.
- One allele-kind classification (SNV, MNP, insertion, deletion, complex)
drives the dispatcher, splice triage's span, the exact-carrier rule's scope
and prep's variant-type label, which each spelled out the same rule.
- Comments. Stale comments are corrected, and ticket labels are removed
from code comments and log text.
Changed — observe_molecules requires a reference FASTA (#204) — breaking¶
observe_molecules()takesreference_fastafrom the argument or fromconfig. Neither (or an empty string) raisesValueError, as the CLI does; so does a path that is not a file. Without a reference, the variants were neither normalized nor given a reference context. The indel and complex-variant rules then ran degraded, with nothing said:- carriers written elsewhere in a repeat were not recognised;
- reads were judged against the bare event;
- complex variants went to the previous classifier.
ObservationResult.variant_statusis always set (a list; it wasNonewithout a reference).- The CLI's config refuses a reference that is not a file. An empty
--fasta ""is., a directory, which passed the existence check and failed in the engine with a FASTA-index error.
Fixed — missing data never changes a count silently (#204)¶
- ALT calls without a reference to read against. An ALT call on a pure indel from a read that spans neither window stood on placement alone when the reference around the event was unavailable. At a contig end the ALT side's reading stretch failed to fetch, so carriers whose bases fit both alleles counted ALT while the same REF reads were withdrawn. Now:
- the rule reads the reference that exists, stopping at the contig end;
- a read with a side to read from but no prepared reference (an unprepared variant, or one that failed prep) counts toward depth only, and is counted and warned once per variant;
- a read lying inside the tract has nothing to read from either way, so it is depth only;
- a read whose deciding base lies past a contig edge stays depth only (a circular contig continues at its start), but is counted and warned with the others.
- Reads starting on a flank. The ALT read-by-bases rule reads from a flank base the read starts on (rightwards), or ends on (leftwards). The ALT side's windows already started there, and the flank is a base both alleles share. Carriers that start on the anchor and whose bases discriminate count ALT; before, they were withdrawn.
- The read must read that flank base: unmasked (BQ at the threshold) and the reference's. For a deletion sliding through a repeat, the flank is all that tells a carrier from REF placed one base along the run. This now holds for the ALT windows too: a window that starts (or ends) on the read's own end base needs that base read, where before the read's extent alone made it informative. A carrier with a masked flank is no longer ALT.
- The exact-carrier rule's reference. Prep widens the reference for exactly the
variants the rule judges: one predicate,
carrier::judges, shared with the dispatcher. Anchor-changing insertions (C>TA) and the homopolymer twin now get it, so a long run no longer sends them silently to the previous classifier. A read the rule still cannot judge is counted and warned once per variant. - In a long run, the rule's junction windows have a known gap (#199): a read holding only the left junction can decide the call. Both sides are strict xfails.
- Splice triage for anchor-changing insertions. An insertion that also changes its anchor (A>CCC) uses the delins span. An RNA read spliced over the anchor and resuming at the next base leaves depth; it counted toward DP as neither.
- Soft-masked references. Every reference window prep reads is upper case. Over a soft-masked (lower-case) FASTA region, left-alignment could not shift. The SW and PairHMM scorers saw reference bases as mismatches: on a synthetic deletion locus REF went 6 → 12 under SW, and partial 6 → 0 under both. Prepared alleles there are now upper case.
- mFSD determinism. The fragment-size statistics are bit-identical run to run.
The size vectors arrived in hash order, and the float sums depended on it: 17
distinct
mfsd_ref_llrvalues over 20 identical runs. The--mfsd-parquetsize arrays are now in a fixed order. The MAF and VCF columns round far coarser and are unchanged. - Three older tests expected ALT from 10bp all-A reads inside an all-A context. Their bases fit both alleles, so they are depth only.
- Measured on RC DNA, FORTE RNA and WES (develop vs branch, every MAF cell compared):
- 140 of 144 files are byte-identical; the other fixes target cases this data does not hold;
- four rows change, each adjudicated read by read:
- a 66bp tandem duplication gains 11 ALT reads (8 fragments). Each starts on the anchor and holds the whole 66bp insert past the first base where the alleles differ, so they are carriers by their bases;
- three deletions (14bp, and 1bp in two G runs) lose 5 ALT reads in all; no fragment counts change. Each read starts or ends on a flank base read at BQ 9–15, so its bases fit both alleles.
Changed — binning invariance and a read census replace the legacy parity path (#170, #171)¶
- The per-variant
count_bam(withcount_single_variant, about 700 lines that duplicated the binned loop) is removed, along with thelegacy-parityCargo feature. Production never called it, so output is unchanged: the RC, FORTE and WES acceptance runs are byte-identical. - It shared the classifier, so it could not see classification bugs. Every one found this cycle came from judging reads by their bases.
- Its one bin bug, the anchor's fetch end, was found in review. Synthetic fixtures fit in one bin, so parity could not have caught it.
- It cost every counting change a second edit.
count_bam_binnedandcount_bam_binned_observationstake two optional test arguments,bin_windowandbin_max_variants(default: the production constants; below 1:ValueError). Counts must not depend on them:- A Rust property test checks that every variant lands in exactly one bin whose fetch holds its read window.
tests/test_binning_invariance.pycompares every field under the per-variant fetch (window 1, cap 1), tiny windows, small caps, one call per row, shuffled input on 4 threads, and decoy rows. Each geometry is asserted to split the bins. Its fixtures are synthetic DNA (plain, and BAQ+UMI+mFSD; siblings, a decomposed twin, overlapping mates), clip carriers past a long anchor, an RNA locus with a GTF and antisense reads, and the real test BAM.- Every counting test now runs through
count_checked, which also counts with one variant per bin, so the 49 tests that ran only the legacy path test production. Most of those calls hold one variant and so check the per-variant fetch window; the binning suite covers multi-variant bins. tests/census.pyjudges each read by its own bases across a pure indel's tract, with the decided rules: REF needs one base past the first difference, ALT only that base. The census finds each tract by its own slide, and a test checks that prep agrees.tests/test_read_census.pychecks the engine against it on reads generated around ten pure indels (ending anywhere past the anchor, some soft-clipped). The open decisions are strict xfails: #200, #201, #202.- Both new suites were mutation-checked. Re-introducing the bin-end bug fails the property test and the clip-carrier fixture. Disabling the ALT-side window, or the equivalent-placement rule, fails the census test.
Fixed — pure-indel reads count ALT only where their own bases hold the ALT (#188, #191, #192, #121)¶
- C10 (#160) takes REF only from reads whose bases settle the allele. ALT reads
had no such check, so a read the CIGAR calls ALT that ends inside a repeat,
where its bases fit both alleles, counted ALT. An ALT read now stands on its
CIGAR when it spans C10's windows as seen from the ALT haplotype. For a
deletion these are an insertion's windows, because a carrier's reference extent
counts its own gap: a −AA carrier in a run of six A's could span the REF
windows while holding only
G AAAA. Otherwise the read keeps ALT only when it reads, unmasked, a base where the alleles differ, with every unmasked base it reads fitting the ALT. Failing both, it counts as depth only, as on the REF side. The check reads bases, not reference coordinates: 2,738 RC carriers, mostly of long insertions, span neither reference window, but their bases discriminate, and they keep ALT. There is no margin base. C10's margin guards a REF call made from the CIGAR alone, whereas this check reads the deciding base itself. - An indel written at the junction counts ALT only when the read has no other
insertion or deletion across the variant's discrimination window. A read with
+A and −T for a +A row, or a split +AA, is another allele (neither + partial in
a repeat). At 50bp or more the large-deletion band's length tolerance applies,
and the band now counts changes across the whole window. Before, it stopped
short of the window for a deletion sliding through a long repeat. The order an
aligner writes an I/D pair in no longer matters. Before, a deletion written
right after an insertion (
M I D M), or an insertion right after a deletion (M D I M), was not inspected, and the read counted REF. A same-length insertion of other bases beside another indel in the window is another allele. Phase 3 called such reads ALT, although their length change is never the ALT's. A deleted anchor followed by an insertion is judged by the read's bases. - A same-length deletion placed elsewhere that gives another haplotype now counts ALT when its bases spell the ALT across the window. Compensating mismatches can make the read the ALT allele even though its gap sits elsewhere. The read is anchored on its nearest aligned bases outside the window. Wherever the haplotype its own CIGAR proposes differs from the ALT, the read base must be unmasked and match the ALT: an N where two deletion alleles differ does not count. Otherwise, at 5bp or more, it is a distinct allele. These reads went to Phase 3, which called them ALT; on the RC set every read Phase 3 reached was another allele by its bases. Under 5bp the read stays REF as before. A deletion of another length within the ≥50bp band keeps Phase-3 arbitration (#191).
- A distinct allele counts neither + partial where the event slides (its shift region is wider than the event) or where the read has an indel inside the discrimination window. It counts REF + partial only in unique sequence with the window clear. Before, only a repeat of a ≤6bp motif gave neither + partial, so carriers of a long-period duplication with another allele counted REF (#192).
- A one-base-REF variant whose ALT changes the anchor base (
A>CCC) went to the insertion check, which matched the inserted bases and ignored the substituted anchor. A read keeping the REF anchor and carrying only the insertion counted ALT. Such variants now go to the exact-carrier rule (#141): a carrier holds the whole ALT, anchor included, across a window that grows through repeats and is padded to the ALT's length. Reads that do not hold that window are depth only, so a read's call does not depend on where it ends (#121). - The Phase-3 context for a slid indel is not changed (#159). Sizing it by the shift region changed 0 ALT calls on the RC, WES and RNA arms, so it was closed with the measurement.
- Measured on the RC panels (realigned), WES without realignment (80 loci from paired IMPACT/TEMPO libraries) and FORTE RNA; develop vs branch, every changed read adjudicated by its own bases:
- DNA: 15 of 1,060 rows change (ALT −39, REF −107, partial −15, depth
unchanged).
- ALT: carriers that end inside the repeat or before the deciding base (7 at a 1bp deletion, 11 at a 66bp duplication ending with the insert, 5 at two 14bp deletions). Also reads carrying other deletions (24–54bp, or a 33bp one 15 bases off) that Phase 3 had called ALT for a 33bp deletion, and 6 reads at a 6bp deletion whose bases fit both alleles.
- REF: −118 at the two anchor-changing rows (#121), from reads that do not carry the REF anchor or do not hold the exact-carrier window.
- At a +AAG insertion co-annotated with that 33bp deletion (two samples), 16 reads move from partial to REF. They carry a large deletion across the anchor, or a D1 inside the window, and only the 33bp row's false ALT kept them out of REF. The insertion row's own rules count them (#200, #202).
- At a 6bp deletion, 5 reads whose bases are REF across the window lose REF to partial. They carry a same-length deletion of other bases outside the window (#200).
- WES: 10 of 80 rows change, all anchor-changing (#121): ALT −12, REF −399, partial +152, depth +62. At one locus 146 reads keep the REF anchor and carry only the insertion, so they count partial, not ALT. At others, reads that carry the whole ALT gain ALT.
- FORTE RNA: the truth set and the STAR repeat-insertion probes are unchanged. One 4bp deletion probe loses 2 ALT reads (a 3bp deletion ending just after).
- Three adversarial reviews; every changed read was adjudicated. Follow-ups: C26 #200 (which of a read's other indels decide its REF call), C28 #202 (a read deleting the anchor falls back to Phase 3's closer haplotype), C27 #201, C25 #199 and R5 #198.
Fixed — carriers of an indel written elsewhere in its repeat count ALT, not REF (#189)¶
- An indel in a repeat can be written at any junction of its shift region with the same haplotype. The input is left-aligned, but aligners may not be: near the FORTE truth set's indel rows STAR placed 162 of 170 repeat insertions away from the left-aligned position (the DNA panels, realigned: 37 of 15,047; WES without realignment: 0 of 620). The windowed checks accepted a shifted placement only by a proxy:
- an insertion when the reference base before it equalled the anchor base. That
is never true inside the repeat, so carriers written elsewhere counted REF
(in a synthetic
G AAAAA Tlocus, REF 10 / ALT 0 instead of 5 / 5), and it was true by chance for some placements that give another haplotype, which counted ALT; - a deletion when the bases it removes equalled the given ones, so a rotated STR
placement (removing
ACforCA) under 5bp counted REF and the same bases deleted outside the repeat counted ALT; - the scan reached
max(5, repeat_span + 2)bases from the anchor, andrepeat_spancounts motifs of up to 6 bases only, so a longer duplication's carriers written past that reach counted REF. - A placement is now accepted when it gives the variant's haplotype (the same bases, or a rotation, elsewhere in the shift region) and is the read's only change across the variant's discrimination window. The scan reaches every placement: every junction of an insertion, a deletion's starts up to its last placement. Other placements:
- the variant written elsewhere with another gap, insertion or splice across the window (a +AA read for a +A row, a split −4 read for a −2 row, a deletion cancelled by an insertion): a distinct allele (neither + partial in a repeat). A read spliced over the anchor with a deletion written just after the junction shows only where the aligner ended the splice: depth only (it counted ALT);
- the variant's inserted bases where they give another haplotype, inside the variant's discrimination window: a distinct allele, as a wrong-length insertion is (neither + partial in a repeat, REF + partial in unique sequence). Outside the window the read shows the window as reference: REF, as before;
- the variant's bases inserted before its anchor base were counted ALT by the
backward-boundary check; they are now another haplotype (REF). An
anchor-substituting ALT (
A>CCC) is never matched by a placement elsewhere (C8, #121, covers the strict path); - a same-length deletion that gives another haplotype: REF under 5bp, as before; at 5bp or more it still goes to Phase 3 (C22, #191, revisits that route).
- Measured on the RC set, develop vs branch, every changed read adjudicated by its own bases:
- DNA: 13 of 1,060 rows change (ALT +46, partial −62, REF +13, depth unchanged). Two insertion rows gain 44 and 10 ALT from carriers written up to 6 junctions right, each holding the ALT haplotype (their last inserted base is an N). Two 2bp deletion rows move 6 and 18 reads from partial to REF: the reads delete the same two bases 6bp before the anchor, another haplotype outside the window. Their co-annotated 33bp rows regain the same reads as REF, which develop had claimed for the 2bp row. Seven 1bp indel rows lose 8 ALT reads that delete or insert the same base outside the run (one ends before the anchor).
- FORTE RNA: the 33 truth samples and the T9 probes are unchanged (the truth set's 3 insertion rows have no carriers). At 23 probes built from STAR's repeat insertions near the truth loci, ALT goes from 11 to 183; an independent census counts 176 shifted equivalent carriers, all ALT on the branch.
- This also changes grouped rows (#99): a tract-mate deleting the same bases at another position is another haplotype, not this row's allele, so where its change lies outside this row's window its carriers are REF here instead of being matched and then demoted to partial. Tract-cluster grouping now reaches as far as the scan.
Fixed — indels near a contig end are left-aligned and get their reference windows (#142)¶
- Prep pads its reference windows on both sides but clamped them only at the contig start. The FASTA reader rejects a window that passes the contig end, so near a contig end each window failed within its own reach:
- left-alignment within about 100bp (a WARN; the variant kept its input position, so reads aligned at the left-aligned position could fall out of depth);
ref_contextwithin its padding (5–50bp), so there were no Phase-3 haplotypes;- the shift region within 256bp (up to 16kb in long repeats), and the event reference within 60bp, for every variant type;
- a complex variant fell back to the tolerant classifier that the exact-carrier rule (#141) replaced, so the same reads were judged differently near a contig end. In a synthetic case the same bases and reads gave REF 0 / ALT 0 mid-contig but REF 5 / ALT 7 at the contig end, two reads carrying another allele among the ALT.
- Every window (left-align,
ref_context, the adaptive repeat scan, the shift region, the event reference) now holds the part of the contig it covers. SNVs and MNPs near a contig end therefore also get their event reference, used by the observed-allele diagnostic. The contig is resolved by the same name the fetch reads, so a FASTA holding a contig under two names cannot mix their lengths. Exact fetches (REF validation, a MAF anchor) stay exact, so a REF running past the end still fails validation instead of being "corrected" to a shorter one.
Changed — rna_antisense_depth counts antisense reads at RNA defaults (#114)¶
- With strandedness enforced (the RNA default), antisense reads were dropped
before the sense/antisense tally, so
rna_antisense_depth(VCFANT) was always 0. Such a read is now classified as a sense read would be, and tallied when it is a first-class REF or ALT read over the anchor. It is then dropped, so REF, ALT, depth and every other count are unchanged. The column means the same with--no-strandedness. STRAND_DISCORDANTis documented as a--no-strandednessdiagnostic: under enforcement, antisense reads never reach the junction tally wherever the gene strand is resolved.- Docs: the read-filters strand table is corrected for the default
reverseprotocol, and the gene strand is documented as coming from the--gtfexons (not the MAF). - The per-read trace line names each excluded antisense read
(
antisense_excluded=true).
Fixed — records stored without bases (SEQ *) are not counted (#172)¶
- A BAM record with no sequence, such as a secondary alignment kept with
--no-filter-secondaryor a primary in a stripped BAM, is now dropped where every read enters counting. It counts in neither depth, fragments normq0_count. Each counting pass logs one WARNING when any were skipped. - Before: the SNV check panicked on such a record, and so did heuristic BAQ (RNA,
or DNA with
--apply-baq). The insertion and deletion checks counted it as REF and depth (or as a fragment) from its CIGAR alone. - Counts change only where such records reach counting. With the default filters, that means primary records without bases.
Changed — the exon-edge BAQ rule and exon_boundary_dist measure from the REF span (#106)¶
exon_boundary_dist(VCFEBD) is now the distance from the nearest annotated exon boundary to any base of the variant's REF span, and0when a boundary lies inside the span (as in Ensembl VEP's overlap test). It was measured from the first base only. The span is the normalized (left-aligned, VCF-style) REF: SNVs and insertions (a one-base REF) are unchanged, and a pure deletion's span starts at its anchor base (VEP drops the anchor, so its distance to a boundary on the left is one more). This is a column change for multi-base variants: the distance is smaller for about 4 in 10 signed-out multi-base variants and indels.- RNA mode with
--gtf: the BAQ exception at exon edges uses the same distance, so a multi-base variant reaching into an exon's last five bases skips BAQ, as an SNV there does. The main counts, per-transcript counts, ASJD and each--rescue-mnpcomponent re-count resolve the rule the same way: a component is counted over its MNP's span, and a rescued row reports the MNP's distance (it reported the adopted component's). - Why: heuristic BAQ lowers the five bases before every splice junction, below
--min-baseqat Q30 or Q37. An MNP starting more than 5bp from a right exon edge but reaching into those bases lost its edge-side bases in every spliced read. So no spliced carrier showed the whole MNP (--rescue-mnpthen adopted a component where the reads show the annotated haplotype), and a spliced read carrying only the edge-side change voted REF on its other bases. - Counts change only for multi-base variants that reach within 5bp of an edge while their first base does not. DNA is unaffected (no GTF).
Changed — a row counts the given allele; the homopolymer twin is opt-in (#163)¶
- The homopolymer twin is no longer dual-counted by default. It was the corrected
allele gbcms counted for a delins like
CCCCCC>T(CCCCCT), reporting whichever form had more ALT reads. The row now counts the allele it is given.--rescue-homopolymer(Nextflow--rescue_homopolymer) restores the dual count, flaggedWARN_HOMOPOLYMER_DECOMPas before. - Why: both forms accept near-matches, so the winner could report another
allele's reads under the row's label. At the case the twin was built for
(SOX2), the reads carry
CCCCT, not the twin. The twin won by tolerance. - It won at 2 of 11 real twin loci, and on 0 of the 6.5.0 RC rows, so counts change only where it used to win.
observe_molecules()takesrescue_homopolymerand threads the twin the same way.
Changed — complex variants count exact carriers only (#141)¶
- A delins, a deletion whose anchor also changes (Del+SNV), or an MNP read with
an indel now counts a read as REF or ALT only when the read's own bases carry
that allele across the whole event, with two reference flank bases each side.
Soft clips count; bases below
--min-baseqmatch anything, the only tolerance. - Why: local alignment, likelihoods and the edit-distance margin credited near-matches, so reads carrying another allele, or ending inside the event, were counted for the given one.
- The windows are read at the event's own position, grown through repeats touching the event on either allele, and padded to equal length so neither allele is favoured by where reads start. Events over 50 bases are judged at both junctions, reading a short ALT whole; a mismatch at either rules a read out.
- A read that cannot hold both windows is depth only (no allele, no
partial_alt, no mFSD class). A read that holds them, matches neither and is closer to ALT counts inpartial_alt. - An MNP read with an indel in or right beside the block is judged the same way (an aligner may write a shifted block as an insertion and a deletion); an indel further off leaves the base-by-base comparison in place.
- Prep fetches enough reference to hold an event grown through a long repeat.
- RNA: a spliced read's windows end at its own junction, so a delins at an exon's first or last bases counts spliced reads; a read spliced through the event counts toward depth only.
- Complex variants (DNA) count reads whose allele lies in soft-clipped bases. An aligner clips an ALT read near its end while the REF reads beside it align in full, so those carriers used to fall outside depth and VAF read low. A read the exact-carrier rule decides from its own bases, inside a well-defined fragment (past the fragment end a clip is adapter), now counts in DP and REF/ALT; undecided clipped reads stay out. Not in RNA mode (#167).
- BAQ (the RNA default, and DNA with
--apply-baq) no longer penalizes a read's own insertion or deletion when that indel is the variant being counted. At Q37 the penalty put every ALT read of a small indel or delins below min BQ while REF reads kept full quality; FORTE's Q40 bins hid it. Splice junctions and indels elsewhere in the read are penalized as before (#166). - In a multi-allelic group, a read that matches this row's ALT and a sibling's
ALT exactly (the two differ only at bases the read has masked) is neither
row's AD: it counts in
partial_alt. --alignment-backend pairhmmandswnow give identical counts for these variants: the rule uses no alignment scoring.
Added¶
- Two
gbcms_diagnosticflags name the allele the reads carry when it is not the given one (canonical VCF form; n reads carry it exactly, m the given allele): OBSERVED_ALLELE(chrom:pos:REF>ALT:n/0): no read carries the given allele exactly, so the input is likely mis-described.COEXISTING_ALLELE(chrom:pos:REF>ALT:n/m): the given allele is present, but a different allele in the same stretch is more frequent (e.g. a germline indel or stutter in a repeat). A caveat for reading the VAF.- Both need n ≥ 3, n > m, at least 5% of the scanned reads, and an allele that is not already an input row.
- The scan covers every variant type: each spanning read is compared over the event plus one base each side, so a germline SNP beside the event does not count.
- Counts are unchanged; the flag says what the reads show so the input can be checked.
- Prep stores the event's reference bases on
Variant.event_ref.
Fixed¶
- Indel REF counts use only reads that can tell the alleles apart (#157).
A read that starts or ends inside an indel's repeat tract matches both
alleles: the aligner places no gap either way. It was counted REF from anchor
coverage alone, which biased repeat-indel VAF down (a 50% homopolymer deletion
read 25% in a synthetic test; on the 6.5.0 RC data the median gbcms-to-
informative VAF ratio was 0.91 for STRs and 0.92 for homopolymers). Such reads
now count toward depth only, as in GATK's AD. A deletion longer than a read
still gets REF reads from either junction. A tandem duplication (an ITD)
slides over its whole duplicated segment, so REF needs a read across all of
it. Prep measures that region over its own fetch (
Variant.shift_region); the repeat context kept for alignment is often too short (a 30bp duplication had an 11-base context). See "Informative Reads for Indels" in the allele-classification reference. - Grouped rows: REF reads and REF fragments exclude the same molecules (#119).
The sibling-ALT guard dropped a read from
ref_countwhen it carried any co-annotated sibling's ALT, even one far from this row, and only after its fragment had been recorded as REF. The guard now applies only to siblings whose change lies inside the row's discrimination window. It runs before fragment evidence in every counting path, soref_countandref_count_fragmentagree. - Carriers of a sibling outside the window count REF again: IGV shows them as REF at this row.
- Carriers of a sibling inside the window now also leave
ref_count_fragment. - What moves with these fixes:
ref_count,ref_count_fragmentand their strand forms, VAF, strand bias, and VCFAD/ADF/ADR/FAD;- values built from REF + ALT: the merged
simplex_duplex_total_count, and RNArna_sense_depth/rna_antisense_depth(these count REF and ALT reads; the docs now say so); - per-transcript REF counts and the ASJD REF partition;
partial_alt: sibling carriers outside the window count REF, not partial. A carrier of an in-window sibling still counts partial, even when it ends inside the tract.- mFSD: a fragment whose reads all end inside the tract lands in no class. It
used to land in REF, and must not look like a third allele (NonREF). In the
observation export it is
OTHER. alt_count_fragmentcan rise by a molecule or two: a mate whose REF call was vacuous (it ended in the tract, or carried a sibling) no longer contests the other mate's ALT. This happened on 3 of 1,060 rows in the 6.5.0 RC DNA runs, by +1 to +2.- Unchanged:
alt_count,total_count(DP) andtotal_count_fragment. SNV and MNP rows change only when they share a site with a co-annotated indel. None did in the RC runs.
[6.5.0] - 2026-09-25¶
⚠️ Breaking Changes — VCF ↔ MAF representation follows vcf2maf / maf2vcf (#110)¶
VCF-input MAF output changes coordinates, alleles and
Variant_Typefor many indel and MNP shapes and gains two columns; MAF-input VCF output changesPOS/REF/ALTfor indels. Counts are unchanged for every countable allele shape; only the edge rows below (Allele1 fallback,ALT_EQUALS_REF, skipped ALTs) change. Re-genotype every flavor (e.g. duplex and simplex) with the same version beforegbcms merge.
- VCF input → MAF writes each record exactly as vcf2maf does. It trims the
leading bases REF and ALT share, never trailing ones;
Start_Position,End_PositionandVariant_Typefollow the trimmed alleles. Before, the conversion followed a length-only type label, so 9 of 17 allele shapes differed from vcf2maf. For example: TTAC>Abecame702–704 TAC>-(vcf2maf:701–704 TTAC>A, DEL);C>TAbecame->A(vcf2maf:821 C>TA, INS);TCT>TCGstayed a 3bp TNP (vcf2maf: SNPT>Gat the changed base);TC>TCGGstayed untrimmed (vcf2maf:- > GG).
Bases are compared case-insensitively, so only mixed-case alleles can differ
from vcf2maf.
- New MAF columns vcf_ref / vcf_alt keep the VCF record itself, alongside
vcf_pos (vcf2maf's names). Each row's vcf_alt is its own allele;
multi-allelic records give one row per ALT. These columns appear only for VCF
input.
- MAF input → VCF output writes each row as maf2vcf does. When an allele is -,
or the alleles differ in length and first base, the reference base before
them is prepended from --fasta. At position 1, the base after the event is
appended instead (VCF spec). Before, - alleles were written into the VCF
(462 AA>-), which is not valid VCF.
- MAF alleles are read as maf2vcf reads them:
- Tumor_Seq_Allele1 is the variant allele when Tumor_Seq_Allele2 is empty
or the reference (older MAFs). Such rows were genotyped as REF against
itself; the reader now logs how many rows it read this way.
- Alleles made only of -, ? or 0 are read as -.
- A variant whose ALT equals its REF (any case; - for both in a MAF) is a
FAIL row with the new reason ALT_EQUALS_REF. Before, it passed and was
counted, meaninglessly.
- --show-normalization norm_* MAF columns follow the left-aligned alleles,
not a type label (a delins was written with its first base stripped).
- Type labels come from the alleles in both readers and in variant preparation.
INSERTION / DELETION now requires a shared anchor base; a delins such as
TTAC>A or C>TA is COMPLEX. gbcms normalize reports these labels.
Counting never read them.
- The VCF reader skips alleles it cannot count, with a WARNING and per-reason
totals: ALT *, symbolic <...>, breakends, a missing ., other
non-sequence alleles, and a REF that is not a base sequence. Before, * and
<DEL> were genotyped as nonsense rows, . was dropped silently, and an
empty REF (read as .) was genotyped. ALT N still reaches preparation,
which reports it as a FAIL row.
- gbcms merge adds the VCF record (vcf_pos / vcf_ref / vcf_alt) to its
join key when every input is VCF-derived, because two VCF records can trim
to one MAF record. It also warns when an input repeats a join key; each such
row joined every matching row of the other inputs, silently.
- The mFSD HTML report finds each variant's MAF row by the key its Parquet uses
(the VCF record for VCF input). VCF-input indels, and some other shapes, were
reported without their MAF row (no gene, no statistics).
- Counting is otherwise unchanged: every countable allele shape counts
identically from VCF and MAF input.
- Docs: for VCF input, Strand, Variant_Classification and
Tumor_Seq_Allele1 are empty, and vcf_id is empty for a . ID. They had
been documented otherwise. End_Position is listed as a required MAF column.
Added — gbcms convert¶
gbcms convertconverts VCF → MAF (vcf2maf's coordinates, withvcf_pos/vcf_ref/vcf_alt) or MAF → VCF (maf2vcf's records; needs an indexed--fasta) without counting. It uses the same conversion as thedna/rnaoutput.
Fixed — homopolymer-decomposed twin: strand and per-sample flag (#107)¶
WARN_HOMOPOLYMER_DECOMPis per sample again. The prepared variants are shared by every sample of a run, so once one sample's corrected allele won, every later sample's row carried the flag, whatever its reads showed. Output depended on sample order.- In RNA with a GTF, the corrected allele takes its original's gene strand, so
--enforce-strandednessapplies to it. Before, where it won, its counts included antisense reads (measured: 0.065% of its depth at synthetic probes). - The docs describe the corrected allele as built: the run with its last base
replaced, the same length as REF (
CCCCCC→CCCCCT). They had described a 1bp deletion plus the change (CCCCCTwas writtenCCCCT). The SOX2 example's chromosome is corrected to 3.
Fixed — output keeps the input's contig naming (#103)¶
chr-named input (e.g. hg38 with UCSC names) no longer loses its naming in the output. VCF input → MAF writesChromosome/vcf_regionas the input names them (was the stripped1), and VCF output writesCHROMin the input's naming with##contiglines declared under the same names (reference lengths kept) — before, records read1under a##contig=<ID=chr1>header, a malformed VCF that htslib rejects (Contig '1' is not defined in the header). Rescue labels and the mFSD Parquet follow the same naming. Counting is unchanged: contigs are still reconciled internally between the variant file, FASTA and BAM. One INFO line per distinct naming pair names the two conventions when they differ. Unprefixed (b37, Ensembl) input against a reference and BAM that use the same naming is unaffected.- The reference FASTA now reconciles the mitochondrion's spellings as the BAM side
already did. Before, a
chrM(orM,MT,chrMT) variant against a FASTA naming it differently was rejectedFAIL/FETCH_FAILEDwith zero counts. gbcms mergejoins inputs whoseChromosomenaming differs (chr1~1,chrM~MT) into one row instead of two half-empty ones. It writes each contig one way, the first input's name where it has the contig, and logs each later input's difference.- An input that names one contig two ways now gets a WARN, since a variant listed under both names is repeated.
- Input columns that collide with merge's join helpers are rejected with the column named.
Fixed — gbcms merge row order is deterministic¶
- Merged rows came out in a different order on every run: the full outer join guarantees no order (5 runs on the same real inputs gave 5 orders), so merged MAFs could not be diffed across runs. Rows now follow the inputs: the first input's rows as it lists them, then rows only a later input has, in that input's order. Row contents are unchanged.
Fixed — one BAQ rule across RNA views; deterministic ASJD junctions¶
- Per-transcript counts and ASJD now apply the main counts' exon-boundary BAQ exception: BAQ is skipped at variants within 5bp of an annotated exon boundary, where its CIGAR-N penalty lands on exactly the reads that splice there. Before, both applied BAQ regardless. Measured on three junction-rich RNA samples at 2010 exon-edge probes:
- per-transcript REF counts were ~3% low overall, and some transcripts lost all their spliced REF reads. The median gap to the main counts drops from 2.5% to 0.05%, and the per-transcript mismatch rate now equals the main counts';
- ASJD saw no junction at some edge variants, including known junctions carried by thousands of fragments.
Main counts are unchanged. Away from exon edges, per-transcript counts are
unchanged. ASJD rows change there only where the tie rule below applies, or
through asjd_qval, which is corrected across the sample.
- ASJD's dominant junction no longer depends on hash order. On a tie, the same
input could report a significant divergence on one run and none on the next.
A tie is now not a divergence: when a REF top junction and an ALT top junction
are the same splice event (both ends within 5bp, ASJD's tolerance
throughout), each allele reports its own of the pair and no test is run.
Otherwise each allele reports the tied junction the other allele's fragments
use most, and the leftmost only on a further tie. Coordinates alone could
otherwise decide the Fisher cells and asjd_flag.
- exon_boundary_dist is found in any contig naming. A chrM variant against
an MT-named GTF read 2147483647, and so did any contig the GTF does not
annotate. The first now reads the real distance, so the exon-edge rule
applies there. The second is now empty. The column is documented as the
unsigned distance it has always been; it was previously documented as
signed.
Added — observability for silent fallbacks (no count changes)¶
SW_FALLBACK(n)ingbcms_diagnostic, plus one WARN per affected variant naming the reason and outcome: under the defaultpairhmmbackend, n depth-contributing reads could not be evaluated by the pangenomic haplotype matrix because the variant's reference context is missing (prep's fetch failed, e.g. an indel near a contig end) or does not contain it, or the ALT haplotype exceeds 400bp (very long insertions). They are scored by the Smith-Waterman fallback where it can run and otherwise end NEITHER — previously with no trace on the row. It never fired on the traced real runs. Never set under--alignment-backend sw.CLIP_CANDIDATES(n)ingbcms_diagnostic: an insertion with no confirmed ALT where n (≥ 2) reads carry a ≥ 8bp soft clip within the insert's duplication reach — clip-represented carriers (typically a tandem-duplication ITD) the engine cannot claim.--umi-tag TAGthat no processed read carries now logs one WARN per counting pass over a BAM (a--rescue-mnprecount can repeat it): fragment grouping silently fell back to read names.
Changed — Smith-Waterman gap-extend is a documented constant¶
dynamic_sw_gap_extendrounded to −1 for every repeat span, so its repeat relaxation never engaged. It is replaced bySW_GAP_OPEN = -5/SW_GAP_EXTEND = -1constants; alignment scores are unchanged.
Added — ASJD-2 splice-disruption markers (RNA + GTF)¶
asjd_diagnosticgains two markers that read the population the splice-aware evidence rule excludes — fragments whose CIGARNspans the variant — at variants within two bases of an annotated intron boundary on the gene's strand, at or above ASJD's own junction-evidence floors:RETENTION_DOMINANT(n)(spliced-over fragments dominate a junction-free classified population, sovafis the retention-population VAF) andNOVEL_JUNC_AT_SPLICE_LOSS(n@start-end)(an anchored, unannotated junction on those fragments outnumbers confirmed ALT — the mutant allele's exon-skip or alternative-site outcome). Both are population comparisons with no tuned rates; they previously surfaced only asLOW_REF_JUNC;LOW_ALT_JUNC(issue #97). No new columns.- Docs now state the
asjd_*_junctioncoordinate convention (0-based, half-open intron).
⚠️ Changed — exclusive assignment at co-annotated tract clusters (user-visible)¶
- Tract-cluster grouping.
assign_multi_allelic_groupsgains a second membership criterion: length-changing variants (indels/delins — never SNV/MNP) whose scan windows (max(5, repeat_span+2)each side) overlap join a group transitively, even when their REF spans never touch. Members whose spans truly intersect keep theMULTI_ALLELICreason tag; window-only members get the newTRACT_CLUSTERtag. - AD-claiming contest. At a grouped locus, anchor-exact (Phase 0)
evidence is never contested; every other ALT read is demoted to
partial_alt/any_alt(excluded from AD and ADF) when any of three tests fires: the read's window does not favor the row's ALT haplotype strictly over its REF haplotype (foreign flank events); the call is alignment-phase on an anchor-preserved pure indel (no matching structural op anywhere — unannotated ladder absorption in tracts; complex/MNP rows exempt); or a co-annotated sibling's ALT haplotype explains the read strictly better over a window covering both spans (equal cost = equivalent representations, both rows keep the read). True carriers — pure indels, delins/complex, shifted self representations, noisy but real reads — keep AD, except a pure-indel carrier resolved only by alignment (e.g. soft-clipped), which surfaces aspartial_alt. - REF-side symmetry. Sibling-claimed reads are excluded from RD on every
path (now including per-transcript); in the main counts they surface as
partial_altinstead of vanishing silently. - Validated against a signed-out ACCESS hypermutation cluster: per-row fragment ALT counts land exactly on (or within counting-basis of) sign-out where the previous windowed counting over-attributed 2–5×; panels without co-annotated clusters are byte-identical.
⚠️ Changed — MNP rescue reports a coherent component genotype (opt-in --rescue-mnp)¶
- Candidate gate is partial dominance. Rescue now considers PASS
MNP_RESCUE_ELIGIBLEMNPs withpartial_alt > alt_countinstead of onlyalt_count == 0. Masked per-position evaluation counts a component carrier whose other discriminating base is low-BQ as full ALT, so a single such read blocked rescue of MNPs whose carriers hold only one component (a TERT promoter GAGGG>AAGGA row sat at alt 1 / partial 88). - Rescued rows adopt the component's full counts. Previously only
alt_count(and read VAF) changed, leaving strand, fragment, strand-bias, mFSD and diagnostic columns at MNP values and breakingalt_count = alt_count_forward + alt_count_reversein the output. A rescued row now carries the winning component SNV's counts in every column (all counting invariants hold, includingany_alt = ad + partial_alt), andgbcms_diagnosticis recomputed from them. Fragment-level consumers (ACCESS,gbcms merge --add-combined) now see rescued counts. gbcms_rescueformat (downstream parsers): every entry hasoutcome=(rescued,skipped_grouped,haplotype_confirmed,no_improvement,ref_validation_failed) and the MNP's ownoriginal_ref/original_alt/original_partial(original_altwas hard-coded0); rescued rows addadopted=. A position whose synthetic SNV failed preparation readsref_failinstead of a silent0.outcome=no_signalis gone.- MNPs whose haplotype the BAM shows are kept (
outcome=haplotype_confirmed, no re-count): the engine now counts MNP ALT reads in which every discriminating base was read (internalBaseCounts.mnp_confirmed_alt, not an output column), and rescue fires only when that count is zero — no read in the BAM shows the annotated MNP. Validation with matched normals found rescue adopting germline het SNPs where reads carried the real MNP (IMPACT: 36 such reads; ACCESS duplex: 6); such rows now keep the MNP's counts. The audit gainsoriginal_confirmed. - Rescue labels use the output's contig naming (an input MAF's own
Chromosome, e.g.chr1, rather than the stripped internal name). gbcms mergewarns on mixed rescue: when duplex and simplex rescue outcomes differ (one rescued, or different components), each row is named in a WARNING with a per-run count — with or without--add-combined, since the two flavors' columns then describe different alleles in one row (with it, thesimplex_duplex_*columns add them); counts are unchanged. When only one flavor was genotyped with--rescue-mnp, merge says so once and names every row rescued in that flavor (previously that case was silently skipped).- Grouped MNPs are skipped (
outcome=skipped_grouped) so rescue cannot hand back reads that exclusive assignment gave a co-annotated sibling. - Fixed: rescue audit leaked across samples. In a multi-BAM run a later
sample's row could show an earlier sample's
gbcms_rescue. - Rescued rows say so:
gbcms_diagnostic(MAF column, VCFGD) gainsRESCUED_COMPONENT(chrom:pos:REF>ALT)on every rescued row, each rescue logs a WARNING (the component and the MNP's own counts), and enabling--rescue-mnplogs a WARNING that rescued rows report a component. - Fixed: VCF
GRwas split by parsers. The audit's positions list was comma-separated and VCF parsers (htslib/pysam) split a comma-bearing Number=1 INFO value, silently truncating the audit. Positions are now joined with+; the MAF column and VCFGRcarry the same content (VCF writes;as|). - Per-sample INFO outcome summary; warnings for anomalies (a component SNV
failing preparation). With
--mfsd-parquet, a rescued row's record keeps the MNP's coordinates but carries the adopted component's fragment sizes (logged per sample). BaseCounts.with_ad()removed from the Python bindings (no remaining callers).- Default runs (
--rescue-mnpoff) are unchanged.
[6.4.0] - 2026-09-22¶
The changes below alter reported counts at wrong-length indel loci and at spliced RNA positions — the minor version bump this release delivers.
⚠️ Changed — span-aligned REF testimony at spliced deletion loci (user-visible)¶
-
Structural REF evidence survives fragment consensus at quality zero. Span-aligned REF's carried quality is the first exon base after the junction — exactly where BAQ can zero it — and
FragmentEvidencegated REF existence on quality > 0, silently dropping such fragments fromrdf(the mirror of the structural-ALT divergence fixed in the splice-aware change below). Span-REF observations now carry a structural flag that keeps them alive in consensus; a structural ALT in the same fragment still wins. On the FORTE locus this recovers one real fragment (rdf726→727 =dpf). -
Junction reads that observe every deleted-span base now count REF at a pure-deletion locus whose anchor base is spliced out (typical for deletion annotations left-aligned onto the last intronic base at an acceptor). The anchor base is not the discriminating fact for a deletion — the span is — and these reads demonstrate the span is present. Previously they counted DP as neither (conservative cluster-A behavior): at a validated FORTE acceptor locus, 823 of the 824 anchor-spliced junction reads convert (
rd24→847 ofdp848, fragmentrdf23→727 ofdpf727, DP unchanged); the one residual read's exon2 alignment covers only one of the two deleted-span bases (95M92N1M) and honestly stays neither. Guards: the FULL span must be aligned (partial coverage stays neither), and reads carrying a competing shifted/wrong-length indel candidate keep their existing arbitration paths. Read- and fragment-level, mirrored in both engine paths via the shared checker.
⚠️ Changed — consensus intron snipping of ref_context removed (user-visible)¶
ref_contextis always genomic now. The RNA-mode step that drained consensus introns from the variant's reference context before Phase 3 had no coordinate map: the context shrank whileref_context_startstayed genomic, so shifted-indel sequence verification, the large-deletion band's context guard, and haplotype offsets all read garbage right of a snipped intron — so exon-contained reads were scored against haplotypes whose offsets no longer matched their genomic coordinates, and pre-mRNA / intron-retention reads against a haplotype missing bases their sequence genuinely contains. Removing it fixes windowed shifted-deletion matching near acceptors (contract testtest_windowed_deletion_after_junction_in_repeat_rna, committed red) and erases the binned-vs-legacy RNA divergence around it; junction reads needing alignment-based scoring stay conservativelyneitherunder the splice-aware evidence rule. Re-validated on local (non-repo) clinical RNA data — the b37 dedup and FORTE hg38 smoke loci report unchanged counts. A coordinate-mapped spliced-haplotype rework remains tracked in issue #94, gated on real-data measurement; it must keep pre-mRNA / intron-retention reads genomically scored.mq0_countnow tallies before the RNA strandedness filter in the binned path, matching the legacy path: an antisense MAPQ-0 read is still a physical read at the locus, and the two paths previously diverged on this diagnostic in stranded RNA mode.
⚠️ Changed — splice-aware evidence in RNA counting (user-visible)¶
- Reads spliced over a variant no longer count depth or REF. A CIGAR
Nspanning every discriminating position (a deletion's deleted span, an insertion's junction flanks, an SNV/MNP's REF bases) is asserted splicing — no observation — so the read classifies neither AND is excluded from DP/fragment depth, matching samtools pileup's zero coverage inside an N gap. Previously such reads counted definitive REF (deletions) or inflated DP with zero aligned bases (intronic positions): on a junction-dense real RNA locus, DP at an intronic SNV drops from ~826 span-overlapping reads to the 2 with aligned bases there. Excluded totals are logged per variant at debug level (Phase stats … splice_skip_excluded=). - The REF/ALT call no longer flips on the aligner's D-vs-N representation
choice. A
D(100)at the expected span is deletion evidence (ALT); the same gap asN(100)is splicing (excluded). - Indels directly after a splice junction are now examined. An indel op
whose only neighbor is a splice
N(M-N-D-M/M-N-I-M— an event at an exon boundary reached through the junction) was structurally invisible to the strict and windowed CIGAR scans; carriers now classify through the same anchor/windowed inspection asM-adjacent ops (evidence attributed to the nearest aligned/inserted base when the anchor itself is spliced out). - Phase 3 no longer scores across a splice. Raw-window extraction refuses
windows that an
Noverlaps, andcheck_complex's reconstruction classifies such reads neither instead of stitching exon arms into a junction-chimeric sequence in which the missing intron reads as deletion evidence. - The exclusion is not silent. Deletion-type loci where splice-skip
exclusions exceed confirmed ALT are flagged
SPLICE_SKIP_DOMINANT(n)ingbcms_diagnostic— RNA aligners write large deletions as splices (STAR: ≥alignIntronMin, default 21bp), soalt_count=0 there may mean the carriers exist as junction reads. Per-variant totals also appear in the debug-levelPhase statsline (splice_skip_excluded=). - Shifted representations across a junction stay in classification. A read whose N covers the annotated span but which carries an I/D op inside the scan window (repeat-tract shift) is deferred to the windowed scans instead of being excluded.
- Fragment consensus recognizes structural ALT independent of base
quality. BAQ can stack splice and indel penalties to quality 0 on the
base carried with a junction-adjacent I/D op;
FragmentEvidence::resolvepreviously treated qual-0 ALT evidence as no evidence, reportingad > 0withadf = 0at the same locus. check_complexrefuses string comparison for structurally anomalous junction reads (splice N inside the context window): with the indel shifted outside the variant span, the span reconstruction is clean REF sequence and Phase 2 would absorb an ALT carrier intord.- ASJD junction tallies at intronic splice-region variants now reflect only
allele-informative reads (spliced reads with no observation at the locus
no longer contribute fabricated REF junctions); expect
LOW_REF_JUNCwhere the REF tally was previously fed by such reads. - Contract battery:
tests/test_rna_splice_contract.py(13 cases incl. a legacy-parity case with N-CIGAR reads and one strict xfail pinning the known D6 ref_context coordinate corruption for the cluster-B fix). DNA-mode classification is untouched (reads withoutNops never enter the triage), and the legacy parity oracle mirrors every engine-loop change.
⚠️ Changed — orchestration fail-fast (user-visible)¶
--bam-listentries that do not exist now fail the run (exit 1) unless--lenient-bamis given — matching the long-documented fail-fast promise. An unreadable list file is equally fatal: previously both cases silently ran a partial sample set and exited 0.- Duplicate sample names are now a hard error (from
--bam,--bam-list, or a mix): the later BAM silently replaced the earlier one, which was then never processed. Usesample_id:path(or two-column list entries) to disambiguate deliberate same-stem inputs. - Ragged MAF rows now raise in the batch readers (
gbcms mergeinputs) instead of having their overflow fields silently truncated — the gbcms count columns are the trailing columns, exactly what truncation dropped.
🔧 Fixed¶
- A rejected (FAIL) variant no longer crashes
--mfsd, RNA--gtf, or--mfsd-parquetruns: the zero-count stub now carries every column the writers read (mFSD q-value and nucleosomal fractions, RNA/GTF annotation and ASJD fields), and the mFSD Parquet excludes rejected variants (they have no fragment data) with the exclusion logged. gbcms mergeno longer coerces float-formatted count strings to 0: a pandas/R round-trip renders integers as12.0, which the combined-column sums silently nulled to 0; genuinely non-numeric values (e.g.NA) are counted and warned about per column.- Re-genotyping warns about replaced columns: an input MAF already carrying
gbcms output column names (e.g.
ref_countfrom a previous run) has those values refreshed — now with a warning naming every colliding column, and the writer's docstring no longer claims originals are never overwritten. - Failed-sample reports name the exception type and log the traceback
(
KeyError: some_keyinstead of a bare'some_key').
⚠️ Changed — wrong-length pure-indel evidence is a distinct allele (issue #91)¶
- A read whose CIGAR proves a pure indel of a DIFFERENT length at the variant anchor now
counts as
partial_alt, never as REF or the queried ALT. In repeat tracts, coexisting distinct-length indel populations are distinct slippage alleles; the previous behavior routed them to the length-blind Phase-3 haplotype window, which promoted them to ALT and inflated VAF several-fold at tract loci (a homopolymer locus with 7 true-ALT reads reported 259).alt_countnow reflects exact support; the distinct-allele evidence surfaces inpartial_alt/any_alt, andPARTIAL_DOMINANTflags loci where it dominates. - Placement-aware ≥50bp deletion band replaces the reciprocal-overlap tier: a
wrong-length D at the anchor is the annotated event only when the read deletes essentially
the whole expected span (≤3 retained bases, ≤3 changed outside) — accepting breakpoint
wobble and split
D+M+Drepresentations while rejecting displaced net-matches. The old ≥50% overlap rule accepted any deletion sharing half the annotated length (its sequence check compared reference against reference, identically true by construction). - Insertion truncation containment: a shorter insert that is a ≥90%-identity slice of a non-low-complexity expected insert still counts as ALT (sequencer truncation smear of long insertions); both sequences must be non-low-complexity so repeat-tract slippage is never mistaken for truncation.
- Same-length insertions with confidently mismatching bases (≥
--min-baseq) are a third allele →partial_alt; they were previously absorbed intoref_countwith no signal. Unverifiable cases (all inserted bases low-quality) go to Phase-3 arbitration. - Windowed wrong-length ops (deletions ≥5bp — 1–4bp windowed Ds remain
CIGAR-definitive alignment noise; insertions at any size): repeat tract →
partial_alt; unique context → REF with the stray op surfaced aspartial_alt. Same-length S3-fail candidates keep the Phase-3 left-alignment rescue (TP53-class), now via the splithas_shifted_same_lengthflag. - Repeat scan anchors at the first changed base (not the shared VCF anchor, which sits one base left of a left-aligned tract), and adaptive context padding covers the whole tract plus flank — Phase-3 haplotype windows in repeat regions are no longer too narrow to distinguish tract lengths.
- Delins/complex variants are unaffected: they route to
check_complex, whose Phase-3 realignment correctly resolves split and mismatch-absorbed representations of one event.
🔧 Fixed¶
--tracenever emitted a single per-read Rust trace line: pyo3-log's default filter capped forwarding at DEBUG, and the Python side configured a logger name (gbcms_rs) that pyo3-log never uses (_rs.…). Both fixed;_rs.reset_log_caching()exposed so enabling trace after Rust code has logged is not silently ignored.- Left-alignment failures are loud: wide-window FASTA fetch failure warns with the
error (reachable near contig ends) instead of silently skipping normalization; the
2500bp expansion cap binding without convergence warns; non-UTF-8 alleles from a corrupt
reference keep the variant fully unnormalized instead of emitting a shifted position with
reverted alleles; a stray
println!on ref_context fetch failure removed. --enforce-strandednesswarns when variants have no resolved gene strand (no GTF, uncovered locus, contig mismatch) instead of silently not enforcing.- GTF annotation warns per variant chromosome with zero loaded exons, on both the
text-parse and the
--gtf-cache-dircache-hit paths (a warm cohort cache previously silenced the warning after the first sample). - Per-read
debug!diagnostics demoted totrace!(WFA router, marginalized PairHMM).
✨ Added¶
- Wrong-length contract battery (
tests/test_wrong_length_contract.py, 20 tests through the binned↔legacy parity oracle) and an end-to-endPARTIAL_DOMINANTreporting-chain test driving the full CLI. - Documentation: "Wrong-Length Pure Indels" rule reference in
docs/reference/allele-classification.mdand a worked case study (Case 4 indocs/reference/complex-indels.md);partial_altsemantics updated in output docs; fictionalRUST_LOG/GBCMS_LOG_LEVELcontrols removed from docs (logging is--verbose/--trace).
[6.3.1] - 2026-08-25¶
🔧 Fixed¶
- Nextflow: turning off a default-on read filter now actually reaches the engine.
--filter_duplicates false,--filter_secondary false,--filter_supplementary false, and--filter_qc_failed falsewere silent no-ops: the modules omitted the flag when false, and the CLI default (on) won. The DNA/RNA modules now always pass the explicit on/off form (--filter-x/--no-filter-x) for all six read filters. - Nextflow ≥26.04: boolean and numeric CLI params are coerced explicitly. The strict
parser hands CLI overrides to the script as Strings (
--flag falsearrives as the truthy String"false"; 25.x coerced it toBoolean false), which silently inverted every boolean param check. A sharedasBool()helper (nextflow/modules/local/utils/main.nf) now guards every boolean param decision, andprocess.resourceLimitscastsmax_cpus/max_memory/max_time. Verified end-to-end on 25.10 and 26.04: identical generated commands both ways for all six filters.
✨ Added¶
- Contract tests pinning filter and phasing behavior (no production code changes):
test_qc_failed_filter_contract— three-way qc-failed filter proof at read and fragment level: flagged reads dropped by default,--no-filter-qc-failedrestores byte-identical counts, no effect on unflagged BAMs.test_partial_haplotype_contract— the cis-phasing contract for DNP/ONP variants: a multi-nt ALT counts towardalt_countonly when the full haplotype co-occurs on one read; partial-haplotype reads are surfaced viapartial_alt/any_alt(any_alt == ad + partial_altasserted end-to-end), and a spurious multimer with no read support reports all three as 0.
[6.3.0] - 2026-08-07¶
✨ Added¶
observe_molecules()takes the settings it actually uses. New keyword argumentsfilters,quality,alignment,umi_tag,threads,apply_baqandlibrary_typesit alongside the existingconfig=, which keeps working; an individual argument overrides the matchingconfigfield.
Previously the only way to change a setting was to build a whole GbcmsDnaConfig, whose
four required fields — variant_file, bam_files, reference_fasta, output — have
zero overlap with the seven this entry point reads. Two of them must name files that
exist on disk. So adjusting one filter meant fabricating an output directory and a variant
path that are never touched: paths that look load-bearing and are not.
Found by the first real consumer (mulligan) needing a non-default read filter.
umi_tag is the one argument where None is an explicit choice rather than an omission —
it means "group by read pair, not UMI family" and overrides config accordingly. Every
other argument treats None as "not supplied". Without that distinction, a caller passing
umi_tag=None alongside a config that sets one would silently inherit it and change what
counts as a molecule.
🐛 Fixed¶
--filter-secondaryand--filter-supplementarynow change the result when turned off. They previously could not change any output: an unconditional skip in the counting loop discarded those records before fragment evidence, regardless of the flags. Measured across all six read filters on real MSK-ACCESS data, these were the only two that did nothing (duplicates,improper_pairandindelall worked). (#82)
Default runs are byte-identical — both filters still default to on, so such records never reach the cache. Verified on real data: total depth unchanged at 1202 across 40 loci, and binned↔legacy parity holds.
Turning a filter off now admits those records to fragment-level evidence
(dpf/rdf/adf) and to the per-molecule observation export. They still never
contribute to read-level dp/rd/ad: a supplementary shares a QNAME with its primary
and is the same physical read, so counting both would report depth 2 where one read
exists. Fragments are immune either way — hash_molecule keys on QNAME, so a primary and
its supplementary collapse into one.
What this fixes concretely: a locus reached only by a supplementary segment used to
report dpf=0 — not a filtered read but a wrong answer, since a molecule demonstrably
covers it. That is what made a molecule spanning a large deletion invisible to
cross-locus phasing.
New consequence to be aware of: with the filter off, an admitted supplementary raises
dpf while leaving dp unchanged, so the two stop moving together. The observation
export reconciles with dpf, and that invariant is verified under both settings.
[6.2.0] - 2026-08-06¶
✨ Added¶
min_mapqonObservationand in the observations Parquet schema — the worst MAPQ among the reads backing each molecule. Minimum rather than best, deliberately: a molecule is only as trustworthy as its least confidently placed read, so a paired fragment reports its worse mate and one cleanly-placed read cannot launder an ambiguous one.255is the SAM spec's unavailable, distinct from0, which is a real value meaning the read mapped ambiguously.
Exported rather than left to inference because the --min-mapq threshold bounds mapping
confidence without measuring it: assuming every surviving molecule sat at the gate would
charge them all the same error, swamping the base-quality term and flattening exactly the
per-molecule differences a weighted linkage statistic exists to express. Consumers doing
read-backed phasing need the measured value.
FragmentEvidence gained a matching min_mapq field, tracked for every read reaching
fragment evidence — including reads that are neither REF nor ALT, which still count toward
dpf and still describe the fragment's placement.
🔧 Changed¶
alignment_backendnow defaults topairhmmingbcms._rs, matching the CLI,Pipeline, andobserve_molecules, which all already defaulted to it. A direct_rscaller that omitted the argument previously got Smith-Waterman — a different classifier than the identical call made through any supported entry point (measured: 5 of 104 molecule calls differ on real ACCESS deletions). No production path changes: the CLI, Nextflow, and the Python API have always passed the backend explicitly, so counts are unchanged.
🐛 Fixed¶
- An unrecognized
alignment_backendtoken now raisesValueErrorinstead of silently falling back to Smith-Waterman."PairHMM","smith-waterman", or any typo previously produced plausible counts from a classifier the caller never asked for, with nothing logged. Validation now happens before any I/O, so a bad token fails immediately rather than after a full read pass. Unreachable from the CLI (typer validates the enum first); reachable from_rs. (#79) _rs.pyicorrected to match the extension, and the match is now enforced.count_bamandcount_bam_binnedeach declared nine defaults (min_mapq…threads) that the pyo3 signature does not grant, so mypy accepted calls that fail at runtime with missing 9 required positional arguments;prepare_variantsandwrite_fsd_parquetnamed parameters that do not exist at runtime (reference_fasta→fasta_path,output_path→path), so any keyword call using the documented name raisedTypeError. Newtests/test_rs_stub_parity.pycompares the stub against pyo3's__text_signature__, turning AGENTS.md invariant 5 from a review item into a CI gate — it catches all four of the above.- Removed the
Defaultimpl onAlignmentBackend, which still namedSmithWatermanlong afterpairhmmbecame the real default everywhere else. Nothing called it; it served only to mislead.
[6.1.0] - 2026-08-06¶
✨ Added¶
--observations-parquetCLI flag ongbcms dnaandgbcms rna, writing<sample>.observations.parquetalongside the normal counts output. Follows the--mfsd-parquetpattern (flag →OutputConfig→ pipeline calls the native writer). This is what makes the capability reachable for Nextflow / ACCESS-Pipeline / CWL consumers, which invoke gbcms through the CLI rather than the Python API. Counts output is unchanged, and nothing is written unless the flag is passed.- Parquet sink for observations.
observations_path=oncount_bam_binned_observations()/gbcms.observe_molecules()writes the rows to Parquet from Rust, so they never cross the FFI boundary. A panel- or genome-wide run produces 10^6–10^7 observations, where materializing that many Python objects — not the counting — is the bottleneck; the same reasoning behindwrite_fsd_parquet. The file is self-describing (variant_index, chrom, pos, ref, alt, molecule_hash, allele, best_qual), since a bare index means nothing once the data outlives the call that produced it. Verified on real data: identical row content to the in-memory path. - Per-molecule observation export.
count_bam_binned_observations()returns the same counts ascount_bam_binned()plus the per-molecule allele calls the engine normally aggregates away — oneObservationper fragment per variant, carrying the molecule identity, the resolved allele (REF / ALT / N / OTHER), and its best supporting base quality. Because a molecule observed at two variants in one call shares amolecule_hash, callers can link alleles across loci (read-backed phasing, allelic imbalance, duplex-masking QC); gbcms itself does no such linking. - Public API:
gbcms.observe_molecules()→ObservationResult. This is the supported surface;gbcms._rsremains internal and is not covered by SemVer. - Counting is untouched — both entry points share one core,
count_bam_binned's signature and return are unchanged, and observations are off unless requested (no allocation when off). Binned↔legacy parity is unaffected. - Rows are sorted by
(variant_index, molecule_hash), so output is deterministic despite parallel bin processing and hash-map iteration order. - Decomposed variants emit one set of rows: observations follow the same
higher-
adarbitration as the counts, so the losing allele form is never exported.
🔄 Changed¶
- Nextflow pipeline is now compatible with the strict syntax parser that is the
default from Nextflow 26.04, while still running on 25.x. Verified with
nextflow linton 26.04.0 (0 errors) and config resolution on both 25.10 and 26.04. nextflow.config: replaced the legacycheck_max()helper (function definitions are illegal in strict config) with the nativeprocess.resourceLimitsdirective;--max_cpus/--max_memory/--max_timestill apply. Bumped thenextflowVersionfloor to>=24.04.0(whenresourceLimitslanded).main.nf: moved input validation into theworkflowbody (top-level statements are disallowed) usingerrorinstead of the removedexit; rewrote thehasDatahelper without awhileloop (removed in strict syntax);Channel→channel.- Removed the remote nf-core institutional-config include. The strict config
syntax forbids the
try/catch(andif) guards that made a fetch failure non-fatal. The localiris/slurmprofiles are self-sufficient; users who want an institutional config can still supply one with-c. Thecustom_config_base/custom_config_versionparams were removed.
[6.0.0] - 2026-07-01¶
[!WARNING] Breaking output changes — downstream parsers of gbcms output must review before upgrading. This major release changes several output behaviours: -
gbcms_statusis now two fields — a verdict (gbcms_status=PASS/FAIL) and a|-separatedgbcms_status_reason. VCF gains aGSRINFO key. Filters onFAIL_FETCH_FAILEDetc. must switch togbcms_status == "FAIL"+ a reason check. - Per-transcript count columns use|between transcripts, not;(RNA +--gtf). - Supplementary/secondary alignments no longer count toward read-level depth —DPmay drop for BAMs with those alignments and the corresponding filters disabled. - Empty-allele variants are rejected at prep time (verdictFAIL, reasonEMPTY_ALLELE). - Nextflow RNA now defaultsmin_mapq=1(viarna_min_mapq) instead of inheriting the DNA20, matching thegbcms rnaCLI — RNA depths increase vs. the 5.x pipeline. ---mfsdMAF/VCF schema grew to 41 MAF columns / 13 VCFMFSD_*INFO fields (gated behind--mfsd, absent otherwise).
🐛 Fixed¶
-
Contig-naming mismatches no longer silently produce zero counts. When the BAM's contigs were named differently from the variants (UCSC
chr1vs Ensembl/b371), the binning step found no matching contig, built 0 bins, and returned zero counts for every variant while the run still exited 0 — a systematic failure masked as success.resolve_tidnow reconciles the naming vianormalize_contig(sochr1↔1,chrM↔MTmatch), and a genuinely-absent chromosome now logs a loudWARN(once per chromosome) instead of being skipped silently. (MSK ACCESS runs were unaffected — BAMs and MAFs both use b37 naming.) Surfaced by the new end-to-end MAF test. -
A run now exits non-zero when a sample fails (HI-1).
Pipeline.run()catches per-sample errors and returns normally, sogbcms dna/rnapreviously exited0even when every BAM failed (e.g. a Rust panic surfaced asPyErr) — masking systematic failure as success under Nextflow. The CLI now exits code 1 when any sample actually fails (all-failed or partial). An empty variant set is not a failure: a sample that legitimately has no variants called still exits0, so per-sample workflows don't fail on it. Successful runs are unchanged (exit 0). -
Rejected variants keep their reason in the output. When every variant is rejected during preparation (e.g. a contig mismatch →
FAIL_FETCH_FAILED), the run no longer short-circuits with no output — it now writes each variant with itsFAIL_*reason in thegbcms_statuscolumn (and zero counts), so the reasons are in the output file, not only the log. (Partial rejections already did this; this closes the all-rejected gap.) -
Nextflow RNA runs no longer silently drop STAR multi-mapper reads. The pipeline used a single global
min_mapq = 20for both modes, so RNA counting ran at MAPQ 20 instead of thegbcms rnaCLI default of 1 — silently dropping STAR's 2–4-locus multi-mapper primaries (which STAR encodes as MAPQ 3/1, vs 255 for uniquely-mapped reads). On a real FORTE RNA sample that was ~16% of reads genome-wide (chr7) and up to ~6% of depth at an individual locus. A newrna_min_mapqparam (default 1) now drives the RNA module; DNA keepsmin_mapq = 20. Set--rna_min_mapq 20to restore unique-only RNA counting.
🔄 Changed¶
-
gbcms_statusis split into a verdict + a reason field (breaking output change). The status is now two fields with a consistent grammar in both formats: a verdict (gbcms_status= exactlyPASSorFAIL) and a|-separated reason list (gbcms_status_reason, empty for a clean PASS). MAF gains thegbcms_status_reasoncolumn; VCF gains aGSRINFO key. Reason tags dropped their verdict prefix (FAIL_REF_MISMATCH→ verdictFAIL+ reasonREF_MISMATCH;PASS;WARN_REF_CORRECTED→ verdictPASS+ reasonWARN_REF_CORRECTED). Reasons now stack with|(WARN_REF_CORRECTED|WARN_HOMOPOLYMER_DECOMP), which also fixes a bug where a REF-correction warning was silently dropped on the success path and whereWARN_HOMOPOLYMER_DECOMPoverwrote any existing reason.|is used (never;/,, both VCF-INFO-unsafe), so the reason string is byte-identical in the MAF column and the VCFGSR— no format-specific conversion. Consumers filtering onFAIL_FETCH_FAILEDetc. must switch togbcms_status == "FAIL"+ agbcms_status_reasoncheck. -
Fragment-count consensus is labeled accurately (ME-12). The stale "Majority Rule" comment on the
BaseCountsfragment fields (DPF/RDF/ADF) is replaced with "quality-weighted consensus with an INDEL structural-priority override and a discard band," pointing atFragmentEvidence::resolve— which is what the code has always done. No behavior change. The alternative of gating a structural-ALT INDEL win on the REF mate's base quality was considered and rejected (anchor BQ is orthogonal to INDEL-detection confidence; seeREJECTED.md). -
Per-transcript count columns now use
|between transcripts, not;(ME-2).transcript_read_counts/transcript_fragment_counts(MAF) andTXRC/TXFC(VCF) separate transcripts with|— e.g.ENST…:AD,RD,DP|ENST…:AD,RD,DP.;is the VCF INFO field separator, so the VCF path already converted to|while MAF emitted raw;; the engine now emits|directly so all three (MAF, VCF, the documented header) agree. Consumers that split the MAF column on;must switch to|. -
--threadsis now a hard, validated thread budget. It is the total worker budget for one gbcms process (multi-sample parallelism is Nextflow's job — N concurrent processes, each pinned totask.cpus).--threads 0is now rejected loudly instead of silently becoming rayon's all-cores default (which would oversubscribe a small SLURM allocation); all rayon pools are sized throughresolve_thread_budget, and the resolved budget is logged. Counts are unaffected.
📖 Documentation¶
- Repo-wide documentation accuracy sweep. A four-way audit against the current code
corrected the docs where they had drifted: the mFSD schema (now 41 MAF columns / 13 VCF
MFSD_*INFO fields, up from a stale 34/7, with the q-value + nucleosomal-fraction + CH-flag columns added); the mono-nucleosomal range (150–200 bp);gbcms_statusvalues (the realFAIL_*prefixes + the two missing statuses); crashing examples (normalizeuses--output <file.tsv>, not--output-dir; corrected RNA MAF column names in the quickstart); the Nextflow README defaults (on-by-default filters,pairhmmbackend, Nextflow ≥22.10.1) and its missing param surface (--strandedness,--gtf,--library_type,--mfsd*,--merge_*); and previously-undocumented features (build-gtf-cache/--gtf-cache-dir, exit-code semantics, the--threadshard budget, and contig auto-reconciliation). Also fixed the--rna-editing-dbCLI help (it loads a REDIportal TABLE1, not a VCF) and several developer-doc references to files/paths that had moved.
🧹 Internal¶
- Single type stub for the Rust extension (LO-1). Removed the orphan
src/gbcms_rs.pyi, which stubbed a top-levelgbcms_rsmodule that isn't importable (the extension isgbcms._rs) and had already drifted from the real one (missing the nucleosomal fields andfisher_exact_2x2). The co-locatedsrc/gbcms/_rs.pyi(shipped viapy.typed) is now the single source of truth, and itscount_bamstub gained the missingreference_fastaparameter so it matches the real#[pyo3(signature)].
📦 Packaging¶
- The shipped wheel no longer exports the legacy
count_bamparity oracle. The per-variantcount_bam(the binned↔legacy parity oracle) is now behind a defaultlegacy-parityCargo feature; release builds use--no-default-featuresto omit it. Production always usedcount_bam_binned, so this only trims a test-only symbol from the wheel. Dev/test builds keep it (default on) so the parity suite still runs.
⚡ Performance¶
-
mFSD is now computed only when requested (
--mfsd/--mfsd-parquet/--mfsd-report). Previously the engine always built the per-fragment size arrays and ran the full KS/LLR/delta statistics for every variant, then discarded them unless an mFSD output was selected. The binned production path now gates this work on themfsdflag (plumbed fromOutputConfig.mfsd), so the default--dnarun no longer holds the per-variantref_sizes/alt_sizesarrays across the whole sample — the dominant mFSD memory cost, and the one that multiplies under Nextflow fan-out (N concurrent processes). The size-array statistics are also factored into a single sharedcompute_mfsd_statshelper used by both the binned and legacy paths. Counts are unaffected: validated on 3,040 real cfDNA variants — all 246 non-mFSD columns are byte-identical with mFSD on vs off, and the 41 mFSD columns appear only when enabled. -
GTF annotation can be cached on disk (
--gtf-cache-dir). Parsing a full Ensembl GTF takes ~9s, and under a Nextflow cohort that cost was paid once per sample. When--gtf-cache-diris set, the parsed intermediate (exon records, splice sites, introns, chrom map) is serialized once and reused on later runs over the same GTF + variant set, dropping the annotation load from ~9s to ~0.05s (validated on the full GRCh38.111 GTF; cold-vs-warm output byte-identical across 47 variants and all annotation columns). The architecture-specific interval trees are not cached — they are rebuilt from the cached records on load, so a cache file is portable across x86/ARM. Caching is best-effort: a missing/corrupt/stale-version/unwritable cache logs and falls back to a normal parse, never affecting correctness.
✨ Added¶
-
VCF now emits the mFSD sub-/mono-nucleosomal fields (ME-1).
MFSD_SUB_NUC_REF_FRAC,MFSD_SUB_NUC_ALT_FRAC,MFSD_SUB_NUC_ENRICHMENT,MFSD_MONO_NUC_REF_FRAC,MFSD_MONO_NUC_ALT_FRACwere computed under--mfsdand written to MAF but silently dropped from VCF — they're now in the VCF INFO too (VCF↔MAF parity), taking the VCF mFSD surface to 13 INFO fields. Also corrected long-standing count drift in the docs/CLI help:--mfsdadds 41 MAF columns (not 34) and 13 VCF INFO fields (not 7). -
gbcms build-gtf-cache— pre-warm the GTF index cache for a cohort. A dedicated command (--gtf --variants --gtf-cache-dir, no BAM) that parses the GTF once and writes the cache, so a fan-out of per-samplegbcms rna --gtf-cache-dir <same-dir>jobs all start warm. This is what makes the cache pay off under concurrency: without a pre-warm step, samples launched together all cold-miss and each re-parses the GTF. Run it as a single Nextflow process upstream of the per-sample fan-out so the whole cohort parses the GTF exactly once. Seedocs/nextflow/parameters.md. -
--strandednessfor RNA mode (reverse|forward|unstranded). The read→transcript-strand fold was previously hardcoded to dUTP/reverse (fr-firststrand, featureCounts-s 2); it is now selectable to support forward (fr-secondstrand,-s 1) and unstranded (-s 0) libraries. The fold drives both--enforce-strandednessfiltering and ASJD strand-discordance detection;unstrandeddisables both (as do amplicon libraries). Default isreverse, preserving prior behavior and matching the FORTE pipeline default. Unknown values are rejected loudly at both the model and FFI layers. Validated on a real reverse-stranded RNA sample:reverse + forward = unstrandedread counts exactly (100% strand partition) at ACTB and GAPDH.
🔄 Changed¶
- ASJD junction evidence is now counted per fragment, not per mate. A molecule
whose R1 and R2 overlap on a short cDNA insert can have both mates carry the same
splice junction (the genomic insert looks large only because it spans the intron) —
on real reverse-stranded RNA this is 35.6% of fragment×junction incidences and is
not a UMI/PCR duplicate. Counting both mates inflated the per-junction strand
tally ~1.38× and could fire spurious
STRAND_DISCORDANTat low depth.detect_asjdnow dedups each fragment (by QNAME) to one vote per allele-total and per junction, matching the per-fragment treatment the rest of the engine already uses. Mates always fold to the same transcript strand (verified 0/319k disagreements), so the dedup is unambiguous. This shiftsasjd_n_*_junc/asjd_n_*_totaland a small number of low-countSTRAND_DISCORDANTflags (18/58,240 junctions genome-wide on the test sample), always in the corrective direction. - Empty-allele variants are now rejected at prep time. A structurally empty REF
or ALT (
"") is malformed input — the internal representation is VCF-style (anchor-based), and MAF dash alleles arrive as the literal-, never empty. Such variants previously fell through to counting and silently produced zero counts; they are now rejected duringprepare_variantswith aFAIL_EMPTY_ALLELEstatus and a warning, so they are surfaced in the output rather than quietly dropped to all-zero. Legitimate MAF-alleles are unaffected. - Supplementary/secondary alignments no longer count toward read-level depth.
They share a QNAME with their primary, so counting them at read level
double-counts DP/RD/AD at the anchor. They are now skipped unconditionally in
both the binned and legacy counting paths, independent of
--no-filter-secondary/--no-filter-supplementary(those flags now only govern whether such records enter the read cache, not whether they are counted). Affects only the non-default flag combination; default behavior is unchanged.
🐛 Fixed¶
- Read-level supplementary/secondary double-count under
--no-filter-secondary/--no-filter-supplementary(see Changed above). check_complextrailing-insertion handling clarified and guarded. The Phase-1 reconstruction deliberately includes an insertion at the exclusive REF end (a trailing insertion that belongs to the ALT, e.g. REF=AB, ALT=ABC); documented the rationale and added regression tests so the boundary condition isn't "fixed" into a misclassification.
[5.3.0] - 2026-05-16¶
✨ Added¶
- CRAM Support: Full CRAM file support across all commands (
dnaandrna). The--bamargument now transparently accepts CRAM files. - CRAM Index Auto-Discovery: The Nextflow pipeline automatically discovers
.craior.cram.craiindexes in the same directory as the CRAM files. - Reference Binding: Rust engine's
IndexedReadercorrectly initializes CRAM references using the provided--fasta.
🔄 Changed¶
- Provenance Comment Lines: Output MAF files now include
#gbcms vX.Y.Zand#command ...provenance lines before the header row in both DNA and RNA modes. - VCF Header Provenance: VCF headers now include
##source,##gbcms_command,##reference, and##contigmetadata in both DNA and RNA modes. - Strand Bias Sanitization: Handled edge cases in Fisher's exact test where structural strand arrays contain all zeros, returning
.,1.0, orNAinstead ofNaN/Inf. - Merge Engine Robustness:
gbcms mergeuses Polarscomment_prefixnatively to skip provenance comment lines without failure.
📚 Documentation¶
- Updated
cli/rna.mdto note CRAM and provenance support for RNA mode. - Updated
cli/merge.mdto explain MAF comment line compatibility. - Updated
cli/index.mdand diagrams to explicitly mentionBAM/CRAM Files. - Expanded
reference/input-formats.mdwith explicit CRAM and.craiindex requirements. - Added DNA vs RNA comparison snippet in
reference/output-formats.mddemonstrating provenance headers.
🧪 Tests¶
- Expanded testing suite (now 329 tests), providing integration coverage for RNA provenance headers and CRAM pipeline data flow.
- Added test validation for merge engine skipping MAF provenance comment lines safely.
[5.2.0] - 2026-05-15¶
✨ Added¶
gbcms mergecommand: New CLI command for merging per-BAM-type genotyped MAFs (e.g., duplex + simplex) into a single output with type-prefixed count columns and optional additive combined metrics.- Merge engine (
src/gbcms/merge.py): Polars-based lazy join engine with 3-phase combined column computation: - Phase 1: Additive sums (12 read + fragment + strand count columns)
- Phase 2: Derived totals and VAFs
- Phase 3: Fisher's exact strand bias (read + fragment level) via Rust
- Batch I/O module (
src/gbcms/io/batch.py): Centralized Polars-basedread_maf,scan_maf,read_parquet,write_maffor batch operations. MergeConfigPydantic model: Validated configuration with ≥2 input enforcement, file existence checks, andadd_combined/legacy_namingoptions.fisher_exact_2x2PyO3 wrapper: Exposed the existing Rust Fisher's exact test asgbcms._rs.fisher_exact_2x2()for Python-side strand bias computation in the merge engine, ensuring numerical parity with the primary counting engine.- Nextflow
MERGE_COUNTSprocess: New module atnextflow/modules/local/gbcms/merge/main.nfthat callsgbcms mergewith--input type:patharguments built from grouped BAM type channels. - Nextflow
bam_typesamplesheet column: Optional column enabling automatic--column-prefixderivation andgroupTuple-based merge orchestration. - Nextflow merge parameters:
merge_counts,merge_add_combined,merge_legacy_naminginnextflow.config. - Polars dependency:
polars>=1.0.0added as a core dependency.
🔧 Fixed¶
- Fragment consensus: INDEL structural evidence priority — When R1 and R2 of a duplex fragment disagree on an insertion or deletion, the read with direct CIGAR evidence (I/D op) now wins the fragment consensus unconditionally, instead of comparing anchor base qualities (which are identical for both reads and caused systematic discard). This recovers ~2-5% of INDEL fragment-level evidence that was previously lost, critical for low-VAF cfDNA detection.
- Insertions:
adfincreases by ~5% for conflict fragments (validated on RB1 INS) - Deletions:
adfincreases by ~2% for conflict fragments (validated on DNMT3A DEL) - Single-molecule events:
adf=0→1recovery (validated on RUNX1 INS, ad=1) - SNP behavior: unchanged (no structural flags on SNP classifications)
- Phase 3 alignment returns: unchanged (non-structural, quality comparison retained)
🧪 Tests¶
- [NEW]
tests/test_merge.py— 24 tests covering variant key join, multi-type merge, combined columns (read + fragment + strand), Fisher strand bias (biased + balanced), column order validation, asymmetric row counts, null-fill, legacy naming, and CLI input format validation. - [NEW]
tests/test_batch_io.py— 8 tests coveringread_maf,scan_maf,write_maf,read_parquet, and error handling for missing files. - [NEW]
tests/test_indel_fragment_consensus.py— 11 tests covering INS/DEL conflict recovery (structural ALT priority), wrong-length INDEL Phase 3 dispatch, agreement paths, singleton reads, SNP regression (tie behavior unchanged), and REF agreement at INDEL sites. All 4 counting invariants asserted per test. - [NEW]
rust/src/shared/fragment.rs#[cfg(test)]— 11 Rust unit tests forFragmentEvidence::resolve()structural priority andobserve()sticky flag logic. - 298 Python tests (up from 255): 32 merge, 8 batch, 11 INDEL consensus tests.
- 161 Rust tests (up from 150): 11 new fragment consensus unit tests, 0 Clippy warnings.
📚 Documentation¶
- [NEW]
docs/cli/merge.md— CLI reference forgbcms mergecovering usage, combined columns schema, Nextflow integration, and Fisher strand bias. - Updated
docs/cli/index.md— Addedmergecommand to commands table. - Updated
docs/reference/output-formats.md— Added Merged MAF Output section with type-prefixed columns and 20 combined metrics. - Updated
docs/reference/architecture.md— Merge engine in system overview diagram,batch.pyandmerge.pyin module tree,MergeConfigin config diagram. - Updated
src/gbcms/_rs.pyi— Addedfisher_exact_2x2type stub. - Updated
src/gbcms/__init__.py— Exportedmerge_mafsandMergeConfig.
[5.1.0] - 2026-05-11¶
⚠️ Breaking Changes¶
- MAF column order reordered (v5.1 schema):
any_alt,partial_alt,n_countmoved from end to immediately afteralt_countfor discoverability. Read strand counts (ref_count_forward, etc.) now precede derived strand bias statistics. Read and fragment metric layers fully separated (no interleaving). Downstream MAF parsers using positional indexing must be updated. - VCF FORMAT fields restructured (VCF 4.2 spec compliance):
DPis now a single integer (total depth), wasref,altpair.ADis nowNumber=Rwithref,alttotals (VCF spec), wasfwd,rev.RDandRDFremoved. Replaced byADF(forward strandref_fwd,alt_fwd) andADR(reverse strandref_rev,alt_rev) following bcftools convention.- New
FAD,FADF,FADRfor fragment-level strand-by-allele counts. FAFrenamed from position afterVAFto after fragment group.- Downstream VCF parsers expecting old
GT:DP:RD:AD:RDF:ADF:VAF:FAF:...must be updated toGT:DP:AD:ADF:ADR:VAF:FAD:FADF:FADR:FAF:.... - VCF INFO field order changed:
AAD,PAD,NADnow appear immediately afterGR(before strand bias fields), matching the diagnostic proximity principle.
🔧 Fixed¶
- Wrong-length insertion Phase 3 fallback (PAX5-class discordance fix):
check_insertionnow routes wrong-length insertions at the strict anchor position to Phase 3 (SW/PairHMM) for haplotype arbitration, mirroringcheck_deletion's existing behavior. Previously, a read withI(1)at the anchor for an expectedI(2)was silently classified as REF with no diagnostic signal. Now classified aspartial_altviahas_nearby_evidence, triggering thePARTIAL_DOMINANTdiagnostic flag. - Wrong-length insertion windowed scan tracking: Wrong-length insertions
within the ±window range now set
has_nearby_length_match, routing to Phase 3 fallback. Previously silently ignored. - Insertion
!found_ref_coveragehaplotype fallback: Added Phase 3 fallback for insertion reads where the CIGAR walk found no evidence but the read spans the anchor (e.g., unusual CIGAR geometry, soft-clip at anchor). Mirrorscheck_deletion's existing!found_ref_coveragepath. - Wrong-length deletion windowed scan tracking: Wrong-length deletions
in the windowed scan now set
has_nearby_length_matchfor Phase 3 fallback. Covers two previously silent-drop cases:- Small deletions (≥5bp, <50bp): always flagged (1-4bp excluded as homopolymer noise)
- Large deletions (≥50bp) with low reciprocal overlap (<50%): previously silently dropped, now flagged for Phase 3 arbitration
[5.0.0] - 2026-05-11¶
⚠️ Breaking Changes¶
- New RNA output columns: When
--gtfis provided, 17 additional columns are appended to RNA output (1 exon boundary, 2 per-transcript, 14 ASJD). Downstream parsers expecting a fixed column count must be updated. - Fragment counts in amplicon mode: When
--library-type ampliconis used, fragment counts (dpf,rdf,adf) will approximate read counts (dp,rd,ad) because R1/R2 fragment consensus is bypassed. This is expected behavior, not a bug.
✨ Added¶
- GTF-based transcript annotation (
--gtf): RNA mode can now load an Ensembl/GENCODE GTF file to enable:- Exon boundary distance (
exon_boundary_dist): Signed distance to the nearest exon boundary for splice-proximal variant filtering. - Per-transcript counting (
transcript_read_counts,transcript_fragment_counts): ALT counts stratified by overlapping transcript, resolving ambiguity at multi-transcript loci. - Aberrant Splice Junction Detection (ASJD): 14 columns comparing observed read splice junctions against annotated transcript splice sites. Flags novel vs annotated junctions with REF/ALT stratification.
- Exon boundary distance (
--library-typeCLI flag: RNA library preparation method selector.capture(default, IDT xGen-style) oramplicon. Available ongbcms rnaonly.- Amplicon mode — fragment consensus bypass: When
--library-type ampliconis set, the Rust counting engine XORsmol_hashwith a read-specific tag (0x1for R1,0x2for R2), treating each read as an independent molecule. This prevents incorrect R1/R2 merging in amplicon libraries where both reads are PCR duplicates of the same template. - Amplicon auto-strandedness override: Both the CLI (
cli.py) and model validator (GbcmsRnaConfig.model_validator) auto-disableenforce_strandednesswhenlibrary_type="amplicon", with a warning. This ensures safe defaults for both CLI and API users. - Per-read strand tracking in junction accumulator: RNA reads now carry per-read strand orientation through the splice junction accumulator, enabling accurate sense/antisense counting at splice-spanning positions.
- COITree annotation index (
rust/src/annotation/): New Rust module providing O(log n + k) exon overlap queries via cache-oblivious interval trees. Built once at startup, shared immutably across Rayon threads viaArc<AnnotationIndex>. - Splice mask construction: Per-transcript
HashSet<(chrom, start, end)>for O(1) splice site lookup during ASJD computation. - BH multiple-testing correction (
shared/stats.rs): Benjamini-Hochberg FDR correction added for strand bias p-values across variants. - Nextflow
library_typeparameter: Registered innextflow.configand threaded throughrna/main.nffor amplicon mode support in pipeline runs.
🔧 Fixed¶
lib.rsdead-code comment: Updated annotation type comment to reflect that annotation types are now fully wired (previously noted as unused).
🧪 Tests¶
- 255 Python tests (up from 238): 17 new tests covering:
test_config_isolation.py: 11 tests forlibrary_typefield/validator, GTF field, DNA isolation, amplicon auto-strandedness, andrescue_mnp_thresholdrange validation (default, shared, >1.0 rejection, <0.0 rejection).test_cli_dna_rna.py: 4 tests for--library-typeand--gtfoption isolation between DNA and RNA commands.test_diagnostic_flags.py: 2 tests forMNP_RESCUE_ELIGIBLEthreshold-based eligibility (conservative 0.50 mode).
- 143 Rust tests (up from 119): 24 new tests covering:
- GTF parsing and chromosome normalization
- COITree overlap queries
- Splice distance computation
- BH FDR correction
- Fisher's exact test edge cases
- 0 Clippy warnings (strict
-D warningsmode).
📚 Documentation¶
- [NEW]
docs/reference/rna-annotation.md— GTF requirements, annotation index architecture, exon boundary distance, per-transcript counting, and ASJD detection reference. - Updated
docs/cli/rna.md—--gtfand--library-typeoptions, updated pipeline diagram with annotation layer, amplicon example tab, amplicon override warning, updated DNA vs RNA comparison table. - Updated
docs/reference/output-formats.md— GTF-aware MAF columns (17), amplicon mode behavioral note. - Updated
docs/reference/architecture.md— AnnotationIndex in system overview diagram,annotation/module in tree, config diagram withlibrary_typeandgtffields,stats.rsBH correction note. - Updated
docs/development/developer-guide.md—annotation/module in project structure diagram. - Updated
mkdocs.yml— "RNA Annotation" added to navigation.
[4.2.0] - 2026-05-10¶
⚠️ Breaking Changes¶
validation_status→gbcms_status: MAF column and Python API parameter renamed.gbcms_statusnow uses semicolon-separated multi-value format (e.g.,PASS;WARN_REF_CORRECTED,PASS;MULTI_ALLELIC). The first token is alwaysPASSorFAIL_*.- VCF
VS→GS/GD/GR: VCF INFO keyVSreplaced byGS(status),GD(diagnostic), andGR(rescue). Downstream VCF parsers must update field references. - Status value format: Old underscore-joined statuses
(
PASS_WARN_HOMOPOLYMER_DECOMP,PASS_MULTI_ALLELIC) replaced with semicolon-separated (PASS;WARN_HOMOPOLYMER_DECOMP,PASS;MULTI_ALLELIC).
✨ Added¶
gbcms_diagnosticcolumn (MAF) +GDINFO key (VCF): Post-counting diagnostic flags computed automatically. Flags include:ZERO_ALT: No confirmed ALT reads despite successful counting.PARTIAL_DOMINANT: More structural/partial evidence than confirmed ALT.MNP_DISC_RATIO(n/m): MNP discriminating position ratio (always emitted for MNPs).MNP_RESCUE_ELIGIBLE: MNP qualifies for rescue (disc/len ≤--rescue-mnp-threshold).HIGH_N_FRACTION(f): N-base fraction exceeding 5% at discriminating positions.
--rescue-mnpCLI flag: Enables MNP rescue pass for multi-base substitutions. Whenad=0andMNP_RESCUE_ELIGIBLEis flagged, decomposes the MNP into individual SNP positions and re-counts viacount_bam_binned. Available in bothgbcms dnaandgbcms rnamodes.--rescue-mnp-thresholdCLI flag: Maximum disc/len ratio for MNP rescue eligibility (0.0–1.0, default: 1.0). At 1.0, all MNPs are eligible (C++ gbcms compatible). Set to 0.5 for conservative sparse-only mode.gbcms_rescuecolumn (MAF) +GRINFO key (VCF): Conditional — only present when--rescue-mnpis enabled. Contains structured audit trail:method=decomposed;original_alt=0;positions=chr:pos(R>A):count,.... Failed rescues includeoutcome=no_signal.has_nearby_evidence(Rust): NewClassifyResultfield propagating structural evidence from variant checkers (INS/DEL/Complex) and alignment backends (SW/PairHMM). Enablespartial_altcounting for INDELs.partial_altnow populated for INDELs: Previously always 0 for SNP/INDEL. Now fired when the counting engine detects nearby structural evidence (right-length INDEL, non-zero ALT alignment score).- Diagnostic flag summary logging:
info-level log of diagnostic flag distribution per sample (e.g.,ZERO_ALT=12, PARTIAL_DOMINANT=3).
🔧 Fixed¶
partial_altdescription: Documentation corrected to reflect thatpartial_altis now populated for all variant types, not just MNP/Complex.
📝 Notes¶
- Invariant 1 breakage (rescue only): After rescue,
any_alt = ad + partial_altno longer holds for rescued variants.adis updated with the best decomposed SNP count whileany_altandpartial_altretain original MNP-level values as forensic evidence. - Rescue strategy: Python-side post-processing using decomposed SNP re-counting. Coordinate shift strategy is reserved for a future release.
🧪 Tests¶
- [NEW]
tests/test_diagnostic_flags.py— 17 tests covering all 4 diagnostic flags, multi-flag combinations, FAIL exclusion, boundary conditions, parametric formatting, andrescue_mnp_threshold-based eligibility gating (permissive 1.0 and conservative 0.50 modes). - [NEW]
tests/test_rescue_mnp.py— 13 tests covering config defaults, conditional column/INFO presence, candidate identification, guard rails (skip non-MNP, skip ad>0, skip FAIL), audit trail format, no-signal cases, column count with rescue, and invariant breakage verification. - Updated
test_normalization.py(12 refs),test_pipeline_v2.py(4 refs),test_phase2_output.py(column count 24→26),test_multi_allelic.pyforgbcms_statusrename. - All 238 Python tests pass; all 119 Rust tests pass.
📚 Documentation¶
- Updated
docs/reference/output-formats.md—gbcms_status,gbcms_diagnostic,gbcms_rescuecolumns;partial_altdescription corrected; prefix behavior updated. - Updated
docs/reference/architecture.md— MNP rescue architecture, data flow diagram, Python-vs-Rust design rationale, invariant impact. - Updated
docs/development/developer-guide.md— Rescue debugging, extension guidelines, test fixture requirements. - Updated
docs/reference/variant-normalization.md— status field name and multi-value format. - Updated
docs/reference/allele-classification.md— multi-allelic status format. - Updated
docs/cli/normalize.md—gbcms_statuscolumn name. - Updated
docs/resources/troubleshooting.md— status field references.
[4.1.0] - 2026-05-04¶
⚠️ Breaking Changes¶
gbcms runcommand removed: The deprecatedgbcms runalias (introduced in v4.0.0 as a transitional shim) has been removed. Usegbcms dnainstead — all arguments are identical.- Nextflow config defaults aligned with CLI:
filter_secondary,filter_supplementary,filter_qc_failedchanged fromfalsetotrue;enforce_strandednesschanged fromfalsetotrue;alignment_backendchanged from'hmm'to'pairhmm'. Pipelines relying on the old Nextflow defaults may see behavior changes.
✨ Added¶
- Physical fragment sizing: Rust-native fragment size calculation using aligned read positions instead of TLEN, improving accuracy for supplementary alignments and soft-clipped reads.
--mfsd-reportflag: Generates an interactive HTML report with per-variant fragment size distribution analysis, dual-axis histograms, and Fragment Origin Signal classification (TUMOR-LIKE / CH-LIKE / AMBIGUOUS / INSUFFICIENT). Implies--mfsdand--mfsd-parquet.- Variant navigator: STRiDE-inspired sticky navigation bar for multi-variant mFSD reports — dropdown selector, prev/next buttons, Focus/Show All toggle, keyboard shortcuts (←/→). Automatically hidden for single-variant reports.
--mfsd-report-min-alt: Minimum ALT fragment count to include a variant in the report (default: 3).--mfsd-report-max-variants: Maximum variants per report (default: 20).- Theme toggle: Light/dark mode switching in HTML reports.
- Print compliance: Reports render audit-ready when printed (navigator hidden, all variants at full opacity, branded footer included).
- RNA BAQ default:
--apply-baqnow defaults toTrueforgbcms rna. RNA pipelines typically lack upstream BQSR, so BAQ penalizes bases near splice junctions and indels to reduce false-positive variant calls. - BAQ trace logging: Per-read BAQ adjustments logged at
tracelevel, showing indel and splice junction counts per read. - Nextflow
cache = 'lenient': Ensures-resumeworks correctly on GPFS/Spectrum Scale filesystems where inode metadata changes during file pool migration. - Nextflow
manifestblock: Pipeline metadata (name, version, author, homepage) for Nextflow Tower and nf-core registry compatibility. - Nextflow SLURM job naming:
clusterOptionsadds descriptive job names (nf-GBCMS_DNA_sampleid) forsqueuereadability. - Nextflow
executor.queueSize: Caps concurrent SLURM submissions at 100. - nf-core institutional configs: Auto-loads site-specific profiles (iris,
jax, sanger, etc.) via
nf-core/configs. - Extended trace fields: IO diagnostics (
rchar,wchar,syscr,syscw,read_bytes,write_bytes) added to execution trace.
🔧 Fixed¶
- Dual-axis gridline artifact: Fixed Plotly
yaxis2overlay creating a duplicate x-axis line in mFSD histograms by standardizingmirror: false,rangemode: 'tozero', andshowlinecontrols. - BAQ RefSkip early-exit bug:
apply_heuristic_baq()silently skipped reads containing only splice junctions (CIGARN) but no indels. The early-exit gate now includesCigar::RefSkip, ensuring splice-spanning reads receive the BAQ quality penalty. - Nextflow config defaults: Aligned
nextflow.configwith CLI defaults —filter_secondary,filter_supplementary,filter_qc_failedcorrected totrue;enforce_strandednesscorrected totrue;alignment_backendcorrected topairhmm. - Nextflow RNA BAQ wiring: Added
--no-baq/--apply-baqargument to RNA module (rna/main.nf), which was previously missing entirely. - Nextflow RNA strandedness: Fixed
strandedness_arglogic — now passes--no-strandednessonly when disabled (was incorrectly passing--enforce-strandednessas an additive flag).
📚 Documentation¶
- [NEW]
docs/reference/mfsd-report.md— mFSD interactive report reference covering Fragment Origin Signal classification, interactive features, output columns, and print compliance. - [NEW]
docs/reference/rna-splice-handling.md— RNA splice-junction handling guide with dual-mechanism comparison (consensus intron snipping vs BAQ), GATK SplitNCigarReads comparison, defense-in-depth analysis (5 layers), and visual splice bleed examples. - Updated BAQ documentation across 7 files:
read-filters.md,cli/dna.md,cli/rna.md,glossary.md,abbreviations.md,nextflow/parameters.md,architecture.md— all now include BAQ_RADIUS/BAQ_PENALTY constants, mode-specific defaults, and guidance on when to enable BAQ for DNA. - Updated
mkdocs.yml— added "RNA Splice Handling" to navigation. - Updated
docs/cli/dna.md— added--mfsd-report,--mfsd-report-min-alt,--mfsd-report-max-variantsto CLI reference. - Updated
docs/nextflow/parameters.md— added Nextflow params for mFSD report; BAQ default now shows mode-specific values. - Updated
docs/development/release-guide.md— version locations table corrected from 5 to 7 references (reflecting v4.0.0 Nextflow module split). - Updated
nextflow/nextflow.config— addedmfsd_report,mfsd_report_min_alt,mfsd_report_max_variantspipeline parameters. - Updated
nextflow/modules/local/gbcms/dna/main.nf— wired--mfsd-reportflags through the DNA module with HTML report output channel (emit: mfsd_report).
🧪 Tests¶
- [NEW]
tests/test_mfsd_report.py— 13 unit tests covering report creation, navigator presence/absence, Plotly integration, theme toggle, branding, summary cards, min_alt/max_variants filtering, and error handling. Uses synthetic test fixtures with no patient identifiers. mfsd_report.pycoverage: 0% → 91%.- [NEW]
tests/test_config_isolation.py— BAQ default assertions for RNA (apply_baq=True) and DNA (apply_baq=False) modes.
🧹 Chores¶
- Deleted ad-hoc analysis scripts from
scripts/(compare_tlen_vs_physical, concordance, plot_fsd_distributions, plot_fsd_histogram). - Deleted
scripts/*_test/directories containing test artifacts. - Added
.gitignorepatterns for*.parquet,*.mfsd_report.html, andscripts/*_test/directories. - Ruff B904 fix:
raise ... from Noneinmfsd_report.py. - Ruff UP015 fix: removed unnecessary
"r"mode argument fromopen().
🧬 N-Base Diagnostic Integration (Phases 0–3)¶
✨ Added¶
- Diagnostic output columns:
any_alt,partial_alt,n_countappended to all MAF output (24 gbcms DNA columns total, up from 21). - VCF diagnostic tags:
AAD(Any ALT Depth),PAD(Partial ALT Depth),NAD(N-base Depth) emitted in both INFO and FORMAT sections. - N-base defense-in-depth: Explicit N-base guards in
check_snp,check_mnp, andcheck_complex— N bases are classified as uninformative regardless of reported base quality, preventing silent evidence inflation from duplex-masked positions (fgbio) or sequencer failure. - Structural invariants:
any_alt = AD + partial_alt,any_alt >= AD,DP >= RD + AD + partial_alt + n_countenforced and documented. trace!-level diagnostics: N-base detection, n_count accumulation, and partial_alt counting logged at trace level for production debugging.- ALT-contains-N rejection: Variants where the ALT allele contains N are
rejected with
FAIL_ALT_CONTAINS_Nvalidation status andwarn!-level log.
🔄 Changed¶
- MNP quality strategy: Replaced all-or-nothing
min(BQ across block)gate with masked per-position evaluation — each discriminating position (REF ≠ ALT) is independently assessed; low-BQ and N bases are masked but unmasked positions still vote. This recovers reads in GC-rich regions (e.g., TERT promoter) where a single low-quality position previously dropped the entire read. - MNP ThirdAllele handling: Mixed-vote reads now track
positions_matching_altfor diagnostic partial_alt counting instead of being silently discarded.
🧪 Tests¶
- [NEW]
tests/test_mnp_concordance.py— 6 tests for MNP concordance with C++ gbcms on production duplex BAMs. - [NEW]
tests/test_phase2_output.py— 5 tests for diagnostic column presence, invariant validation, and N-count sanity on fixture data. - 26+ Rust unit tests for N-base masking, MNP per-position evaluation, partial match tracking, invariant enforcement, and edge cases (all-N reads, mixed BQ/N).
- Updated
tests/test_column_count_delta_is_threeto assert 24 gbcms DNA columns.
📚 Documentation¶
- Updated
docs/reference/counting-metrics.md— diagnostic columns, invariant tables, VCF tag definitions, N-base handling section. - Updated
docs/reference/output-formats.md— AAD/PAD/NAD in VCF header, INFO/FORMAT tables, annotated example; any_alt/partial_alt/n_count in MAF table. - Updated
docs/reference/allele-classification.md— SNP flowchart with N guard, MNP section rewritten for masked per-position algorithm, Complex N-base note. - Updated
docs/reference/architecture.md— structural invariants and diagnostic output fields in Formulas section. - Updated
docs/development/developer-guide.md— regression invariant checklist. - Updated
docs/development/testing-guide.md— Phase 2 test suite, silent failures matrix, invariant table, updated test counts.
[4.0.1] - 2026-03-24¶
🔧 Fixed¶
was_normalizedflag accuracy: Split into granularwas_anchor_resolvedandwas_left_alignedflags with backward-compatiblewas_normalizedgetter. Fixes 1150 false negatives (anchor resolution not tracked) and 58 false positives (unnecessary anchor+trim round-trip for non-dash complex variants). No impact on BAM counting — display/logging only.- Left-alignment false positives: Fixed case-sensitive
modifiedcheck inleft_align_variantto useeq_ignore_ascii_case, preventing soft-masked FASTA bases from triggering spurious normalization flags. - Non-dash anchor resolution: Narrowed MAF anchor resolution guard to
dash-allele-only variants. Non-dash complex/deletion MAF variants
(e.g.,
GG>A) no longer enter the unnecessary anchor+trim cycle. - PairHMM pangenome panic: Fixed unsigned integer underflow
(
range end index 18446744073709551615) in pangenomic haplotype construction caused by left-to-right delta-adjusted coordinate math. Rewrotebuild_haplotype_matrixwith right-to-left variant application algorithm that eliminates coordinate drift by construction, plus power-set sibling combinatorics for true multi-haplotype evaluation. Only affects--alignment-backend hmm.
🔄 Changed¶
- Normalization logging now shows granular breakdown:
"X normalized (Y anchor-resolved, Z left-aligned)"in both Rust engine and Python pipeline/normalize logs. gbcms normalizeTSV output now includeswas_anchor_resolvedandwas_left_alignedcolumns before the existingwas_normalizedcolumn.- CLI and reference docs updated with new column descriptions.
[4.0.0] - 2026-03-20¶
⚠️ Breaking Changes¶
- Nextflow module split: Single
run/main.nfreplaced by three dedicated modules —dna/main.nf,rna/main.nf,normalize/main.nf. Consumer pipelines must updateincludepaths and useGBCMS_DNA,GBCMS_RNA, orGBCMS_NORMALIZEprocess names. - Rust
shared/module: Common BAM utilities, BAQ, filters, fragment logic, and statistics extracted fromcounting/intoshared/. Any Rust consumer crate linking against gbcms internals must update import paths.
✨ Added¶
- Phase 3 WFA+PairHMM unification (
feat: unify check_complex Phase 3): Complex indel classification now routes through a unified pangenomic pipeline — fast-path WFA alignment with PairHMM fallback. Haplotype matrix construction viapangenome.rs, WFA routing viawfa_router.rs. Significantly improves classification accuracy on complex multi-allelic variants. - RNA mode output columns (
fix(rna): pass mode= to VcfWriter/MafWriter):gbcms rnanow correctly emits RNA-specific columns in both VCF and MAF output. Previously, all RNA columns (SEN,ANT,ASEN,RED,SPLin VCF;rna_sense_depth,rna_antisense_depth,rna_alt_sense_count,rna_editing_site,rna_splice_spanningin MAF) were silently absent regardless of mode. Regression tests added. gbcms normalizeNextflow module: Newnormalize/main.nffor standalone variant normalization without counting in Nextflow pipelines.- Output Formats reference doc:
docs/reference/output-formats.md— complete column-level schema reference for VCF and MAF output under all mode/flag combinations (DNA vs RNA, with mFSD, with normalization columns).
🔧 Fixed¶
- Complex indel classification (
fix: correctly classify complex indels): Fixes for Phase 3 dispatch cases 2, 3, and 4 — previously misclassified complex variants whereref_len ≠ alt_lenand the CIGAR structure doesn't map cleanly to pure insertion or deletion. rna_editing_dblog leakage (fix: gate rna_editing_db from DNA mode): DNA mode no longer emits a log line referencingrna_editing_db.- CliRunner terminal width (
fix: widen CliRunner terminal): CI help output truncation resolved —CliRunner(mix_stderr=False, terminal_width=120)prevents Typer/Rich from hiding options in the middle of the params list. - Filter defaults documentation: Secondary, supplementary, and QC-failed
filters are on by default for DNA mode — corrected in
cli/dna.md,cli/rna.md, andread-filters.md. - RNA mode output pipe wiring (critical silent bug):
Pipeline._write_output()now passesmode=self.config.modeto bothVcfWriterandMafWriter.
🏗️ Refactored¶
- Rust
shared/module extraction:bam_utils,baq,filters,fragment,statsextracted from thecounting/directory into a new top-levelshared/module, enabling reuse acrosscounting/andnormalize/. parquet_writer.rsrelocated: Moved fromcounting/intoshared/during module extraction.
📚 Documentation¶
Major documentation overhaul — 28+ files updated:
- Complete MkDocs plugin utilization pass (tabbed, details, admonitions,
mermaid, code annotations, glightbox) across all reference pages
- WFA fast-path Phase 3 documented in allele-classification.md
- Complex indels guide: RNA compatibility and exon-boundary limitation (D6)
documented with cross-links
- Architecture module tree corrected for shared/ and new Nextflow modules
- Filter defaults corrected across cli/dna.md, cli/rna.md, read-filters.md
- Mermaid diagrams fixed: raw unicode escapes removed, \\n → <br/>,
backslash-escaped quotes removed
- NEW: docs/reference/output-formats.md — authoritative output schema reference
- Versioned docs assets: old opaque-named binary files replaced with
{name}_{version}.{ext} convention (overview_4.0.0.pdf,
allele_classification_4.0.0.pdf, read_filter_4.0.0.jpg); 5 stale files
deleted; poster references corrected to match page content
🧹 Chores¶
- All Clippy
-D warningsresolved acrossengine.rs,rna.rs,variant_checks.rs,pairhmm.rs ruff,black,mypyall pass with 0 errors (38 source files checked)- Auto-generated mermaid SVGs removed from git (added to
.gitignore) fallback_to_build_date=trueadded togit-revision-date-localizedplugintest_cli_dna_rna.py: mypyattr-definedfixed by annotating_click_appasclick.Group;Set[str | None]→Set[str]viaif p.name is not None
🧪 Tests¶
tests/test_pipeline_rna.py(NEW): 7 pipeline-level integration tests for RNA mode output — VCF INFO headers, INFO values, FORMAT field, MAF column headers, MAF values, and negative DNA assertionstests/test_rna_output.py: 4 write round-trip tests added (VCF INFO values, RED flag on/off, MAF RNA column values)tests/test_maf_preservation.py:test_vcf_to_maf_always_uses_sample_nameaddedtests/test_cli_dna_rna.py: mypy fixes (click.Group cast, None guard)- All test
MockCountshelpers updated with RNA fields
[3.0.0] - 2026-03-05¶
⚠️ Breaking Changes¶
- Package renamed:
py-gbcms→gbcms. Update your dependencies: A finalpy-gbcms==3.0.0deprecation stub on PyPI re-exportsgbcmsand issues aDeprecationWarningfor smooth migration.
✨ Added¶
- mFSD native integration:
--mfsdand--mfsd-parquetflags output a Parquet file with 31 MAF columns + 7 VCF INFO fields via the Rust native Parquet writer. Compression: ZSTD level 1. - CLI validation hardening: 12 validation gaps resolved — fail-fast BAM
accessibility check,
--lenient-bamflag for permissive mode,.vcf.bgzaccepted as a valid variant input extension. py-gbcmsdeprecation stub:compat/py-gbcms/shim published to PyPI aspy-gbcms==3.0.0for backwards compatibility during migration.
🔧 Fixed¶
- Parquet output compression switched from SNAPPY to ZSTD(1).
- Resolved
mypyAlignedSegment.cigarattribute errors in test suite. - Lint and stale-comment cleanup post Phase 6 audit.
📚 Documentation¶
- Full documentation sync with codebase after Phase 6 (mFSD) merge.
- PDF generation guide added to developer documentation.
.agent/rules/directory added; stale.antigravitydocs removed.
🏗️ CI¶
- Added
mkdocs-print-site-pluginto CI docs install step andpyproject.toml.
[2.8.0] - 2026-02-23¶
✨ Added¶
- PairHMM alignment backend: Alternative Phase 3 alignment via
--alignment-backend hmmwith probabilistic scoring using base quality probabilities. Configurable LLR threshold (default 2.3 ≈ ln(10)) and gap probabilities for repeat/non-repeat regions. 6 new CLI options. Exposed as first-class Nextflow params innextflow.config - MNP min-BQ-across-block quality strategy: MNP quality now assessed using min(BQ) across the entire block, matching C++ GBCMS
baseCountDNP. Low-quality MNP reads now fall through tocheck_complexfor masked comparison instead of being silently skipped - Per-phase ClassifyResult counters: Diagnostic counters track how many reads are resolved in each classification phase (Phase 1/2/2.5/3)
--traceflag: Two-tier Rust logging —--verbosefor debug,--tracefor per-read classification diagnostics via pyo3-log
🔧 Fixed¶
- Phase 2/2.5 overcounting: Complex variants with
REF >> ALTnow skip Phase 2 Case B and Phase 2.5 whenref_len > 2 × alt_len— short ALT trivially matches, edit distance is biased toward shorter allele - Phase 3 bypass for pure DEL/INS: Removed
is_worth_realignmentprefilter — CIGAR structure is ground truth for pure deletions/insertions, prefilter was overcounting - DP anchor overlap: Now uses single-position check (
read_start ≤ variant.pos), matching Mutect2, VarDictJava, and samtools mpileup standard - MNP LowQuality routing: LowQuality reads now fall through to
check_complexfor masked comparison instead of being skipped entirely - S3 underflow guard: Guards against negative
ctx_offsetin deletion S3 validation
🏗️ Refactored¶
- Rust module structure: Split
counting.rs(1904 LOC) andnormalize.rs(1221 LOC) into idiomatic module directories:counting/(7 modules) andnormalize/(7 modules) - AlignmentBackend threading:
AlignmentBackendenum threaded through all Phase 3 call sites
📚 Documentation¶
- Comprehensive audit: 28 fixes across 23 files — all GitBook URLs → MkDocs, version templating (X.Y.Z), undocumented CLI options, 5 mermaid diagrams updated for post-2.7.0 logic, architecture module tree refreshed, PairHMM documented end-to-end
- Deleted stale
nextflow/CHANGES.md
🧹 Chores¶
- Fix all cargo clippy warnings
- Fix ruff lint issues, black formatting
- Update test expectations for new behavior
- CI: skip test workflow for docs-only changes
🧪 Tests¶
- 8 Python alignment backend integration tests
- 5 SW-vs-PairHMM concordance tests
- 10 MNP unit tests
- Multi-allelic isolation, DP/neither, fragment consensus, normalization tests
[2.7.0] - 2026-02-19¶
✨ Added¶
- Phase 2.5 edit distance fallback: When read reconstruction length matches neither REF nor ALT (e.g., incomplete MAF definition), Levenshtein distance discriminates the closest allele with >1 edit margin safety guard
- Phase 3 local SW fallback: Complex variants where semiglobal alignment produces confident-but-wrong calls (e.g., EPHA7
TCC→CT) are now rescued via local Smith-Waterman that soft-clips mismatched flanks. Dual-trigger requires both score reversal and ≥2-point margin
🔧 Fixed¶
- Allele-based dispatch (
check_allele_with_qual): Routes byref_len × alt_leninstead of unreliablevariant_typestring labels. SNP (1×1), insertion (1×N), deletion (N×1), MNP (N×N equal), complex (N×M unequal) — eliminates misrouting when callers emit inconsistent type annotations - SW semiglobal argument order: Fixed
ref_hap/alt_hapargument swap in semiglobal alignment that was scoring reads against the wrong haplotype - Haplotype trimming removed: Eliminated shared symmetric trim that caused
slice index starts at 7 but ends at 6panics on asymmetric indels; replaced with validated per-haplotype bounds - MNP fallback: MNP reads now correctly fall through to SW alignment instead of silently returning "neither" on partial mismatches
- Dual-count guard: Prevents a single read from being counted as both REF and ALT when SW scores are exactly equal
- Soft-clip restriction: Soft-clipped bases no longer incorrectly contribute to variant region reconstruction
- Strand bias orientation: Strand bias (Fisher's exact) now couples to the winning allele, not the raw alignment orientation
- Interior REF quality proxy: Reads falling entirely within a large deletion (>50bp) now use median base quality instead of 0
- Interior REF guard removed: Eliminated the
has_large_cigar_delguard that massively overcounted REF for large deletions by misclassifying ALT-supporting reads
🧹 Chores¶
- Clippy: Removed unused
has_large_cigar_delvariable - Tests: Updated
test_fuzzy_complex::TestLengthMismatchexpectation to reflect Phase 3 local SW fallback behavior
[2.6.1] - 2026-02-19¶
🔧 Fixed¶
- Per-haplotype trimming: Fixed
slice index starts at 7 but ends at 6panic incounting.rson asymmetric indels. Replaced shared symmetric trim with independent per-haplotypetrim_haplotype()function that calculates bounds safely for each allele
✨ Added¶
- Tolerant REF validation: Variants with ≥90% REF match against the FASTA are now counted (status
PASS_WARN_REF_CORRECTED) instead of being silently rejected. The FASTA REF is used for haplotype construction. Variants with <90% match are still rejected asREF_MISMATCH
📚 Documentation¶
- Visual posters: Added overview, normalization, and read-filter/counting-metrics posters (JPG) to reference documentation pages with lightbox support
- Embedded PDFs: Added inline PDF viewer for allele classification guide and detailed overview presentation via
mkdocs-pdfplugin - Variant normalization: Updated REF validation docs with 3-tier flowchart,
PASS_WARN_REF_CORRECTEDstatus, and EGFR exon 19 real-world example
🔧 CI¶
deploy-docs.yml: Addedmkdocs-pdfto docs CI pip install dependencies
[2.6.0] - 2026-02-18¶
✨ Added¶
- Adaptive context padding: Dynamically increases
ref_contextflanking in tandem repeat regions (homopolymer through hexanucleotide). Formula:max(default, repeat_span/2 + 3), capped at 50bp. Enabled by default (--adaptive-context/--no-adaptive-context) gbcms normalizecommand: Standalone variant normalization (left-align + REF validate) without counting, outputs TSV with original and normalized coordinates- Nextflow parameters:
fragment_qual_threshold,context_padding,show_normalization,adaptive_contextnow configurable innextflow.config - Docs restructure: Split monolithic
variant-counting.mdinto 4 focused pages: Variant Normalization, Allele Classification, Counting Metrics, Read Filters - HPC install docs: Micromamba-based source install with Python 3.13
🔧 Fixed¶
- Interior REF guard for large deletions (>50bp): Reads falling entirely within a deleted region are now correctly classified as REF instead of ALT by Smith-Waterman
- Windowed reciprocal overlap: Improved shifted indel detection using bidirectional overlap scoring
- Complex variant counting (EPHA7
TCC→CT): Fixed base quality extraction for all variant-type handlers (check_insertion,check_deletion,check_mnp,check_complex) - MAF VCF-style conversion: Corrected complex variant handling in MAF→internal coordinate conversion
- Lint: Fixed ruff I001/E402/B905 and black formatting in
pipeline.py
🔄 Changed¶
- Dead code removed:
GenomicIntervalclass,Variant.intervalproperty,fragment_countingconfig field fetch_single_base()refactored: Delegates tofetch_region(), removing 33 lines of duplicated chr-prefix retry logic- Release guide: Updated version locations table with exact line numbers and verification command
- Nextflow pipeline diagram added to docs index
[2.5.0] - 2026-02-12¶
✨ Added¶
--preserve-barcodeflag: Keeps originalTumor_Sample_Barcodefrom input MAF instead of overriding with BAM sample name (MAF→MAF workflows)--column-prefixparameter: Controls prefix for gbcms count columns in MAF output (default: none; use--column-prefix t_for legacy compatibility) ⚠️CoordinateKernel: Centralized MAF↔internal 0-based coordinate conversion with variant-type-aware logic for SNP, insertion, deletion, and complex variants- Nextflow
FILTER_MAFmodule: Per-sample MAF variant filtering byTumor_Sample_Barcodesupporting exact match, regex, and multi-select (comma-separated) modes - Nextflow
PIPELINE_SUMMARYmodule: Aggregated per-sample filtering statistics with formatted console output - Nextflow
--filter_by_sampleparameter and samplesheettsbcolumn for multi-sample MAF workflows - Nextflow documentation: Samplesheet
tsbcolumn guide,--filter_by_sampleparameter reference, multi-sample MAF filtering examples
🔧 Fixed¶
- Fragment quality extraction (critical): All variant-type handlers (
check_insertion,check_deletion,check_mnp,check_complex) now return actualbase_qualfrom CIGAR walk instead of 0 — fixes systematic ALT undercount for indels in fragment-level consensus - FILTER_MAF heredoc conflict: Restructured script from Python shebang to
python3 << 'PYEOF'pattern, resolvingSyntaxErrorfrom bash syntax in Python context - FILTER_MAF string quoting: Changed to single-quoted Python strings for Nextflow variable interpolation to prevent CSV-parsed double-quote conflicts
splitCsvquote handling: Addedquote:'"'parameter for correct RFC 4180 parsing of comma-separated TSB values within quoted CSV fields- mypy
no-redeferror: Removed redundant type annotation inoutput.pyelse branch
🔄 Changed¶
- MAF output column prefix default: Changed from
t_to empty string (no prefix). Use--column-prefix t_for legacyt_ref_count/t_alt_countstyle columns ⚠️ MafWriterrefactored: MAF→MAF path preserves all original columns verbatim; VCF→MAF path builds row from GDC fieldnames withCoordinateKernelcoordinate conversion- Nextflow
GBCMS_RUNinput: Variants bundled into sample tuple(meta, bam, bai, variants)instead of separate channel - Nextflow
GBCMSworkflow: Simplified to 2-channel interface (ch_samples,ch_fasta) from 3 channels - Nextflow config: Added
column_prefix,preserve_barcode,filter_by_sampleparameters
[2.4.0] - 2026-02-10¶
✨ Added¶
- Fragment Consensus Engine:
FragmentEvidencestruct with u64 QNAME hashing and quality-weighted R1/R2 consensus; ambiguous fragments are discarded (not assigned to REF) --fragment-qual-threshold: New CLI option (default 10) controlling consensus quality difference for fragment conflict resolution- Windowed Indel Detection: ±5bp positional scan with 3-layer safeguards (sequence identity, closest match, reference context validation)
- Quality-Aware Complex Matching: Masked comparison that ignores bases below
--min-baseq; 3-case comparison (equal-length, ALT-only, REF-only) with ambiguity detection - Variant Counting Guide: New
docs/reference/variant-counting.mdwith algorithm diagrams for all variant types (~700 lines) - MAF Normalization Docs: Added indel normalization and coordinate handling to
docs/reference/input-formats.md - 47 Tests: Up from 16 — added
test_shifted_indels.py(15),test_fuzzy_complex.py(14),test_fragment_consensus.py(4)
🔄 Changed¶
--min-baseqdefault:0→20(Phred Q20) — activates quality masking by default for improved accuracy on low-coverage samples ⚠️--versionflag: Added to CLI (gbcms --version)- Deploy-Docs Workflow: Replaced
mkdocs gh-deploywithmikefor multi-version documentation; deploysstable(tagged version) from main anddevfrom develop branch; addedextra.version.provider: miketomkdocs.ymlwith version switcher widget
🔧 Fixed¶
- Fragment double-counting bug: R1+R2 pairs previously counted as two independent observations; now collapsed via quality-weighted consensus
- MAF Input Hardening: Graceful handling of missing/malformed fields with warnings instead of crashes
- CI Release Pipeline: Stabilized manylinux builds — migrated from manylinux_2_28 to manylinux_2_34, resolved OpenSSL/CURL vendor conflicts via
docker-optionspattern - Type stubs:
_rs.pyiandgbcms_rs.pyisynced with Rust bindings (addedref_context,ref_context_start,fragment_qual_threshold) - Linting: All files pass black, ruff, and mypy
📚 Documentation¶
- Architecture comparison table updated with windowed indels and masked comparison
- Nextflow config and docs updated with new
min_baseqdefault - Testing guide expanded with Phase 2a/2b test files
- HPC/RHEL 8 installation instructions updated with
clangdevheader management - Release guide updated with docs version locations
.antigravityproject files updated with current Rust LOC (~1270) and test counts (47)
[2.3.0] - 2026-02-06¶
✨ Added¶
- Nextflow BAI Auto-Discovery: Checks
.bam.baiand.baiextensions automatically - Documentation Modernization: Hierarchical navigation, glightbox, panzoom, abbreviations
- Performance Benchmarks: cfDNA duplex sample metrics in documentation
- RHEL 8 Installation Guide: Conda-based source installation for legacy Linux
🔄 Changed¶
- Dockerfile: Added
procps,bash, OCI labels,maturin[patchelf], selective COPY - Nextflow Config:
--platform linux/amd64, shell config, local profile, observability (trace/report/timeline/dag) - MkDocs: Switched to
navigation.sections, 20+ abbreviations with hover tooltips - GitHub Actions: Consolidated deploy-docs workflows, added caching and PR validation
- CI Wheels: Migrated from
manylinux_2_28tomanylinux_2_34(AlmaLinux 9 with OpenSSL 3.0+)
🔧 Fixed¶
- Nextflow: Empty
--suffixargument no longer causes failures - Admonitions: Converted GitHub-style alerts to MkDocs syntax
- CI Build: Resolved
curl-sysOpenSSL version conflict by switching to manylinux_2_34
[2.2.0] - 2026-02-04¶
✨ Added¶
- Multi-platform Wheel Publishing: Maturin-based CI builds for Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), and Windows
- Structured Logging: New
utils/logging.pymodule with Rich console output, timing utilities, and log file support - Mermaid Diagrams: Architecture documentation with interactive flowcharts
- Release Guide: Comprehensive
docs/RELEASE.mdwith git-flow workflow
🔄 Changed¶
- Folder Restructure: Moved Rust code to
rust/(bundled asgbcms._rs) - Config Hierarchy: Nested Pydantic models (
ReadFilters,QualityThresholds,OutputConfig) for better organization - Code Quality: Added
__all__exports, docstrings, and type hints across all modules - StrEnum: Modern enum pattern with Python 3.10 backport
📚 Documentation¶
- New
docs/ARCHITECTURE.mdwith system diagrams - New
docs/DEVELOPMENT.md(developer guide) - New
docs/TESTING.md(testing guide) - Updated MkDocs with mermaid2 plugin and snippet includes
[2.1.2] - 2025-11-25¶
🔧 Fixed¶
- PyPI Distribution: Fixed source distribution size issue by correctly excluding large files (tests, docs, etc.) via
pyproject.tomlconfiguration.
[2.1.1] - 2025-11-25 [YANKED]¶
Yanked Release
This release was yanked from PyPI due to a source distribution size limit error. Use 2.1.2 instead.
🔧 Fixed¶
- PyPI Distribution: Added MANIFEST.in (failed to work with Hatchling) to reduce source distribution size
- Documentation: Added comprehensive Installation guide
- Documentation: Unified Contributing guide (merged code + docs contributions)
- Documentation: Added Changelog to documentation navigation
[2.1.0] - 2025-11-25¶
✨ Added¶
Nextflow Workflow¶
- Production-ready Nextflow workflow for processing multiple samples in parallel
- SLURM cluster support with customizable queue configuration
- Per-sample suffix support via optional
suffixcolumn in samplesheet - Docker and Singularity profiles for containerized execution
- Automatic BAI index discovery with validation
- Resume capability for failed workflow runs
- Resource management with automatic retry and scaling
- Comprehensive documentation in
docs/NEXTFLOW.mdandnextflow/README.md
Documentation¶
- Usage pattern comparison guide (
docs/WORKFLOWS.md) for choosing between CLI and Nextflow - MkDocs integration for beautiful GitHub Pages documentation
- Local documentation preview with live reload (
mkdocs serve) - Staging deployment from
developbranch for testing docs - Production deployment from
mainbranch - Reorganized documentation structure with clear CLI vs Nextflow separation
- CLI Quick Start guide (
docs/quick-start.md)
🔧 Changed¶
- Documentation workflow: docs now live on
mainbranch with automated deployment - GitBook integration: configured to read from
mainbranch - Nextflow module: improved parameter passing with meta.suffix support
📝 Documentation¶
- Complete Nextflow workflow guide with SLURM examples
- Per-sample suffix usage examples
- Git-flow documentation workflow guide
- Local preview instructions
- Updated README with clear usage pattern separation
[2.0.0] - 2025-11-21¶
🚀 Major Rewrite¶
Version 2.0.0 represents a complete rewrite of py-gbcms with a focus on performance, correctness, and modern architecture.
✨ Added¶
Core Features¶
- Rust-based Counting Engine: Hybrid Python/Rust architecture for 20x+ performance improvement
- Strand Bias Statistics: Fisher's exact test p-values and odds ratios for both reads (
SB_PVAL,SB_OR) and fragments (FSB_PVAL,FSB_OR) - Fragment-Level Counting: Majority-rule fragment counting with strand-specific counts (
RDF,ADF) - Variant Allele Fractions: Read-level (
VAF) and fragment-level (FAF) allele fraction calculations - Thread Control: Explicit control over parallelism via
--threadsargument (default: 1)
Input/Output¶
- VCF Output Format: Standard VCF with comprehensive INFO and FORMAT fields
- MAF Output Format: Extended MAF with custom columns for strand counts and statistics
- Column Preservation: Input MAF columns are preserved in output
- Multiple BAM Support: Process multiple samples via
--bam-listor repeated--bamarguments - Sample ID Override: Explicit sample naming via
--bam sample_id:pathsyntax
Filters¶
--filter-duplicates: Filter duplicate reads (default: enabled)--filter-secondary: Filter secondary alignments--filter-supplementary: Filter supplementary alignments--filter-qc-failed: Filter reads that failed QC--filter-improper-pair: Filter improperly paired reads--filter-indel: Filter reads with indels in CIGAR
CLI & Usability¶
- Modern CLI: Built with Typer and Rich for beautiful terminal output
- Progress Tracking: Real-time progress bars and status indicators
- Direct Invocation: Use
gbcms runinstead ofpython -m gbcms.cli - Output Customization:
--suffixflag for output filename customization - Flexible Input: Support for both VCF and MAF input formats
Infrastructure¶
- Docker Support: Production-ready multi-stage Dockerfile with optimized layers
- Type Safety: Full type annotations with mypy support
- Type Stubs: Provided
.pyistub file for Rust extension - Comprehensive Tests: Extended test suite with accuracy and filter validation
- CI/CD: GitHub Actions workflows for testing, linting, and releases
🔄 Changed¶
Architecture¶
- Migrated from pure Python to hybrid Python/Rust architecture
- Core counting logic implemented in Rust using
rust-htslib - Data parallelism over variants with per-thread BAM readers
Output Formats¶
- VCF FORMAT fields: Strand-specific counts now use comma-separated values (e.g.,
RD=5,3for forward,reverse) - MAF columns: Standardized column names (
t_ref_count_forward,t_alt_count_reverse, etc.) - Coordinate System: Internal 0-based indexing with correct conversion for VCF (1-based) and MAF output
Performance¶
- Speed: 20x+ faster than v1.x on typical datasets
- Memory: Efficient per-thread BAM readers with minimal overhead
- Scalability: Configurable thread pool for optimal resource usage
Dependencies¶
- Python: Updated to require Python ≥3.10
- Rust: pyo3 0.27.1, rust-htslib 0.51.0, statrs 0.18.0
- Python Packages: pysam ≥0.21.0, typer ≥0.9.0, rich ≥13.0.0, pydantic ≥2.0.0
🗑️ Removed¶
- Legacy Python Counting: Pure Python implementation removed in favor of Rust
- Old CLI: Deprecated
python -m gbcms.clientry point - Unused Dependencies: Removed
cyvcf2andnumba(no longer needed) - Pre-commit Hooks: Removed in favor of explicit linting in CI
🐛 Fixed¶
- Correct handling of complex variants (MNPs, DelIns)
- Proper strand assignment for fragment counting
- Reference validation against FASTA for all variant types
- Thread-safe BAM access with per-thread readers
📚 Documentation¶
- Complete rewrite of all documentation
- New guides:
INSTALLATION.md,CLI_FEATURES.md,INPUT_OUTPUT.md - Comprehensive API documentation
- Docker usage examples
- Contributing guidelines updated
🔧 Technical Details¶
Rust Components¶
gbcms._rs: PyO3-based Rust extension (bundled in wheel)- Fisher's exact test via
statrscrate - Rayon-based parallelism with configurable thread pools
- Safe memory management with Rust's ownership model
Testing¶
- 16 comprehensive test cases
- Accuracy validation with synthetic BAM files
- Filter validation for all read flag combinations
- Integration tests with real-world data
⚠️ Breaking Changes¶
Version 2.0.0 is not backward compatible with 1.x. Key breaking changes:
- CLI syntax: Use
gbcms runinstead ofpython -m gbcms.cli - Output format: VCF/MAF column structures have changed
- Default behavior: Only duplicate filtering enabled by default (was: all filters)
- Dependencies: Requires Rust toolchain for installation from source
- Python version: Minimum Python 3.10 (was: 3.8)
📦 Installation¶
# From PyPI (includes pre-built wheels)
pip install gbcms
# From source (requires Rust)
pip install git+https://github.com/msk-access/gbcms.git
# Docker
docker pull ghcr.io/msk-access/gbcms:2.0.0
🙏 Acknowledgments¶
This rewrite was designed and implemented with a focus on correctness, performance, and modern best practices in bioinformatics software development.
[1.x] - Legacy¶
Previous versions (1.x) used a pure Python implementation. See git history for details.
abbreviations