Statistical Evaluation

Before feeding extracted feature matrix numbers into ensemble models, kreview utilizes native scipy.stats functionality executed in evaluate_feature().

Our data almost never follows a clean parametric distribution, so we strictly use non-parametric tests.

flowchart LR
    classDef step fill:#8b5cf6,stroke:#5b21b6,color:#fff;
    A["Feature Matrix"]:::step --> B["Kruskal-Wallis\n(4-group omnibus)"]:::step
    B --> C["Mann-Whitney U\n(pairwise)"]:::step
    C --> D["Cohen's d\n(effect size)"]:::step
    D --> E["Spearman Rank\n(confounders)"]:::step
    E --> F["Univariate AUC\n(per-feature LR)"]:::step
    F --> G["Mutual Information\n(non-linear)"]:::step
    G --> H["mRMR / Hybrid Union\n(Selection)"]:::step
    H --> I["Pass to ML\nModels"]:::step

The 4-Way Omnibus Test (Kruskal-Wallis)

We group the sample's generated feature matrices by the 4 primary CtDNALabeler labels (True ctDNA+, Possible ctDNA+, Possible ctDNA−, Healthy Normal).

Because we are checking more than two independent samples to determine if they originate from the same distribution, we employ the Kruskal-Wallis H-test (stats.kruskal).

If the omnibus \(p\)-value is significant (\(p < 0.05\)), it suggests that the feature is stratifying at least one label group. Pairwise analysis then identifies which groups differ.

Pairwise Separation (Mann-Whitney U)

We run five independent 2-sample Mann-Whitney U rank-sum tests:

We additionally compute a Rank-Biserial correlation to understand the direction and magnitude of separation.

Benjamini-Hochberg FDR Correction

Because kreview executes five independent pair-wise checks simultaneously, it introduces a significant multiple-testing problem. To prevent artificially inflated False Positive rates (p-hacking), the engine natively applies the Benjamini-Hochberg Method to wrap all 5 raw \(p\)-values. The generated fdr_pvalue arrays are what you should evaluate for true significance.

Effect Size (Cohen's d)

As an accompaniment to strict \(p\)-values (which easily become inflated by large sample cohorts), we compute Cohen's d. This represents the standardized difference between two means (True+ vs Healthy):

d = \frac{M_{1} - M_{2}}{SD_{pooled}}

Confounder Tracking (Spearman Rank)

Fragmentomics logic is notorious for being accidentally driven by sequencing depth rather than actual biological shedding signals.

To prevent this, evaluate_feature independently extracts Spearman Rank Correlations mapping the generated feature against:

1. max_vaf — Is the feature actually scaling linearly with structural tumor burden?
2. total_fragments — Is the feature artificially inflating simply because a sample was sequenced to 4,000x depth?

Per-Feature QC Metrics

In addition to statistical tests, evaluate_feature() computes three data quality fields for each feature:

These metrics are saved to *_eval_stats.parquet and surfaced in the dashboard's Cohort & QC page.

Feature Selection Scoring (v0.0.9+)

After descriptive statistics are computed, kreview scores every feature using complementary methods to decide which features enter the downstream ML models. The default strategy is mRMR.

Minimum Redundancy Maximum Relevance (mRMR) [Default]

The strategy="mrmr" method (via mrmr-selection) iteratively selects features that maximize correlation with the target label (relevance) while minimizing correlation with already-selected features (redundancy).

Optimization Objective

At each step \(i\), mRMR selects the feature \(f_i\) that maximizes:

where:

• \(\mathcal{F}\) is the full feature set
• \(S\) is the set of already-selected features
• \(y\) is the binary ctDNA label

Relevance and Redundancy Metrics

kreview's mRMR implementation uses:

Algorithm Steps

1. Impute NaNs in the feature matrix (median/mean/zero).
2. Compute F-statistic(feature, target) for ALL features → relevance scores.
3. Initialize S = {} (selected set).
4. For i = 1 to K (where K = top_percentile% of total features):
   a. For each candidate f not in S:
      - Score(f) = F(f, y) − (1/|S|) × Σ |Pearson(f, s)| for s in S
   b. Select f_i = argmax Score(f)
   c. Add f_i to S
5. Apply variance guard: drop any selected feature with std = 0.
6. Return selected feature set + QC metadata.

Why mRMR Over Hybrid Union?

Hybrid Union Selection [Legacy/Fallback]

The strategy="hybrid_union" method determines the feature set by:

where \(X\) is controlled by --top-percentile (default: 10%). The union ensures that features strong in either metric are retained — a feature with high AUC but low MI (linear predictor) or high MI but low AUC (non-linear predictor) will both be included.

Multimodal Selection (v0.0.11+)

When aggregating multiple feature sets in kreview eval multimodal, the pipeline offers two higher-order selection strategies via --multimodal-selection. These operate on the super-matrix (all evaluators fused), which can contain hundreds of features.

Mutual Information (mi) [Default]

Rapidly selects the top \(K\) features using sklearn's mutual_info_classif ranking across all concatenated features in the super-matrix.

• Strengths: Fast (\(O(N)\)), captures non-linear dependencies, no model training required.
• Weakness: Does not consider feature-feature interactions or redundancy.

Boruta-SHAP (boruta_shap)

An interaction-aware selection method using the BorutaShap algorithm, which wraps a tree-based model (XGBoost) with SHAP importance values and a rigorous statistical test.

How Boruta-SHAP Works

flowchart TB
    classDef step fill:#8b5cf6,stroke:#5b21b6,color:#fff;
    A["Original Features
(N features)"]:::step --> B["Create Shadow Features
(N random permutations)"]:::step
    B --> C["Train XGBoost on
Real + Shadow Features"]:::step
    C --> D["Compute SHAP Values
for All 2N Features"]:::step
    D --> E{"Is real feature's SHAP
> max shadow SHAP?"}:::step
    E -->|Yes, consistently| F["✅ Accept Feature"]:::step
    E -->|No| G["❌ Reject Feature"]:::step
    E -->|Inconclusive| H["🔄 Repeat Trial
(50 iterations)"]:::step
    H --> C

Key Concepts

1. Shadow Features: For each real feature, a "shadow" copy is created by randomly permuting its values. This destroys any relationship with the target while preserving the distribution. Shadow features serve as a null hypothesis baseline — if a real feature is no more important than its shuffled copy, it carries no genuine signal.

2. SHAP Importance: Instead of using impurity-based importance (which is biased toward high-cardinality features), Boruta-SHAP uses SHAP values — a game-theoretic measure of each feature's marginal contribution to predictions. This captures non-linear effects, interactions, and is model-agnostic in interpretation.

3. Statistical Testing: Over 50 trials (n_trials=50), each feature's SHAP importance is compared against the maximum shadow feature importance. Features that consistently exceed this threshold are Accepted; those that don't are Rejected. Features with borderline performance remain Tentative and are ultimately rejected.

4. Graceful Fallback: If Boruta-SHAP rejects all features (e.g., extremely noisy super-matrix), the pipeline falls back to MI-based top-\(K\) selection automatically.

kreview's Configuration

When to Choose Each Strategy