Sklearn Models & Metrics

Every feature evaluator undergoes automated ensemble classification to measure how well a cfDNA feature set discriminates ctDNA-positive from ctDNA-negative samples.

Inside single_feature_model(), three CPU classifier families are evaluated using stratified cross-validation against a binary label derived from the ctDNA labeling engine. Optional GPU foundation models add up to four more.

The Predictive Ensemble

Three CPU classifier families are trained on every feature set:

# 1. Linear Baseline (Logistic Regression)
lr = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000, random_state=random_state, class_weight="balanced"))
])

# 2. Tree-Based (Random Forest)
rf = RandomForestClassifier(
    n_estimators=100,
    max_depth=5,
    min_samples_leaf=max(1, min(10, len(y) // 10)),
    random_state=random_state,
    class_weight="balanced",
)

# 3. Gradient Boosting (XGBoost)
xgb = XGBClassifier(
    n_estimators=100,
    max_depth=5,
    learning_rate=0.1,
    random_state=random_state,
    eval_metric="logloss",
    use_label_encoder=False,
)

Graceful Degradation

XGBoost is imported dynamically with a try/except. If xgboost is unavailable (common in restricted HPC environments), the engine safely falls back to Random Forest and Logistic Regression only.

Class Imbalance & CV Strategy

Clinical cohorts are typically imbalanced (e.g. 4,000 cancer patients vs 300 healthy controls). To prevent majority-class bias, the pipeline uses Stratified K-Fold Cross Validation:

cv = StratifiedKFold(n_splits=folds, shuffle=True, random_state=random_state)
rf_probs = cross_val_predict(rf, X, y, cv=cv, method="predict_proba")

The number of folds is dynamically bounded by the minimum class size to prevent n_splits errors. All downstream metrics use out-of-fold predictions — the model never evaluates samples it trained on.

Metric Categories (10 total)

1. ROC-AUC & Bootstrap CIs

The primary scoring metric. To quantify uncertainty, kreview computes bootstrap 95% confidence intervals (n_resamples=1000) for every AUC score. Results are stored as auc_rf, auc_rf_ci_lower, auc_rf_ci_upper.

2. Precision-Recall (PR) Curves

PR curves are more informative than ROC when the positive class is rare. kreview computes precision-recall curves and average precision (AP) for all trained models (3 CPU + up to 4 GPU). PR data is stored as {prefix}_pr_curve and {prefix}_avg_precision.

3. Probability Calibration

Tree-based models (RF, XGBoost) tend to produce poorly calibrated probabilities. kreview computes sklearn calibration curves (prob_true vs prob_pred) in 10 uniform bins. Stored as rf_calibration.

4. Optimal Thresholding (Youden's J)

A continuous probability doesn't translate to a clinical decision. kreview calculates the binary cutoff that maximizes the separation between True Positive Rate and False Positive Rate:

\[J = \max(TPR - FPR)\]

optimal_idx = np.argmax(tpr - fpr)  # Youden's J Statistic
optimal_threshold = float(thresholds[optimal_idx])

This threshold generates the classification_report and confusion_matrix for each model.

5. Threshold Sensitivity Sweep

A single optimal threshold may be fragile. kreview evaluates sensitivity, specificity, and PPV across 50 thresholds (0.01–0.99) for the RF model. This helps clinicians choose an operating point based on their risk tolerance. Stored as rf_threshold_sweep.

6. Decision Curve Analysis (DCA)

DCA computes the net clinical benefit of using the model vs treating all or treating none. See the full DCA methodology guide.

Stored as rf_dca and xgb_dca.

7. Fold-Level AUC Tracking

To assess model stability, kreview runs cross_val_score separately and stores per-fold AUC for all trained models. A high standard deviation (>0.05) suggests the model is data-sensitive. Stored as {prefix}_fold_aucs and {prefix}_auc_std.

8. Sensitivity at Fixed Specificity (v0.0.16+)

In clinical ctDNA detection, zero false positives on healthy controls is paramount. kreview computes sensitivity at multiple specificity thresholds:

Metric	Key	Clinical Meaning
Sensitivity @ 100% spec	`{name}_sensitivity_at_100spec`	At zero FPR: how many cancers detected?
Sensitivity @ 99% spec	`{name}_sensitivity_at_99spec`	At 1% FPR
Sensitivity @ 95% spec	`{name}_sensitivity_at_95spec`	At 5% FPR
Healthy-normal specificity	`{name}_sensitivity_at_100spec_healthy`	At threshold where no Healthy Normal is FP
Threshold @ 100% spec	`{name}_threshold_at_100spec`	Decision boundary for zero FPR
Detected @ 100% spec	`{name}_n_detected_at_100spec`	Number of true positives at zero FPR

The healthy-normal variant uses the max predicted probability among Healthy Normal samples as the threshold — clinically this means "no healthy person is called positive."

9. SHAP Explainability

For RF and XGBoost, kreview generates SHAP (SHapley Additive exPlanations) values using TreeExplainer:

Beeswarm Plot: Global feature importance ranked by mean |SHAP|, colored by feature value
Dependence Scatter: Feature value vs SHAP impact, colored by an interacting feature
Waterfall Plots: Per-sample prediction decomposition for the most confident TP and FP

Interpretation

SHAP explains how the model decides, not biological causality. A feature with high SHAP impact may reflect a data artifact rather than a biological mechanism.

10. Subgroup Analysis (Out-Of-Fold)

After training, the pipeline evaluates model sensitivity across biological subgroups using out-of-fold predictions (preventing optimistic bias):

Cancer Type Stats: Sensitivity per cancer type (top 10 by sample count)
Assay Stats: Sensitivity stratified by ACCESS panel version (e.g. ACCESS129 vs ACCESS146)

Serialized into the results JSON under cancer_type_stats and assay_stats.

11. Nested CV Feature Group Ablation (v0.0.20+)

When feature group ablation is enabled (--run-ablation), kreview runs a nested cross-validation loop to identify the optimal feature group subset before final evaluation:

Outer CV (5-fold):
  For each outer fold k:
    Inner CV (3-fold on train_k):
      For each subset s in {ALL, solo_groups, leave-one-out}:
        Score s using sensitivity_at_100spec_healthy
      Winner_k = argmax(scores)
    Store Winner_k features for fold k

Final eval:
  For each fold k: train on Winner_k features, predict OOF
  Stitch OOF predictions → compute metrics with _compute_oof_metrics()
  Refit on full data using majority-vote feature subset

Key design choices:

Optimization metric: sensitivity_at_100spec_healthy — clinically, this means "how many cancers are detected when zero healthy controls are false positive?"
Subset generation: identify_feature_groups() maps feature columns to suffix-based groups (e.g., _mean_size, _count). generate_subsets() creates ALL + solo + leave-one-out combinations.
Per-fold communication: Different folds may select different winning subsets. The nested CV path in cpu_models() and gpu_models() manually iterates folds instead of using cross_val_predict().
Majority-vote refit: The final model is refit on the most commonly selected features across folds. This is stored as {model}_refit_features in the results JSON.
Backward compatibility: When per_fold_features is None, the standard cross_val_predict() path runs unchanged.

Results carry a nested_cv: true flag, and the scoreboard surfaces best_sens_100spec_healthy (the best sensitivity across all models).

Metric	Key	Description
Nested CV flag	`nested_cv`	`true` when ablation was used
Refit features	`{model}_refit_features`	Feature names used for final refit
Best sensitivity	`best_sens_100spec_healthy`	Max sensitivity across models (scoreboard)

GPU Foundation Models (v0.0.13+)

In addition to the CPU ensemble, kreview can optionally train GPU-accelerated foundation models when a CUDA-capable device is available. As of v0.0.18, four GPU model variants are supported (2 zero-shot + 2 fine-tuned).

Model Registry

Model Key	Class	Mode	Configurable Params
`tabpfn`	`TabPFNClassifier`	Zero-shot (frozen weights)	—
`tabpfn_ft`	`FinetunedTabPFNClassifier`	Fine-tuned	`--finetune-epochs` (50), `--finetune-lr` (1e-5)
`tabicl`	`TabICLClassifier`	Zero-shot (frozen weights)	—
`tabicl_ft`	`FinetunedTabICLClassifier`	Fine-tuned	`--finetune-epochs` (50), `--finetune-lr` (1e-5)

TabPFN (Tabular Prior-Data Fitted Network)

TabPFN is a prior-data fitted network that performs in-context learning on tabular classification tasks. It was pre-trained on millions of synthetic datasets and can achieve strong performance without hyperparameter tuning.

# v8.0.3 API (updated import paths)
from tabpfn import TabPFNClassifier

tabpfn = TabPFNClassifier(device="cuda")  # Zero-shot

TabICL (Tabular In-Context Learning)

TabICL is a large-scale in-context learning model for tabular data that uses transformer-based architectures.

# v2.1 API
from tabicl import TabICLClassifier

tabicl = TabICLClassifier(device="cuda")  # Zero-shot

Fine-Tuned Variants (v0.0.18+)

Fine-tuned variants adapt the pre-trained foundation model weights to the specific dataset during training. This can improve performance on domain-specific data (e.g., cfDNA fragmentomics) at the cost of increased training time.

# Fine-tuned TabPFN
from tabpfn import FinetunedTabPFNClassifier

tabpfn_ft = FinetunedTabPFNClassifier(
    device="cuda",
    n_epochs=50,        # Configurable via --finetune-epochs
    learning_rate=1e-5, # Configurable via --finetune-lr
)

# Fine-tuned TabICL
from tabicl import FinetunedTabICLClassifier

tabicl_ft = FinetunedTabICLClassifier(
    device="cuda",
    n_epochs=50,
    learning_rate=1e-5,
)

Zero-shot vs Fine-tuned

To run only zero-shot models (faster, no gradient computation), pass --gpu-models tabpfn,tabicl. The default (tabpfn,tabpfn_ft,tabicl,tabicl_ft) includes all four variants.

GPUModelCVAdapter (v0.0.18+)

TabPFN's classes_ attribute is a @property that raises AttributeError on unfitted models. Since sklearn's cross_val_predict probes classes_ before fitting, kreview wraps all GPU models in GPUModelCVAdapter:

class GPUModelCVAdapter:
    """sklearn-compatible wrapper for GPU foundation models."""
    def fit(self, X, y):
        self.model_.fit(X, y)
        self.classes_ = np.unique(y)  # Plain attribute, not @property
        return self

Exposes classes_ as a plain attribute set during fit()
Delegates predict(), predict_proba(), and get_params() to the inner model
Makes GPU models fully compatible with sklearn's CV infrastructure (cross_val_predict, cross_val_score)

GPU SHAP with shapiq

SHAP values for GPU models cannot use TreeExplainer (designed for tree models). Instead, kreview uses the shapiq package, which provides model-agnostic Shapley value computation via kernel-based approximation:

import shapiq

explainer = shapiq.TabularExplainer(
    model=model.predict_proba,
    data=X_background,
    index="SV",
)
interaction_values = explainer.explain(X_explain)

Performance

shapiq is computationally expensive. The --shap-samples flag controls how many background samples are used. Default is 500. GPU SHAP is only computed when --shap flag is passed to eval gpu.

GPU Model Persistence

By default, fitted GPU models are saved as .joblib files (like CPU models) for downstream SHAP rendering in reports. Since these files can be large (>200MB), use --skip-gpu-joblib to opt out. When joblib is skipped, reports will render mean |SHAP| bar charts from pre-computed values in the JSON instead of interactive beeswarm plots.

GPU Feature Capping (v0.0.16+)

TabPFN has practical VRAM limits (~150-200 features). When --max-gpu-features is set (default: 150 in Nextflow), kreview caps features using a priority hierarchy:

Score-based (from eval_stats): Top features by mutual_info or univariate_auc
Variance fallback: If no scores available, top features by variance

The same cap indices are applied to both training and holdout data to ensure dimension consistency.

80/20 Holdout Evaluation (v0.0.16+)

In addition to cross-validation, kreview performs holdout evaluation on the 20% test set that was never seen during feature selection or CV:

Full Data → 80/20 split (stratified by label)
  80% TRAIN → feature selection → 10-fold CV → development AUC
  20% TEST  → never touched → holdout AUC (honest estimate)

After CV completes, the best model is refit on the full training set and evaluated on the holdout:

Metric	Key	Description
Holdout AUC	`holdout_{name}_auc`	AUC on unseen test data
Holdout AUC CI	`holdout_{name}_auc_ci_lower/upper`	Bootstrap 95% CI
Holdout Sensitivity	`holdout_{name}_sensitivity_at_100spec`	Sensitivity at zero FPR on test
AUC Drop	`auc_drop` (scoreboard)	CV AUC − Holdout AUC (overfit diagnostic)

Interpreting AUC Drop

A positive auc_drop indicates the CV AUC was optimistic (overfitting). A negative value means the holdout actually performed better than CV (common with small test sets).

QC Metrics

For every feature, evaluate_feature() also computes data quality metrics:

Metric	Field	Purpose
Missing count	`n_missing`	Number of NaN values
Missing percentage	`pct_missing`	Percentage of samples with NaN
Zero variance	`is_zero_variance`	Whether `std == 0` (constant feature)

These are surfaced in the dashboard's Cohort & QC page and the statistical ledger.

Feature Stability

To assess whether the same features are consistently ranked as important across different data splits, kreview trains a fresh RF on each CV fold's training set and records which features appear in the top-10 by Gini importance. The result is a score from 0.0 (never in top-10) to 1.0 (always in top-10). Stored as feature_stability.

Visualization Themes

CVD-Safe Mode

Use the --cvd-safe flag to switch from the default neon palette to the Okabe-Ito color scheme, which is accessible for red-green colorblindness. The default palette uses curated colors optimized for dark backgrounds.

JSON Output Schema Summary

cpu_models() + gpu_models() + evaluate_holdout() produce 100+ JSON fields per feature set (when all 7 models are trained):

Category	Fields	Count
AUC & CI	`auc_{lr,rf,xgb}`, `auc__ci_lower`, `auc__ci_upper`	9
GPU AUC & CI	`auc_{tabpfn,tabpfn_ft,tabicl,tabicl_ft}`, `auc_*_ci_lower/upper`	12
Classification	`_classification_report`, `_confusion_matrix`	6 (CPU) + 8 (GPU)
Thresholds	`*_optimal_threshold`	3 (CPU) + 4 (GPU)
Sensitivity @ Spec	`_sensitivity_at_{100,99,95}spec`, `_threshold_at_100spec`, `_n_detected_at_100spec`, `_n_total_positive`	18 (CPU) + 24 (GPU)
Healthy-Normal Spec	`_sensitivity_at_100spec_healthy`, `_threshold_at_100spec_healthy`	6 (CPU) + 8 (GPU)
Holdout Metrics	`holdout__auc`, `holdout__sensitivity_at_100spec`, `holdout_n_train/test`	12+ (CPU) + 16+ (GPU)
PR Curves	`_pr_curve`, `_avg_precision`	6 (CPU) + 8 (GPU)
DCA	`rf_dca`, `xgb_dca`	2
Fold AUCs	`_fold_aucs`, `_auc_std`	6 (CPU) + 8 (GPU)
Calibration	`rf_calibration`	1
Threshold Sweep	`rf_threshold_sweep`	1
Feature Stability	`feature_stability`	1
Training Time	`*_training_time_sec`	3 (CPU) + 4 (GPU)
Feature Importances	`rf_feature_importances`, `top_features`	2
AUC Deltas	`auc_delta_rf_lr`, `auc_delta_xgb_rf`	2
Subgroups	`cancer_type_stats`, `assay_stats`	2
Selection QC	`selection_qc` (method, overlap stats, feature counts)	1
GPU OOF Probs	`{gpu_model}_oof_probs`	4
GPU SHAP	`{gpu_model}_shap_values` (when `--shap` enabled)	4
Nested CV	`nested_cv` (boolean), `{model}_refit_features` (list)	2 + 3–7

Reproducibility

All randomization points in kreview are parameterized via --seed (default: 42) and --deterministic (default: True).

CLI Flags

# Default (seed=42, deterministic GPU ops)
kreview run ... --seed 42 --deterministic

# Custom seed, faster GPU (non-deterministic cuDNN)
kreview eval gpu ... --seed 123 --no-deterministic

# Nextflow
nextflow run main.nf --seed 42 --deterministic true

What Gets Seeded

Library	Mechanism	Deterministic?
Python `random`	`random.seed(seed)`	✅ Always
PyTorch	`torch.manual_seed(seed)` + `cuda.manual_seed_all(seed)`	✅ When `--deterministic`
cuDNN	`cudnn.deterministic=True`, `benchmark=False`	✅ When `--deterministic`
scikit-learn	`random_state=` on all models + `StratifiedKFold`	✅ Always
XGBoost	`random_state=` on `XGBClassifier`	✅ Always
TabPFN	`TabPFNClassifier(random_state=)`	✅ Always
TabICL	`TabICLClassifier(random_state=)`	✅ Always
shapiq	`TabularExplainer(random_state=)`	✅ Always
shap	`PermutationExplainer(seed=)`	✅ Always
BorutaShap	`selector.fit(random_state=)`	✅ Always
Mutual Information	`mutual_info_classif(random_state=)`	✅ Always

NumPy Policy

NumPy randomness is never set globally via np.random.seed(). Instead, all functions accept a random_state parameter and create local RandomState objects. This prevents side effects on third-party libraries.

Deterministic Mode Trade-off

--deterministic (default True) enables torch.backends.cudnn.deterministic=True and disables autotuning (benchmark=False). This ensures bit-exact GPU results across runs but incurs a ~10–20% speed penalty. Use --no-deterministic for faster training when exact reproducibility is not required.