Evaluation Engine API Reference

The kreview.eval_engine module contains the statistical testing functions, ML model training, visualization generators, and clinical utility computations.

For conceptual explanations, see:

`kreview.eval_engine`

`FeatureEvaluator`

Base class for all feature evaluators. Defines the extraction contract that transforms raw DuckDB queries into 1D arrays.

Subclasses must implement extract() for per-sample Python extraction. Optionally, override extract_sql() to return a DuckDB SQL query that performs full-cohort extraction in a single pass (no Python loop).

Class Attributes

name: Evaluator identifier (used in file names, logging). source_file: Parquet suffix (e.g. '.WPS.parquet'). tier: Feature importance tier (1=core, 2=supplementary). category: Feature category for grouping (e.g. 'epigenetics'). extract_columns: If set, DuckDB will SELECT only these columns instead of *. Reduces memory for wide parquet files. Must include 'sample_id' (or it will be auto-appended). max_chunk_rows: Target rows per DuckDB chunk (default 15M). Lower values reduce peak memory for heavy features.

`supports_sql` `property`

True if this evaluator provides a SQL pushdown query.

`extract(df)`

Transform the loaded raw dataframe into meaningful scalar metrics. Called per sample-group or per sample.

`extract_sql()`

Return a DuckDB SQL query for full-cohort extraction, or None.

If implemented, the query should: - Accept a read_parquet(?, ...) placeholder for file paths - GROUP BY sample_id to produce one row per sample - SELECT all extracted feature columns with their final names

Returning None (the default) means this evaluator does not support SQL pushdown and will use the chunked Python path.

`GPUModelCVAdapter`

Bases: BaseEstimator, ClassifierMixin

sklearn-compatible wrapper for GPU foundation models.

Sidesteps two TabPFN sklearn-compatibility issues:

classes_ is a read-only @property on FinetunedTabPFNClassifier that raises AttributeError when unfitted. cross_val_predict tries to read classes_ on the fitted clone, but sklearn.clone() copies the unfitted object first and the property breaks. This adapter exposes classes_ as a plain attribute set during fit().
sklearn.base.clone() is unreliable for TabPFN (PriorLabs/TabPFN#327). By accepting a model_factory callable, each fit() creates a fresh instance without relying on clone.

Attributes:

Name	Type	Description
`model_factory`		Zero-argument callable returning a fresh model instance.
`name`		Human-readable model name for logging.
`model_`		The fitted model instance (set after `fit()`).
`classes_`		Unique sorted class labels (set after `fit()`).

`getstate()`

Exclude model_factory (a lambda closure) from pickle serialization.

The factory is only needed for training (creating fresh model instances during CV). Deserialized adapters are inference-only — predict_proba() uses the already-fitted model_ attribute.

`setstate(state)`

Restore adapter in inference-only mode (no model_factory).

Calling fit() on a deserialized adapter will raise TypeError because model_factory is None. This is intentional — loaded models are for prediction/SHAP only.

`fit(X, y)`

Create a fresh model via factory and fit it.

Parameters:

Name	Type	Description	Default
`X`	`ndarray`	Feature matrix (n_samples, n_features).	required
`y`	`ndarray`	Binary label array (n_samples,).	required

Returns:

Type	Description
`'GPUModelCVAdapter'`	self

Raises:

Type	Description
`TypeError`	If called on a deserialized adapter (`model_factory` is None). Loaded models are inference-only.

`predict_proba(X)`

Predict class probabilities.

Raises:

Type	Description
`NotFittedError`	If `fit()` has not been called.

`predict(X)`

Predict class labels.

Raises:

Type	Description
`NotFittedError`	If `fit()` has not been called.

`parse_array(s)`

Parse a numeric array value into a list of floats.

Handles multiple input formats: - numpy arrays (native parquet list<float>, DuckDB FLOAT[]) - Python lists/tuples - String-encoded arrays ('[1.0 2.0 3.0]') from legacy parquet

Returns empty list on any parse failure (no silent corruption).

`univariate_auc(feature_col, y, n_folds=5, random_state=42)`

Compute cross-validated AUC for a single feature using univariate LR.

Parameters:

Name	Type	Description	Default
`feature_col`		pandas Series or array-like of a single feature.	required
`y`		binary label array (0/1).	required
`n_folds`	`int`	number of CV folds.	`5`
`random_state`	`int`	random seed.	`42`

Returns:

Type	Description
`float`	Cross-validated AUC (float). Returns 0.5 if the feature is constant,
`float`	there are too few samples per class, or CV fails.

`mutual_info_score(feature_col, y, random_state=42)`

Compute mutual information between a single feature and binary target.

Uses sklearn's mutual_info_classif with k=3 nearest neighbors to estimate the non-linear dependency between a feature and the label. Unlike AUC, mutual information captures arbitrary (non-monotonic) relationships.

Parameters:

Name	Type	Description	Default
`feature_col`		pandas Series or array-like of a single feature.	required
`y`		binary label array (0/1).	required
`random_state`	`int`	random seed for reproducibility.	`42`

Returns:

Type	Description
`float`	Mutual information score (float, >= 0). Higher means more informative.
`float`	Returns 0.0 if the feature is constant or computation fails.

`set_theme(cvd_safe=False)`

Dynamically updates the global label and model colors based on CVD preference.

`evaluate_feature(feature_values, labels, total_fragments=None, max_vaf=None)`

Run all statistical tests for a single feature in one stratum. Outputs metrics directly to scoring dict.

`plot_violin(df, feature_col, label_col='label', title='')`

4-group violin with overlaid box plot and individual points for small groups.

`plot_density(df, feature_col, label_col='label', title='')`

Overlaid density curves per group — shows distribution shape differences.

`plot_feature_vs_vaf(df, feature_col, vaf_col='max_vaf', label_col='label', title='')`

Continuous relationship between feature and tumor burden (VAF proxy).

`plot_roc_curves(y_true_dict, y_score_dict, title='')`

Overlay ROC curves for multiple comparisons.

`plot_feature_importance(importances, title='')`

Bar plot of RF feature importances.

`decision_curve_analysis(y_true, y_prob, thresholds=None)`

Compute Decision Curve Analysis (DCA) net benefit data.

For each threshold, calculates the net benefit of using the model vs treating all or treating none. This helps clinicians choose an operating threshold that balances false positives against missed detections.

Parameters:

Name	Type	Description	Default
`y_true`	`ndarray`	Binary ground truth labels (0/1).	required
`y_prob`	`ndarray`	Predicted probabilities for positive class.	required
`thresholds`	`ndarray \| None`	Array of decision thresholds to evaluate. Defaults to `np.linspace(0.01, 0.99, 99)`.	`None`

Returns:

Type	Description
`dict`	Dictionary with keys `thresholds`, `net_benefit_model`, and
`dict`	`net_benefit_treat_all`.

`identify_feature_groups(columns, meta_cols=None)`

Map selected feature column names → feature groups by statistic type.

Groups are identified by suffix pattern matching against known kreview evaluator output patterns (see _SUFFIX_TO_GROUP). Columns that don't match any known suffix go into the "other" group.

Parameters

columns : list[str] Feature column names from a selected matrix. meta_cols : set[str] | None Columns to exclude (e.g. LABEL_META_COLS). If None, uses kreview.core.LABEL_META_COLS.

Returns

dict[str, list[str]] Mapping of group name → list of column names belonging to that group.

`generate_subsets(groups)`

Generate feature subsets for ablation from feature groups.

Returns a dict of named subsets:

"ALL": all features from all groups
"{group}": solo — only features from that group (one per group)
"no_{group}": leave-one-out — everything except that group

If only 1 group exists, returns only {"ALL": all_features} since ablation is meaningless with a single group.

Parameters

groups : dict[str, list[str]] Output of :func:identify_feature_groups.

Returns

dict[str, list[str]] Named subsets mapping subset name → list of feature column names.

`ablate_feature_groups(matrix_path, models=('lr', 'rf', 'xgb'), n_outer_folds=5, n_inner_folds=3, random_state=42, output_path=None, device='cpu', max_gpu_features=None, eval_stats_path=None)`

Run feature group ablation with per-fold nested cross-validation.

For each outer fold, evaluates all feature subsets using inner CV to find the best subset per model. Results include per-fold winners and the fold assignment array for downstream reproducibility.

Data flow:

Load matrix, filter to split == "train" (prevent holdout leakage).
Build binary target via :func:kreview.selection.build_binary_target.
Identify feature groups via :func:identify_feature_groups.
Generate subsets via :func:generate_subsets.
For each outer fold: a. Split into outer-train / outer-val using StratifiedKFold. b. For each model × subset:
- Run :func:_inner_cv_sensitivity on outer-train.
- Record sensitivity@100%spec_healthy score. c. Identify winning subset per model for this fold.
Aggregate results across folds.

Parameters

matrix_path : Path | str Path to *_matrix.parquet file. models : tuple[str, ...] Model names to evaluate. CPU: "lr", "rf", "xgb". GPU: "tabpfn", "tabicl" (zero-shot only). n_outer_folds : int Number of outer CV folds (must match downstream EVAL step). n_inner_folds : int Number of inner CV folds for subset selection. random_state : int Random seed for reproducibility. output_path : Path | str | None Directory to write ablation JSON results. device : str PyTorch device for GPU models (default "cpu"). max_gpu_features : int | None Feature cap for GPU models (TabPFN limit). eval_stats_path : Path | str | None Path to *_eval_stats.parquet for score-based feature capping.

Returns

dict Ablation results containing per-fold per-model scores, winners, fold assignments, and feature group metadata.

`merge_ablation(cpu_json_path, gpu_json_path=None, matrix_path=None, output_path=None)`

Merge CPU and GPU ablation results into a unified best_subset.json.

For each model across all folds, determines the per-fold winning feature subset. The output JSON is consumed by downstream kreview eval cpu / kreview eval gpu via --best-subset.

Handles GPU error JSONs gracefully: if the GPU JSON contains an "error" key (from a failed GPU ablation), the merge proceeds with CPU-only results.

Parameters

cpu_json_path : Path | str Path to *_ablation_cpu.json from :func:ablate_feature_groups. gpu_json_path : Path | str | None Path to *_ablation_gpu.json. None or a path to a file containing "error" triggers CPU-only mode. matrix_path : Path | str | None Path to the original matrix parquet (for fold assignment verification). output_path : Path | str | None Directory to write {evaluator}_best_subset.json.

Returns

dict Merged result with per_model_per_fold_features and fold_assignment.

`evaluate_model(model, X, y, cv, name, feature_names=None, refit=True, compute_shap=False, shap_samples=500, random_state=42, sample_labels=None)`

Evaluate any sklearn-compatible model via stratified cross-validation.

Shared primitive used by cpu_models(), gpu_models(), and multimodal_eval(). The model must implement fit() and predict_proba().

Parameters:

Name	Type	Description	Default
`model`		sklearn-compatible estimator (Pipeline, RF, XGB, TabPFN, etc.).	required
`X`	`ndarray`	Feature matrix, shape (n_samples, n_features).	required
`y`	`ndarray`	Binary labels (0/1), shape (n_samples,).	required
`cv`	`StratifiedKFold`	Pre-configured StratifiedKFold splitter.	required
`name`	`str`	Prefix for all result keys (e.g. "lr", "rf", "tabpfn").	required
`feature_names`	`list[str] \| None`	Optional feature names for importance extraction.	`None`
`refit`	`bool`	If True, refit model on full data after CV.	`True`
`compute_shap`	`bool`	If True, compute SHAP values (requires refit=True).	`False`
`shap_samples`	`int`	Max samples for SHAP computation.	`500`
`random_state`	`int`	Random seed for bootstrap CI.	`42`
`sample_labels`	`ndarray \| None`	Original 4-tier labels (e.g. "Healthy Normal", "Possible ctDNA+") for computing sensitivity@100%spec on healthy normals only. Shape (n_samples,). Optional.	`None`

Returns:

Type	Description
`dict`	(result_dict, fitted_model_or_None). result_dict keys are prefixed
`object \| None`	with `name` (e.g. `auc_rf`, `rf_oof_probs`, `rf_fold_aucs`,
`tuple[dict, object \| None]`	`rf_sensitivity_at_100spec`).

`evaluate_holdout(fitted_model, X_test, y_test, name, sample_labels=None)`

Evaluate a fitted model on the holdout test set.

This is called AFTER evaluate_model() has refitted the model on the full training set. The holdout set was never seen during CV or feature selection, providing an unbiased estimate of generalization.

Parameters:

Name	Type	Description	Default
`fitted_model`		Model already fitted on training data.	required
`X_test`	`ndarray`	Holdout feature matrix.	required
`y_test`	`ndarray`	Holdout binary labels.	required
`name`	`str`	Model name prefix for result keys.	required
`sample_labels`	`ndarray \| None`	Original label strings for the holdout set (for healthy-normal specificity).	`None`

Returns:

Type	Description
`dict`	Dict with holdout metrics, all prefixed with `holdout_{name}_`.

`cpu_models(X, y, feature_names=None, cancer_types=None, assays=None, n_folds=5, random_state=42, compute_shap=False, shap_samples=500, sample_labels=None, per_fold_features=None, fold_assignment=None)`

Train LR, RF, and XGB on a feature matrix with stratified CV.

Delegates per-model evaluation to evaluate_model(), then adds cross-model diagnostics (AUC deltas, top features, threshold sweep, DCA, feature stability, subgroup analysis).

When per_fold_features and fold_assignment are provided (from nested CV ablation), bypasses evaluate_model() and uses manual fold iteration with per-fold feature filtering.

Parameters:

Name	Type	Description	Default
`sample_labels`	`ndarray \| None`	Original 4-tier labels for computing healthy-normal specificity. Passed to `evaluate_model()`.	`None`
`per_fold_features`	`dict \| None`	Dict from best_subset.json with per-model per-fold feature lists. Structure: `{model_name: {"folds": {"0": {"features": [...]}, ...}}}`. When `None`, uses standard path (all features, all folds).	`None`
`fold_assignment`	`list \| ndarray \| None`	Per-sample fold index array from ablation. Must align with the samples in X/y.	`None`

Returns (results_dict, lr_pipeline, rf_model, xgb_model).

Audit fixes preserved

C-01: LR uses Pipeline(scaler+lr) to prevent data leakage
C-02: Subgroup metrics use out-of-fold predictions (unbiased)
H-01: LR has class_weight="balanced", XGB has scale_pos_weight
M-02: Bootstrap 95% CI on AUC values

`gpu_models(X, y, feature_names=None, cancer_types=None, assays=None, n_folds=5, random_state=42, models=('tabpfn',), device='cuda', finetune_epochs=50, finetune_lr=1e-05, compute_shap=False, shap_samples=500, sample_labels=None, max_gpu_features=None, eval_stats=None, per_fold_features=None, fold_assignment=None)`

Train GPU foundation models on a feature matrix.

Same output schema as cpu_models() for each model, using the shared evaluate_model() primitive. Each model name encodes its variant: "tabpfn" = zero-shot, "tabpfn_ft" = fine-tuned, etc.

Parameters:

Name	Type	Description	Default
`X`	`ndarray`	Feature matrix, shape (n_samples, n_features).	required
`y`	`ndarray`	Binary labels (0/1).	required
`feature_names`	`list[str] \| None`	Optional feature names.	`None`
`cancer_types`	`ndarray \| None`	Optional cancer type array for subgroup analysis.	`None`
`assays`	`ndarray \| None`	Optional assay array for subgroup analysis.	`None`
`n_folds`	`int`	Number of CV folds.	`5`
`random_state`	`int`	Random seed.	`42`
`models`	`tuple[str, ...]`	Tuple of GPU model names. Valid values: `"tabpfn"` (zero-shot), `"tabpfn_ft"` (fine-tuned), `"tabicl"` (zero-shot), `"tabicl_ft"` (fine-tuned).	`('tabpfn',)`
`device`	`str`	PyTorch device (`"cuda"`, `"cpu"`).	`'cuda'`
`finetune_epochs`	`int`	Epochs for fine-tuned variants (default 50).	`50`
`finetune_lr`	`float`	Learning rate for fine-tuned variants.	`1e-05`
`compute_shap`	`bool`	If True, compute SHAP values.	`False`
`shap_samples`	`int`	Max samples for SHAP computation.	`500`
`sample_labels`	`ndarray \| None`	Original 4-tier labels for computing healthy-normal specificity. Passed to `evaluate_model()`.	`None`
`max_gpu_features`	`int \| None`	If set and n_features > max_gpu_features, cap to the top N features using eval_stats scores. Default None (no cap). CLI default is 150.	`None`
`eval_stats`	`DataFrame \| None`	DataFrame from `score_features()` containing feature scoring columns. Used for intelligent feature capping when max_gpu_features is set.	`None`
`per_fold_features`	`dict \| None`	Dict from best_subset.json with per-model per-fold feature lists (from ablation). When provided, uses manual fold iteration with per-fold feature filtering.	`None`
`fold_assignment`	`list \| ndarray \| None`	Per-sample fold index array from ablation.	`None`

Returns:

Type	Description
`dict`	(results_dict, fitted_models_dict). fitted_models_dict maps
`dict[str, object]`	model name to fitted model object.

`load_model_results(directory, evaluator_name)`

Load and merge CPU + GPU model results for a single evaluator.

Looks for:

{evaluator_name}_model_results.json (CPU results)
{evaluator_name}_gpu_model_results.json (merged GPU results)
{evaluator_name}_*_gpu_model_results.json (scattered per-model GPU results from NF scattering, e.g. FSC_tabpfn_gpu_model_results.json)

All found GPU files are merged into the CPU dict. GPU keys take precedence for overlapping model keys; CPU metadata keys (evaluator, matrix_path, etc.) are preserved.

Used by report templates where the evaluator name is known.

Parameters:

Name	Type	Description	Default
`directory`	`Path`	Directory containing the JSON result files.	required
`evaluator_name`	`str`	Evaluator name (e.g., `"FSCOnTarget"`).	required

Returns:

Type	Description
`dict \| None`	Merged dict, or None if no JSON exists for this evaluator.

`load_all_model_results(directory)`

Load and merge all CPU + GPU model results from a directory.

Scans for *_model_results.json and *_gpu_model_results.json, groups by evaluator name, and merges GPU keys into CPU dicts.

Used by scoreboard and multimodal engine for directory scanning.

Parameters:

Name	Type	Description	Default
`directory`	`Path`	Directory containing model result JSON files.	required

Returns:

Type	Description
`dict[str, dict]`	Dict keyed by evaluator name → merged model results dict.
`dict[str, dict]`	Empty dict if no results found.

`multimodal_eval(results_dir, super_matrix_path=None, *, models=('rf', 'xgb'), gpu_models=(), device='cuda', finetune_epochs=50, finetune_lr=1e-05, n_folds=5, top_percentile=10.0, random_state=42, multimodal_selection='mi')`

Cross-evaluator multimodal evaluation.

Implements three complementary strategies:

Stacking: Meta-learner trained on OOF predictions from all per-evaluator models. Each column is one evaluator's OOF probability. This measures how much combining multiple fragmentomics signals improves classification.
Raw features (optional): If super_matrix_path is provided, trains directly on the fused feature matrix with multimodal_selection-based feature selection.
Ablation: Leave-one-evaluator-out analysis on the stacking matrix, showing each evaluator's marginal contribution.

Parameters:

Name	Type	Description	Default
`results_dir`	`str \| Path`	Directory with `*_model_results.json` files.	required
`super_matrix_path`	`str \| Path \| None`	Optional path to `super_matrix.parquet`.	`None`
`models`	`tuple[str, ...]`	CPU model names to use (`rf`, `xgb`, `lr`).	`('rf', 'xgb')`
`gpu_models`	`tuple[str, ...]`	GPU model names (`tabpfn_ft`, `tabicl_ft`). Empty = CPU only.	`()`
`device`	`str`	PyTorch device string for GPU models.	`'cuda'`
`finetune_epochs`	`int`	Epochs for fine-tuned GPU variants (default 50).	`50`
`finetune_lr`	`float`	Learning rate for fine-tuned GPU variants.	`1e-05`
`n_folds`	`int`	Cross-validation folds.	`5`
`top_percentile`	`float`	Top N% features for feature selection.	`10.0`
`random_state`	`int`	Reproducibility seed.	`42`
`multimodal_selection`	`str`	Feature selection strategy ('mi', 'boruta_shap', 'leshy', or 'grootcv').	`'mi'`

Returns:

Type	Description
`dict`	A comprehensive results dict with keys for each strategy.

`multimodal_prep(results_dir, super_matrix_path=None, *, multimodal_selection='mi', top_percentile=10.0, random_state=42, output_dir='.')`

Stage 1: Build stacking + optional raw-feature matrices.

Loads per-evaluator model results, builds a stacking matrix from OOF probabilities, optionally loads and selects from the super matrix, and writes outputs as parquet files + a metadata JSON.

Parameters:

Name	Type	Description	Default
`results_dir`	`str \| Path`	Directory with `*_model_results.json` files.	required
`super_matrix_path`	`str \| Path \| None`	Optional path to `super_matrix.parquet`.	`None`
`multimodal_selection`	`str`	Feature selection strategy for raw features ('mi', 'boruta_shap', 'leshy', or 'grootcv').	`'mi'`
`top_percentile`	`float`	Top N% features to retain after MI ranking.	`10.0`
`random_state`	`int`	Random seed.	`42`
`output_dir`	`str \| Path`	Directory to write output files.	`'.'`

Returns:

Type	Description
`dict`	Metadata dict with keys: `stacking_columns`, `n_evaluators`,
`dict`	`evaluators`, `single_evaluator_aucs`, `best_single_evaluator`,
`dict`	`best_single_auc`, `stacking_shape`, `raw_shape` (optional).

Raises:

Type	Description
`FileNotFoundError`	If `results_dir` does not exist.
`ValueError`	If no valid evaluator baselines found, or stacking matrix has 0 columns.

`multimodal_single(stacking_matrix_path, model_name, *, raw_features_path=None, n_folds=5, random_state=42, device='cuda', finetune_epochs=50, finetune_lr=1e-05, best_single_auc=0.0, output_dir='.')`

Stage 2: Train a single model on the stacking (+ optional raw) matrix.

Reads the stacking matrix parquet, trains the requested model via :func:evaluate_model, and writes a per-model result JSON.

Parameters:

Name	Type	Description	Default
`stacking_matrix_path`	`str \| Path`	Path to `stacking_matrix.parquet` from prep.	required
`model_name`	`str`	Model to train (e.g. `"rf"`, `"xgb"`, `"tabpfn_ft"`).	required
`raw_features_path`	`str \| Path \| None`	Optional path to `raw_features_matrix.parquet`.	`None`
`n_folds`	`int`	Cross-validation folds.	`5`
`random_state`	`int`	Random seed.	`42`
`device`	`str`	PyTorch device for GPU models.	`'cuda'`
`finetune_epochs`	`int`	Fine-tuning epochs for `_ft` models.	`50`
`finetune_lr`	`float`	Fine-tuning learning rate for `_ft` models.	`1e-05`
`best_single_auc`	`float`	Best single-evaluator AUC (for delta computation).	`0.0`
`output_dir`	`str \| Path`	Directory to write output JSON.	`'.'`

Returns:

Type	Description
`dict`	Results dict with stacking and optional raw-feature metrics.

Raises:

Type	Description
`FileNotFoundError`	If stacking_matrix_path doesn't exist.
`ValueError`	If model build fails.

`multimodal_ablation(stacking_matrix_path, stacking_results_dir, *, n_folds=5, random_state=42, output_dir='.')`

Stage 3: Leave-one-evaluator-out ablation study.

Finds the best stacking model from partial result JSONs, then for each evaluator drops its columns and retrains to measure marginal contribution.

Parameters:

Name	Type	Description	Default
`stacking_matrix_path`	`str \| Path`	Path to `stacking_matrix.parquet`.	required
`stacking_results_dir`	`str \| Path`	Directory with `stacking_*_results.json` files from :func:`multimodal_single`.	required
`n_folds`	`int`	Cross-validation folds.	`5`
`random_state`	`int`	Random seed.	`42`
`output_dir`	`str \| Path`	Directory to write output JSON.	`'.'`

Returns:

Type	Description
`dict`	Ablation results dict with per-evaluator deltas.

Raises:

Type	Description
`FileNotFoundError`	If inputs don't exist.
`ValueError`	If no stacking results found.

`multimodal_merge(stacking_results_dir, prep_metadata_path, *, ablation_path=None, output_dir='.')`

Stage 4: Merge partial JSONs into unified multimodal_results.json.

Combines the prep metadata, per-model stacking results, and ablation results into a single output JSON matching the schema produced by the monolithic :func:multimodal_eval.

Parameters:

Name	Type	Description	Default
`stacking_results_dir`	`str \| Path`	Directory with `stacking_*_results.json` files from :func:`multimodal_single`.	required
`prep_metadata_path`	`str \| Path`	Path to `prep_metadata.json` from prep.	required
`ablation_path`	`str \| Path \| None`	Optional path to `ablation_results.json`.	`None`
`output_dir`	`str \| Path`	Directory to write `multimodal_results.json`.	`'.'`

Returns:

Type	Description
`dict`	Unified results dict matching :func:`multimodal_eval` schema.

Raises:

Type	Description
`FileNotFoundError`	If prep_metadata_path doesn't exist.

Evaluation Engine API Reference

kreview.eval_engine

FeatureEvaluator

supports_sql property

extract(df)

extract_sql()

GPUModelCVAdapter

__getstate__()

__setstate__(state)

fit(X, y)

predict_proba(X)

predict(X)

parse_array(s)

univariate_auc(feature_col, y, n_folds=5, random_state=42)

mutual_info_score(feature_col, y, random_state=42)

set_theme(cvd_safe=False)

evaluate_feature(feature_values, labels, total_fragments=None, max_vaf=None)

plot_violin(df, feature_col, label_col='label', title='')

plot_density(df, feature_col, label_col='label', title='')

plot_feature_vs_vaf(df, feature_col, vaf_col='max_vaf', label_col='label', title='')

plot_roc_curves(y_true_dict, y_score_dict, title='')

plot_feature_importance(importances, title='')

decision_curve_analysis(y_true, y_prob, thresholds=None)

identify_feature_groups(columns, meta_cols=None)

Parameters

Returns

generate_subsets(groups)

Parameters

Returns

ablate_feature_groups(matrix_path, models=('lr', 'rf', 'xgb'), n_outer_folds=5, n_inner_folds=3, random_state=42, output_path=None, device='cpu', max_gpu_features=None, eval_stats_path=None)

Parameters

Returns

merge_ablation(cpu_json_path, gpu_json_path=None, matrix_path=None, output_path=None)

Parameters

Returns

evaluate_model(model, X, y, cv, name, feature_names=None, refit=True, compute_shap=False, shap_samples=500, random_state=42, sample_labels=None)

evaluate_holdout(fitted_model, X_test, y_test, name, sample_labels=None)

cpu_models(X, y, feature_names=None, cancer_types=None, assays=None, n_folds=5, random_state=42, compute_shap=False, shap_samples=500, sample_labels=None, per_fold_features=None, fold_assignment=None)

load_model_results(directory, evaluator_name)

load_all_model_results(directory)

multimodal_eval(results_dir, super_matrix_path=None, *, models=('rf', 'xgb'), gpu_models=(), device='cuda', finetune_epochs=50, finetune_lr=1e-05, n_folds=5, top_percentile=10.0, random_state=42, multimodal_selection='mi')

multimodal_prep(results_dir, super_matrix_path=None, *, multimodal_selection='mi', top_percentile=10.0, random_state=42, output_dir='.')

multimodal_single(stacking_matrix_path, model_name, *, raw_features_path=None, n_folds=5, random_state=42, device='cuda', finetune_epochs=50, finetune_lr=1e-05, best_single_auc=0.0, output_dir='.')

multimodal_ablation(stacking_matrix_path, stacking_results_dir, *, n_folds=5, random_state=42, output_dir='.')

multimodal_merge(stacking_results_dir, prep_metadata_path, *, ablation_path=None, output_dir='.')

`kreview.eval_engine`

`FeatureEvaluator`

`supports_sql` `property`

`extract(df)`

`extract_sql()`

`GPUModelCVAdapter`

`getstate()`

`setstate(state)`

`fit(X, y)`

`predict_proba(X)`

`predict(X)`

`parse_array(s)`

`univariate_auc(feature_col, y, n_folds=5, random_state=42)`

`mutual_info_score(feature_col, y, random_state=42)`

`set_theme(cvd_safe=False)`

`evaluate_feature(feature_values, labels, total_fragments=None, max_vaf=None)`

`plot_violin(df, feature_col, label_col='label', title='')`

`plot_density(df, feature_col, label_col='label', title='')`

`plot_feature_vs_vaf(df, feature_col, vaf_col='max_vaf', label_col='label', title='')`

`plot_roc_curves(y_true_dict, y_score_dict, title='')`

`plot_feature_importance(importances, title='')`

`decision_curve_analysis(y_true, y_prob, thresholds=None)`

`identify_feature_groups(columns, meta_cols=None)`

`generate_subsets(groups)`

`ablate_feature_groups(matrix_path, models=('lr', 'rf', 'xgb'), n_outer_folds=5, n_inner_folds=3, random_state=42, output_path=None, device='cpu', max_gpu_features=None, eval_stats_path=None)`

`merge_ablation(cpu_json_path, gpu_json_path=None, matrix_path=None, output_path=None)`

`evaluate_model(model, X, y, cv, name, feature_names=None, refit=True, compute_shap=False, shap_samples=500, random_state=42, sample_labels=None)`

`evaluate_holdout(fitted_model, X_test, y_test, name, sample_labels=None)`

`cpu_models(X, y, feature_names=None, cancer_types=None, assays=None, n_folds=5, random_state=42, compute_shap=False, shap_samples=500, sample_labels=None, per_fold_features=None, fold_assignment=None)`

`load_model_results(directory, evaluator_name)`

`load_all_model_results(directory)`

`multimodal_eval(results_dir, super_matrix_path=None, *, models=('rf', 'xgb'), gpu_models=(), device='cuda', finetune_epochs=50, finetune_lr=1e-05, n_folds=5, top_percentile=10.0, random_state=42, multimodal_selection='mi')`

`multimodal_prep(results_dir, super_matrix_path=None, *, multimodal_selection='mi', top_percentile=10.0, random_state=42, output_dir='.')`

`multimodal_single(stacking_matrix_path, model_name, *, raw_features_path=None, n_folds=5, random_state=42, device='cuda', finetune_epochs=50, finetune_lr=1e-05, best_single_auc=0.0, output_dir='.')`

`multimodal_ablation(stacking_matrix_path, stacking_results_dir, *, n_folds=5, random_state=42, output_dir='.')`

`multimodal_merge(stacking_results_dir, prep_metadata_path, *, ablation_path=None, output_dir='.')`