Skip to content

Evaluation CLI API Reference

The kreview.cli_eval module implements the kreview eval command family — the CPU and GPU model stages, the nested-CV ablation stages, and the multimodal stacking stages that the Nextflow DAG scatters over. Each subcommand is a pipeline stage boundary, not a convenience wrapper: there is one implementation per behaviour and the DAG calls it.

For conceptual explanations, see:

kreview.cli_eval

Evaluation CLI subcommands: kreview eval cpu|gpu|multimodal {run,prep,single,ablation,merge}

Docs: https://msk-access.github.io/kreview/cli_eval.html.md

__all__ = ['log', 'eval_app', 'multimodal_app', 'ablate_app', 'eval_ablate_cpu', 'eval_ablate_gpu', 'eval_ablate_merge', 'eval_cpu', 'eval_gpu', 'eval_multimodal', 'eval_multimodal_prep', 'eval_multimodal_single', 'eval_multimodal_ablation', 'eval_multimodal_merge'] module-attribute

Evaluation CLI subcommands: kreview eval cpu|gpu|multimodal {run,prep,...}.

Registered in the main CLI via::

from kreview.cli_eval import eval_app
app.add_typer(eval_app, name="eval")

This keeps evaluation-specific code out of the 1,500-line cli.py.

eval_ablate_cpu(matrix=typer.Option(..., help='Path to selected matrix parquet'), output=typer.Option('.', help='Output directory for ablation JSON'), n_outer_folds=typer.Option(5, help='Outer CV folds (must match eval)'), n_inner_folds=typer.Option(3, help='Inner CV folds for subset selection'), seed=typer.Option(42, help='Random seed'))

CPU feature group ablation (LR, RF, XGB) with nested CV.

Evaluates feature subsets across outer folds using inner CV to find the best feature group combination per model. Output: {evaluator}_ablation_cpu.json

eval_ablate_gpu(matrix=typer.Option(..., help='Path to selected matrix parquet'), output=typer.Option('.', help='Output directory for ablation JSON'), models=typer.Option('tabpfn,tabicl', help='Comma-separated GPU model names'), n_outer_folds=typer.Option(5, help='Outer CV folds (must match CPU ablation)'), n_inner_folds=typer.Option(3, help='Inner CV folds for subset selection'), seed=typer.Option(42, help='Random seed'), device=typer.Option('cuda', help='PyTorch device'), max_gpu_features=typer.Option(150, help='Feature cap for GPU models'), eval_stats=typer.Option(None, '--eval-stats', help='Path to eval_stats parquet for score-based feature capping'))

GPU feature group ablation (TabPFN, TabICL) with nested CV.

Uses ZERO-SHOT inference only in the inner CV loop (no fine-tuning). Output: {evaluator}_ablation_gpu.json

eval_ablate_merge(cpu_json=typer.Option(..., help='Path to *_ablation_cpu.json'), gpu_json=typer.Option(None, help='Path to *_ablation_gpu.json (optional)'), output=typer.Option('.', help='Output directory for best_subset.json'))

Merge CPU + GPU ablation results into best_subset.json.

Produces a unified per-model per-fold feature list consumed by kreview eval cpu --best-subset and kreview eval gpu --best-subset.

eval_cpu(matrices_dir=typer.Option(..., '--matrices-dir', help='Directory containing *_matrix.parquet files from kreview extract'), output=typer.Option('output/', help='Output directory'), models=typer.Option('lr,rf,xgb', '--models', help='Comma-separated CPU models: lr,rf,xgb'), cv_folds=typer.Option(5, '--cv-folds', help='Cross-validation folds'), resume=typer.Option(False, '--resume', help='Skip evaluators with existing results'), seed=typer.Option(42, '--seed', help='Random seed for reproducibility.'), deterministic=typer.Option(True, '--deterministic/--no-deterministic', help='Enable PyTorch deterministic mode (slower but reproducible).'), best_subset=typer.Option(None, '--best-subset', help='Path to *_best_subset.json from ablation merge (enables nested CV)'))

Per-evaluator evaluation using LR, RF, XGBoost (CPU).

Iterates over all _matrix.parquet files in --matrices-dir, trains the specified models, and writes _model_results.json to --output.

eval_gpu(matrices_dir=typer.Option(..., '--matrices-dir', help='Directory containing *_matrix.parquet files from kreview extract'), output=typer.Option('output/', help='Output directory'), models=typer.Option('tabpfn,tabicl', '--models', help='Comma-separated GPU models: tabpfn,tabpfn_ft,tabicl,tabicl_ft'), cv_folds=typer.Option(5, '--cv-folds', help='Cross-validation folds'), finetune_epochs=typer.Option(50, '--finetune-epochs', help='Fine-tuning epochs for _ft variants'), finetune_lr=typer.Option(1e-05, '--finetune-lr', help='Fine-tuning learning rate for _ft variants'), device=typer.Option('cuda', '--device', help='PyTorch device: cuda, cpu'), compute_shap=typer.Option(False, '--shap', help='Compute SHAP values'), shap_samples=typer.Option(500, '--shap-samples', help='Max SHAP samples'), resume=typer.Option(False, '--resume', help='Skip evaluators with existing results'), skip_gpu_joblib=typer.Option(False, '--skip-gpu-joblib', help='Skip saving GPU model joblib files (can be >200MB each)'), seed=typer.Option(42, '--seed', help='Random seed for reproducibility.'), deterministic=typer.Option(True, '--deterministic/--no-deterministic', help='Enable PyTorch deterministic mode (slower but reproducible).'), max_gpu_features=typer.Option(150, '--max-gpu-features', help='Maximum features for GPU models. If feature count exceeds this, the top N are selected by mutual information from eval_stats. Set to 0 to disable capping.'), best_subset=typer.Option(None, '--best-subset', help='Path to *_best_subset.json from ablation merge (enables nested CV)'))

Per-evaluator evaluation using TabPFN, TabICL (GPU).

Each model name encodes its variant: 'tabpfn' = zero-shot, 'tabpfn_ft' = fine-tuned, 'tabicl' = zero-shot, 'tabicl_ft' = fine-tuned. Iterates over all *_matrix.parquet files and writes results JSONs.

eval_multimodal(results_dir=typer.Option(..., '--results-dir', help='Directory with *_model_results.json files from eval cpu/gpu'), super_matrix=typer.Option(None, '--super-matrix', help='Optional path to super_matrix.parquet for raw-feature strategy'), output=typer.Option('output/', help='Output directory'), models=typer.Option('rf,xgb', '--models', help='Comma-separated CPU models for multimodal evaluation (lr,rf,xgb)'), gpu_models=typer.Option('', '--gpu-models', help='Comma-separated GPU models: tabpfn_ft,tabicl_ft. Empty = CPU only.'), top_percentile=typer.Option(10.0, '--top-percentile', help='Top N%% features for MI selection (matches per-evaluator pipeline)'), multimodal_selection=typer.Option('grootcv', '--multimodal-selection', help='Multimodal feature selection: grootcv (default since #96 — most stable all-relevant selector, measured), mi (fast exploration), leshy, or boruta_shap (DEPRECATED — needs kreview[legacy-boruta], conflicts with the arfs extra). grootcv/leshy require: pip install kreview[arfs]'), selection_cutoff=typer.Option(3.0, '--selection-cutoff', help='GrootCV shadow-importance divisor; higher admits more features (#96 measured default)'), selection_n_iter=typer.Option(10, '--selection-n-iter', help='GrootCV shadow-test iterations (#96: 0.96 agreement with 50 at 6x speed)'), selection_n_jobs=typer.Option(0, '--selection-n-jobs', help='LightGBM threads for GrootCV (0 = library default; set to the scheduler allocation on HPC)'), cv_folds=typer.Option(5, '--cv-folds', help='Cross-validation folds'), device=typer.Option('cuda', '--device', help='PyTorch device: cuda, cpu'), finetune_epochs=typer.Option(50, '--finetune-epochs', help='GPU fine-tuning epochs for _ft variants'), finetune_lr=typer.Option(1e-05, '--finetune-lr', help='GPU fine-tuning learning rate'), seed=typer.Option(42, '--seed', help='Random seed for reproducibility.'), deterministic=typer.Option(True, '--deterministic/--no-deterministic', help='Enable PyTorch deterministic mode (slower but reproducible).'))

Cross-evaluator multimodal evaluation with stacking and ablation.

Reads per-evaluator model_results.json files for OOF probabilities and combines them into a stacking matrix. Three strategies are run:

  1. Stacking: Meta-learner on OOF probabilities across evaluators
  2. Raw features (if --super-matrix provided): MI or Boruta-SHAP selected features
  3. Ablation: Leave-one-evaluator-out importance analysis

eval_multimodal_prep(results_dir=typer.Option(..., '--results-dir', help='Directory with *_model_results.json files from eval cpu/gpu'), super_matrix=typer.Option(None, '--super-matrix', help='Optional path to super_matrix.parquet for raw-feature strategy'), multimodal_selection=typer.Option('grootcv', '--multimodal-selection', help='Feature selection for raw features: grootcv (default since #96 — most stable all-relevant selector, measured), mi (fast exploration), leshy, or boruta_shap (DEPRECATED — needs kreview[legacy-boruta], conflicts with the arfs extra). grootcv/leshy require: pip install kreview[arfs]'), top_percentile=typer.Option(10.0, '--top-percentile', help='Top N%% features for MI selection'), selection_cutoff=typer.Option(3.0, '--selection-cutoff', help='GrootCV shadow-importance divisor; higher admits more features (#96 measured default)'), selection_n_iter=typer.Option(10, '--selection-n-iter', help='GrootCV shadow-test iterations (#96: 0.96 agreement with 50 at 6x speed)'), selection_n_jobs=typer.Option(0, '--selection-n-jobs', help='LightGBM threads for GrootCV (0 = library default; set to the scheduler allocation on HPC)'), seed=typer.Option(42, '--seed', help='Random seed'), output=typer.Option('output/', help='Output directory'))

Stage 1: Build stacking + raw-feature matrices from evaluator results.

Produces stacking_matrix.parquet, optionally raw_features_matrix.parquet, and prep_metadata.json.

eval_multimodal_single(stacking_matrix=typer.Option(..., '--stacking-matrix', help='Path to stacking_matrix.parquet from multimodal prep'), model=typer.Option('rf', '--model', help='Model to train: rf, xgb, lr, tabpfn_ft, tabicl_ft'), raw_features_matrix=typer.Option(None, '--raw-features-matrix', help='Optional path to raw_features_matrix.parquet from prep'), cv_folds=typer.Option(5, '--cv-folds', help='Cross-validation folds'), device=typer.Option('cuda', '--device', help='PyTorch device'), finetune_epochs=typer.Option(50, '--finetune-epochs', help='GPU fine-tuning epochs'), finetune_lr=typer.Option(1e-05, '--finetune-lr', help='GPU fine-tuning learning rate'), best_single_auc=typer.Option(0.0, '--best-single-auc', help='Best single-evaluator AUC for delta computation'), seed=typer.Option(42, '--seed', help='Random seed'), deterministic=typer.Option(True, '--deterministic/--no-deterministic', help='Enable PyTorch deterministic mode.'), output=typer.Option('output/', help='Output directory'))

Stage 2: Train one model on the stacking (+ optional raw) matrix.

Produces stacking_{model}_results.json.

eval_multimodal_ablation(stacking_matrix=typer.Option(..., '--stacking-matrix', help='Path to stacking_matrix.parquet from multimodal prep'), stacking_results_dir=typer.Option(..., '--stacking-results-dir', help='Directory with stacking_*_results.json from multimodal single'), cv_folds=typer.Option(5, '--cv-folds', help='Cross-validation folds'), seed=typer.Option(42, '--seed', help='Random seed'), output=typer.Option('output/', help='Output directory'))

Stage 3: Leave-one-evaluator-out ablation analysis.

Uses the best stacking model to measure each evaluator's marginal contribution. Produces ablation_results.json.

eval_multimodal_merge(stacking_results_dir=typer.Option(..., '--stacking-results-dir', help='Directory with stacking_*_results.json from multimodal single'), prep_metadata=typer.Option(..., '--prep-metadata', help='Path to prep_metadata.json from multimodal prep'), ablation_results=typer.Option(None, '--ablation-results', help='Optional path to ablation_results.json from multimodal ablation'), output=typer.Option('output/', help='Output directory'))

Stage 4: Merge partial results into unified multimodal_results.json.

Combines prep metadata, per-model stacking results, and optional ablation into a single output matching the monolithic schema.