fig -- NumPy Implementation¶

The primary implementation of FIG-V and FIG-C. Functions in this module accept NumPy arrays (or any array-like input) and return Python floats and NumPy arrays.

This module is the reference implementation. The PyTorch version in fig.fig_torch mirrors this API but returns tensors and supports gradient computation.

Main Functions¶

fractional_information_gain_validation¶

fractional_information_gain_validation ¶

fractional_information_gain_validation(
    y_pred_eval: ndarray | Sequence[float],
    y_eval: ndarray | Sequence[float],
    item_id_eval: ndarray | Sequence[Any],
    student_id_eval: ndarray | Sequence[Any],
    y_train: ndarray | Sequence[float],
    item_id_train: ndarray | Sequence[Any],
    eps: float = 1e-12,
    use_shrinkage: bool = True,
    alpha: float = 2.0,
    beta: float = 2.0,
    calibration: bool = False,
    calib_n_bins: int = 10,
    calib_warn_threshold: float = 0.1,
) -> FigVResult

Compute Fractional Information Gain for Validation (FIG-V).

FIG-V measures how much better a model's predictions are than per-item base rates, using cross-entropy against ground truth.

FIG-V = 1 - BCE(y_pred, y) / H(baseline[item_id])

where:

H(baseline[item_id]) = sum of binary entropies of per-item base rates (prior uncertainty), looked up by item ID
BCE(y_pred, y) = sum of binary cross-entropies of model predictions against ground truth
BCE(p, y) = -[y*log(p) + (1-y)*log(1-p)]

Parameters:

Name	Type	Description	Default
`y_pred_eval`	`array - like`	Predicted probabilities (0 to 1). Shape: (n_eval,)	required
`y_eval`	`array - like`	Ground truth on eval split. Shape: (n_eval,) Values: 1=correct, 0=incorrect.	required
`item_id_eval`	`array - like`	Item identifiers for eval observations. Shape: (n_eval,)	required
`student_id_eval`	`array - like`	Student identifiers for eval observations. Shape: (n_eval,)	required
`y_train`	`array - like`	Ground truth on train split. Shape: (n_train,) Values: 1=correct, 0=incorrect.	required
`item_id_train`	`array - like`	Item identifiers for train observations. Shape: (n_train,)	required
`eps`	`float`	Small constant for clipping probabilities to avoid log(0).	`1e-12`
`use_shrinkage`	`bool`	If True, apply empirical Bayes shrinkage to item baselines.	`True`
`alpha`	`float`	Prior pseudo-count for successes in shrinkage.	`2.0`
`beta`	`float`	Prior pseudo-count for failures in shrinkage.	`2.0`
`calibration`	`bool`	If True, attach calibration metrics (ECE) to output.	`False`
`calib_n_bins`	`int`	Number of bins for ECE computation.	`10`
`calib_warn_threshold`	`float`	Warn if ECE exceeds this value.	`0.1`

Returns:

Type	Description
`FigVResult`	Dictionary with keys: `fig_v_pooled`: Observation-weighted FIG-V (scalar) `fig_v`: Student-weighted FIG-V (scalar) `fig_v_by_student`: Per-student FIG-V values (np.ndarray) `student_ids`: Unique student IDs in sorted order (np.ndarray) `calibration`: None unless `calibration=True`, in which case a dict of ECE metrics from `check_calibration`

Notes

FIG-V = 0 when model predictions match item base rates (no information gain).
FIG-V = 1 when model predictions are perfect (zero cross-entropy).
FIG-V can be negative if model is worse than the baseline.
FIG-V > 1 usually indicates data leakage; the library warns when this happens.
Items in eval that don't appear in train use the global training mean as baseline.

Examples:

>>> y_pred = [0.8, 0.6, 0.9, 0.5, 0.7]
>>> y_eval = [1, 0, 1, 0, 1]
>>> items = [1, 1, 2, 2, 1]
>>> students = [1, 1, 2, 2, 3]
>>> y_train = [1, 1, 0, 1, 0]
>>> items_train = [1, 1, 2, 2, 2]
>>>
>>> results = fractional_information_gain_validation(
...     y_pred_eval=y_pred,
...     y_eval=y_eval,
...     item_id_eval=items,
...     student_id_eval=students,
...     y_train=y_train,
...     item_id_train=items_train,
... )
>>> print(f"FIG-V: {results['fig_v']:.3f}")

fractional_information_gain_confidence¶

fractional_information_gain_confidence ¶

fractional_information_gain_confidence(
    y_pred_eval: ndarray | Sequence[float],
    item_id_eval: ndarray | Sequence[Any],
    student_id_eval: ndarray | Sequence[Any],
    y_train: ndarray | Sequence[float],
    item_id_train: ndarray | Sequence[Any],
    eps: float = 1e-12,
    use_shrinkage: bool = True,
    alpha: float = 2.0,
    beta: float = 2.0,
) -> FigCResult

Compute Fractional Information Gain for Confidence (FIG-C).

FIG-C measures the reduction in uncertainty about student responses based on model confidence, without requiring ground truth labels.

FIG-C = 1 - H(y_pred) / H(baseline[item_id])

where:

H(baseline[item_id]) = sum of binary entropies of per-item base rates (prior uncertainty), looked up by item ID
H(y_pred) = sum of binary entropies of model predictions (remaining uncertainty)
H(p) = -[p*log(p) + (1-p)*log(1-p)]

Parameters:

Name	Type	Description	Default
`y_pred_eval`	`array - like`	Predicted probabilities (0 to 1). Shape: (n_eval,)	required
`item_id_eval`	`array - like`	Item identifiers for eval observations. Shape: (n_eval,)	required
`student_id_eval`	`array - like`	Student identifiers for eval observations. Shape: (n_eval,)	required
`y_train`	`array - like`	Ground truth on train split. Shape: (n_train,) Values: 1=correct, 0=incorrect.	required
`item_id_train`	`array - like`	Item identifiers for train observations. Shape: (n_train,)	required
`eps`	`float`	Small constant for clipping probabilities to avoid log(0).	`1e-12`
`use_shrinkage`	`bool`	If True, apply empirical Bayes shrinkage to item baselines.	`True`
`alpha`	`float`	Prior pseudo-count for successes in shrinkage.	`2.0`
`beta`	`float`	Prior pseudo-count for failures in shrinkage.	`2.0`

Returns:

Type	Description
`FigCResult`	Dictionary with keys: `fig_c_pooled`: Observation-weighted FIG-C (scalar) `fig_c`: Student-weighted FIG-C (scalar) `fig_c_by_student`: Per-student FIG-C values (np.ndarray) `student_ids`: Unique student IDs in sorted order (np.ndarray)

Notes

FIG-C = 0 when model predictions match item base rates (no information gain).
FIG-C = 1 when model is perfectly confident (entropy = 0).
FIG-C can be negative if model adds uncertainty beyond the baseline.
For well-calibrated models, FIG-C approximates FIG-V in expectation.
Items in eval that don't appear in train use the global training mean as baseline.

Examples:

>>> y_pred = [0.95, 0.05, 0.9, 0.1, 0.8]
>>> items = [1, 1, 2, 2, 1]
>>> students = [1, 1, 2, 2, 3]
>>> y_train = [1, 1, 0, 1, 0]
>>> items_train = [1, 1, 2, 2, 2]
>>>
>>> results = fractional_information_gain_confidence(
...     y_pred_eval=y_pred,
...     item_id_eval=items,
...     student_id_eval=students,
...     y_train=y_train,
...     item_id_train=items_train,
... )
>>> print(f"FIG-C: {results['fig_c']:.3f}")

Result Types¶

FigVResult¶

FigVResult ¶

Bases: TypedDict

Return type of fractional_information_gain_validation.

Attributes:

Name	Type	Description
`fig_v_pooled`	`float`	Observation-weighted FIG-V.
`fig_v`	`float`	Student-weighted FIG-V (macro average).
`fig_v_by_student`	`ndarray`	Per-student FIG-V values, aligned with `student_ids`.
`student_ids`	`ndarray`	Unique student IDs in sorted order.
`calibration`	`CalibrationResult or None`	Calibration metrics. None unless `calibration=True` was passed.

FigCResult¶

FigCResult ¶

Bases: TypedDict

Return type of fractional_information_gain_confidence.

Attributes:

Name	Type	Description
`fig_c_pooled`	`float`	Observation-weighted FIG-C.
`fig_c`	`float`	Student-weighted FIG-C (macro average).
`fig_c_by_student`	`ndarray`	Per-student FIG-C values, aligned with `student_ids`.
`student_ids`	`ndarray`	Unique student IDs in sorted order.