fig.fig_torch -- PyTorch Implementation¶

Differentiable versions of FIG-V and FIG-C that accept and return PyTorch tensors. Gradients flow through y_pred_eval only; all other inputs (training data, item/student IDs) are treated as constants. Baseline computation always runs in NumPy.

The API mirrors fig.fig with the following differences:

Inputs can be tensors, NumPy arrays, or sequences.
Outputs are tensors (except student_ids, which is always a NumPy array).
A baseline parameter allows reusing pre-computed baselines across batches (strongly recommended -- use precompute_baseline_torch once before the training loop).
Device is inferred from y_pred_eval if it is a tensor, or defaults to CPU.

Note

Import this module directly -- it is not re-exported from the top-level fig package:

from fig.fig_torch import fractional_information_gain_validation_torch

Main Functions¶

fractional_information_gain_validation_torch¶

fractional_information_gain_validation_torch ¶

fractional_information_gain_validation_torch(
    y_pred_eval: Tensor | ndarray | Sequence[float],
    y_eval: Tensor | ndarray | Sequence[float],
    item_id_eval: Tensor | ndarray | Sequence[Any],
    student_id_eval: Tensor | ndarray | Sequence[Any],
    y_train: Tensor
    | ndarray
    | Sequence[float]
    | None = None,
    item_id_train: Tensor
    | ndarray
    | Sequence[Any]
    | None = None,
    eps: float = 1e-12,
    use_shrinkage: bool = True,
    alpha: float = 2.0,
    beta: float = 2.0,
    calibration: bool = False,
    calib_n_bins: int = 10,
    calib_warn_threshold: float = 0.1,
    baseline: BaselineData | None = None,
) -> FigVTorchResult

Compute FIG-V (Validation) using PyTorch.

FIG-V measures how much better a model's predictions are than per-item base rates, using cross-entropy against ground truth. Differentiable with respect to y_pred_eval.

FIG-V = 1 - BCE(y_pred, y) / H(baseline[item_id])

where:

H(baseline[item_id]) = sum of binary entropies of per-item base rates (prior uncertainty), looked up by item ID
BCE(y_pred, y) = sum of binary cross-entropies of model predictions against ground truth
BCE(p, y) = -[y*log(p) + (1-y)*log(1-p)]

Parameters:

Name	Type	Description	Default
`y_pred_eval`	`array - like or Tensor`	Predicted probabilities (0 to 1). Shape: (n_eval,) This is the only input that retains gradients.	required
`y_eval`	`array - like or Tensor`	Ground truth on eval split. Shape: (n_eval,) Values: 1=correct, 0=incorrect.	required
`item_id_eval`	`array - like or Tensor`	Item identifiers for eval observations. Shape: (n_eval,)	required
`student_id_eval`	`array - like or Tensor`	Student identifiers for eval observations. Shape: (n_eval,)	required
`y_train`	`array - like or Tensor`	Ground truth on train split. Shape: (n_train,) Required if baseline is not provided.	`None`
`item_id_train`	`array - like or Tensor`	Item identifiers for train observations. Shape: (n_train,) Required if baseline is not provided.	`None`
`eps`	`float`	Small constant for clipping probabilities to avoid log(0).	`1e-12`
`use_shrinkage`	`bool`	If True, apply empirical Bayes shrinkage to item baselines.	`True`
`alpha`	`float`	Prior pseudo-count for successes in shrinkage.	`2.0`
`beta`	`float`	Prior pseudo-count for failures in shrinkage.	`2.0`
`calibration`	`bool`	If True, attach calibration metrics (ECE) to output.	`False`
`calib_n_bins`	`int`	Number of bins for ECE computation.	`10`
`calib_warn_threshold`	`float`	Warn if ECE exceeds this value.	`0.1`
`baseline`	`BaselineData`	Pre-computed baseline from `precompute_baseline_torch()`. If provided, `y_train`, `item_id_train`, `use_shrinkage`, `alpha`, and `beta` are ignored. If not provided, `y_train` and `item_id_train` are required and the baseline is computed on each call.	`None`

Returns:

Type	Description
`FigVTorchResult`	Dictionary with keys: `fig_v_pooled`: Observation-weighted FIG-V (scalar tensor) `fig_v`: Student-weighted FIG-V (scalar tensor) `fig_v_by_student`: Per-student FIG-V values (1D tensor) `student_ids`: Unique student IDs in sorted order (np.ndarray) `calibration`: None unless `calibration=True`, in which case a dict of ECE metrics from `_check_calibration_torch`

Notes

FIG-V = 0 when model predictions match item base rates (no information gain).
FIG-V = 1 when model predictions are perfect (zero cross-entropy).
FIG-V can be negative if model is worse than the baseline.
FIG-V > 1 usually indicates data leakage; the library warns when this happens.
Items in eval that don't appear in train use the global training mean as baseline.

fractional_information_gain_confidence_torch¶

fractional_information_gain_confidence_torch ¶

fractional_information_gain_confidence_torch(
    y_pred_eval: Tensor | ndarray | Sequence[float],
    item_id_eval: Tensor | ndarray | Sequence[Any],
    student_id_eval: Tensor | ndarray | Sequence[Any],
    y_train: Tensor
    | ndarray
    | Sequence[float]
    | None = None,
    item_id_train: Tensor
    | ndarray
    | Sequence[Any]
    | None = None,
    eps: float = 1e-12,
    use_shrinkage: bool = True,
    alpha: float = 2.0,
    beta: float = 2.0,
    baseline: BaselineData | None = None,
) -> FigCTorchResult

Compute FIG-C (Confidence) using PyTorch.

FIG-C measures the reduction in uncertainty about student responses based on model confidence, without requiring ground truth labels. Differentiable with respect to y_pred_eval.

FIG-C = 1 - H(y_pred) / H(baseline[item_id])

where:

H(baseline[item_id]) = sum of binary entropies of per-item base rates (prior uncertainty), looked up by item ID
H(y_pred) = sum of binary entropies of model predictions (remaining uncertainty)
H(p) = -[p*log(p) + (1-p)*log(1-p)]

Parameters:

Name	Type	Description	Default
`y_pred_eval`	`array - like or Tensor`	Predicted probabilities (0 to 1). Shape: (n_eval,) This is the only input that retains gradients.	required
`item_id_eval`	`array - like or Tensor`	Item identifiers for eval observations. Shape: (n_eval,)	required
`student_id_eval`	`array - like or Tensor`	Student identifiers for eval observations. Shape: (n_eval,)	required
`y_train`	`array - like or Tensor`	Ground truth on train split. Shape: (n_train,) Required if baseline is not provided.	`None`
`item_id_train`	`array - like or Tensor`	Item identifiers for train observations. Shape: (n_train,) Required if baseline is not provided.	`None`
`eps`	`float`	Small constant for clipping probabilities to avoid log(0).	`1e-12`
`use_shrinkage`	`bool`	If True, apply empirical Bayes shrinkage to item baselines.	`True`
`alpha`	`float`	Prior pseudo-count for successes in shrinkage.	`2.0`
`beta`	`float`	Prior pseudo-count for failures in shrinkage.	`2.0`
`baseline`	`BaselineData`	Pre-computed baseline from `precompute_baseline_torch()`. If provided, `y_train`, `item_id_train`, `use_shrinkage`, `alpha`, and `beta` are ignored. If not provided, `y_train` and `item_id_train` are required and the baseline is computed on each call.	`None`

Returns:

Type	Description
`FigCTorchResult`	Dictionary with keys: `fig_c_pooled`: Observation-weighted FIG-C (scalar tensor) `fig_c`: Student-weighted FIG-C (scalar tensor) `fig_c_by_student`: Per-student FIG-C values (1D tensor) `student_ids`: Unique student IDs in sorted order (np.ndarray)

Notes

FIG-C = 0 when model predictions match item base rates (no information gain).
FIG-C = 1 when model is perfectly confident (entropy = 0).
FIG-C can be negative if model adds uncertainty beyond the baseline.
For well-calibrated models, FIG-C approximates FIG-V in expectation.
Items in eval that don't appear in train use the global training mean as baseline.

Loss Wrappers¶

Convenience functions that negate the FIG metric for use as a minimization objective with standard optimizers.

fig_v_loss¶

fig_v_loss ¶

fig_v_loss(*args: Any, **kwargs: Any) -> Tensor

Return FIG-V negated for use as a minimization loss.

Equivalent to: -fractional_information_gain_validation_torch(...)['fig_v']

fig_c_loss¶

fig_c_loss ¶

fig_c_loss(*args: Any, **kwargs: Any) -> Tensor

Return FIG-C negated for use as a minimization loss.

Equivalent to: -fractional_information_gain_confidence_torch(...)['fig_c']

Baseline Pre-computation¶

precompute_baseline_torch¶

precompute_baseline_torch ¶

precompute_baseline_torch(
    y_train: Tensor | ndarray | Sequence[float],
    item_id_train: Tensor | ndarray | Sequence[Any],
    use_shrinkage: bool = True,
    alpha: float = 2.0,
    beta: float = 2.0,
) -> BaselineData

Pre-compute baseline statistics from training data for reuse across calls.

This avoids recomputing per-item baselines on every call to fractional_information_gain_validation_torch or fractional_information_gain_confidence_torch. Useful when evaluating the same training set against multiple eval batches (e.g., during a training loop).

The computations in this function always happen on the CPU.

Parameters:

Name	Type	Description	Default
`y_train`	`array - like or Tensor`	Ground truth on train split. Shape: (n_train,)	required
`item_id_train`	`array - like or Tensor`	Item identifiers for train observations. Shape: (n_train,)	required
`use_shrinkage`	`bool`	If True, apply empirical Bayes shrinkage to item baselines.	`True`
`alpha`	`float`	Prior pseudo-count for successes in shrinkage.	`2.0`
`beta`	`float`	Prior pseudo-count for failures in shrinkage.	`2.0`

Returns:

Type	Description
`BaselineData`	Pre-computed baseline. Pass this as the `baseline` argument to the FIG-V/FIG-C functions to avoid recomputing per training batch.

Result Types¶

FigVTorchResult¶

FigVTorchResult ¶

Bases: TypedDict

Return type of fractional_information_gain_validation_torch.

Attributes:

Name	Type	Description
`fig_v_pooled`	`Tensor`	Observation-weighted FIG-V (scalar tensor).
`fig_v`	`Tensor`	Student-weighted FIG-V (scalar tensor).
`fig_v_by_student`	`Tensor`	Per-student FIG-V values, aligned with `student_ids`.
`student_ids`	`ndarray`	Unique student IDs in sorted order.
`calibration`	`CalibrationResult or None`	Calibration metrics. None unless `calibration=True` was passed.

FigCTorchResult¶

FigCTorchResult ¶

Bases: TypedDict

Return type of fractional_information_gain_confidence_torch.

Attributes:

Name	Type	Description
`fig_c_pooled`	`Tensor`	Observation-weighted FIG-C (scalar tensor).
`fig_c`	`Tensor`	Student-weighted FIG-C (scalar tensor).
`fig_c_by_student`	`Tensor`	Per-student FIG-C values, aligned with `student_ids`.
`student_ids`	`ndarray`	Unique student IDs in sorted order.