renate.evaluation.metrics.performance_regression_metrics module

class renate.evaluation.metrics.performance_regression_metrics.NegativeFlipRateMetric

Bases: Metric

Compute Negative Flip Rate (NFR) between new and old models’ predictions,

NFR = len((pred_old == labels) and (pred_new != labels)) / len(test_set).

full_state_update: bool = False

higher_is_better: bool = False

is_differentiable: bool = False

update(new_pred, old_pred, labels)

Update the metric.

Parameters:

• new_pred (Tensor) – a 1-D torch tensor contains new model’s predicted labels.

• old_pred (Tensor) – a 1-D torch tensor contains old model’s predicted labels.

• labels (Tensor) – a 1-D torch tensor contains ground truth labels.

compute()

Override this method to compute the final metric value from state variables synchronized across the distributed backend.

class renate.evaluation.metrics.performance_regression_metrics.NegativeFlipImpactMetric

Bases: Metric

Compute Negative Flip Impact (NFI) between new and old models’ predictions,

NFI = len((pred_old == labels) and (pred_new != labels)) / len(pred_new != labels).

full_state_update: bool = False

higher_is_better: bool = False

is_differentiable: bool = False

update(new_pred, old_pred, labels)

Updates the metric.

Parameters:

• new_pred (Tensor) – a 1-D torch tensor contains new model’s predicted labels.

• old_pred (Tensor) – a 1-D torch tensor contains old model’s predicted labels.

• labels (Tensor) – a 1-D torch tensor contains ground truth labels.

compute()

Override this method to compute the final metric value from state variables synchronized across the distributed backend.

class renate.evaluation.metrics.performance_regression_metrics.PositiveFlipRateMetric

Bases: Metric

Compute Positive Flip Rate (PFR) between new and old models’ predictions,

PFR = len((pred_old != labels) and (pred_new == labels)) / len(test_set).

full_state_update: bool = False

higher_is_better: bool = True

is_differentiable: bool = False

update(new_pred, old_pred, labels)

Updates the metric.

Parameters:

• new_pred (Tensor) – a 1-D torch tensor contains new model’s predicted labels.

• old_pred (Tensor) – a 1-D torch tensor contains old model’s predicted labels.

• labels (Tensor) – a 1-D torch tensor contains ground truth labels.

compute()

Override this method to compute the final metric value from state variables synchronized across the distributed backend.

class renate.evaluation.metrics.performance_regression_metrics.PositiveFlipImpactMetric

Bases: Metric

Compute Positive Flip Impact (PFI) between new and old models’ predictions,

PFI = len((pred_old != labels) and (pred_new == labels)) / len(pred_new == labels).

full_state_update: bool = False

higher_is_better: bool = True

is_differentiable: bool = False

update(new_pred, old_pred, labels)

Updates the metric.

Parameters:

• new_pred (Tensor) – a 1-D torch tensor contains new model’s predicted labels.

• old_pred (Tensor) – a 1-D torch tensor contains old model’s predicted labels.

• labels (Tensor) – a 1-D torch tensor contains ground truth labels.

compute()

Override this method to compute the final metric value from state variables synchronized across the distributed backend.