renate.evaluation.metrics.performance_regression_metrics module#
- class renate.evaluation.metrics.performance_regression_metrics.NegativeFlipRateMetric[source]#
Bases:
Metric
Compute Negative Flip Rate (NFR) between new and old models’ predictions,
NFR = len((pred_old == labels) and (pred_new != labels)) / len(test_set).
-
full_state_update:
bool
= False#
-
higher_is_better:
bool
= False#
-
is_differentiable:
bool
= False#
-
full_state_update:
- class renate.evaluation.metrics.performance_regression_metrics.NegativeFlipImpactMetric[source]#
Bases:
Metric
Compute Negative Flip Impact (NFI) between new and old models’ predictions,
NFI = len((pred_old == labels) and (pred_new != labels)) / len(pred_new != labels).
-
full_state_update:
bool
= False#
-
higher_is_better:
bool
= False#
-
is_differentiable:
bool
= False#
-
full_state_update:
- class renate.evaluation.metrics.performance_regression_metrics.PositiveFlipRateMetric[source]#
Bases:
Metric
Compute Positive Flip Rate (PFR) between new and old models’ predictions,
PFR = len((pred_old != labels) and (pred_new == labels)) / len(test_set).
-
full_state_update:
bool
= False#
-
higher_is_better:
bool
= True#
-
is_differentiable:
bool
= False#
-
full_state_update:
- class renate.evaluation.metrics.performance_regression_metrics.PositiveFlipImpactMetric[source]#
Bases:
Metric
Compute Positive Flip Impact (PFI) between new and old models’ predictions,
PFI = len((pred_old != labels) and (pred_new == labels)) / len(pred_new == labels).
-
full_state_update:
bool
= False#
-
higher_is_better:
bool
= True#
-
is_differentiable:
bool
= False#
-
full_state_update: