qumphy.metrics module

File: qumphy/metrics.py Project: 22HLT01 QUMPHY Contact: nando.hegemann@ptb.de Gitlab: https://gitlab.com/qumphy Description: Evaluation metrics for model performance.

qumphy.metrics.all_binary_metrics(target, prediction)[source]

Evaluate all binary classification metrics.

Given a target and a prediction array, this function computes all metrics as decided for the QUMPHY common evaluation framework.

The metrics are returned as a dictionary with the following keys:

  • auc: Area under the curve calculated with raw probabilities

  • f1: F1-score calculated with a classification threshold of 0.5

  • mcc_sens: Matthews correlation coefficient calculated with a threshold achieving a sensitivity of 0.8

  • mcc_spec: Matthews correlation coefficient calculated with a threshold achieving a specificity of 0.8

  • sens: Sensitivity (with a threshold achieving a sensitivity of 0.8)

  • spec: Specificity (with a threshold achieving a specificity of 0.8)

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions (raw probability of positive class).

Returns:

Dictionary with all metrics.

Return type:

Dict[str, float]

qumphy.metrics.all_regression_metrics(target, prediction, baseline_mae=None)[source]

Evaluate all regression classification metrics.

Return type:

dict[str, float]

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

  • baseline_mae (float) – Baseline mean absolute error.

Returns:

Dictionary with all metrics.

Return type:

dict[str, float]

qumphy.metrics.auc_score_binary(target, prediction, axis=0)[source]

Compute the area und curve (AUC) score for binary classification.

Return type:

float | np.ndarray

Parameters:

  • target (np.ndarray) – Binary ground truth values for different samples.

  • prediction (np.ndarray) – Binary model output predictions (raw prob.) associated with the positive class.

  • axis (int, optional) – Axis to compute AUC over, by default 0.

Returns:

Array of AUC values.

See also

multiclass_auc_score

AUC score for more then two classes.

Examples

>>> target = np.array([0, 1, 0, 1, 1, 0, 1, 0, 1, 0])
>>> prediction = np.array([.99, .8, .6, .63, .77, .23, .3, .78, .2, 0.01])
>>> auc_score_binary(target, prediction)
xx

>>> target = np.random.randint(0, 2, (100, 2))
>>> auc_score_binary(target, target, axis=0)
(1.0, 1.0)

>>> target = np.random.randint(0, 2, (50, 100, 2, 5))
>>> auc_score_binary(target, target, axis=1).shape
(50, 2, 5)
qumphy.metrics.auc_score_multiclass(target, prediction, comparison_type='ovr')[source]

Compute the area und curve (AUC) score.

Return type:

float | np.ndarray

Parameters:

  • target (np.ndarray) – Multiclass ground truth values.

  • prediction (np.ndarray) – Array of model output probabilities of different classes for different samples. If target shape is (n_samples, ...) with n_classes different class values, then prediction needs to have shape (n_samples, n_classes, ...). Axis 1 (n_classes) needs to sum to one.

  • comparison_type (str) –

    Comparison type for multiclasses, by default “ovr”.

    ovr : Stands for one-vs-rest. Computes the AUC for each class against the rest of the classes.

    ovo : Stands for one-vs-one. Computes the average AUC of all possible pairwise combinations of classes.

Returns:

Array of AUC values.

See also

auc_score_binary

AUC score for exactly two classes.

Examples

>>> target = np.array([0, 1, 2, 1, 2, 0])
>>> prediction = np.array([[0.8, 0.1, 0.1],
>>>                        [0.2, 0.5, 0.3],
>>>                        [0.8, 0.1, 0.1],
>>>                        [0.7, 0.2, 0.1],
>>>                        [0.4, 0.3, 0.3],
>>>                        [0.5, 0.4, 0.1]])
>>> auc_score_multiclass(target, prediction, comparison_type="ovo")
0.6875

>>> target = np.random.randint(0, 3, (100, 10, 5, 2))
>>> prediction = np.random.uniform(0, 1, (100, 3, 10, 5, 2))
>>> prediction /= np.expand_dims(np.sum(prediction, axis=1), 1)
>>> auc_score_multiclass(target, prediction).shape
(10, 5, 2)
qumphy.metrics.balanced_accuracy_score(target, prediction)[source]

Compute balanced accuracy score for binary or multi-class classification.

For the binary case the balanced accuracy score \(\operatorname{Acc}_b\) is given by the arithmetic mean of sensitivity (Se) and specificity (Sp), i.e. :rtype: float

\[\operatorname{Acc}_b = \frac{1}{2}(\operatorname{Se} + \operatorname{Sp}) = \frac{1}{2}\Bigl( \frac{\operatorname{TP}}{\operatorname{TP}+\operatorname{FN}} + \frac{\operatorname{TN}}{\operatorname{TN}+\operatorname{FP}} \Bigr),\]

expressed by true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). In general, balanced accuracy is computed by

\[\operatorname{Acc}_b(y_{\mathrm{true}}, y_{\mathrm{pred}}) = \frac{\sum_{i=1}^N w_i \, \delta(y_{\mathrm{true},i} = y_{\mathrm{pred}, i})}{\sum_{i=1}^N w_i}\]

with weights

\[w_i = \frac{1}{\sum_{j=1}^N \delta(y_{\mathrm{true},i} = y_{\mathrm{true},j})},\]

where \(\delta(y_i = y_j)\) denotes the Kronecker delta function.

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

Returns:

Balanced accuracy score for binary or multi-class classification.

Return type:

float

Examples

Computation of balanced accuracy score for binary classification, i.e., a one dimensional array with only 2 classes

>>> target = np.array([0, 1, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = np.array([0, 1, 0, 1, 1, 1, 0, 1, 1, 1])
>>> balanced_accuracy_score(target, prediction)
0.6

Computation of balanced accuracy score for a multi-class scenario, i.e., a 1D array with more then two classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> balanced_accuracy_score(target, prediction)
0.55
qumphy.metrics.brier_score(target, prediction)[source]

Compute Brier score for binary classification.

Return type:

float

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

Returns:

Brier score for binary classification.

Return type:

float

Examples

>>> target = np.array([0, 1, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = prediction = np.linspace(0,1,10)
>>> brier_score(target, prediction)
0.34074074074074073
qumphy.metrics.f1_score(target, prediction, average=None)[source]

Compute F1-score of binary, multi-class or multi-label classification.

The \(F_1\) score is computed using the true positives (TP), false positives (FP) and false negatives (FN) via :rtype: float | np.ndarray

\[F_1 = \frac{2\operatorname{TP}}{2\operatorname{TP} + \operatorname{FP} + \operatorname{FN}}.\]
Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

  • average (str | None, optional) – Averaging of the F1-scores (default None). For binary classification, average=binary is the default case. For multi-class and multi-lable classification, average=None is the default case, which results in F1-scores for each individual class. For more detail about averaging see the documentation of sklearn.metrics.f1_score.

Returns:

F1 score(s) for binary, multi-class or multi-lable classification.

Return type:

float | np.ndarray

Examples

Computation of F1-score for binary classification, i.e., a one dimensional array with only 2 classes

>>> target = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = np.array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1])
>>> f1_score(target, prediction)
0.5

Computation of F1-score for a multi-class scenario, i.e., a 1D array with more then two classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> f1_score(target, prediction)
array([0.4, 0.44444444, 0.33333333])

Computation of F1-score for a multi-lable scenario, i.e., a 2D array with with columns representing different labels and values 0 or 1 as entries

>>> target = np.array([[0, 1, 0], [1, 0, 1], [1, 1, 0], [1, 1, 1]])
>>> prediction = np.array([[1, 0, 1], [1, 1, 1], [0, 0, 1], [1, 1, 0]])
>>> f1_score(target, prediction)
array([0.66666667, 0.4, 0.4])

Computation of F1-score for a multi-class scenario with averaging of F1-scores over the different classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> f1_score(target, prediction, average='micro')
0.4
qumphy.metrics.false_discovery_rate(target, prediction, average=None)[source]

Compute false discovery rate (FDR).

The false discovery rate (FDR) is given by \(\operatorname{FDR} = 1 - \operatorname{PPV}\), where \(PPV\) is the precision (positive predicted value).

Return type:

float | np.ndarray

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

  • average (str | None, optional) – Averaging type (default None). For more detail about averaging see the documentation of sklearn.metrics.precision_score.

Returns:

False discovery rate for binary, multi-class or multi-lable classification.

Return type:

float | np.ndarray

See also

precision_score, f1_score

qumphy.metrics.general_threshold(target, prediction, metric, metric_value, greater_than=True)[source]

Find the threshold that sets the given metric closest to metric_value.

Return type:

float

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

  • metric (Callable[[np.ndarray, np.ndarray], float]) – A metric function that takes target and prediction arrays as input.

  • metric_value (float) – The value of the metric to be achieved.

  • greater_than (bool) – True if the metric is supposed to be higher than metric_value, false otherwise.

Returns:

The threshold that achieves the metric.

Return type:

float

qumphy.metrics.ieee_grades(target, prediction)[source]

Compute the IEEE grades of the predicted values.

The grades are calculated by comparing the difference between the target and the prediction. Returned are the percentage of samples that fall within each grade. The grading scores follow the IEEE Std 1708a™-2019 scheme, where instead of a mean absolute difference of two measurements with the standard device, we use only one measurement.

Return type:

ndarray

The grading for each sample is determined as follows

  • Grade A for error <5 mmHg

  • Grade B for error between 5-6 mmHg

  • Grade C for error between 6-7 mmHg

  • Grade D for error >7 mmHg

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

Return type:

np.ndarray

qumphy.metrics.ieee_grades_torch(target, prediction)[source]
qumphy.metrics.l1_norm(array, axis=0)[source]

Compute the \(L^1\)-norm of an array along an axis.

The \(L^1\)-norm of an array \(x\in\mathbb{R}^N\) is given by \(\Vert x \Vert_{L^1} = \frac{1}{N}\sum_{j=1}^N \vert x_j \vert\).

Return type:

float | np.ndarray

Parameters:

  • array (np.ndarray) – Data array.

  • axis (int, optional) – Axis, by default 0.

Returns:

Array of \(L^1\)-norms over the specified axes.

See also

mean_absolute_error

Wrapper for l1_norm(target - prediction).

l2_norm, root_mean_square_error

Examples

>>> l1_norm(np.array([1, 2, 3, 4]))
10

>>> array = np.random.normal(0, 1, (10, 5, 3, 2))
>>> l1_norm(array, axis=1).shape  # norm over second axis
(10, 3, 1)
qumphy.metrics.l2_norm(array, axis=0)[source]

Compute the \(L^2\)-norm along an axis.

The \(L^2\)-norm of an array \(x\in\mathbb{R}^N\) is given by \(\Vert x \Vert_{L^2} = \sqrt{\frac{1}{N}\sum_{j=1}^N x_j^2 }\).

Return type:

float | np.ndarray

Parameters:

  • array (np.ndarray) – Data array.

  • axis (int, optional) – Axis, by default 0.

Returns:

Array of \(L^2\)-norms over the specified axes.

See also

root_mean_square_error

Wrapper for l2_norm(target - prediction).

l1_norm, mean_absolute_error

Examples

>>> l2_norm(np.array([1, 2, 3, 4]))**2
30.0

>>> array = np.random.normal(0, 1, (10, 5, 3, 2))
>>> l2_norm(array, axis=1).shape  # norm over second axis
(10, 3, 1)
qumphy.metrics.matthews_correlation_coefficient(target, prediction)[source]

Compute Matthews correlation coefficient (Mcc) of binary or multi-class task.

The Matthews correlation coefficient (Mcc) is computed using the true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) via :rtype: float

\[\operatorname{Mcc} = \frac{\operatorname{TP}\cdot\operatorname{TN} - \operatorname{FP}\cdot\operatorname{FN}}{\sqrt{(\operatorname{TP}+\operatorname{FP})(\operatorname{TP}+\operatorname{FN})(\operatorname{TN}+\operatorname{FP})(\operatorname{TN}+\operatorname{FN})}}.\]

For the multi-class case, let \(C\) be the confusion matrix for \(K\) classes and define the number of times class \(k\) truly occurs \(t_k = \sum_{i=1}^K C_{ik}\), the number of times class \(k\) was predicted \(p_k = \sum_{i=1}^K C_{ki}\), the total number of samples correctly predicted \(c = \sum_{k=1}^K C_{kk}\) and the total number of samples \(s = \sum_{i,j=1}^K C_{ij}\). Then the multiclass Mcc is defined as

\[\operatorname{Mcc} = \frac{c \cdot s -\sum_{k=1}^K p_k \cdot t_k}{\sqrt{ (s^2 - \sum_{k=1}^K p_k^2)(s^2 - \sum_{k=1}^K t_k^2)}}.\]

Note

When there are more than two labels, the value of the MCC will no longer range between -1 and +1. Instead the minimum value will be somewhere between -1 and 0 depending on the number and distribution of ground true labels. The maximum value is always +1.

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

Returns:

Mcc for binary and multi-class classification.

Return type:

float | np.ndarray

Examples

Computation of Mcc for binary classification, i.e., a one dimensional array with only 2 classes

>>> target = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = np.array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1])
>>> matthews_correlation_coefficient(target, prediction)
-0.10206207261596577

Computation of Mcc for a multi-class scenario, i.e., a 1D array with more then two classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> matthews_correlation_coefficient(target, prediction)
0.13130643285972254
qumphy.metrics.mean_absolute_error(target, prediction, axis=0)[source]

Compute the mean absolute error (MAE) between target and prediction.

Return type:

float | np.ndarray

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

  • axis (int, optional) – Axis to sum over, by default 0.

Returns:

Array of MAE values (\(L^1\)-norms) over the specified axes.

See also

l1_norm

This is a wrapper for l1_norm(target - prediction, axis=axis).

l2_norm, root_mean_square_error

Examples

>>> mean_absolute_error(np.array([1, 2, 3]), np.array([1, 2, 3]))
0.0

>>> target = np.random.normal(0, 1, (10, 5, 3, 2))
>>> prediction = np.random.normal(0, 1, (10, 5, 3, 2))
>>> mean_absolute_error(target, prediction, axis=1).shape  # norm over second axis
(10, 3, 1)
qumphy.metrics.mean_absolute_scaled_error(baseline_mae, model_mae)[source]

Compute mean absolute scaled error (MASE).

The MASE is a measure of the magnitude of the error relative to a baseline error. It is defined as the mean absolute error divided by the baseline error.

Return type:

float

Parameters:

  • baseline_mae (float) – Mean absolute scaled error of the baseline.

  • model_mae (float) – Mean absolute error.

Returns:

Mean absolute scaled error.

Return type:

float

qumphy.metrics.negative_predictive_value(target, prediction)[source]

Compute negative predictive value (NPV) of binary, multi-class or multi-label classification.

The negative predictive value is computed using the true positives (TN) and false positives (FN) via :rtype: float | np.ndarray

\[\operatorname{NPV} = \frac{\operatorname{TN}}{\operatorname{TN} + \operatorname{FN}}.\]
Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

Returns:

Precision scores for binary, multi-class or multi-lable classification.

Return type:

float | np.ndarray

See also

f1_score, false_discovery_rate, precision_score

Examples

Computation of negative predictive value score for binary classification, i.e., a one dimensional array with only 2 classes

>>> target = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = np.array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1])
>>> negative_predictive_value(target, prediction)
0.5
qumphy.metrics.precision_score(target, prediction, average=None)[source]

Compute precision (PPV) of binary, multi-class or multi-label classification.

The precision score (positive predictive value, PPV) is computed using the true positives (TP) and false positives (FP) via :rtype: float | np.ndarray

\[\operatorname{PPV} = \frac{\operatorname{TP}}{\operatorname{TP} + \operatorname{FP}}.\]
Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Predicted values.

  • average (str | None, optional) –

    Averaging type for score (default None). For more detail about averaging see the documentation of sklearn.metrics.precision_score.

Returns:

Precision scores for binary, multi-class or multi-lable classification.

Return type:

float | np.ndarray

See also

f1_score, false_discovery_rate, negative_predictive_value

Examples

Computation of precision score for binary classification, i.e., a one dimensional array with only 2 classes

>>> target = np.array([0, 0, 1, 0, 1, 0, 0, 1, 0, 1])
>>> prediction = np.array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1])
>>> precision_score(target, prediction)
0.375

Computation of precision score for a multi-class scenario, i.e., a 1D array with more then two classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> precision_score(target, prediction)
array([0.25, 0.5, 0.5])

Computation of precision score for a multi-lable scenario, i.e., a 2D array with with columns representing different labels and values 0 or 1 as entries

>>> target = np.array([[0, 1, 0], [1, 0, 1], [1, 1, 0], [1, 1, 1]])
>>> prediction = np.array([[1, 0, 1], [1, 1, 1], [0, 0, 1], [1, 1, 0]])
>>> precision_score(target, prediction)
array([0.33333333, 0.5, 0.66666667])

Computation of precision score for a multi-class scenario with averaging of F1-scores over the different classes

>>> target = np.array([1, 2, 2, 2, 1, 2, 1, 0, 1, 1])
>>> prediction = np.array([1, 1, 2, 0, 0, 1, 1, 0, 0, 2])
>>> precision_score(target, prediction, average='micro')
0.4
qumphy.metrics.recall_score_threshold(target, prediction, recall_value, pos_label=1, greater_than=True, dtype=numpy.float32)[source]

Compute the classification threshold so that the recall score is closest to the specified value, but greater (or lower).

Default: The threshold is computed for the sensitivity score.

The threshold is set as the next floating point number after (before) the value of the prediction that needs to be classified positive (negative) to achieve the desired recall score.

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

  • recall_value (float) – The desired recall score.

  • pos_label (1 | 0, optional) – 1 to compute the threshold for sensitivity, 0 for specificity.

  • greater_than (bool, optional) – True to let the recall score be higher than recall_value, false to let the recall score be lower than recall_value.

  • dtype (np.dtype, optional) – The data type of the threshold, by default np.float32

Returns:

The classification threshold.

Return type:

float

qumphy.metrics.root_mean_square_error(target, prediction, axis=0)[source]

Compute the root mean square error (RMSE) between target and prediction.

Return type:

float | np.ndarray

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

  • axis (int, optional) – Axis to sum over, by default 0.

Returns:

Array of RMSE values (\(L^2\)-norms) over the specified axes.

See also

l2_norm

This is a wrapper for l2_norm(target - prediction, axis=axis).

l1_norm, mean_absolute_error

Examples

>>> root_mean_square_error(np.array([1, 2, 3]), np.array([1, 2, 3]))
0.0

>>> target = np.random.normal(0, 1, (10, 5, 3, 2))
>>> prediction = np.random.normal(0, 1, (10, 5, 3, 2))
>>> root_mean_square_error(target, prediction, axis=1).shape  # norm over second axis
(10, 3, 1)
qumphy.metrics.sensitivity(target, prediction)[source]

Compute the sensitivity (or true positive rate). Sensitivity is also known as the recall score of the positive class.

Return type:

float

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

Returns:

Sensitivity score.

Return type:

float

See also

specificity

qumphy.metrics.specificity(target, prediction)[source]

Compute the specificity (or true negative rate). Specificity is also known as the recall score of the negative class.

Return type:

float

Parameters:

  • target (np.ndarray) – Ground truth values.

  • prediction (np.ndarray) – Model output predictions.

Returns:

Specificity score.

Return type:

float

See also

sensitivity