qumphy.misc.output_conversions module

File: qumphy/output_conversions.py Project: 22HLT01 QUMPHY Contact: vivek.desai@npl.co.uk Gitlab: https://gitlab.com/qumphy Description: Model output conversion functions.

qumphy.misc.output_conversions.convert_prediction_intervals(intervals, converter, confidence_level=0.95)

Convert a list of prediction intervals to distributions, giving a list of distributions, predictions, and uncertainties as variances.

Return type:

tuple

Parameters:

intervals (np.ndarray):

Prediction interval outputs from a model. Expected shape of (n_samples, 2) for [lower_bound, upper_bound] pairs.

converter (Callable):

Conversion function used to convert intervals, given as lower and upper bounds, to distributions. Means and variances for the distributions are also returned.

confidence_level (float, optional):

The confidence level at which conformal prediction was evaluated. Used in the sampling process for converting the interval to a distribution. Defaults to 0.95.

Returns:

: distributions (List[distribution objects]):

List of distribution objects.

predictions (List[float]):

List of predictions, which are the means of the distributions, and should be asymptotically equivalent to the midpoint of the prediction intervals.

uncertainties (List[float]):

List of uncertainties, given as variances of the distributions.

qumphy.misc.output_conversions.intervals_to_probs(intervals, method='jaccard')

Convert prediction intervals to probabilities by taking the mean of lower and upper bounds.

Return type:

ndarray

Parameters:

intervals (np.ndarray): Prediction intervals with shape (n_samples, n_classes, 2), where last dimension represents [lower_bound, upper_bound].

Returns:

:

probs (np.ndarray): Probabilities derived from interval means

qumphy.misc.output_conversions.kde_convert_interval(lower_bound, upper_bound, num_samples=10000, confidence_level=0.95, integrate=False)

Convert a prediction interval to a distribution object using Kernel Density Estimation (KDE).

Return type:

tuple

References:

KDE documentation for statsmodel package can be found here: <https://www.statsmodels.org/dev/generated/statsmodels.nonparametric.kde.KDEUnivariate.html>.

Documentation for scipy quad integration method can be found here: <https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.quad.html>.

Parameters:

lower_bound (float):

Lower bound of the prediction interval.

upper_bound (float):

Upper bound of the prediction interval.

num_samples (int, optional):

Number of samples to generate between the interval bounds. Defaults to 10000.

confidence_level (float, optional):

Confidence level associated with prediction interval. Defaults to 0.95.

integrate (bool, optional):

Option to evaluate mean and standard deviation using numerical integration instead of discrete summation. Defaults to False.

Returns:

:

mean (float):

The mean of the output distribution.

var (float):

The variance of the output distribution - used as the uncertainty.

kde (statsmodel.nonparametric.KDEUnivariate object):

The output distribution approximated using KDE.

qumphy.misc.output_conversions.norm_convert_interval(lower_bound, upper_bound, confidence_level=0.95)

Assuming the quantiles are from a Gaussian distribution, convert to a full distribution object. Return the distribution object, as well as the mean and standard deviation.

Parameters:

lower_bound (float): Lower quantile. upper_bound (float): Upper quantile. confidence_level (float, optional): Confidence level associated with the quantiles. Defaults to 0.95.

Returns:

:

mean (float): The mean of the normal distribution. var (float): The variance of the normal distribution. distribution (scipy.stats.norm object): The full scipy.stats.norm distribution object, which has useful attributes e.g. CDF.

qumphy.misc.output_conversions.probs_to_pred_sets(probabilities, alpha)

Wrapper function to convert probabilities to prediction sets given an array of model confidences per class.

Return type:

ndarray

Parameters:

probabilities (np.ndarray): Array of model outputs as probabilities for each class. alpha (float): Defines the confidence/coverage level at which to set threshold for top-k selection.

Returns:

:

np.ndarry: Array of lists, where each list is a prediction set determined through top-k selection at 1-alpha confidence level.