Generalized Aggregations

[jahall]

Hi team, this is a great library, thanks for putting it together!

I was wondering if you would consider extending for the more general case of (weighted) aggregations rather than just counts/prevelances? i.e. where each instance has a weight or a cost associated with it e.g. maybe you're trying to estimate total $ value of the positive class where each instance has a price associated with it. You can recover the standard count definition by setting all weights to 1.

I have some code to do this for CC, ACC, HDy, etc - but it seems like the count/prevalence notion is fairly baked in to this lib...

[AlexMoreo]

Hi Hoe!

That sounds interesting. Could you please elaborate a bit more on the idea? Maybe one concrete example, or a link to your code, would be helpful.

It is my impression, though, that your suggestion is related to cost-sensitive quantification (or simply cost quantification), which is not (yet) covered in QuaPy. You might want to have a look at Chapter 5.5 of the (open access) quantification book https://link.springer.com/book/10.1007/978-3-031-20467-8

Let me know!

Regards
Alex

[jahall]

Thanks Alex,

Here is a link to the weighted quantification helper function I've been using...basically weighting the adjusted predictions before summation.

def quantify(
    predictions: np.ndarray,
    *,
    confusion: Optional[np.ndarray] = None,
    weights: Optional[np.ndarray] = None,
    n_labels: Optional[int] = None,
) -> np.ndarray:
    """Aggregate a vector of labels and adjust based on a confusion matrix.
    
    :param predictions: individual predictions, can either be an n_samples
        vector of integer labels or an n_samples x n_labels matrix of one-hot
        encoded labels or probabilities
    :param confusion: optional n_labels x n_labels confusion matrix with true
        labels as rows and predicted labels as columns
    :param weights: optional n_samples vector of weights to apply to each sample
        before aggregation
    :param n_labels: optionally explicit specification of number of labels, if not
        provided is inferred from predictions
    :returns: an n_labels vector of aggregate counts for each label
    """
    # convert predictions to n_labels x n_samples matrix of values from 0 to 1
    predictions = _conform_predictions(predictions, n_labels)
    n_labels = predictions.shape[1]
    
    if confusion is None:
        inv_confusion = np.eye(n_labels)
    else:
        # ensure confusion is normalized to the true labels, for binary
        # classification the confusion will be of the form:
        # [
        #     [ TNR (ie Specificity),  FPR                            ],
        #     [ FNR,                   TPR (ie Recall or Sensitivity) ],
        # ]
        confusion = confusion / confusion.sum(axis=1, keepdims=True)

        # again for binary case the inverse would look like this:
        # [
        #     [TPR / det, -FPR / det],
        #     [-FNR / det, TNR / det],
        # ]
        # where det = TNR * TPR - FPR * FNR
        #           = TPR - FPR  (due to TNR=1-FPR and FNR=1-TPR)
        #
        inv_confusion = np.linalg.inv(confusion)

    adjusted = predictions @ inv_confusion

    if weights is not None:
        adjusted *= weights[:, None]

    # in the binary case the (un-weighted) adjusted count of the
    # positive class would be:
    #
    #   (n_predicted_positives - FPR * n_samples) / (TPR - FPR)
    #
    return adjusted.sum(axis=0)


def _conform_predictions(preds: np.ndarray, n_labels: Optional[int]) -> np.ndarray:
    if len(preds.shape) == 1:
        labels = preds
        n_samples = preds.shape[0]
        if n_labels is None:
            n_labels = len(np.unique(labels))
        preds = np.zeros((n_samples, n_labels))
        preds[np.arange(n_samples), labels] = 1.0
    return preds

[jahall]

...and to apply weights in the distribution-matching context I construct the empirical distribution as follows:

def to_empirical_dist(
    scores: np.ndarray,
    n_bins: int = 50,
    weights: Optional[np.ndarray] = None,
    bounds: Tuple[float, float] = (0.0, 1.0),
) -> np.ndarray:
    """Convert array of scores to an empirical distribution.
    
    :param scores: array of scores that lie within certain bounds
    :param n_bins: number of bins to use for empirical distribution
    :param weights: optional weights for each score
    :param bounds: upper and lower bounds for the scores
    :returns: n_bins distribution vector
    """
    if scores.size == 0:
        return np.ones(n_bins) / n_bins
    
    if weights is None:
        weights = np.ones(scores.shape)
    
    probs = (scores - bounds[0]) / (bounds[1] - bounds[0])
    dist = np.zeros(n_bins)
    bins = _to_bins(probs, n_bins)
    for bin, w in zip(bins, weights):
        dist[bin] += w
        
    return dist / dist.sum()

def _to_bins(probs: np.ndarray, n_bins: int) -> np.ndarray:
    idx = (probs * n_bins).astype(int)
    return np.minimum(idx, n_bins - 1)

[jahall]

From looking at chapter 5.5. I think my quantify corresponds to Classify & Total (weights != None, confusion = None) and Grossed-Up Total (weights != None, confusion != None)

[aesuli]

Hello Joe,

we looked at your code and we agree with you that quantify corresponds to Classify & Total (weights != None, confusion = None). For the case with confusion!=None, I'm not convinced that it matches Grossed-Up Total.
Your code applies ACC correction (right? yet I see no clipping to [0,1]) to the single instances and the spreads the weight of each instance according to its corrected distribution. Then it sums up by label to get the final cost prediction.
Grossed Up Total seems to be simpler, as it increases or decreases the Classify & Total estimate by the ratio of a CC estimate and any other quantifier deemed to be a better one (it should be easy to implement it as a wrapper that works using any other quantifier to estimate the correction factors).
So I think your method is an original idea, and you should try to compare it with Grossed Up Total.
A limitation of Grossed Up Total is that its output likely will not sum up to the original total weight.

The to_empirical_dist method is sound to me, as it determines the mass in each bin with respect to the weight and not the instance count.

With respect to the support for weights in Quapy, the instance-based weight model is more related to classification than to quantification, so it got less attention from quantification researchers, and from us.
The few existing methods are dated and very empirical and we have not (yet) found relevant applications that made it worth the effort to consider the weighted case.
We will consider adding it the next releases. If you think you can contribute on this we can have a chat.

[jahall]

Thank you @aesuli for your considered reply! I would indeed be interested in contributing!

As for my grossed-up-total, let me take a look. I believe my method should respect the total sum. Also, I think a simpler implementation (which I believe leads to the same output) is:

def quantify(predictions, *, confusion=None, weights=None, n_labels=None):
    """Aggregate a vector of labels and adjust based on a confusion matrix."""
    # convert predictions to n_samples x n_labels matrix of values from 0 to 1
    predictions = _conform_predictions(predictions, n_labels)
    n_labels = predictions.shape[1]
    if weights is not None:
        predictions *= weights[:, None]  # <- only difference between normal count methods
    if confusion is None:
        confusion = np.eye(n_labels)
    confusion /= confusion.sum(axis=1, keepdims=True)
    return np.linalg.solve(confusion.T, predictions.T).sum(axis=1)