Implementing Class Rectification Loss in fast.ai

amqdn-crl.ipynb

@micimize

Thanks for taking the time to do that! I like the idea of using torch.unique to accomplish the sample counting. As you can see, I published this over a year ago, and I notice my relative inexperience with PyTorch and ML back then shows.

Re: mining results being a function of outputs and targets... If I understand your question: You're asking why it is that I included the mining operation in a callback_fn instead of inside the loss.forward. In looking at my code and trying to remember why, I think I couldn't come up with a way to retain the indices of the majority/minority classes using just the loss module. Since the majority/minority class designation changes from batch-to-batch dynamically in this paper, it's necessary to keep track of the indices (I think) in order to calculate the loss properly. That doesn't mean it's not possible, but I hadn't found a way.

Great explanation, but I don't think the counter usage works, as the targets are tensors and not plain numbers. That's why your Majority and Minority Indices are the same.

Here is some of the preliminary work I did that fixes this using torch.unique and and generalizes up to SortMinorityClass:

from dataclasses import dataclass

def indices_of(occurences, given_class):
    "extract a tensor of indices of the given instance class"
    return tensor([
        i for i, encoding in enumerate(occurences)
        if torch.all(torch.eq(encoding, given_class))
    ])

@dataclass
class BatchClassification:
    __slots__ = ['encoding', 'indices', 'frequency']
    encoding: Tensor # class encoding as a tensor
    indices: Tensor # indices within the batch
    frequency: int # number of occurrences in the class

def classifications_of(targets: Tensor, descending_frequency=False) -> t.Iterable[BatchClassification]:
    class_encodings, class_indices, class_counts = torch.unique(targets, dim=0, return_counts=True, return_inverse=True)
    return sorted([
        BatchClassification(encoding, indices_of(class_indices, encoding), frequency)
        for encoding, frequency in zip(class_encodings, class_counts)
    ], key=lambda bc: bc.frequency, reverse=descending_frequency)

class SortMinorityClass(LearnerCallback):
    def __init__(self, learn:Learner):
        super().__init__(learn)
        self.iters = 0  # Manage the number of printouts we get
    def on_batch_begin(self, last_target:Tensor, **kwargs:Any) -> Tensor:
        if self.iters < 2:
            for index, bc in enumerate(classifications_of(last_target)):
                print(f'frequency group {index}, count {str(bc.frequency)}: {bc}')
            self.iters += 1

Also, since the mining results are a function of outputs and targets anyways, couldn't we implement the entire mining operation within the loss function? I'm kinda new to ML so maybe I'm missing something

Actually, the Counter works well here.
Because when passing the tensors to Counter, the author calls the tolist() method of torch.tensor object.
Check this bellow demo👇:

@2foil ah you're right – but it seems we're both right, because I was looking at the output of SortMinorityClass, which uses Counter(targets), which is later fixed in the final ClassLevelHardMining callback

@micimize Okay, I got it. Thanks for your explanation 😊.

@amqdn Thanks for your tutorial ❤️, it helps me a lot when implementing CRLloss.

Here I have one question.
Now I'm dealing with one training dataset, which has multiple majority and minority classes.
So how to compute the omega in CRLloss?

Glad it's helping! From the paper: "Given different individual class data sample sizes, we define Ωimb as the minimum percentage count of data samples required over all classes in order to form an overall uniform (i.e. balanced) class distribution in the training data." The authors did not provide an explicit formulation for Ωimb, so I had to infer it. There are two ways to think about this: 1) What percentage of the overall number of samples does the class with the least amount of samples represent? E.g., if you have 5 classes with [100, 100, 50, 50, 25] sample counts, 25 / (100 + 100 + 50 + 50 + 25) ≈ 0.077. The problem with this is: Taking 7.7% of each class will not result in a balanced class distribution. 2) Between the majority number of samples 100 in the above example and the minority number of samples 25, 25 constitutes 25% of the maximum amount for any given class in this example set. If you take 25% of 100, you will get 25 for the majority classes. It's better, but then you see that if you take 25% of 50, you have a similar problem of not having balanced class distribution. Still, I preferred this alternative, and you see that in the code. These were my best guesses. Maybe it'll inspire you to find something better!

…

On Tue, Sep 22, 2020 at 1:19 AM 2foil ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ @amqdn <https://github.com/amqdn> Thanks for your tutorial ❤️, it helps me a lot when implementing CRLloss. Here I have one question. Now I'm dealing with one training dataset, which have multiple majority and minority classes. So how to compute the omega in CRLloss? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://gist.github.com/f3ba1ea30e4e21c24617f6d7aec75212#gistcomment-3462729>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJTKJWWYD4DLTTP2CCIZQKDSHBMZPANCNFSM4OQPT3KA> .

@amqdn Got it 😊, thanks for your explanation. ❤️