modules.losses

Losses module contains implementations for various losses used generally in vision and language space. One can register custom losses to be detected by MMF using the following example.

from mmf.common.registry import registry
from torch import nn


@registry.register_loss("custom")
class CustomLoss(nn.Module):
    ...

Then in your model’s config you can specify losses attribute to use this loss in the following way:

model_config:
    some_model:
        losses:
            - type: custom
            - params: {}

class mmf.modules.losses.AttentionSupervisionLoss

Loss for attention supervision. Used in case you want to make attentions similar to some particular values.

forward(sample_list, model_output)

Calculates and returns the multi loss.

Parameters

• sample_list (SampleList) – SampleList containing targets attribute.

• model_output (Dict) – Model output containing scores attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.BCEAndKLLoss(weight_softmax)

binary_cross_entropy_with_logits and kl divergence loss. Calculates both losses and returns a dict with string keys. Similar to bce_kl_combined, but returns both losses.

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.BinaryCrossEntropyLoss

forward(sample_list, model_output)

Calculates and returns the binary cross entropy.

Parameters

• sample_list (SampleList) – SampleList containing targets attribute.

• model_output (Dict) – Model output containing scores attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.CaptionCrossEntropyLoss

forward(sample_list, model_output)

Calculates and returns the cross entropy loss for captions.

Parameters

• sample_list (SampleList) – SampleList containing targets attribute.

• model_output (Dict) – Model output containing scores attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.CombinedLoss(weight_softmax)

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.ContrastiveLoss

This is a generic contrastive loss typically used for pretraining. No modality assumptions are made here.

forward(sample_list: Dict[str, torch.Tensor], model_output: Dict[str, torch.Tensor])

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.CosineEmbeddingLoss

Cosine embedding loss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.CrossEntropyLoss(**params)

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.InBatchHinge(margin: float = 0.0, hard: bool = False)

Based on the code from https://github.com/fartashf/vsepp/blob/master/model.py

forward(sample_list: Dict[str, torch.Tensor], model_output: Dict[str, torch.Tensor])

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.LabelSmoothingCrossEntropyLoss(label_smoothing=0.1, reduction='mean', ignore_index=- 100)

Cross-entropy loss with label smoothing. If label_smoothing = 0, then it’s canonical cross entropy. The smoothed one-hot encoding is 1 - label_smoothing for true label and label_smoothing / (num_classes - 1) for the rest.

Reference: https://stackoverflow.com/questions/55681502/label-smoothing-in-pytorch

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.LogitBinaryCrossEntropy

Returns Binary Cross Entropy for logits.

Attention

Key: logit_bce

forward(sample_list, model_output)

Calculates and returns the binary cross entropy for logits

Parameters

• sample_list (SampleList) – SampleList containing targets attribute.

• model_output (Dict) – Model output containing scores attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.LossConfig(type: str = '???', params: Dict[str, Any] = '???')

class mmf.modules.losses.Losses(loss_list: List[Union[str, mmf.modules.losses.LossConfig]])

Losses acts as an abstraction for instantiating and calculating losses. BaseModel instantiates this class based on the losses attribute in the model’s configuration model_config. loss_list needs to be a list for each separate loss containing type and params attributes.

Parameters

loss_list (ListConfig) – Description of parameter loss_list.

Example:

# losses:
# - type: logit_bce
# Can also contain `params` to specify that particular loss's init params
# - type: combined
config = [{"type": "logit_bce"}, {"type": "combined"}]
losses = Losses(config)

Note

Since, Losses is instantiated in the BaseModel, normal end user mostly doesn’t need to use this class.

losses

List containing instantiations of each loss passed in config

forward(sample_list: Dict[str, torch.Tensor], model_output: Dict[str, torch.Tensor])

Takes in the original SampleList returned from DataLoader and model_output returned from the model and returned a Dict containing loss for each of the losses in losses.

Parameters

• sample_list (SampleList) – SampleList given be the dataloader.

• model_output (Dict) – Dict returned from model as output.

Returns

Dictionary containing loss value for each of the loss.

Return type

Dict

class mmf.modules.losses.M4CDecodingBCEWithMaskLoss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.MMFLoss(params=None)

Internal MMF helper and wrapper class for all Loss classes. It makes sure that the value returned from a Loss class is a dict and contain proper dataset type in keys, so that it is easy to figure out which one is the val loss and which one is train loss.

For example: it will return {"val/vqa2/logit_bce": 27.4}, in case logit_bce is used and SampleList is from val set of dataset vqa2.

Parameters

params (type) – Description of parameter params.

Note

Since, MMFLoss is used by the Losses class, end user doesn’t need to worry about it.

forward(sample_list: Dict[str, torch.Tensor], model_output: Dict[str, torch.Tensor])

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.MSELoss

Mean Squared Error loss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.MSLoss(alpha=50, beta=2, margin=0.5, hard_mining=True, is_multilabel=False)

A Multi-Similarity loss between embeddings of similar and dissimilar labels is implemented here.

Reference:

“Multi-similarity loss with general pair weighting for deep metric learning”

Parameters

• alpha – parameters used in loss function calculation

• beta – parameters used in loss function calculation

• margin – parameters used in loss function calculation

• hard_mining – if true, select only the hardest examples (defined based on margin)

• is_multilabel – True if there are more than two labels, false otherwise

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.MultiLoss(params)

A loss for combining multiple losses with weights.

Parameters

params (List(Dict)) – A list containing parameters for each different loss and their weights.

Example:

# MultiLoss works with config like below where each loss's params and
# weights are defined
losses:
- type: multi
  params:
  - type: logit_bce
    weight: 0.3
    params: {}
  - type: attention_supervision
    weight: 0.7
    params: {}

forward(sample_list, model_output, *args, **kwargs)

Calculates and returns the multi loss.

Parameters

• sample_list (SampleList) – SampleList containing attentions attribute.

• model_output (Dict) – Model output containing attention_supervision attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.NLLLoss

Negative log likelikehood loss.

forward(sample_list, model_output)

Calculates and returns the negative log likelihood.

Parameters

• sample_list (SampleList) – SampleList containing targets attribute.

• model_output (Dict) – Model output containing scores attribute.

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.RefinerContrastiveLoss(sim_thresh=0.1, epsilon=1e-16)

A contrastive loss between the decoder outputs of a given embedding size and its targets

This loss can be used in lieu of a reconstruction loss, wherein the goal is to get a decoded signal closer to its target than other targets. As long as the reconstructed signal of a given input is closer to its target than any other target, the loss will remain zero.

Reference:

Sankaran, S., Yang, D. and Lim, S.N., “Multimodal Fusion Refiner Networks”

Parameters

• sim_thresh – similarity threshold used to consider only samples beyond

• threshold (# this) –

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.RefinerMSLoss(alpha: float = 50, beta: float = 2, base: float = 0.5, margin: float = 0.1, epsilon: float = 1e-16)

A Multi-Similarity loss between the decoder outputs of a given embedding size and its targets

This loss pulls the decoded signal of a sample closer to its target, while simultaneously pushing it away from other targets

References:

1) Wang et al., Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning, CVPR 2019 2) Sankaran, S., Yang, D. and Lim, S.N., “Multimodal Fusion Refiner Networks”

Parameters

ms_loss (same as) –

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.SoftLabelCrossEntropyLoss(ignore_index=- 100, reduction='mean', normalize_targets=True)

compute_loss(targets, scores)

for N examples and C classes - scores: N x C these are raw outputs (without softmax/sigmoid) - targets: N x C or N corresponding targets

Target elements set to ignore_index contribute 0 loss.

Samples where all entries are ignore_index do not contribute to the loss reduction.

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.SoftmaxKlDivLoss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.TripleLogitBinaryCrossEntropy

This is used for Three-branch fusion only. We predict scores and compute cross entropy loss for each of branches.

forward(sample_list, model_output)

Calculates and returns the binary cross entropy for logits :param sample_list: SampleList containing targets attribute. :type sample_list: SampleList :param model_output: Model output containing scores attribute. :type model_output: Dict

Returns

Float value for loss.

Return type

torch.FloatTensor

class mmf.modules.losses.WeightedSoftmaxLoss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmf.modules.losses.WrongLoss

forward(sample_list, model_output)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.