pol_emo_mDeBERTa2: Discrete-Emotional-Classifier

DOWNLOAD

Start by downloading v1.0.0 on the right side under "Releases". Please download the zip-file which contains all necessary files to apply the fine-tuned transformer model.

Model description

The model is based on the data described in the following paper:

Widmann, Tobias, and Maximilian Wich. 2022. "Creating and Comparing Dictionary, Word Embedding, and Transformer-Based Models to Measure Discrete Emotions in German Political Text." Political Analysis, June, 1--16. https://doi.org/10.1017/pan.2022.15.

However, different to the model described in the paper, pol_emo_mDeBERTa2 has been trained (80%) and tested (20%) on the whole set of crowd-coded data: 19620 sentences from political communication. Furthermore, the base model used is a multilingual mDeBERTa trained on data from 100 languages (mDeBERTa-v3-base from Microsoft). Thus, in comparison to the model presented in the paper, the fine-tuned model pol_emo_mDeBERTa2, can be applied to more languages than German.

You can learn more about the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing in their research paper. You can learn more about the CC100 dataset that has been used to train the base model here.

The model classifies text into to the following eight emotions: anger, fear, disgust, sadness, joy, enthusiasm, pride, and hope.

Python code is provided below.

Eval results

The model was evaluated using 20% of the test dataset described in the research paper. The metrics described below are macro precision, recall, and F1 scores.

Emotion	Precision	Recall	F1 Score
anger	0.782	0.775	0.778
fear	0.725	0.664	0.689
disgust	0.697	0.768	0.727
sadness	0.776	0.72	0.744
joy	0.819	0.84	0.829
enthusiasm	0.691	0.693	0.692
pride	0.79	0.666	0.71
hope	0.647	0.603	0.62
Macro Avg	0.741	0.716	0.724

In addition, to test its applicability to other languages than German, I applied the model to more than 1,200 test sentences in English, Spanish, and French. The sentences have been created by an generative AI model, resembling stereotypical statements from political communication (with and without discrete emotional appeals). The performance metrics for this exercise can be seen below. As indicated, the model achieves high performance in languages other than German:

Emotion	Precision	Recall	F1 Score
anger	0.837	0.842	0.84
fear	0.866	0.803	0.831
disgust	0.92	0.604	0.657
sadness	0.876	0.88	0.878
joy	0.863	0.963	0.906
enthusiasm	0.782	0.622	0.663
pride	0.95	0.852	0.894
hope	0.91	0.707	0.768
Macro Avg	0.875	0.784	0.805

Citation

Please always cite the following paper when using the model:

Widmann, Tobias, and Maximilian Wich. 2022. "Creating and Comparing Dictionary, Word Embedding, and Transformer-Based Models to Measure Discrete Emotions in German Political Text." Political Analysis, June, 1--16. https://doi.org/10.1017/pan.2022.15.

Ideas for cooperation or questions?

If you have questions or ideas for cooperation, contact me at widmann{at}ps{dot}au{dot}dk

Model Use

# torch libraries
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

# pytorch libraries
import pytorch_lightning as pl
from torchmetrics import F1Score
from torchmetrics.functional import accuracy, auroc #F1Score #f1
from pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping
from pytorch_lightning.loggers import TensorBoardLogger

# transformers libraries
from transformers import AutoTokenizer, DebertaV2Model, AdamW, get_linear_schedule_with_warmup

import tqdm

import pandas as pd

#set working directory to the downloaded folder
import os
os.chdir('/path/to/your/directory')

#define functions necessary to apply mDeBERTa model
LABEL_COLUMNS = ['anger_v2', 'fear_v2', 'disgust_v2', 'sadness_v2', 'joy_v2', 'enthusiasm_v2', 'pride_v2', 'hope_v2']
BASE_MODEL_NAME = "microsoft/mdeberta-v3-base"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME)
batch_size = 8
device = "cuda" if torch.cuda.is_available() else "cpu"


class CrowdCodedTagger(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.bert = DebertaV2Model.from_pretrained(BASE_MODEL_NAME, return_dict=True)
    self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)
    self.n_training_steps = n_training_steps
    self.n_warmup_steps = n_warmup_steps
    self.criterion = nn.BCELoss()

  def forward(self, input_ids, attention_mask, labels=None, token_type_ids=None):
    output = self.bert(input_ids, attention_mask=attention_mask)
    output = self.classifier(output.last_hidden_state[:, 0])
    output = torch.sigmoid(output)
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output

  def training_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("train_loss", loss, prog_bar=True, logger=True)
    return {"loss": loss, "predictions": outputs, "labels": labels}

  def validation_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("val_loss", loss, prog_bar=True, logger=True)
    return loss

  def test_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("test_loss", loss, prog_bar=True, logger=True)
    return loss

  def training_epoch_end(self, outputs):

    labels = []
    predictions = []
    for output in outputs:
      for out_labels in output["labels"].detach().cpu():
        labels.append(out_labels)
      for out_predictions in output["predictions"].detach().cpu():
        predictions.append(out_predictions)

    labels = torch.stack(labels).int()
    predictions = torch.stack(predictions)

    for i, name in enumerate(LABEL_COLUMNS):
      class_roc_auc = auroc(predictions[:, i], labels[:, i])
      self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)

  def configure_optimizers(self):

    optimizer = AdamW(self.parameters(), lr=2e-5) #DEFINING LEARNING RATE

    scheduler = get_linear_schedule_with_warmup(
      optimizer,
      num_warmup_steps=self.n_warmup_steps,
      num_training_steps=self.n_training_steps
    )

    return dict(
      optimizer=optimizer,
      lr_scheduler=dict(
        scheduler=scheduler,
        interval='step'
      )
    )

# Define function for inference
def predict_labels(df):
    input_text = df['sentence'].tolist()
    num_inputs = len(input_text)
    num_batches = (num_inputs - 1) // batch_size + 1
    batched_input = [input_text[i * batch_size:(i + 1) * batch_size] for i in range(num_batches)]
    batched_output = []

    for i, batch in enumerate(tqdm.tqdm(batched_input)):
        encoded_input = tokenizer(batch, padding=True, truncation=True, max_length=512, return_tensors='pt')
        outputs = model(**encoded_input.to(device))

        # Extract the decimal numbers from the tensor
        tensor_values = outputs[1].tolist()
        decimal_numbers = [[num for num in sublist] for sublist in tensor_values]

        # Create Pandas DataFrame
        output_df = pd.DataFrame(decimal_numbers, columns=LABEL_COLUMNS)

        # Apply threshold function to DataFrame
        threshold = 0.65
        threshold_fn = lambda x: 1 if x >= threshold else 0
        output_df = output_df.applymap(threshold_fn)

        # Concatenate input DataFrame with output DataFrame
        input_df = df.iloc[i * batch_size:(i + 1) * batch_size].reset_index(drop=True)
        output_df = pd.concat([input_df, output_df], axis=1)

        batched_output.append(output_df)


    # Concatenate all batches into a single output DataFrame
    output_df = pd.concat(batched_output, ignore_index=True)

    return output_df

#provide an input dataframe, for example by loading in a csv file
#by default, the column in the dataframe including sentences to be classified should be called "sentence" (can be adjusted above)
input_df = pd.read_csv("./example_data.csv")
input_df

# putting model into evaluation mode and load local fine-tuned model
model = CrowdCodedTagger(n_classes=8)
model.load_state_dict(torch.load("./model/pytorch_model.pt"), strict = False)
model.to(device)
model.eval() 

#apply function to the input_df
results = predict_labels(input_df)
print(results)

#save results as a csv file in the working directory
result.to_csv('example_results.csv', index=False)