Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add LegalBench datasets - 11 #661

Merged

Conversation

awinml
Copy link
Contributor

@awinml awinml commented May 10, 2024

Checklist for adding MMTEB dataset

Reason for dataset addition: Showcases the model's performance on the LegalBench datasets, enabling evaluation on domain-specific legal texts.

Follow up to #657

Changes Made:

The following 10 datasets have been added:

  1. Ability To Consummate Concept Is Subject To MAE Carveouts
  2. Accuracy Of Fundamental Target RWS Bringdown Standard
  3. Accuracy Of Target Capitalization RW Outstanding Shares Bringdown Standard Answer
  4. Accuracy Of Target General RW Bringdown Timing Answer
  5. Additional Matching Rights Period For Modifications Cor
  6. Application Of Buyer Consent Requirement Negative Interim Covenant
  7. Buyer Consent Requirement Ordinary Course
  8. Change In Law Subject To Disproportionate Impact Modifier
  9. Changes In GAAP Or Other Accounting Principles Subject To Disproportionate Impact Modifier
  10. COR Permitted In Response To Intervening Event

The datasets have been included in #680, only points for them are being added here.


  • I have tested that the dataset runs with the mteb package.
  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.
  • I have added points for my submission to the points folder using the PR number as the filename (e.g. 438.jsonl).

@awinml awinml mentioned this pull request May 11, 2024
10 tasks
@awinml awinml mentioned this pull request May 15, 2024
10 tasks
@awinml
Copy link
Contributor Author

awinml commented May 15, 2024

These datasets have been included in MAUDLegalBenchClassification and added as part of #680. This PR just add points for the MAUD datasets that have been added.

@awinml
Copy link
Contributor Author

awinml commented May 16, 2024

@KennethEnevoldsen This PR is similar to #678 and ready to merge.

@KennethEnevoldsen KennethEnevoldsen merged commit c3acf03 into embeddings-benchmark:main May 21, 2024
6 checks passed
dokato pushed a commit to dokato/mteb that referenced this pull request May 24, 2024
* Add LegalBench datasets

* Add points

* Remove files added in embeddings-benchmark#680

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants