XLMRoberta embeddings not differentiating between different sentences

[robfuller7]

Is there an existing issue for this?

• I have searched the existing issues and did not find a match.

Who can help?

No response

What are you working on?

As part of a dissertation, I am attempting to use cosine similarity to compare embeddings in order to assess the quality of machine translated text. Specifically I want to compare pairs of Russian original and English translated sentences. I am trialling various models for this including XLMRoberta. I am running this in Google Colab.

Current Behavior

The difference in cosine similarity between a pair of sentences with a validated, good quality translation and a translation which has no relation to the original at all is currently negligible.

Expected Behavior

Cosine similarity for a good translation is high and low for a poor one.

Steps To Reproduce

https://colab.research.google.com/drive/10rtJmAkDgZLp5sVHsmeqMPZpKQ4NUkfu?usp=drive_link

Spark NLP version and Apache Spark

See notebook

Type of Spark Application

Python Application

Java Version

No response

Java Home Directory

No response

Setup and installation

See notebook

Operating System and Version

No response

Link to your project (if available)

No response

Additional Information

No response