增加Sentence-Bert论文讲解 #858
Open
增加Sentence-Bert论文讲解 #858
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
请参考： |
w5688414
reviewed
Feb 20, 2022
|
|
||
| 并且在对bert模型进行微调时，设置了三个目标函数，用于不同任务的训练优化，具体如下： | ||
|
|
||
| ### 2.1分类目标函数（Classification Objective Function） |
|
|
||
|  | ||
|
|
||
| ### 2.2回归目标函数（Regression Objective Function） |
|
|
||
|  | ||
|
|
||
| ### 2.3三元目标函数（Triplet Objective Function） |
| ## 1介绍 | ||
| [Bert模型](https://paddlepedia.readthedocs.io/en/latest/tutorials/pretrain_model/bert.html#)已经在NLP各大任务中都展现出了强者的姿态。在语义相似度计算（semantic textual similarity）任务上也不例外，但是，由于bert模型规定，在计算语义相似度时，需要将两个句子同时进入模型，进行信息交互，这造成大量的计算开销,使得它既不适合语义相似度搜索，也不适合非监督任务，比如聚类。 | ||
|
|
||
| 例如，有10000个句子，我们想要找出最相似的句子对，需要计算（10000*9999/2）次，需要大约65个小时。 |
|
|
||
| ### 2.3三元目标函数（Triplet Objective Function） | ||
|
|
||
| 在这个目标函数下，将模型框架进行修改，将原来的两个输入，变成三个句子输入。给定一个锚定句（anchor sentence）$a$，一个肯定句(positive sentence)$p$和一个否定句(negative sentence)$n$，模型通过使$a到p$的距离小于$a到n$的距离，来优化模型。使其目标函数$o$最小，即 |
| 其中，$s_{a}$、$s_{p}$和$s_{n}$分别表示句子$a$、$p$和$n$的向量，$||·||$表示距离度量，$\varepsilon$表示边距。在论文中，距离度量为欧式距离，边距大小为1。 | ||
|
|
||
| ### 2.4训练参数 | ||
| 模型训练过程中，批次大小为16，学习率为2e-5，采用Adam优化器进行优化，并且默认的池化策略为平均池化。 |
There was a problem hiding this comment.
Adam还是AdamW？现在的BERT的优化器应该都是AdamW
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
增加Sentence-Bert论文讲解