You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add an optional stop_regex parameter to GenerateParameters. Stop generating tokens if stop_regex matches the generated text.
Motivation
Many applications such as inline code completion require fine-grained control over generated text. For example, you may want to generate a full "block" of code in which case you may desire to stop generation once a "dedent" is matched. Another example is generating till a matching a closing brace, quote or parenthesis is generated by the model. This is sometimes hard to express using TGI's "stop sequences" feature.
Your contribution
I have a working small demo, I'm open to making a PR if needed.
A few notes:
Security and performance must be considered. Regexes from Python's re package can sometimes be very slow, especially in the presence of adversarial patterns. I think the best approach to this is to disable the feature by default (similar to how grammars work) and/or use a more restrictive Regex engine.
The text was updated successfully, but these errors were encountered:
Apologies, I didn't realise this was a duplicate of #1130. I see the suggested approach is to use streaming and drop the connection. Nonetheless, is it something that you would still consider?
Feature request
Add an optional
stop_regex
parameter toGenerateParameters
. Stop generating tokens ifstop_regex
matches the generated text.Motivation
Many applications such as inline code completion require fine-grained control over generated text. For example, you may want to generate a full "block" of code in which case you may desire to stop generation once a "dedent" is matched. Another example is generating till a matching a closing brace, quote or parenthesis is generated by the model. This is sometimes hard to express using TGI's "stop sequences" feature.
Your contribution
I have a working small demo, I'm open to making a PR if needed.
A few notes:
re
package can sometimes be very slow, especially in the presence of adversarial patterns. I think the best approach to this is to disable the feature by default (similar to how grammars work) and/or use a more restrictive Regex engine.The text was updated successfully, but these errors were encountered: