[Bug]: Specifying a seed dramatically slows down performance

[Ben-Epstein]

moved from GoogleCloudPlatform/generative-ai#428

I'm not really sure this is the right place but i was told to move this here. We do have a service contract with google but have yet to get a response for this issue.

See also https://issuetracker.google.com/issues/327699486

Contact Details
ben.epstein@evolutioniq.com

What happened?
You can't directly set a seed in the python SDK (see #3186), but looking through the code im doing this, which works fine.

But you can see the performance slows dramatically.

import vertexai
from vertexai.language_models import TextGenerationModel

model = TextGenerationModel.from_pretrained("text-bison@002")
prompt = "tell me a story and make it very creative"

def predict(prompt, seed=None):
kwargs = {"maxDecodeSteps": 1024, "temperature": 0.0, "topP": 1, "topK": 40}
if seed is not None:
kwargs["seed"] = seed
instance = {"content": prompt}
raw_pred = model._endpoint.predict(instances=[instance], parameters=kwargs)
response = vertexai.language_models._language_models._parse_text_generation_model_multi_candidate_response(raw_pred)
return response.text.strip()

%timeit predict(prompt, seed=None); # 3.96 s ± 531 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit predict(prompt, seed=42); # 7.42 s ± 797 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Also, setting a seed isn't guaranteeing determinism. I ran this 500 times, both with and without a seed

without a seed: 52 unique responses
with a seed: 7 unique responses
Relevant log output
No response

Code of Conduct
[X] I agree to follow this project's Code of Conduct