How does Temperature fallback with beam search work?

[tom-huntington]

whisper/whisper/transcribe.py

Lines 102 to 128 in 02aa851

	def decode_with_fallback(segment: torch.Tensor) -> DecodingResult:
	temperatures = [temperature] if isinstance(temperature, (int, float)) else temperature
	decode_result = None
	for t in temperatures:
	kwargs = {**decode_options}
	if t > 0:
	# disable beam_size and patience when t > 0
	kwargs.pop("beam_size", None)
	kwargs.pop("patience", None)
	else:
	# disable best_of when t == 0
	kwargs.pop("best_of", None)
	options = DecodingOptions(**kwargs, temperature=t)
	decode_result = model.decode(segment, options)
	needs_fallback = False
	if compression_ratio_threshold is not None and decode_result.compression_ratio > compression_ratio_threshold:
	needs_fallback = True # too repetitive
	if logprob_threshold is not None and decode_result.avg_logprob < logprob_threshold:
	needs_fallback = True # average log probability is too low
	if not needs_fallback:
	break
	return decode_result

If we need_fallback, then we try again with the next temperature value

temperature: Union[float, Tuple[float, ...]] = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),

But the temprature values only seems to affect the GreadyDecoder, not the BeamSearchDecoder.

Does the fallback just do exactly the same thing again when decoding with beam search?

[tom-huntington]

--beam_size BEAM_SIZE
number of beams in beam search, only applicable when temperature is zero (default: 5)

So the fallback must always go to greedy decoding.

[jongwook]

Both beam search and greedy decoding are deterministic algorithms and make sense only with temperature 0. With nonzero temperature, the implementation becomes nondeterministic and uses the best_of parameter, which defaults to 5 in the CLI, where it makes best_of independent samples and select the one with the highest log probability.

[tom-huntington]

Thanks for clearing this up! The code isn't written to make this clear

whisper/whisper/decoding.py

Lines 477 to 482 in eff383b

	if options.beam_size is not None:
	self.decoder = BeamSearchDecoder(
	options.beam_size, tokenizer.eot, self.inference, options.patience
	)
	else:
	self.decoder = GreedyDecoder(options.temperature, tokenizer.eot)

[jongwook]

Good point! I should consider renaming it to something like TemperatureSampler..

[tom-huntington]

It's doing both the greedy decoding and the indeterministic temperature decoding

[HAJEKEL]

@jongwook when you say "Both beam search and greedy decoding are deterministic algorithms and make sense only with temperature 0." I believe you are trying to demonstrate that the temperature does not affect the outcome of the algorithms right? Because for the example of beam search, you pick the n most likely logits, and these specific n logits will not change for a change in temperature.