Configurable feedback level

[arn-tru]

Instead of providing rating for llm generated feedback between 0.1 to 1 in increments of .1, offer options to do it increments of 0.2 or 0.33

[joshreini1]

The prompting still needs to be changed in concert with the normalization. Now (for llm-based feedback) we prompt the LLM to score 0-10. If we go with fewer levels, that prompting should also change to 0-4, 0-2, etc.

[piotrm0]

The prompting still needs to be changed in concert with the normalization. Now (for llm-based feedback) we prompt the LLM to score 0-10. If we go with fewer levels, that prompting should also change to 0-4, 0-2, etc.

Perhaps the current PR could instead refer to score range then? Normalizing by something less than 10 with current prompts will produce scores with ranges outside of 0-1 but unsure if this was intention.

[piotrm0]

Figure out phrasing around feature, is it about the normalized score range being different than [0-1] or is it about the resolution of the prompted score (but still normalized to [0-1]). The latter requires the changes Josh mentioned.

[joshreini1]

Figure out phrasing around feature, is it about the normalized score range being different than [0-1] or is it about the resolution of the prompted score (but still normalized to [0-1]). The latter requires the changes Josh mentioned.

It's the latter - offering variable resolutions is the goal here. For many cases, clients have observed that LLMs are unable to properly distinguish 11 levels (0-10) of feedback and find that 3-5 levels work better. The goal of the PR is to support that workflow.