Use test vars in function schema

[tomquist]

I'm trying to evaluate a prompt that uses function calls. However, the function schema depends on one of the variables in the test output. It looks like function calls currently are part of the provider config and not of the prompt, so I don't see a way to inject variables into a function definition. Is there any other way to adjust the function schema per test case?

[typpo]

Thanks for the suggestion @tomquist! #792 implements this and will be in the next release.

[typpo]

Released in 0.59.0!

[tomquist]

@typpo Awesome! Thanks for the quick implementation. 🎉

I have one use-case which still wouldn’t be possible with this implementation:

I want to pass in a JSON Array as a var so I can use it in a string schema as the enum property. Would it be possible to support a function at any level of the template object? This waz you could do something like this:

tools: [
  {
    type: …, parameters: (context) => ({ enum: context.vars.options, …})
  }
]

Sorry for the minimalistic example, I’m on mobile right now

I think this could just be handled as another case in renderVarsInObject().

Also I’m happy to open a new issue if you prefer.

[tomquist]

@typpo I submitted a PR to implement this: #813