0.61.0

What's Changed

• feat: moderation assert type by @typpo in #821
• feat: general purpose http/https provider by @typpo in #822
• feat: add portkey provider by @typpo in #819
• feat: Add Cloudflare AI Provider by @jacobdr in #817
• fix: Remove duplicate logging line by @efung in #825
• fix: The ‘defaultTest’ option has no effect during evaluation. by @big-camel in #829
• fix: Improve Error Handling in Python Script Execution by @mldangelo in #833
• docs: How to red team LLMs by @typpo in #828
• chore(mistral): add codestral by @typpo in #831

New Contributors

• @big-camel made their first contribution in #829
• @jacobdr made their first contribution in #817
• @mldangelo made their first contribution in #833

Full Changelog: 0.60.0...0.61.0

0.60.0

What's Changed

• feat: redteam testset generation by @typpo in #804
• feat: support for deep equality check in equals assertion by @mikanfactory in #805
• feat(webui): Add image viewer by @typpo in #816
• feat: Allow functions in renderVarsInObject by @tomquist in #813
• feat: ability to reference previous llm outputs via storeOutputAs by @typpo in #808
• feat: support for prompt objects by @typpo in #818
• fix: huggingface api key handling by @typpo in #809
• docs: Restore ProviderResponse class name by @alexclaydon in #806
• docs: Fix typo in local build command by @chrismaltais in #811

New Contributors

• @mikanfactory made their first contribution in #805
• @alexclaydon made their first contribution in #806
• @chrismaltais made their first contribution in #811
• @tomquist made their first contribution in #813

Full Changelog: 0.59.1...0.60.0

0.59.1

What's Changed

• fix: handle null result timestamp when writing to db. #800

See also: 0.59.0 release notes

Full Changelog: 0.59.0...0.59.1

0.59.0

What's Changed

• fix: python prompts break when using whole file by @typpo in #784
• feat(webui): add --filter-description option to promptfoo view by @typpo in #780
• Langfuse need to compile variables by @albertpurnama in #779
• chore(webui): display prompt and completion tokens by @typpo in #794
• chore: include full error response in openai errors by @typpo in #791
• chore: add logprobs to assertion context by @typpo in #790
• feat: support var interpolation in function calls by @typpo in #792
• chore: add timestamp to EvaluateSummary by @typpo in #785
• fix: render markdown in variables too by @typpo in #796
• feat(bedrock): add support for embeddings models by @typpo in #797
• fix(vertex): remove leftover dependency on apiKey by @typpo in #798

Full Changelog: 0.58.1...0.59.0

0.58.1

What's Changed

• fix: improve GradingResult validation by @typpo in #772
• fix(langfuse): Check runtime type of getPrompt, stringify the result by @albertpurnama in #774
• fix: update python ProviderResponse error message and docs. #769
• chore(openai): add gpt-4o models (https://github.com/promptfoo/promptfoo/pull/776[)](https://github.com/promptfoo/promptfoo/commit/ff4655d31d3588972522bb162733cb61e460f36f)

See also: 0.58.0 release notes

New Contributors

• @albertpurnama made their first contribution in #774

Full Changelog: 0.58.0...0.58.1

0.58.0

Breaking

rouge-type assertions no longer support multiple reference strings. This is due to an update to the underlying rouge package. To check multiple strings, break them into separate assertions.

What's Changed

• feat: assert-set by @mikkoh in #765
• feat: add comma-delimited string support for array-type assertion values by @typpo in #755
• fix: Resolve JS assertion paths relative to configuration file by @Arkham in #756
• fix: not-equals assertion by @EKranjec in #763
• fix: upgrade rouge package and limit to strings by @typpo in #764

New Contributors

• @Arkham made their first contribution in #756
• @EKranjec made their first contribution in #763

Full Changelog: 0.57.1...0.58.0

0.57.1

What's Changed

• fix: do not serialize js objects to non-js providers by @typpo in #754
• See 0.57.0 release notes

Full Changelog: 0.57.0...0.57.1

0.57.0

Breaking

The eval --first-n option has been renamed to eval --filter-first-n to match other new filtering options.

What's Changed

• feat: ability to override provider per test case by @typpo in #725
• feat: eval tests matching pattern by @mikkoh in #735
• feat: add -n limit arg for promptfoo list by @typpo in #749
• feat: promptfoo import and promptfoo export commands by @typpo in #750
• feat: add support for --var name=value cli option by @typpo in #745
• feat: promptfoo eval --filter-failing outputFile.json by @mikkoh in #742
• fix: eval --first-n arg by @typpo in #734
• chore: Update openai package to 3.48.5 by @matteodepalo in #739
• chore: include logger and cache utils in javascript provider context by @typpo in #748
• chore: add PROMPTFOO_FAILED_TEST_EXIT_CODE envar by @typpo in #751
• docs: Document python: prefix when loading assertions in CSV by @efung in #731
• docs: update README.md by @eltociear in #733
• docs: Fixes to Python docs by @jamesbraza in #728
• docs: Update to include --filter-* cli args by @mikkoh in #747

New Contributors

• @efung made their first contribution in #731
• @eltociear made their first contribution in #733
• @mikkoh made their first contribution in #735
• @matteodepalo made their first contribution in #739

Full Changelog: 0.56.0...0.57.0

0.56.0

What's Changed

• feat: Intergration with Langfuse by @tam0201 in #707
• feat(webui): improved comment dialog by @typpo in #713
• feat: Support IBM Research BAM provider by @abratnap in #711
• fix: Make errors uncached in Python completion. by @grahl in #706
• fix(vertex/gemini): support nested generationConfig by @typpo in #714
• fix: include python tracebacks in python errors by @typpo in #724
• fix: getCache should return a memory store when disk caching is disabled by @typpo in #715
• chore(webui): improve eval view performance by @typpo in #719
• chore(webui): always show provider in header by @typpo in #721
• chore: add support for OPENAI_BASE_URL envar by @typpo in #717

New Contributors

• @grahl made their first contribution in #706
• @tam0201 made their first contribution in #707
• @abratnap made their first contribution in #711

Full Changelog: 0.55.0...0.56.0

0.55.0

What's Changed

• [Docs] Add llama3 example to ollama docs by @chanonroy in #695
• bugfix in answer-relevance by @alexandres in #697
• feat: add support for provider transform property by @typpo in #696
• feat: add support for provider-specific delays by @typpo in #699
• feat: portkey.ai integration by @typpo in #698
• feat: eval -n arg for running the first n test cases by @typpo in #700
• feat: ability to write outputs to google sheet by @typpo in #701
• feat: first-class support for openrouter by @typpo in #702
• Fix concurrent cache request behaviour by @chrisprice in #703

New Contributors

• @chanonroy made their first contribution in #695
• @alexandres made their first contribution in #697
• @chrisprice made their first contribution in #703

Full Changelog: 0.54.1...0.55.0