Request: JSON schema for promptfooconfig.yaml

[jamesbraza]

It would be cool to publish a JSON schema for promptfooconfig.yaml. Then within an IDE one can:

• Hover over fields and get a description of them
• Rely on IDE warnings to iterate on mistakes in a promptfooconfig.yaml

For example PyCharm natively supports JSON schema integrations: https://www.jetbrains.com/guide/dotnet/tips/json-schema/

Then for bonus points the JSON schema could be registered in the JSON Schema Store: https://www.schemastore.org/json/ --> https://github.com/SchemaStore/schemastore/tree/master/src/schemas/json

---

For reference, here's how I personally make JSON schemas:

1. Express schema as Pydantic BaseModel
2. Export a JSON schema via BaseModel.model_json_schema()
3. Add the JSON schema to PyCharm for the file name (e.g. promptfooconfig.yaml)