New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: integrate weave with evaluation pipeline and experiments with finetuning #75

Open

ayulockin wants to merge 6 commits into main from weave-eval

Member

ayulockin commented May 9, 2024

Adds W&B Weave Evaluation in our evaluation pipeline.
Adds LiteLLM to easily configure embedding models.
Use HF Inference Endpoint to host embedding models and use it in the pipeline.
Generate dataset for embedding finetuning (synthetic)
Evaluation of performance with OpenAI embedding, OS embedding and fine-tuned OS embedding
End-to-end evaluation with new embeddings with weave eval.

ayulockin added 6 commits

May 7, 2024 19:10


          chore: remove extra logger.info

0464fb5


          instrument weave eval

7a0c4f0


          chore: improve for better UI rendering

90c5c10


          add weave Model as evaluator

17884a7


          add litellm embedding and evaluated

a14d1d3


          nits

9320a9c

socket-security bot commented May 13, 2024

Removed dependencies detected. Learn more about Socket for GitHub ↗︎

🚮 Removed packages: pypi/appnope@0.1.4, pypi/appnope@0.1.4, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi-bindings@21.2.0, pypi/argon2-cffi@23.1.0, pypi/argon2-cffi@23.1.0, pypi/arrow@1.3.0, pypi/arrow@1.3.0, pypi/async-lru@2.0.4, pypi/async-lru@2.0.4, pypi/babel@2.15.0, pypi/babel@2.15.0, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/debugpy@1.8.1, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/faiss-cpu@1.8.0, pypi/fastjsonschema@2.19.1, pypi/fastjsonschema@2.19.1, pypi/flask-cors@4.0.1, pypi/flask-cors@4.0.1, pypi/fqdn@1.5.1, pypi/fqdn@1.5.1, pypi/ipynbname@2024.1.0.0, pypi/ipynbname@2024.1.0.0, pypi/isoduration@20.11.0, pypi/isoduration@20.11.0, pypi/itsdangerous@2.2.0, pypi/itsdangerous@2.2.0, pypi/jedi@0.19.1, pypi/jedi@0.19.1, pypi/json5@0.9.25, pypi/json5@0.9.25, pypi/jsonschema-specifications@2023.12.1, pypi/jsonschema-specifications@2023.12.1, pypi/jupyter-client@8.6.1, pypi/jupyter-client@8.6.1, pypi/jupyter-core@5.7.2, pypi/jupyter-core@5.7.2, pypi/jupyter-events@0.10.0, pypi/jupyter-events@0.10.0, pypi/jupyter-lsp@2.2.5, pypi/jupyter-lsp@2.2.5, pypi/jupyter-server-terminals@0.5.3, pypi/jupyter-server-terminals@0.5.3, pypi/jupyter-server@2.14.0, pypi/jupyter-server@2.14.0, pypi/jupyterlab-pygments@0.3.0, pypi/jupyterlab-pygments@0.3.0, pypi/jupyterlab-server@2.27.1, pypi/jupyterlab-server@2.27.1, pypi/jupyterlab@4.2.0, pypi/jupyterlab@4.2.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/llvmlite@0.42.0, pypi/mistune@3.0.2, pypi/mistune@3.0.2, pypi/nbconvert@7.16.4, pypi/nbconvert@7.16.4, pypi/notebook-shim@0.2.4, pypi/notebook-shim@0.2.4, pypi/notebook@7.2.0rc0, pypi/notebook@7.2.0rc0, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/numba@0.59.1, pypi/pandocfilters@1.5.1, pypi/pandocfilters@1.5.1, pypi/parso@0.8.4, pypi/parso@0.8.4, pypi/pathspec@0.12.1, pypi/pathspec@0.12.1, pypi/pexpect@4.9.0, pypi/pexpect@4.9.0, pypi/pluggy@1.5.0, pypi/pluggy@1.5.0, pypi/prometheus-client@0.20.0, pypi/prometheus-client@0.20.0, pypi/prompt-toolkit@3.0.43, pypi/prompt-toolkit@3.0.43, pypi/ptyprocess@0.7.0, pypi/ptyprocess@0.7.0, pypi/pynndescent@0.5.12, pypi/pynndescent@0.5.12, pypi/pywinpty@2.0.13, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/pyzmq@26.0.3, pypi/referencing@0.35.1, pypi/referencing@0.35.1, pypi/rfc3339-validator@0.1.4, pypi/rfc3339-validator@0.1.4, pypi/rfc3986-validator@0.1.1, pypi/rfc3986-validator@0.1.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/rpds-py@0.18.1, pypi/send2trash@1.8.3, pypi/send2trash@1.8.3, pypi/terminado@0.18.1, pypi/terminado@0.18.1, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/tornado@6.4, pypi/types-python-dateutil@2.9.0.20240316, pypi/types-python-dateutil@2.9.0.20240316, pypi/umap-learn@0.5.6, pypi/umap-learn@0.5.6, pypi/uri-template@1.3.0, pypi/uri-template@1.3.0, pypi/webcolors@1.13, pypi/webcolors@1.13

View full report↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment