[19/02/2024 - 26/02/2024] New Models/Endpoints/Providers/Improvements

[ishaan-jaff]

Guardrails

• Support llm guard - [Feature]: Support llm guard on proxy  #1960
• Support output moderations #2085
• allow user to turn on / off content moderation #2084
• Allow controlling pre-call-hook + moderation hooks for llm guard #2083
• content moderation streaming support #2082
• Azure content safety integration #2071
• Support nemo guardrails on proxy #2070
• AWS Content Moderation support [Feature]: AWS Comprehend Support (text moderation)  #2036
• Docs for custom instructions for Llama Guard - [Feature]: Enable custom instructions for Llama Guard  #2044
• Google Text Moderation docs #2069
• Admin review on repeated content moderation violations #2068
• Block keys with repeated content moderation violations #2067
• Support Presidio PII Redaction for Images #2076
• add llama guard on anyscale - https://docs.endpoints.anyscale.com/supported-models/Meta-Llama-Llama-Guard-7b/
• enable ad-hoc recognizers for presidio - [Feature]: Enable ad-hoc recognizers for presidio  #2099
• ban competitor / keywords #2121
• caching support for moderation calls #2122
• endpoints for blocking/unblocking users [Feature]: add endpoints for blocking / unblocking user id's  #2167
• Make blocking/unblocking users work across multiple workers #2182

Admin UI / Spend Tracking / Budget Management

• Admin UI - allow admin to create teams on proxy #2181
• Admin UI -allow multiple admins to login with sso #2185
• Admin UI - allow app owners to request access to an LLM [Feature]: Admin UI - allow app owners to request access to an LLM  #2057
• admin UI - for non admins the UI should show all available models and for each if the user has access to the model or no [Feature]: admin UI - for non admins the UI should show all available models and for each if the user has access to the model or not  #2062
• Admin UI - allow the admin to approve or reject requests for a model #2161
• Admin UI SSO Keycloak [Bug]: Generic SSO user attributes #2130
• Admin UI - Set a limited list in the models field of a user #2162
• Tags Block List - block Requests if a tag is in a block list #2136
• Log key alias as part of langfuse key metadata #2137
• allow users to set tpm / rpm limit when creating keys [Feature]: Admin UI - allow users to set tpm / rpm limit when creating keys  #2064 [Feature]: Admin UI - allow users to set tpm / rpm limit when creating keys  #2064
• Allow budget manager to work with proxy [Feature]: enable budget manager to work with proxy  #1976
• Max budget not showing in UI from Azure but displays in localhost Max budget not showing in UI from Azure but displays in localhost #2051
• Table Names should respect standard naming: [Bug]: Allow users to set custom table name #1972
• UI - Allow users to use chat playground : [Feature]: Admin UI - add a playground so proxy users can test / experiment with each model  #1941
• [BUG] Model Spend - on team it's different #2095
• implement /team/update and /team/delete - allow user to add new members #2096
• update team table schema for member roles #2097
• Admin UI - allow Admin to view users, create new users #2117
• Admin UI - allow a user to add new models through the UI [Feature]: Admin UI - allow a user to add new models through the UI  #2059
• Unexpected Model Deletion in POST /key/update When Updating Team ID [Bug]: Unexpected Model Deletion in POST /key/update When Updating Team ID #2091
• Manage rate limits on the team level #2112
• Make sure TPM / RPM rate limiting on virtual keys works with Redis Cache #2113
• Bug Fix - Admin UI - not working when user sets UI Username/UI Password #2116
• Make sure rate limits work for either tpm/rpm (don't require both to be set) #2153
• model based tpm / rpm limits #2156
• make sure tpm / rpm manager works with redis #2155
• tpm/rpm limit alerting #2154
• allow editing limits on ui #2152
• better e2e testing for tpm/rpm limits #2151

Misc

• fix router timeouts + acompletion fallbacks - [Bug]: Fallbacks don't work with acompletion #1959
• reuse vertex ai client (speed improvement) - [Feature]: reuse vertex_ai client #1938
• Deleting a key on one LiteLLM instance, but the key works on the 2nd instance #2149
• Cohere Aya Support - [Feature]: New Model - Cohere AYA #1985
• Google Gemini 1.5 support - [Feature]: DAY 1 SUPPORT - Gemini 1.5  #1982
• Support claude on vertex ai - [Feature]: New Model: Support claude on vertex ai  #1745
• [Bug]: AWS Bedrock: Error with OpenAI Client, working with direct Litellm API #1981
• [Bug]: fix streaming cached response logging #2074
• [Bug]: drop_params not working for vertex ai gemini pro  #2078
• Model Name Syncing for Eval and Completion #2079
• [Bug]: Invokation not found in SageMaker since region is hard-coded #2086
• Add support for caching /moderations #2126
• [Docs] Update Budget Manager point to LiteLLM Proxy #2094
• don't log master api key to langfuse - [Bug]: LITELLM_MASTER_KEY leaked to Langfuse #2114
• don't use http proxy for internal api's - [Bug]: Support no_proxy env to use openai compatible api from internal behind corporate firewall #2127