Skip to content

Latest commit

 

History

History
31 lines (26 loc) · 1.28 KB

feature.md

File metadata and controls

31 lines (26 loc) · 1.28 KB

Feature

  • Support the openai chat completion normal and stream request
  • Get top_k similar search results, it can be set when creating the data manager
  • Support the cache chain, see: Cache#next_cache
bak_cache = Cache()
bak_cache.init()
cache.init(next_cache=bak_cache)
  • Whether to completely skip the current cache, that is, do not search the cache or save the Chat GPT results, see: Cache#cache_enable_func
  • In the cache initialization phase, no cache search is performed, but save the result returned by the chat gpt to cache, see: cache_skip=True in create request
openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=mock_messages,
    cache_skip=True,
)
  • Like Lego bricks, custom assemble all modules, including:
    • Adapter: The user interface to adapt different LLM model requests to the GPTCache protocol
    • Pre-processor: Extracts the key information from the request and preprocess
    • Context Buffer: Maintains session context
    • Encoder: Embed the text into a dense vector for similarity search
    • Cache manager: which includes searching, saving, or evicting data
    • Ranker: Evaluate similarity by judging the quality of cached answers
    • Post-processor: Determine which cached answers to the user, and generate the response