InjectionTester is a framework for iteratively testing
- InjectionTester uses Pantry as a backend to dynamically run LLMs. You can find it here. You'll need version 0.0.2 or later.
cargo install injection-tester- You'll need to run Pantry, the same way you'd run Docker to host containers.
- To run against OpenAI targets you'll have to set Pantry's OpenAI key,
which can be done in the UI or by setting
OPENAI_API_KEYin Pantry's context. - The CLI will run all public and private configurations by default. You can use CLI flags to have it run only private configurations, or only named configurations. Private configurations are automatically detected by rust but ignored by git.
Very early alpha.
I'd like to include a broader set of LLMs and build out a larger number of vulnerabilities. If you use the framework to test out mitigation strategies.
Pantry currently saves your local LLM sessions so you can resume them later. This is a great features, except that your average LLM session is ~1.6GB (this is because rustformers/llm serializes the entire model state), and each run of InjectionTester will make potentially a double digit number of them.
You'll want to check ~/Library/Application Support/com.jmerz.pantry and clean out old sessions, stored in the folders
beginning with llmrs- (don't worry about deleting LLMs, they're stores separately so external programs can access them).