garak

What it is

A command-line scanner that sends sets of probes to a language model and checks the replies for failures. The name stands for Generative AI Red-teaming and Assessment Kit, and NVIDIA contributes to the project.

What it helps with

- Checking a model or chatbot for prompt injection, jailbreaks, and data leakage
- Flagging toxic, false, or made-up output
- Running every probe family or only the ones you pick, then reading a report of what held up and what failed

Who it's for

Security engineers and AI red teamers testing models and chat systems they own or are authorized to assess.

Worth knowing

It tests the model's replies, not the application code around it.

Can it test a model I host myself?
Yes. It can target hosted model APIs as well as models you run yourself.

Does it fix what it finds?
No. It reports the weaknesses, and fixing them is up to the team that runs the model.