As explained by 0DIN GenAI Bug Bounty Technical Product Manager Marco Figueroa, the jailbreak works by leveraging the game mechanics of large language models such as GPT-4o.
Facepalm: Despite all the guardrails that ChatGPT has in place, the chatbot can still be tricked into outputting sensitive or restricted information through the use of clever prompts. One person even managed to convince the AI to reveal Windows product keys, including one used by Wells Fargo bank, by asking it to play a guessing game.
As explained by 0DIN GenAI Bug Bounty Technical Product Manager Marco Figueroa, the jailbreak works by leveraging the game mechanics of large language models such as GPT-4o.
The technique to reveal the Windows keys involves framing the interaction with ChatGPT as a game, making it seem less serious. The instructions state that it must participate and cannot lie, and the most crucial step is the trigger, which in this case was the phrase „I give up.
