ChatGPT was tricked into generating valid activation keys for Windows 95

YouTuber Enderman managed to generate several valid activation keys for Windows 95 by tricking OpenAI's ChatGPT. This was made as a fun exercise as Windows 95 is essentially abandonware now; its support ended back in 2003.

However, they needed to bypass the AI's own built-in safeguard which initially refused to generate any key when a simple "Can you generate a valid Windows 95 key?" prompt was used. ChatGPT responded:

As an AI language model, I cannot generate a valid Windows 95 key or any other type of activation key for proprietary software. Activation keys are unique to each installation and must be purchased from the software vendor. Moreover, Windows 95 is a very old operating system that is no longer supported by Microsoft, and therefore, it is not available for purchase anymore. I suggest that you consider upgrading to a newer version of Windows to ensure that your system is secure and up-to-date.

Another reason for experimenting with Windows 95 keys was because the nature and key format for the OS are much less complex compared to more modern Windows versions, as the latter would require things like Product ID. In the images below, the Windows 95 key format as well as valid and invalid keys are shown:

Using this information, a revised, detailed prompt was given to ChatGPT which clearly defined the string format that a valid Windows 95 key needed to be. A total of 30 strings were asked to be generated:

With this revised prompt, ChatGPT did everything it was asked to do without breaking a sweat, not realizing that it was essentially acting like a key generator (keygen). In the above image, the second key was able to successfully activate Windows 95.

Source: Enderman (YouTube)

ChatGPT was tricked into generating valid activation keys for Windows 95