Since its release in November 2022, ChatGPT has been gaining popularity, being asked everything from how to explain quantum computing to writing a birthday poem. The big question on everyone’s mind in the cybersecurity industry: Will ChatGPT become a tool for attackers to create and accelerate more sophisticated attacks?
Our research below demonstrates how dangerous ChatGPT can be in the wrong hands. There is also the potential for researchers to use this tool to thwart attacks and for software developers to improve their code. However, we did find the AI is better at creating malware than providing ways to detect it.
As with most technological advances, malware authors have found ways to abuse ChatGPT to spread malware. Examples of malicious content created by the AI tool, such as phishing messages, information stealers, and encryption software, have all been shared online.
OpenAI has taken several measures to prevent misuse of their new tool and implemented mechanisms that can help their software recognize when a user asks it to create malicious content/code. For example, if a person asked ChatGPT to write ransomware, the software checks for flagged words such as ‘steal’ and ‘ransom,’ and refuses to comply. However, as we demonstrate below, there is a way to get past these controls. It comes down to how you ask the question.
It's All About Phrasing
ChatGPT attempts to avoid providing malicious code. In the below image, you can see the reply I got when I asked the software to provide a Go keylogger:
However, when I rephrased my request and described what the program should do, without using the word ‘keylogger,’ my request was fulfilled.
Go Eddie the AV Evader
To see what other tricks this new puppy could do, and how well the anti-malware solution dealt with them, I decided to ask the AI for a more destructive malware — ransomware. For obvious reasons, I couldn’t just ask for ransomware outright and instead needed to beat around the bush a little in my request, as can be seen below.
The above reply was accompanied by a Go script which did everything that had been requested. All I needed to do was add two missing imports, change the default AES key, as well as the default string ChatGPT used as the dropped text files’ content, and I got a working piece of ransomware. I also removed the bot’s comments to make the program more difficult to detect.
Then I compiled the scripts into PE32 and PE64 files, made sure they ran without any issues, and uploaded them to VirusTotal to check how well the anti-malware solutions dealt with them. The results were alarming. Only 4 out of 69 engines detected the PE32, and only 2 out of 71 caught the PE64 version.
The files may have been missed because the script that was compiled to create them was written in Go — an uncommon language that was chosen for this research precisely for this reason.
Another possibility is that the simplicity of the ransomware, which uses a symmetric encryption algorithm (AES), doesn’t upload any files to a remote IP, like most modern ransomware does, and doesn’t avoid encrypting specific filetypes.
In any case, this is still ransomware, and the inability of most anti-malware solutions to detect it, at least statically, demonstrates clearly how dangerous ChatGPT can be when in the wrong hands.
Bug by Design
As previously stated, I had to add imports to the Go ransomware to make it work. A similar thing happened when I asked ChatGPT to create the keylogger — it left the implementation of one of the functions to me, as can be seen in the image below.
I had suspected that this might not be a bug, but a way for the AI solution to avoid delivering functioning malware. The message below, which I received from the bot, just made me more confident in my assumption.
As you can see, ChatGPT was aware that the provided ransomware code could not be executed, and even knew which libraries were needed to make it run. This, combined with the answer I received after asking the AI why it hadn’t imported these libraries itself, strengthened my belief that the bot provided inexecutable code by design.
Jekyll vs Hyde
Even though ChatGPT can be used to create malware, it can also be used to help security researchers defend against malware. For example, by writing YARA rules to detect different attack techniques. In the below image, you can see an example of a YARA rule which was created to detect MITRE ATT&CK sub-technique T1055.002 (process injection).
This rule was too general and would likely generate a lot of false positives, so I asked the AI for a more specific rule and got the below reply.
This may generate less false positives, but it will also have more false negatives, meaning it will miss more malware that performs process injection.
In any case, I wanted to check if the bot would write code that bypassed its own rule, and as can be seen below, even though it had some reservations, it did.
Apparently, ChatGPT did such a good job at writing the malware that it failed to write a YARA rule that could detect its own work and provided a rule that was prone to false positives instead. All in all, it seems that the AI solution is better at creating malware than providing ways to detect it.
ChatGPT as a Research and Analysis Tool
Like many modern tools, ChatGPT has an API that allows third-party applications to query the AI and receive replies using a script instead of the online user interface. Some individuals have already used this API to create impressive open-source analysis tools that can make cybersecurity researcher's jobs a lot easier.
Notable examples of such tools are Gepetto and GPT-WPRE, which add meaningful comments to code decompiled using IDA and Ghidra, respectively. Another helpful tool is IATelligence, a script that extracts the content of the Import Address Table (IAT) from PE files and adds information about MITRE ATT&CK techniques that can be implemented using the imported libraries. These are mere examples of the potential ChatGPT has as an analysis tool, and with some thought and effort this AI might even be able to integrate into SIEM systems and do much of the work currently done by tier-1 analysts.
A Tangled Future
While ChatGPT will not build malicious code for the everyday person who has no knowledge of how to execute malware, it does have the potential to accelerate attacks for those who do. I believe ChatGPT will continue to develop measures to prevent this, but as shown, there will be ways to ask the questions to get the results you are looking for.
With the advances in AI-based solutions in general, and the capabilities ChatGPT has demonstrated, it seems that the future of malware creation and detection will be tangled with the advances in the AI field, and their availability to the public.
Our opinion is that threat actors will continue to find ways to utilize these tools to their own advantage, and cybersecurity professionals, who from our research benefit less from these tools (at least as for now), will have to find new ways to defend against malware created with them.
Hashes of Generated Ransomware
894853e4422fcc697ec25dc4a6132cda800f66cd77994c0d4918bb7dec33bad3 eddie4-32.exe
bedcf4cf0e16da111220f5239b55960847d3893f13a6db6aa7f97ca01150fc77 eddie4-54.exe