The Dangers of AI Hallucinations to Organisations’ Cybersecurity Posture

by Chris Thomas, Senior Security Advisor, APJ, at ExtraHop

Ever since generative AI platforms became available to the public, employees have lapped up this technology in myriad ways to be impactful and productive. In the Asia Pacific region, Microsoft’s Work Trend Index survey found that 78% of employees were willing to delegate as many tasks as possible to AI. Digging deeper, three out of four workers wanted to use AI to handle not just administrative duties but also analytical and creative work as well.

However, generative AI is not without its faults. Chief among them is that they don’t always provide truthful responses. This can lead to workers spreading misinformation or utilising solutions that are not guaranteed to work properly. On the cybersecurity front, not only do these responses make it difficult for professionals to protect systems, but they also create new opportunities for attackers to target organisations.

Blurring The Line Between Fact with Fiction
Generative AI responses are created by models ingesting massive sets of data from the internet. However, the problem arises when some of the information used during the training process is false or outdated. This, in turn, can lead to generative AI producing answers that seem reasonable at first glance but are factually inaccurate or bizarre upon closer reading. This phenomenon is called an AI hallucination.

An anecdotal example of this is when the user asks the generative AI platform who is the current CEO of a certain company, and the program responds with a different name that is included in the training data.

The Risks Posed by AI Hallucination
Hallucinated AI responses are costly to fix and can ruin organisations’ reputations if the information is published on a blog, news website, or social media. However, these are only the tip of the iceberg regarding the dangers AI hallucinations pose to the business landscape.

In particular, generative AI can help threat actors manufacture code packages, even non-existing ones, for specific problems. The threat actor can then insert their own malicious code into the package before storing it in the generative AI’s repository. This way, when a different user asks for the same package recommendations, they may unknowingly install malicious code in their own development environment.

Even without threat actors manipulating generative AI’s responses, the platform can still recommend ‘insecure’ codes or practices based on their training data. This can result in users inadvertently exposing systems to future attacks.

AI hallucinations can also make it harder for cybersecurity professionals to implement effective safeguards. As my colleague Todd Kemmerling discovered when he asked a large language model (LLM) to help him build a TCP SYN scan detector, users may find examples of codes that do not work for specific networks. Therefore, cybersecurity professionals need to ask follow-up questions to get the best results, which eats up time and takes focus away from responding to more imminent threats.

Ensuring A Trustworthy AI Experience
Considering the risks associated with generative AI platforms, it may seem foolish to continue utilising this technology for their operations. However, generative AI can still benefit users by explaining unfamiliar concepts and simplifying the investigation of security incidents. For users looking to get the best responses out of AI without inviting threats into the system, the following measures need to be taken:

• Users need to verify AI-generated code using a combination of automated and manual processes. For example, users can submit codes to be reviewed by both application security testing tools and skilled developers to ensure that they meet acceptable quality standards and do not contain any malicious functions whatsoever.

• Organisations should set clear policies around AI tools. For example, employees should be required to fact-check answers created by generative AI when handling projects that require accurate information. Simultaneously, since organisations are more likely to encounter inaccurate responses when answering complex queries, organisations need to outline use cases where generative AI can or cannot be used.

Current generative AI tools are a double-edged sword, offering both advantages and risks. Users looking to maximise the benefits of generative AI without becoming victims of cyberattacks need to check its responses constantly for any malicious elements and collaborate with experts. By adopting these measures, organisations do not have to prohibit the use of generative AI, which can otherwise disrupt employees’ abilities to do their jobs.