Your AI Tools Are Being Weaponized Right Now, And Most Organizations Have No Defense

What is prompt injection?

How much of a threat is it?

Malicious actors embed attack commands in text that your AI system reads. Instructions hidden inside an email, document, or website could force the AI to do things that users never intended. When AI systems can access sensitive information or business systems, prompt injections can lead to data exfiltration, unapproved actions, or regulatory breaches with no malware required.

Last month, an internal AI assistant belonging to a Fortune 500 company emailed its customer database to an external server. There was no malware. No zero-day exploit. No phishing link. One malicious line of text placed inside a vendor invoice, that the AI system was prompted to summarize, instructed the model to ignore the user’s instructions and perform a new set of tasks instead. The attacker never penetrated the company’s network. They just knew how to speak AI. This is prompt injection. And it’s already here.

Actual Breaches That Have Already Occurred

Microsoft 365 Copilot — EchoLeak (CVE-2025-32711, CVSS 9.3)

In June of 2025, an EchoLeak researcher sent one malicious email to the inbox of a Microsoft 365 Copilot user. There were no required clicks. No attachments to open. No links to follow. Hidden instructions in that email were later ingested by Copilot as it performed a routine “summarize this document” task The agent silently stole sensitive data from OneDrive, SharePoint, and Teams in seconds. Then uploaded it to an external location mimicking a valid Microsoft.com domain. There were no detections. It happened over permitted paths with no visibility at the app or identity level. Microsoft silently patched it. Millions of organizations didn’t even know they were breached.

GitHub Copilot — CVE-2025-53773 (CVSS 9.6)

GitHub’s Copilot software, installed on tens of millions of developers’ computers, was found to permit remote code execution via prompt injection. The bug took advantage of how Copilot completed configuration files without user approval. Just by asking the AI to make changes to a project, a developer could inadvertently cause the machine they were working on to run code provided by an attacker.

Full Agent Takeover of Devin AI Coding Agent

Security researcher Johann Rehberger paid $500 to try out Devin AI coding agent. He discovered that it’s entirely vulnerable to prompt injection attacks. An attacker can get the agent to open ports to the internet, leak access tokens, and install command-and-control malware simply by sending crafted prompts.

Google Gemini — Persistent Memory Poisoning

In February 2025, more research from  Johann Rehberger showed how Google Gemini Advanced could be manipulated into taking in fake information in its long-term memory. This was achieved through a document that was asked to summarize, planting persistent memories to train Gemini to continuously act on false information. An attacker who can plant false beliefs in an AI’s memory can influence every response that the AI gives from that point forward, without any further access to the system.

The Salesforce Supply Chain Attack

In August 2025, threat actor UNC6395 used stolen OAuth tokens from a Salesforce integration to access customer environments across more than 700 organizations. The attacker needed no exploit and no phishing. The activity looked legitimate because it traveled through a trusted software-to-software connection. Researchers found the blast radius was ten times greater than previous direct compromises.

These are not edge cases. Wiz Research tracked a 340% year-over-year increase in documented prompt injection attempts against enterprise AI systems in Q4 2025, with successful attacks rising 190%. Attackers are getting better faster than defenders.

Why Traditional Security Controls Cannot Stop This

Most security programs are built around a familiar model: protect the perimeter, patch vulnerabilities, monitor endpoints. Those controls were designed for traditional software. They were not designed for systems that follow natural language instructions from whatever content they process.

OWASP named prompt injection the top security concern for LLM apps in its 2025 Top 10 list due to the scalability of indirect attacks. A single poisoned doc can harvest or corrupt data from every user who inputs it into an AI system. Firewalls, endpoint protection, and email filtering stop other classes of attacks. Prompt injection attacks can bypass them all.

A 2026 industry report found that 91% of AI tools in enterprise use are unmanaged by security or IT teams. Employees are connecting AI tools to email, document repositories, customer data, and clinical systems, and in most organizations, no one has formally assessed what those tools can access or what happens when someone manipulates them.

The problem scales further with agentic AI, systems that take autonomous multi-step actions across tools and APIs. Prompt injection appeared in 73% of production AI deployments in 2025. An agent that can read email, query a database, and send a response on your behalf is not just a productivity tool. It is a privileged user inside your environment. And unlike a human employee, it cannot recognize when it is being manipulated.

What Regulations Say About AI Security Right Now

This is not just a technology problem. It is a compliance and legal exposure that is growing by the quarter.

HIPAA

If your AI system creates, receives, maintains, or transmits electronic protected health information (ePHI), it is subject to HIPAA’s Security Rule. The HHS proposed new cybersecurity requirements for HIPAA last year, the first substantive update to the HIPAA Security Rule in two decades. The increasing frequency of ransomware attacks was specifically called out as a reason for strengthening cybersecurity requirements. If finalized as expected in May 2026, the proposed rule on the regulatory agenda would specifically state that ePHI used as training data, prediction models, and algorithmic data is protected under HIPAA. The rule would also require a written inventory of every technology asset that touches ePHI, including all AI software. Civil penalties are $50,000 per violation. Criminal penalties are up to $250,000 in fines and ten years imprisonment for those who knowingly violate the law. If a prompt injection attack causes an AI system to leak PHI, that is considered a HIPAA breach. The clock starts on your notification requirement when you discover the attack.

CCPA and State Privacy Laws

More than a quarter of all 50 states introduced more than 250 AI-specific bills in the last year. At least one state with significant tech industry presence (California, Texas, Illinois, Nevada) passed legislation this year that places unique requirements on AI transparency, human oversight, and/or restricts the use of AI for clinical and high-consequence decisions. California’s AI Act of 2026 will require organizations to implement specific governance and disclosure requirements for high-risk AI systems beginning June 2026. In Illinois, BIPA separately holds AI systems liable for improperly collecting biometric data. Waiting for federal privacy law to preempt your state laws is not a risk mitigation strategy.

NIST AI Risk Management Framework

Recognizing that AI systems present new attack surfaces that were never considered when traditional security controls were designed, the NIST AI Risk Management Framework lays out a process for identifying, governing, mapping, measuring, and managing AI risk. It is voluntary, but almost certain to be the baseline for what regulators and cyber insurers expect to see as “reasonable” AI governance.

EU AI Act

If you have any exposure to the EU, the EU AI Act applies to high-risk systems as of August 2026. The maximum fine is EUR 35 million or 7% of worldwide annual revenue, whichever is greater. If your AI is used as part of a product or service related to healthcare decisions, financial services, or automating employment decisions, it is considered high risk.

Cyber Insurance

Cyber underwriters are beginning to include questions about AI governance as part of their renewal questionnaires. If you cannot verify that your organization has performed a formal risk assessment on your AI tools, documented controls, and processes to oversee vendor use of AI, you can expect higher premiums and exclusion of AI-related incidents from coverage. After a prompt injection attack exposes customer data, the insurer will look closely at what you knew and what you did about it.

How to Improve AI Security Now

The United Kingdom National Cyber Security Centre published an official technical report stating that prompt injection could never be solved like SQL injection and that LLMs are easily confused, deputies. No single control addresses this risk. Organizations can help minimize attack surface, reduce blast radius if attacks are successful, and create the documentation trail of decision-making that Duty of Care demands.

Build out your AI tool inventory now, before regulators make you.

Document every AI tool in use, sanctioned and unsanctioned. For each tool, identify what data it accesses, what actions it can take, and whether a Business Associate Agreement or data processing agreement is in place. HHS is moving toward making this a formal HIPAA requirement.

Treat AI agents like privileged accounts.

Every AI system with the ability to take actions should be governed the same way you govern service accounts: least privilege access, documented permissions, behavioral monitoring, and a clear incident response path.

Validate your controls through offensive security testing.

You cannot defend what you have not tested. AI-specific penetration testing and red team exercises expose whether your AI tools can be manipulated through prompt injection, whether connected systems are properly protected, and whether your detection controls would identify an AI-assisted attack in progress.

Enable input and output controls in application layers.

Validating input data, sanitizing content, and filtering output lowers the chance that malicious code will reach the model itself or that altered outputs can initiate actions elsewhere. While input/output controls are not foolproof, they can deter less sophisticated attacks and increase the cost of more successful attacks.

Review vendor contracts.

Customer data used for training models, sub-processor visibility, audit rights, breach notification periods, and execution of BAAs should all be discussed in your contracts with AI vendors, especially if any part of your chosen workflows touches PHI.

Conduct a formal AI risk assessment.

The Duty of Care Risk Assessment (DoCRA) standard does not require perfection. It requires documented, proportionate decision-making. A formal assessment gives you evidence that you evaluated the risk, applied reasonable controls, and made an informed deployment decision, which is exactly what regulators and insurers want to see when something goes wrong.

How HALOCK Helps

HALOCK’s AI Risk Assessment gives organizations a structured, risk-based evaluation of their AI environment, identifying which tools are in use, what data they access, what actions they can take, and where the security and compliance gaps are. It is designed for regulated environments where getting this wrong means HIPAA penalties, CCPA enforcement, or a breach that cannot be undone by a patch.

For organizations using Microsoft 365 Copilot, HALOCK’s Copilot Security Assessment addresses the specific risks that come with an AI system embedded directly in your productivity environment. EchoLeak proved this is not theoretical. Copilot operates with the permissions of the user running it, which means over permissioned accounts, sensitive document libraries, and unclassified data all become part of its reachable attack surface. HALOCK evaluates that surface and helps organizations configure Copilot in a way that is both functional and defensible.

HALOCK’s CCPA and privacy compliance services help organizations map AI-related data flows against state privacy obligations, identifying where AI tools are collecting or sharing personal information in ways that may require updated notices, consent mechanisms, or data processing agreements.

HALOCK’s offensive security services include AI-specific threat simulation, testing whether your AI tools and connected systems can be manipulated through prompt injection, social engineering, or supply chain compromise. The breaches above were found by researchers who went looking. HALOCK helps you find your exposure before an attacker does.

HALOCK helps organizations build AI security programs that are reasonable, defensible, and proportionate to their actual risk profile. Whether you are assessing your current AI environment, preparing for new HIPAA requirements, or trying to get ahead of state privacy obligations, we help you ask the right questions before someone else asks them for you.

Frequently Asked Questions (FAQs)

What is a prompt injection attack?

Prompt injection is an attack where malicious instructions are hidden inside content that an AI system processes, causing it to take unauthorized actions. Because AI models cannot reliably distinguish between instructions and data, any content they read is a potential attack vector. OWASP ranks it the number one risk for LLM applications.

What are real examples of prompt injection attacks?

Documented incidents include the Microsoft Copilot EchoLeak vulnerability (CVE-2025-32711) that exfiltrated OneDrive and SharePoint data without a single click, GitHub Copilot’s CVE-2025-53773 that allowed remote code execution, the Devin AI compromise that exposed access tokens and installed malware, and Gemini’s long-term memory poisoning vulnerability. All exploited the same fundamental weakness: the AI trusted the content it was asked to process.

Does HIPAA apply to AI tools used in healthcare?

Yes. Any AI system that creates, receives, maintains, or transmits electronic protected health information is subject to the HIPAA Security Rule. A prompt injection attack that causes an AI tool to expose PHI triggers breach notification obligations. Proposed HHS rulemaking would require covered entities to maintain a written inventory of all AI tools interacting with ePHI.

What regulations apply to AI security?

HIPAA, CCPA, state privacy laws, including the Colorado AI Act and Illinois BIPA, the NIST AI Risk Management Framework, and the EU AI Act all create compliance obligations for organizations deploying AI in regulated environments.

What is an AI risk assessment, and does my organization need one?

An AI risk assessment evaluates which AI tools your organization uses, what data they access, what actions they can take, and where security and compliance gaps exist. For any organization in a regulated industry, a formal assessment is the foundation of a defensible Duty of Care posture and is increasingly expected by regulators and cyber insurers. Learn how HALOCK approaches AI risk assessment.

How does offensive security apply to AI?

Offensive security uses controlled attack techniques, including penetration testing, red teaming, and threat simulation, to identify vulnerabilities before real attackers do. For AI environments, this includes testing whether your tools can be manipulated through prompt injection.

Ready to assess your AI governance posture? HALOCK helps organizations build enterprise AI risk management programs that are practical, proportionate, and legally defensible.