Strengthening AI Security to Match AI Innovation

The Growing AI Security Gap

The race to advance artificial intelligence (AI) is accelerating at an unprecedented pace, but security has struggled to keep up. Organisations are pushing AI capabilities forward rapidly, often prioritising innovation over secure development. The recent exposure of a publicly accessible database belonging to DeepSeek, a Chinese AI startup, underscores the growing tension between rapid AI progress and the need for robust security measures. Yet, AI security is often an afterthought, leaving systems vulnerable to significant risks.

The DeepSeek Data Breach: A Case Study

DeepSeek, a relatively new AI company that rapidly gained attention for its sophisticated models, surprised many with its sudden rise in the industry. However, it soon faced scrutiny when it left a database containing over a million lines of sensitive data exposed to the public internet. This breach included:

• User chat histories with the AI model
• Secret API keys
• Backend service metadata

This case demonstrates how AI companies can fail to implement basic security hygiene, leading to severe data exposure risks. The absence of fundamental security controls raises concerns about the broader AI ecosystem’s security posture.

The Fundamental Flaws in AI Security

Overlooking Traditional Security Principles

AI models rely on existing IT infrastructure, which is subject to traditional vulnerabilities (e.g., unpatched servers, and weak authentication). However, AI providers often neglect technical and non-technical security frameworks in their rush to deploy new models, as evidenced by past breaches such as OpenAI’s API vulnerability incident and research highlighting adversarial attacks against widely used AI systems.

Many AI companies fail to adhere to industry best practices like ISO 27001, NIST Cybersecurity Framework (CSF), and SOC 2, establishing essential governance, risk management, and compliance standards. The DeepSeek breach, for instance, was largely due to a failure to implement basic security hygiene measures, such as secure database configurations, proper access controls, and robust audit mechanisms. These oversights illustrate that AI security is not just a technical challenge. It’s also a governance issue requiring structured risk assessments, security awareness training, and compliance with established frameworks. Many AI systems lack robust security baselines, making them susceptible to common cyber threats, including unauthorised access, data exfiltration, and adversarial manipulations.

Additionally, the rapid iteration cycles in AI development often sideline security best practices. This leads to systemic weaknesses that attackers can exploit at scale. AI-driven applications, such as chatbots and recommendation systems, are particularly vulnerable due to their real-time data interactions. These interactions can be manipulated for misinformation, fraud, and social engineering attacks.

AI-Specific Attack Vectors Are Ignored

• Social Engineering Attacks on AI Chatbots: Attackers have successfully manipulated AI chatbots to reveal sensitive information or execute unintended actions, exposing security gaps in prompt-based AI interactions.
• Adversarial Machine Learning (AML): Attackers can manipulate AI models by feeding them misleading data, which can affect their decision-making.
• Data Poisoning: Malicious actors can introduce harmful data into training sets, corrupting AI behaviour.
• Model Extraction & Theft: Attackers can extract proprietary AI models through API queries, leading to intellectual property theft.
• Bias Exploitation: Attackers can exploit inherent biases in AI models to manipulate decision-making processes, leading to ethical and security concerns.
• Adversarial Machine Learning (AML): Attackers can manipulate AI models by feeding them misleading data, which can affect their decision-making.
• Data Poisoning: Malicious actors can introduce harmful data into training sets, corrupting AI behaviour.
• Model Extraction & Theft: Attackers can extract proprietary AI models through API queries, leading to intellectual property theft.

Also read: LLM security, Top 10 Risks & How To Mitigate Them

Regulatory and Compliance Blind Spots

With increasing global regulatory scrutiny, AI providers and users must comply with emerging frameworks like the EU AI Act, NIS2 Directive, and U.S. AI Executive Order on Trustworthy AI. These regulations introduce strict requirements on AI transparency, data privacy, and cybersecurity, imposing significant obligations on organisations deploying AI technologies.

The EU AI Act, for instance, categorises AI applications by risk level, mandating specific controls for high-risk AI systems. At the same time, the NIS2 Directive expands cybersecurity requirements for essential service providers, including AI-driven platforms. However, many organisations remain unaware or unprepared for compliance. They lack structured governance models and security measures necessary to align with these evolving legal frameworks.

Bridging the AI Security Gap: A Shared Responsibility

AI Providers Must Prioritize Security by Design

• Implement Zero Trust Architecture for AI infrastructure.
• Encrypt sensitive training data and AI model parameters.
• Conduct regular adversarial testing to identify weaknesses.
• Adopt secure APIs with strict access controls.
• Establish transparent AI security policies aligned with regulations.

AI Users Must Enforce Security in AI Deployments

• Conduct comprehensive AI risk assessments to identify data storage, processing, and model output vulnerabilities.
• Assess where AI-generated data is stored and processed, ensuring compliance with data protection laws such as GDPR, CCPA, and other regional frameworks.
• Establish clear data retention policies to define how long AI-related data should be stored and when it should be deleted.
• Ensure AI models do not store or process personally identifiable information (PII) beyond necessary limits and in compliance with legal obligations.
• Collaborate with legal and compliance teams to evaluate the jurisdictional impact of data transfers, particularly for AI models trained across multiple regions.
• Implement privacy-preserving AI techniques such as federated learning and differential privacy to minimise exposure to sensitive information.
• Ensure proper data governance frameworks are in place to track how AI data is collected, processed, and shared with third parties.
• Develop risk mitigation strategies based on assessment results, ensuring that AI models operate within an acceptable risk threshold.
• Assess where AI-generated data is stored and processed, ensuring compliance with data protection laws such as GDPR, CCPA, and other regional frameworks.
• Establish clear data retention policies to define how long AI-related data should be stored and when it should be deleted.
• Ensure AI models do not store or process personally identifiable information (PII) beyond necessary limits and in compliance with legal obligations.
• Collaborate with legal and compliance teams to evaluate the jurisdictional impact of data transfers, particularly for AI models trained across multiple regions.
• Implement privacy-preserving AI techniques such as federated learning and differential privacy to minimise exposure of sensitive information.
• Ensure proper data governance frameworks are in place to track how AI data is collected, processed, and shared with third parties.
• Vet AI providers based on their security and compliance standards.
• Implement AI governance frameworks to monitor AI usage.
• Ensure proper access controls to AI-generated insights and data.
• Train employees on AI-specific security risks and best practices.

Actionable Steps to Improve AI Security

Organisations can enhance their AI security posture by implementing the following measures:

1. Conduct Comprehensive AI Security Audits – Regularly assess AI infrastructure for vulnerabilities, including risks related to training data, model integrity, and deployment security.
2. Deploy Multi-Factor Authentication (MFA) and Role-Based Access Controls (RBAC) – Limit access to critical AI systems and enforce strong identity verification.
3. Encrypt AI Training Data and Model Weights – Prevent unauthorised access to sensitive datasets and model parameters.
4. Develop AI-Specific Risk Assessment Frameworks – Establish structured methodologies to evaluate the risks posed by AI deployments, such as potential bias, adversarial exploitation, and regulatory compliance gaps.
5. Develop AI Incident Response Plans – Establish protocols to quickly contain, investigate, and mitigate AI-specific security incidents, including data poisoning and adversarial attacks.
6. Comply with AI-Specific Regulations – Align security practices with emerging AI governance laws, ensuring accountability and transparency in AI decision-making.

Read our article about AI Security Maturity!

The Balancing Act Between AI Speed and Security

The DeepSeek breach is a stark reminder. While AI innovation is advancing at an unprecedented rate, security measures often lag. Organisations are racing to push AI capabilities forward, but too often, security is a secondary concern. The challenge lies in striking the right balance between speed and safety—accelerating AI development and ensuring it remains secure and trustworthy.

Rushing AI models into production without robust security governance can create systemic vulnerabilities that are difficult to mitigate later. Security must be embedded from the start and integrated into every stage of AI development, from data collection and model training to deployment and continuous monitoring. Organisations must adopt a security-by-design approach. In this approach, risk assessments, compliance with evolving regulations, and adversarial testing become standard practices rather than afterthoughts.

Moreover, collaboration across the industry is essential. AI developers, enterprises, regulators, and cybersecurity professionals must work together to establish and follow best practices that protect against emerging threats. Regulatory frameworks such as the EU AI Act and the NIS2 Directive provide guidance, but companies must proactively align with these standards before enforcement becomes necessary.

The future of AI security depends on the collective efforts of its stakeholders. By fostering a culture of accountability, continuous improvement, and proactive risk management, AI providers and users can build a safer and more resilient AI ecosystem. Those who prioritise both innovation and security will not only protect themselves from cyber threats and regulatory scrutiny but will also lead the way in shaping a responsible and trusted AI-driven future.

The choice is clear—either we integrate security into AI today or pay the price tomorrow.