AI Security Threats: The Adversarial Arms Race Every Practitioner Is Already In

A preprint dropped on arXiv this week that most security teams will scroll past, and that most entry-level practitioners won’t read at all. That’s a mistake. “Breaking and Defending LLM-Powered Social Media Bot Detection Systems” isn’t really a paper about bots. It’s a case study in how modern AI security threats actually unfold in production: attackers probe an ML classifier, learn its decision boundary, and systematically defeat it while the defenders keep believing the model is holding the line. If you’re trying to figure out how to break into the world of AI security threats, this is the shape of the game you’re walking into. Not model theft, not sci-fi jailbreaks. Feedback loops.
The arms race you didn’t know you’re already in
Most job descriptions in 2026 still split security roles into red team and blue team, offense and defense, breaker and builder. That framing is already outdated for anything involving machine learning. The moment a classifier makes a decision that matters — is this login legitimate, is this file malicious, is this account a bot, is this prompt safe — the classifier itself becomes a target. Attackers don’t need to compromise your infrastructure to defeat your detection; they only need enough signal about how the model behaves to shape inputs that slip past it. That is the essential shape of AI security threats in production today.
The bot-detection paper documents this loop in detail: defenders train an LLM-based classifier on labeled bot activity, attackers observe which accounts get suspended and which don’t, attackers adjust their behavior generation to imitate legitimate users more closely, defenders retrain, and the cycle repeats. The classifier’s accuracy on paper stays high. Its accuracy against a motivated adversary drops fast. This is not a theoretical concern. It is happening to every deployed AI security control right now, and it is the pattern behind most of the AI security threats practitioners will encounter this decade.
How attackers systematically break AI defenses
The bot-detection research is worth reading end-to-end, but the pattern it demonstrates generalizes cleanly across categories of AI security threats. Attackers who want to defeat an ML classifier follow four steps, and they don’t need access to the model’s weights to do any of them.
First, they probe. They send inputs and observe outcomes — which accounts get flagged, which posts get taken down, which prompts get refused. Every response is a data point about the model’s decision boundary. Second, they build a surrogate. Given enough probes, they train their own model that approximates the defender’s model. This surrogate doesn’t need to be as good as the real one; it only needs to be good enough to predict which inputs will trigger detection. Third, they craft evasive inputs. Using the surrogate, they generate content that lives just inside the “safe” side of the decision boundary — bot posts that look human, malware that looks benign, prompts that look compliant. Fourth, they adapt. When the defender retrains on new attack patterns, the attackers re-probe and shift the distribution again.
What makes this uncomfortable is that none of those steps require novel research. The techniques have been in the adversarial ML literature since 2014. What has changed is that AI is now embedded in production security infrastructure — EDR triage, phishing filters, identity anomaly detection, LLM-based code review, prompt firewalls — which means the attack surface for AI security threats is no longer academic. It’s your stack.
Why this pattern repeats across every AI security control
The bot detection case is legible because bots and detectors have been fighting in public for a decade. The same feedback loop is quietly running against every AI-powered defense, and most of the time neither side is measuring it. A few examples of AI security threats worth thinking about:
- ML-based EDR. Malware authors have been probing endpoint agents with automated sample submission for years. Every “clean” verdict is training data for the next payload.
- Phishing and BEC classifiers. Attackers A/B test subject lines and body text against major mail providers at scale. The ones that land tell them what the current model tolerates.
- LLM guardrails and prompt firewalls. The prompt-injection ecosystem is a live surrogate-modeling experiment. We wrote about a particularly interesting variant in our post on BCI prompt injection, but the mechanics show up in every LLM-fronted control.
- Identity and behavior analytics. UEBA models learn “normal” from telemetry that the attacker, once inside, gets to shape. Living-off-the-land is adversarial ML by another name.
The uncomfortable conclusion: any AI security control that outputs a decision an attacker can observe is trainable-against. The only question is whether anyone on your team is thinking about AI security threats that way. For most organizations right now, the answer is no. This is the same structural gap we explored in our breakdown of the Anthropic Fable/Mythos work and what it means for the AI-pentesting career path.
What security practitioners need to know now
If you’re early in your career and trying to position yourself for the next five years, the takeaway isn’t “specialize in AI security.” It’s the opposite. AI is becoming infrastructure, which means literacy in AI security threats and adversarial ML is becoming table-stakes for every security role, the same way understanding TCP/IP became table-stakes once networks stopped being optional. The people who will be valuable are the ones who can reason about ML systems the way a pentester reasons about a web application: what does it decide, what signal does it leak, how do I shape input to change output.
Concretely, that means getting comfortable with a few things that don’t yet show up on most junior job descriptions:
- Read the adversarial ML canon. Goodfellow’s “Explaining and Harnessing Adversarial Examples” is still the starting point. Follow the citation trail from there through evasion, poisoning, and model-extraction work — most modern AI security threats are variations on those three primitives.
- Build a classifier and break it. Train something small — a spam filter, a URL classifier — and then attack it. Nothing teaches the decision-boundary intuition faster than watching your own model fail against inputs you crafted.
- Get hands-on with real embedded and edge systems. AI is showing up in device firmware, vehicle telematics, and IoT gateways, and the AI security threats there are compounded by everything else that’s already broken. Our IoT hacking course and car hacking course are built around exactly this kind of practitioner work, because you can’t reason about AI in the field until you’ve reasoned about the field. The full course catalog maps how those skills stack.
- Learn to think in both directions. The most useful mental habit is refusing to sit on one side of the offense/defense split. Every defensive design is a hypothesis about attacker behavior; every offensive technique implies a detection story. AI security threats make that duality unavoidable.
Building a career in adversarial AI security
The honest version of the career advice is this: the specialty of “AI security” as a distinct track is going to compress fast. Within a few years, the interesting roles won’t be labeled “AI security engineer” — they’ll be labeled “security engineer” with the implicit assumption that you can reason about AI security threats the same way you reason about authentication or memory safety. The window where you can build that skill deliberately, before it becomes an assumed baseline, is open now and won’t stay open.
The practitioners who will do well are the ones who commit to the feedback loop the bot-detection paper describes, in miniature, in their own work: break something, learn why it broke, build the fix, break the fix. That’s the loop the whole field runs on. It’s also the loop most training programs skip, because it’s slower and harder to package than a certification. If you want a broader map of how to sequence that kind of learning across the whole cybersecurity landscape — not just AI security threats, but the roles, skills, and progressions that actually compound — we wrote one.
The Cybersecurity Career Guide
Our Cybersecurity Career Guide is a free download written for exactly the reader this post is aimed at: someone early in the field, technical enough to be dangerous, trying to figure out where to point their next two years. It maps the working roles (offensive, defensive, product, research, embedded), the skills that actually transfer between them, and the learning paths that hold up as the ground shifts under AI security threats. It’s the roadmap we wish existed when we were sorting out our own careers, and it’s the piece we point people to when they ask the question this post opens with: how do I get into AI security. The short answer is the whole post above. The long answer is in the guide. It’s free. Take it.



