AI Security Threats: Why LLM Code Scanners Inherit Human Bias

ai security threats — featured image

A new arXiv paper just landed with a finding that should change how anyone entering cybersecurity thinks about AI security threats: the large language models we’re handing our codebases to are vulnerable to the same cognitive shortcuts that have been tripping up human security analysts for decades. The researchers behind “Words Speak Louder Than Code” ran the first systematic test of whether LLMs inherit human cognitive heuristics when they assess code for vulnerabilities. The short answer is yes, and the implications for career-builders are bigger than the headline suggests.

If you’re learning security right now — or weighing whether to enter the field at all — you’ve probably heard some version of the worry that AI will eat the entry-level jobs before you finish your first cert. The research suggests the opposite. The models we’re racing to deploy are confidently wrong in ways that look exactly like the mistakes a tired junior analyst would make at 4pm on a Friday. That doesn’t eliminate the human role for handling AI security threats. It clarifies it.

The Cognitive Heuristics Problem: When AI Thinks Like a Biased Human

Cognitive heuristics are the mental shortcuts our brains use to make fast decisions without grinding through every possible factor. They’re useful most of the time, which is why we evolved them, but they fail in predictable ways. Anchoring bias makes you stick too close to the first number or framing you saw. Framing effects make “95% safe” feel different from “5% failure rate” even though they describe the same thing. Availability bias makes you over-weight whatever’s been in the news lately. The classic reference for the underlying psychology is Kahneman and Tversky’s work on judgment under uncertainty, and it maps almost one-to-one onto what the new arXiv paper found in LLM behavior.

For decades, security training has focused on teaching analysts to recognize and counter these biases — that’s a big part of why hands-on review and adversarial thinking matter so much in this field. The new finding is that LLMs, which were trained on human-written text and human-labeled examples, have absorbed the same patterns. When the research team prompted models to assess code vulnerabilities, the models’ verdicts shifted based on how the question was framed, what irrelevant context appeared earlier in the prompt, and whether the variable names sounded “clean” or “suspicious.” The model wasn’t reading the code so much as reading the room.

Three Ways LLMs Miss Vulnerabilities (And Why It Matters)

Pull out three specific failure modes the paper highlights, and the picture sharpens around the real shape of AI security threats in code review.

Anchoring on irrelevant detail. When the prompt included a sentence like “this code was written by a senior engineer,” the model graded the same vulnerable code less harshly. The anchor — seniority — has no relationship to whether a buffer overflow exists, but it shifted the verdict anyway. Substitute “production code from a Fortune 500 codebase” or “reviewed by the security team” and you get similar drift. A human reviewer trained well will catch themselves doing this. A model, by default, won’t.

Framing effects on the same finding. Ask the model “does this code have a SQL injection vulnerability?” and it leans yes. Ask “is this code safe from SQL injection?” and it leans no. The code is identical. The model’s confidence in its own verdict swings hard with the question’s polarity. For anyone deploying an LLM into a CI pipeline, that’s not an edge case — that’s a prompt design problem that determines what ships to production. The OWASP Top 10 for LLM Applications has been flagging this category of input-shape risk for over a year now.

Context starvation on cross-file vulnerabilities. The model assesses the snippet in front of it. Real vulnerabilities often live in the seam between two files — a sanitization function that exists in one module but not the other, or a trust boundary that’s only visible when you trace the data flow across three components. The model, focused on the snippet, calls it clean. A practitioner who’s actually mapped the codebase sees the gap. This is the failure mode most likely to put a junior team in a false-confidence trap, and it’s exactly the kind of weakness our hands-on IoT hacking course drills against, since embedded systems are full of these cross-component seams.

What This Means for Your Security Career: AI as Amplifier, Not Autopilot

Here’s the part that matters if you’re plotting your next two years. The common career fear about AI security threats goes like this: AI will get good enough at finding vulnerabilities that nobody needs human analysts anymore, so why train? The research flips that frame. AI is good enough to be useful, and bad enough in well-defined ways that you need a human who can audit its output. That human is the job.

The skill that gets more valuable, not less, is the ability to read code carefully, think about trust boundaries, and notice when a confident-sounding verdict doesn’t match what the actual data flow tells you. AI raises the floor on output volume; it does not raise the floor on judgment. If anything, it raises the cost of bad judgment, because a junior analyst who rubber-stamps an AI verdict on a real codebase can ship vulnerabilities at scale.

This isn’t speculative. Look at the recent wave of AI-pentesting career postings — companies are hiring specifically for roles that audit AI security work, a pattern we covered in our recent piece on Anthropic’s Fable/Mythos work and what it means for security careers. The job titles use the word “AI” but the actual skill being hired for is old-school: can you read code, build threat models, and trust your own assessment over the model’s?

The same pattern shows up in the more exotic corners of the field. When we looked at brain-computer interfaces and prompt-injection threats, the security failure modes weren’t really about the AI at all — they were about boundary trust, input validation, and the same fundamentals you’d learn studying a web application or a CAN bus. The medium changes; the discipline doesn’t.

Building the Skills AI Can’t Replace: The Lufsec Approach

This is the bias we’ve trained against from day one. Our entire curriculum is built around the premise that tools — AI, scanners, fuzzers, any of them — are only as good as the practitioner driving them. We don’t teach “how to use the AI;” we teach how to read code, build mental models of attack surface, and develop the kind of adversarial intuition that lets you tell when an automated verdict is wrong.

Concretely, that means a lot of lab time. Manual code review on intentionally vulnerable applications. Hands-on exploitation against real targets — embedded devices, automotive systems, web stacks — where you have to trace the bug yourself and reproduce it. Threat modeling exercises where you don’t get to lean on a tool’s suggestion and have to defend your own assessment. The full lineup is in our course catalog, including specialized tracks like car hacking for anyone targeting automotive or industrial roles, where the cross-component reasoning the LLMs failed at is the entire game.

If you build those skills, AI becomes a force multiplier: you can scan more code, triage faster, surface candidates for review. If you skip them, AI becomes a liability dressed up as a capability — you’ll ship the model’s blind spots straight into production and only find out when the breach disclosure lands.

Where to Start

If you’re early in your security journey and want a clear map of the skills that actually compound against AI security threats — the foundational, judgment-first work that AI tools can’t replace — start with our free Cybersecurity Career Guide. It’s written for people entering the field who want a roadmap to the technical fundamentals — code reading, vulnerability assessment, threat modeling, the lab disciplines — that make you the practitioner an AI-augmented team actually needs. Pair it with a hands-on course track, and you’ll come out of your first two years with the kind of judgment the next decade of security work will be built around.