AI Programming: Supply Chain and Code Security

AI lets you write faster, and it also makes 'things going wrong' arrive faster. When a single prompt can generate hundreds of lines of code, npm install a pile of unfamiliar dependencies, and pull model weights from Hugging Face into your inference service, your supply chain attack surface has quietly expanded—and many teams haven't realized it yet.

This isn't 'AI replacing programmers' anxiety; it's a concrete new attack vector: models write vulnerabilities, fabricate packages that don't exist, copy protected open-source code, and the weights themselves may be deserialization bombs. This article breaks these four categories of risk apart and gives a defense plan you can land in CI right now.

1. The New Supply Chain Risks AI Programming Introduces

The traditional software supply chain focuses on 'whether what you depend on is trustworthy.' AI programming adds another layer on top of this: what you depend on is generated by a model, and the model itself is not trustworthy. In engineering terms, there are four concrete categories of risk:

1. AI-generated vulnerabilities: models write code based on training data and will carry over outdated, insecure patterns from that data verbatim. In 2025, OWASP specifically added 'LLM output abuse/hallucination' to the LLM Top 10.
2. Hallucinated packages: the model recommends a plausible-looking package name that doesn't actually exist—an attacker just has to squat the name and poison it when you or someone else runs pip install.
3. License risk: the model may have 'memorized' GPL/AGPL code and reproduce large chunks verbatim into your private project, planting compliance landmines.
4. Model supply chain: the open-source model weights, adapters, and tokenizer files you use may hide pickle deserialization vulnerabilities in their loading path—this is a recurring incident in the Hugging Face ecosystem.

One key understanding must be established first: AI-generated code should be treated as a third-party dependency—its trustworthiness is no higher than a random unfamiliar package you casually install from npm. With this premise, the defenses below follow naturally.

2. AI-Generated Code Vulnerabilities—Dual Guard with SAST + AI Code Review

The categories of vulnerabilities models most often produce basically map to the classic OWASP Top 10 items:

• SQL/command injection: to 'look correct,' models frequently concatenate strings into SQL or shell rather than using parameterized queries. When asked to write a Node script, child_process.exec(\rm -rf ${dir}`)` shows up fairly often.
• Insecure deserialization: pickle.loads(), yaml.load() (without SafeLoader), Java's ObjectInputStream—models use them casually with essentially no validation.
• Hardcoded secrets: to get things 'running first,' the AI writes OpenAI keys and database passwords directly into source code 'for easy testing,' and commits them to Git along the way.
• Auth bypass / path traversal: for concatenations like path.join(base, req.query.file), models rarely proactively add .. validation; SSRF is also frequently overlooked.

For defense, human review alone isn't enough; you must tool it up in CI.

SAST (Static Application Security Testing)—mandatory in CI:

# Semgrep: many rules, controllable false positives, supports dozens of languages
semgrep ci --config=auto

# Or GitHub Advanced Security / CodeQL, enable it directly in the repo
# .github/workflows/codeql.yml is a ready-made template

AI Code Review—this is a new layer of defense specifically for AI-generated code. The key is to give the review bot a 'security-first' prompt, otherwise it'll only praise how well the code is written. A system prompt snippet I often use:

You are a strict security reviewer. For every piece of AI-generated code, you must check:
1. Are there injections (SQL / command / template / SSRF)?
2. Are secrets or credentials hardcoded or leaked into logs?
3. Are deserialization, file paths, and external redirects validated?
4. Do the introduced dependencies actually exist, are they commonly used, and are their licenses compliant?
If any issue is found, block the PR directly and provide fix recommendations—no pleasantries.

For tooling, GitHub Copilot's code review, CodeRabbit, Greptile, and Snyk Code all support running automatically on PRs. I recommend turning on at least two: a general bot for style and logic, and a security-focused bot (or a custom prompt) for vulnerabilities. Don't mix the two responsibilities—if you do, neither will be done well.

3. Hallucinated Dependencies—The Most Hidden Poisoning Vector

This is a new type of attack unique to AI programming. As early as 2024, academic papers systematically exposed that models 'fabricate' package names. For example, if you ask 'how to read Excel in Python,' it may tell you to pip install excel-utils-fast (a fabricated name)—follow it and you fall into the trap: an attacker just has to squat the name and publish a package with a malicious setup.py, and you're RCE'd. Real name-squatting poisoning cases have appeared on both npm and PyPI, and these attacks need no vulnerability—they rely entirely on your trust.

Three defenses:

1. Dependency existence verification—for any import xxx / require('yyy') the AI gives you, verify before installing:

# PyPI: check the version and release history
curl -s https://pypi.org/pypi/<package>/json | jq '.info.version, (.releases | length)'

# npm: check the first release time and maintenance activity
npm view <package> time --json

# Anything not in the official registry, don't install—full stop

A simple heuristic: package age + download count. A package published last week with single-digit downloads that the AI keeps recommending is almost certainly poisoning. By contrast, mature libraries like pandas and openpyxl won't have this problem.

2. Pin versions + lockfiles:

# Reproducible dependencies
pnpm install --frozen-lockfile
pip install pip-tools && pip-compile requirements.in

3. Trust only official sources: set registry=https://registry.npmjs.org/ in .npmrc, configure index-url=https://pypi.org/simple for PyPI, and use a private mirror (Verdaccio / Artifactory) internally as a proxy + approval gate. Never add some 'faster' unfamiliar mirror on the AI's suggestion.

4. License Compliance—Don't Let AI Copy GPL for You

The model's training data contains large amounts of copyleft code (GPL, AGPL, SSPL, MPL). In certain scenarios, it will 'recall' large chunks of code and output them almost verbatim. Once this enters your closed-source project, it is in theory infringement + license contamination, and the compliance team will explode.

The risk isn't only direct copying: an npm package introduced by the AI may hide a GPL library in its transitive dependencies, which will likewise contaminate your binary. The license file won't proactively tell you about this.

License scanning should run in CI in parallel with SCA:

# JS ecosystem
npx license-checker --production --summary
npx license-checker --production --failOn "GPL-3.0;AGPL-3.0;SSPL-1.0"

# Python
pipx run pip-licenses \
  --fail-on="GPL.*;AGPL.*" \
  --allow-only="MIT;Apache-2.0;BSD-.*;ISC;MPL-2.0"

# Professional tools: FOSSA, ScanCode Toolkit, Snyk License Policy

I recommend a whitelist-based policy: by default only allow permissive licenses like MIT / Apache-2.0 / BSD / ISC; for MPL-2.0 (weak copyleft), check whether it's linked; for LGPL, check whether it's dynamically linked; for GPL / AGPL, always require manual review. Write this policy into CI—new dependencies that aren't compliant are blocked outright, with no room for favors.

5. Model Supply Chain—The Weights Themselves Are Code

This risk is more upstream than 'code the AI wrote.' The .bin / .safetensors / adapters you load—especially older weights in pickle format—can execute arbitrary code during deserialization. Hugging Face has had multiple historical incidents where a user-uploaded 'model' was actually a malicious pickle, and loading it resulted in RCE. In LLM agent scenarios, the model can also read the .huggingface/ cache, local files, and environment variables, making the harm even greater.

Defenses:

• Prefer safetensors: .safetensors is a pure tensor format and cannot execute code. Hugging Face now mandates that new models default to safetensors; pickle weights must have their source explicitly confirmed.
• Trusted sources: use only official organization accounts (meta-llama, mistralai, Qwen) or high-star, long-maintained repositories. Look at commit history, contributors, and download counts.
• Sandboxed loading: when loading unknown weights locally, isolate with a container—

# No network, read-only root, tmpfs workspace
docker run --rm --network=none \
  --read-only --tmpfs /tmp \
  -v ./weights:/w:ro \
  huggingface/transformers-pytorch-gpu \
  python -c "from transformers import AutoModel; AutoModel.from_pretrained('/w')"

• Scan before loading: picklescan and modelscan (ProtectAI) can detect suspicious opcodes.

pipx run picklescan --path ./weights

6. Bundle Defenses into CI—SCA / SAST / License + Dependency Governance

Point tools are useless on their own; you must chain them into a CI pipeline that runs automatically on every PR and fails when non-compliant. Minimum viable pipeline:

# .github/workflows/supply-chain.yml (excerpt)
on: [pull_request]
jobs:
  sca:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm audit --audit-level=high          # known vulnerable dependencies
      - run: npx @snyk/snyk test || true           # or OSV-scanner / Dependabot
  license:
    - run: npx license-checker --failOn "GPL*;AGPL*"
  sast:
    - uses: github/codeql-action/init@v3
    - uses: github/codeql-action/analyze@v3
    # or semgrep ci --config=auto
  ai-review:
    - uses: ./.github/actions/ai-sec-review         # custom: call LLM + security prompt

Dependency governance (a long-term effort)—scanning once isn't enough; you must build a long-term mechanism:

• Principle of minimal dependencies: if you can write ten lines yourself, don't install a library. The AI tends to 'import everything'; during review, apply reverse pressure: is this dependency really necessary?
• Periodic audits: run npm audit / pip-audit / osv-scanner as a cron job; auto-open issues for new CVEs.
• Renovate / Dependabot: automatically upgrade patch versions to reduce the backlog of 'afraid to upgrade.'
• SBOM: generate an SBOM (CycloneDX / SPDX) for every release artifact, so when a vulnerability surfaces, you immediately know 'am I affected?'

# Generate SBOM
npx @cyclonedx/cyclonedx-npm --output-file bom.json

7. The Human Gate—AI-Written Code Is Third-Party Code

All the tools are in place, but the last gate is still human. The most dangerous situation I've seen isn't the AI writing a vulnerability; it's a reviewer clicking approve because 'the code looks clean'—clean is not the same as secure.

A few practical rules:

1. By default, do not trust AI-generated code. During review, treat it like a PR from an unfamiliar contributor: check dependencies, check input validation, check auth.
2. Require authors to flag the AI-generated portions. Add a checkbox to the PR template: 'AI-generated code accounts for ___% of this PR and has passed SAST + license scanning.' Transparency enables accountability.
3. Force human review on high-risk changes: auth, payments, secret handling, and deserialization—whether written by AI or human—must have two-person review, and CI cannot override this.
4. Establish a team AI usage baseline: which scenarios allow AI (scaffolding, tests, documentation) and which require human gatekeeping (core business logic, security modules). Write it into CONTRIBUTING.md, not just an oral agreement.

Summary

• AI programming expands the supply chain attack surface from 'dependencies' to 'generation + dependencies + models'; the four risk categories must be governed separately: generated vulnerabilities, hallucinated dependencies, licenses, and model weights.
• Defense relies on CI automation, not human memory: SCA (npm audit / Snyk / OSV) + SAST (Semgrep / CodeQL) + license scanning + AI code review, one fail-fast pipeline.
• Hallucinated dependencies are a new attack unique to AI, and name-squatting poisoning is real—install only from official sources, verify package age and download counts, use lockfiles + private mirrors.
• Treat model weights as code: prefer safetensors, trusted accounts, sandboxed loading, and picklescan scanning.
• Long-term, rely on governance: minimal dependencies, Renovate auto-upgrades, SBOM, and periodic audits.

Bottom line: AI-written code is not your code—it's third-party code. Review it with the same suspicion and scan it with the same tools, and only then can you confidently enjoy AI's tenfold speed.