{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/ai-supply-chain-attack","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/ai-supply-chain-attack/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/ai-supply-chain-attack/"},"term":{"en":"AI supply chain attack","da":"Angreb på AI-forsyningskæden"},"aka":{"en":["ML supply chain attack","model supply chain attack"],"da":["angreb på ML-forsyningskæden"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"model","status":"current","summary":{"en":"Attacking the ready-made parts an AI system is built from - shared models, datasets or plug-in files - instead of the system itself.","da":"At angribe de færdige dele, et AI-system er bygget af - delte modeller, datasæt eller tilføjelser - i stedet for selve systemet."},"body":{"formal":{"en":"Tampering with third-party parts of an AI system - model weights, add-on files such as LoRA, training data, model files that run hidden code when loaded, or copycat model pages on public hubs; OWASP lists it as LLM03:2025.","da":"Manipulation af tredjepartsdele i et AI-system - modelvægte, tilføjelser som LoRA, træningsdata, modelfiler, der kører skjult kode, når de indlæses, eller efterlignede modelsider på offentlige hubs; OWASP fører det som LLM03:2025."},"plain":{"en":"Like buying a spare part from a shop whose name differs by one letter from your usual one - it fits and works, but someone altered it before it reached you.","da":"Som at købe en reservedel i en butik, hvis navn afviger med ét bogstav fra den, du plejer at bruge - den passer og virker, men nogen har ændret den, før du fik den."},"inPractice":{"en":"A developer in a pension fund's IT department downloads a popular open model from a copycat page; loading its old-style file quietly runs code that opens a back door on the build server.","da":"En udvikler i en pensionskasses IT-afdeling henter en populær åben model fra en efterlignet side; når den gamle filtype indlæses, kører der stille kode, der åbner en bagdør på byggeserveren."},"whyItMatters":{"en":"Most teams build on models and data they did not make and cannot fully inspect, so one poisoned download can reach every product that uses it.","da":"De fleste teams bygger på modeller og data, de ikke selv har lavet og ikke fuldt kan efterprøve, så én forgiftet download kan nå ud til alle produkter, der bruger den."}},"deepDive":{"en":"An ML system inherits a longer dependency graph than ordinary software: pretrained base models, fine-tuned derivatives and adapters (LoRA), tokenizers and config files, datasets and their download URLs, conversion and quantisation tools, inference servers, Python packages, and increasingly plugins, MCP servers and agent tools. OWASP LLM03:2025 (Supply Chain) covers tampered or vulnerable third-party components, outdated or deprecated models, unclear licensing, vulnerable LoRA adapters, and weak model provenance on public hubs; NIST AI 100-2 treats supply-chain compromise as an enabler for poisoning and backdoor attacks.\n\nThe most direct technical vector is unsafe deserialisation. PyTorch's legacy checkpoint format is a ZIP containing a Python pickle, and unpickling can invoke arbitrary callables through __reduce__, so torch.load on an untrusted .pt/.bin file is remote code execution. Similar issues exist with Keras Lambda layers (CVE-2024-3660) and joblib files. Researchers have repeatedly found malicious models on Hugging Face (JFrog reported around 100 in 2024), and ReversingLabs' 2025 \"nullifAI\" samples used deliberately broken pickles that evaded Picklescan. Mitigations are to load only weight-only formats such as safetensors, which stores raw tensors plus a JSON header and executes no code, to rely on torch.load(weights_only=True) (the default since PyTorch 2.6), and to run conversion jobs in sandboxes without credentials.\n\nThe subtler vector is behavioural: weights that execute no code but have been fine-tuned or edited to contain a backdoor or targeted misinformation. Mithril Security's 2023 PoisonGPT demonstration surgically edited one fact in GPT-J with ROME and uploaded it under a look-alike organisation name, and the model still scored normally on standard benchmarks. Such tampering cannot be found by malware scanning; detection requires provenance and behavioural evaluation. Look-alike and hijacked namespaces on model hubs play the role typosquatting plays in npm and PyPI, and agents that suggest non-existent packages open the related slopsquatting vector.\n\nControls follow classic software supply chain practice adapted to models: an internal model registry or proxy with an allow-list, pinning models by immutable commit hash rather than a mutable tag or \"latest\", verifying cryptographic signatures (for example OpenSSF model signing based on Sigstore), recording components in an ML-BOM (CycloneDX 1.5 added machine-learning BOMs; SPDX 3.0 has an AI profile), reviewing model cards and licences, re-running your own evaluation and red teaming on each new version, and treating datasets as versioned, hashed artefacts. Under the EU AI Act, providers of high-risk systems must document third-party components in their technical documentation, and Art. 25(4) requires written agreements with suppliers of AI tools, services, components or processes integrated into a high-risk system.","da":"Et ML-system arver en længere afhængighedsgraf end almindelig software: fortrænede basismodeller, finjusterede afledninger og adaptere (LoRA), tokenizere og konfigurationsfiler, datasæt og deres download-URL'er, værktøjer til konvertering og kvantisering, inferensservere, Python-pakker og i stigende grad plugins, MCP-servere og agentværktøjer. OWASP LLM03:2025 (Supply Chain) dækker manipulerede eller sårbare tredjepartskomponenter, forældede eller udfasede modeller, uklare licenser, sårbare LoRA-adaptere og svag sporbarhed af modellers oprindelse på offentlige hubs; NIST AI 100-2 behandler kompromittering af forsyningskæden som en vej til forgiftnings- og bagdørsangreb.\n\nDen mest direkte tekniske angrebsvej er usikker deserialisering. PyTorchs gamle checkpoint-format er en ZIP-fil med en Python-pickle, og unpickling kan kalde vilkårlige funktioner via __reduce__, så torch.load på en upålidelig .pt- eller .bin-fil svarer til remote code execution. Tilsvarende problemer findes med Keras Lambda-lag (CVE-2024-3660) og joblib-filer. Forskere har gentagne gange fundet ondsindede modeller på Hugging Face (JFrog rapporterede omkring 100 i 2024), og ReversingLabs' \"nullifAI\"-eksempler fra 2025 brugte bevidst ødelagte pickles, der slap uden om Picklescan. Modtræk er kun at indlæse rene vægtformater som safetensors, der gemmer rå tensorer plus en JSON-header og ikke afvikler kode, at bruge torch.load(weights_only=True) (standard siden PyTorch 2.6) og at køre konverteringsjob i sandkasser uden adgangsnøgler.\n\nDen mere subtile angrebsvej er adfærdsmæssig: vægte, der ikke afvikler kode, men er finjusteret eller redigeret til at indeholde en bagdør eller målrettet misinformation. Mithril Securitys PoisonGPT-demonstration fra 2023 redigerede præcist én kendsgerning i GPT-J med ROME og uploadede modellen under et organisationsnavn, der lignede det rigtige, og modellen scorede stadig normalt på standard-benchmarks. Den slags manipulation findes ikke med malwarescanning; det kræver sporbarhed og adfærdsmæssig evaluering. Efterlignede og overtagne navnerum på model-hubs spiller den rolle, typosquatting spiller i npm og PyPI, og agenter, der foreslår pakker, som ikke findes, åbner den beslægtede slopsquatting-vej.\n\nKontrollerne følger klassisk praksis for softwareforsyningskæden, tilpasset modeller: et internt modelregister eller en proxy med allow-list, fastlåsning af modeller på uforanderlig commit-hash i stedet for et foranderligt tag eller \"latest\", verifikation af kryptografiske signaturer (fx OpenSSF model signing baseret på Sigstore), registrering af komponenter i en ML-BOM (CycloneDX 1.5 indførte BOM'er for maskinlæring; SPDX 3.0 har en AI-profil), gennemgang af modelkort og licenser, egen evaluering og red teaming af hver ny version, og at behandle datasæt som versionerede artefakter med hash-værdier. Efter AI-forordningen skal udbydere af højrisikosystemer dokumentere tredjepartskomponenter i den tekniske dokumentation, og art. 25, stk. 4, kræver skriftlige aftaler med leverandører af AI-værktøjer, tjenester, komponenter eller processer, der indgår i et højrisikosystem."},"edges":[{"type":"requires","to":"platform/software-supply-chain","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/model-weights","confidence":"high","strength":"normal"},{"type":"kind-of","to":"security/supply-chain-attack","why":{"en":"It is the same trick of striking through a trusted supplier, aimed at models and data instead of ordinary software.","da":"Det er det samme greb - at ramme gennem en betroet leverandør - rettet mod modeller og data i stedet for almindelig software."},"confidence":"high","strength":"primary"},{"type":"contrasts-with","to":"ai/slopsquatting","why":{"en":"Both abuse trust in downloads, but slopsquatting plants code packages under names an AI invents, while an AI supply chain attack tampers with the AI's own parts.","da":"Begge misbruger tillid til downloads, men slopsquatting planter kodepakker under navne, en AI finder på, mens et angreb på AI-forsyningskæden manipulerer AI'ens egne dele."},"confidence":"medium","strength":"normal"},{"type":"used-with","to":"ai/open-weight-model","why":{"en":"Freely shared model files are downloaded and loaded widely with little checking, making them an easy carrier for tampered weights or hidden code.","da":"Frit delte modelfiler hentes og indlæses bredt uden megen kontrol, hvilket gør dem til en let bærer af manipulerede vægte eller skjult kode."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/data-poisoning","why":{"en":"Poisoned datasets or fine-tuned weights shared publicly are one of the main ways an AI supply chain attack is delivered.","da":"Forgiftede datasæt eller finjusterede vægte, der deles offentligt, er en af de vigtigste måder, et angreb på AI-forsyningskæden leveres på."},"confidence":"high","strength":"normal"}],"depth":3,"sources":[{"title":"OWASP Top 10 for LLM Applications 2025 - LLM03 Supply Chain","url":"https://genai.owasp.org/llmrisk/llm032025-supply-chain/","tier":"reference","publisher":"OWASP"},{"title":"NIST AI 100-2 - Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations","url":"https://doi.org/10.6028/NIST.AI.100-2e2025","tier":"standard","publisher":"NIST"},{"title":"CERT/CC VU#253266 - Keras Lambda layers allow arbitrary code injection (CVE-2024-3660)","url":"https://kb.cert.org/vuls/id/253266","tier":"official-doc","publisher":"CERT/CC"},{"title":"PyTorch 2.6 release blog (torch.load weights_only default changed)","url":"https://pytorch.org/blog/pytorch2-6/","tier":"official-doc","publisher":"PyTorch Foundation"},{"title":"Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor","url":"https://jfrog.com/blog/data-scientists-targeted-by-malicious-hugging-face-ml-models-with-silent-backdoor/","tier":"other","publisher":"JFrog"}],"draft":true}