AI red teaming vs Penetration test

Why they differ

A pentest looks for ways into systems and data; AI red teaming also hunts for harmful, false or rule-breaking answers from the model itself.

AI red teaming

AI risk & governance

Testers attack an AI system on purpose, before and after release, to find ways it can be tricked into harmful or unsafe behaviour.

Formal

A structured, authorised exercise in which people, often helped by automated tools, act as attackers against an AI model or product to find harmful output, prompt injection, rule breaking, data leaks and other failures, and report them so they can be fixed.

In plain English

Like hiring clever troublemakers to spend a week trying to talk a new shop assistant into breaking every rule, so you learn where the training falls short before real customers do.

In practice

Before a municipality opens a chat assistant for citizens, a team spends two weeks trying to make it reveal other citizens' case details, give wrong advice about benefits or ignore its rules, and each trick found is blocked.

Why it matters

AI systems fail in ways ordinary functional tests never try, and the EU AI Act requires this kind of attack testing for the largest general-purpose models, whose failures could cause harm across society.

Penetration test

Controls & technical basics

An agreed, simulated attack in which skilled testers try to break in, to show how well systems really hold up.

Formal

An authorised test, limited in scope and time, in which people use the same methods as real attackers to find and actually use weaknesses, chaining them together to show what an attacker could reach.

In plain English

Like hiring a professional burglar to try to break into your shop, with your permission, and then write down exactly how they got in.

In practice

Testers hired by a Danish region find a forgotten test website, use a weak password there to get onto the network, and within two days show the IT security manager they can read the finance folder.

Why it matters

Lists of weaknesses do not show how they combine; seeing a real path from outside to valuable data convinces management and shows what to fix first.