COAI Research · Threat modeling & evaluation
People keep their agency only as long as they can notice what AI systems do, make sense of it, and still step in or undo it. We study where that ability breaks down and what sustains it.
How we work
Trace how capabilities, access, incentives and shared dependencies could lead to harm. Identify the actors, assumptions and barriers at each step.
Compare alternative explanations. Assess plausibility, severity and uncertainty using existing research, incident evidence and critical review.
Output: a model people can inspect, challenge and use in a decision.
Threat modeling in detailTurn consequential uncertainties into testable questions. Compare safeguards, reproduce findings and report where the evidence stops.
Output: evidence that strengthens, weakens or revises a specific assumption.
Safety evaluation in detailOur internal working environment. See how we use RiskGraph: define events and dependencies, retain evidence and disagreement, and examine what a probabilistic model implies.
Threat modeling in practice
Our proposed programme brings original modeling together with review and synthesis. Each model should make its assumptions, counterarguments and revision history visible.
RiskGraph, our internal threat-modeling platform, makes event definitions, evidence, judgments and uncertainty inspectable. Its versioned workflow supports the cumulative research we want to build.
Pathways, actors, barriers and uncertainty.
Synthesis, counterevidence and independent critique.
Methods, inspectable traces and bounded findings.
What the evidence means for a specific decision.
Evaluation in practice
This pilot tests the first link of the oversight chain: whether a monitor that is not built by the agents themselves can notice a violation in time. Our pilot investigates whether JEV and comparison models can recognize changed permissions and judge an agent’s next action. We examine decision quality and response time to assess their potential for this monitoring use case.
18 scenario families · ten monitor configurations
Independent label review pending
Our enduring mission
COAI is a non-profit research institute in Germany. Our wider purpose is to help people and institutions keep the ability to notice, understand, decide, intervene and reverse what AI systems do.
Threat modeling and targeted evaluation give this mission a concrete starting point. Interpretability and human–AI collaboration remain part of the wider research context.
The argument behind the mission, in one note: When is cognitive offloading benign?
Research community
Events connect our research with people who can challenge and develop it.
Fürth, Nuremberg region
Nuremberg, Germany
Discuss a research question, contribute a critique, or support independent research.
COAI Research — an Affiliated Research Institute of Ansbach University of Applied Sciences
