Module 1: What is the object of assurance?
Safety claims require specifying the intended system under consideration. It is tempting to ask, "Is AI safe?" because discourse often benefits from productive ambiguity. Indeed, AI at a system-level is a bit of a moving target in terms of deployment, capabilities, and operating conditions. A system boundary attempts to separate which parts of the system are the object of assurance and which parts are treated as part of its environment.
The object of assurance is the bounded artifact, system, or institution about which a claim is actually made. This is distinct from the unit of analysis, which provides a representation for generating well-specified claims. System boundaries are then specifications of the variables intended to be modeled in a system, not inherent factorizations of the world. A system boundary helps determine which hazards are plainly visible, which aspects are treated as endogenous, and ultimately which claims are falsifiable.
Key principle: choose a system boundary narrow enough to analyze, but broad enough to contain the variables and interactions relevant to the claim.
In this course, there are three stages of implementation: modeling, learning, and inference. There are also the institutions that develop, evaluate, or regulate them. We consider a previous example system: a content filter trained on crowd-sourced toxicity labels. Let's consider what it means to treat each stage as an object of assurance.
Running example.
A platform trains a classifier to identify toxic comments. Human annotators provide toxicity labels and the resulting data is used to train a classifier model. At inference time, a decision rule is used to choose which comments are shown and which are filtered for moderation. Any user is able to appeal a moderation decision. The platform is able to retrain or adjust the model periodically.
Earlier, we identified a potential hazard. The classifier learns a spurious correlation between dialect and toxicity resulting in a high-false positive rate for comments in that dialect. A potential harm is the systematic flagging of non-toxic comments written by speakers of that dialect.
However, "where" is the relevant safety problem located?
Models as object of assurance.
The most tightly scoped object of assurance is the trained (reflex) model itself. This specifies the system boundary, and represents the object of assurance as a function
which maps a comment \(x \in \mathcal{X}\) to a toxicity score \(y \in \mathcal{Y}\). The annotation process, platform and moderation policy, and users are then treated as part of the environment.
This considers the model through a black-box interface. We might instead assume white-box access to parameters, architecture, or internal activations. Either way, such an object boundary scopes assurance claims, the question is what useful assurances remain.
For example, we might ask for an assurance about the false-positive rate between different linguistic groups, calibration compared to human annotators, or even whether the function is "overfit." All of these require additional assumptions about access to evaluation or training data. However, the key principle is satisfied: the boundary is narrow, but can contain relevant interactions to input/output comparisons.
The reason this distinction matters is that we do not include development or deployment in the object of assurance. So we must make assumptions about the evaluation process through which we obtain human annotations. Often claims lie outside this boundary: evidence that concerns one object of assurance while the claim is about another. These might take the shape of model-level evaluation outcomes used to justify qualities about development or deployment.
Selbst et al. describe these sorts of failures as abstraction traps [2]. Of course, while abstraction is necessary for analysis, it can also erase interactions responsible for the claim under investigation. This is why well-scoped abstractions are key to making justifiable claims.
Development pipelines as objects of assurance.
The object of assurance can be expanded to include activities such as data collection, annotation, or model and objective selection.
In the content filtering example, an observed dialect correlation may originate from several places. Crowd workers may systematically assign higher toxicity scores to particular dialects. The sampling procedure may underrepresent or misrepresent certain associations. The training process itself could also prioritize aggregate metrics over certain subgroups.
Here is a familiar claim that becomes available with learning as the object of assurance: generalization. We can compare performance during training with performance on a validation set. The difference between performance is the generalization gap. Formal claims about a learning procedure become attainable using probabilistic or statistical learning theoretic methods.
However, even at this point, generalization does not yet imply good performance under realistic deployment conditions. Assurance of the model, the process that produced it, and the performance in the intended setting remain distinct problems.
Deployed systems as objects of assurance.
The boundary can be expanded yet again to include the system in which the model is actually used. The content filter is no longer an isolated scoring function or a well-fit classifier. Now the model is used at inference time for decision-making using its predictions. Those predictions feed into decision-rules for moderation, inform the moderation policy and appeal process, and ultimately begin to affect future training data as the implicit norms users rely on are partially determined by the model itself.
Amodei et al.'s Concrete Problems in AI Safety illustrates the importance of preserving awareness of system boundaries when abstracting [1]. They identify five recurring sources of accident risk: negative side effects, reward hacking, scalable supervision, safe exploration, and robustness to distribution shift. Viewed through the lens of system boundaries, these problems do not all live within the same object of assurance. Reward hacking is about the relationship between the objective and the behavior it induces. On the other hand, scalable supervision is concerned with the process through which behavior is evaluated. These problems may interact, but the underlying object of concern nevertheless shifts. The taxonomy therefore can be read as suggesting a useful factorization of an AI system: a collection of objects and interactions about which assurance claims might be made.
Returning to the content-filter example, the relevant system boundary may include not just a classifier model, but also its decision rule, its role in the moderation process, and downstream user effects on future training data. This generates new hazards which are only visible because different components interact.
At this point the AI system is naturally understood as a sociotechnical system: its behavior is produced through technical components interacting with people, social context, and organizational procedures. Whether a classifier's false-positive rate causes harm becomes something that depends on the system the model is embedded within. Is the moderation process consequential i.e. can users be banned or otherwise penalized? Do users respond to moderation through changed engagement e.g. leaving the platform? The deployment context contains feedback loops that lie outside the model or learning boundary.
This is the central concern of Fairness and Abstraction in Sociotechnical Systems. Selbst et al. argue that properties attributed to technical systems can depend on the social context in which the system is embedded [2]. These interactions often vanish when the technical artifact is abstracted from its original context.
The role of institutions.
Institutions, in this context, are organizations and processes that specify, develop, deploy, evaluate, or regulate other organizations or processes. These types of institutions are important because they concern the process by which safety judgements are produced and legitimized. As an example, they may produce and legitimize a claim that a particular evaluation process reliably detects a specified class of safety-critical hazards before they become harmful.
This raises additional questions. Who chooses the evaluation criteria? What incentives does an evaluator or platform have? Are incidents documented and does the organization have accountability for responding? Finally, who has the authority to pause or remediate a deployment after evidence of a hazard?
On the other hand, just because institutions provide coverage of scope does not mean they should always be included in the system boundary. This choice should be made intentionally, and it should be always be kept in mind that claims about an institution cannot be justified solely based on technical artifacts it oversees.
Boundary Misspecification.
Ultimately, there is no universally correct system boundary. Moreover, given an intended claim, a boundary can nonetheless be inadequate. The boundaries we considered, and the problems that arose, are illustrations of what happens when the specified boundary does not contain the necessary mechanisms or sufficient access to observational evidence to support the intended claim.
This returns us to the key principle. Choose a system boundary narrow enough to analyze, but broad enough to contain the variables and interactions relevant to the intended claim. Once that is fixed, the next question becomes more precise: what exactly are we claiming about it, and what evidence would justify that claim?
References.
[1] Amodei, Dario, et al. “Concrete Problems in AI Safety.” arXiv preprint arXiv:1606.06565, 2016.
[2] Selbst, Andrew D., et al. "Fairness and abstraction in sociotechnical systems." Proceedings of the conference on fairness, accountability, and transparency. 2019.