Proxies and Model-Selection Pressure

Module 2: Proxies and model-selection pressure

In the previous note, we introduced the machine-learning pipeline and discussed a harm scenario where a freshness-based predictor contributes to an unwanted purchase through the ordering rule and the timing of the merchant's payment and customer feedback. We keep the same harm \(L_F\), hazard \(H_F\), safety constraint \(SC_F\), and baseline ordering controller. The discussion of underspecification showed seemingly incidental choices can end up determining which predictor is selected. Selection itself is something already encountered regularly with train-test splits and empirical loss minimization.

Now we ask how objectives, learning procedure, and development choices can favor particular predictors contributing to undesired selection properties. Let \(G\) represent the target property and \(M\) the measured criterion used for selection. The target may be observable, but simply too expensive or underrepresented to be effectively available for evaluation. Recall the food-ordering setting. The intended model-level target is accurate preference prediction for this customer between food types at equal prices and comparable, acceptable freshness, while the development criterion measures prediction of recorded choices, which may confound food type with freshness. A gap here creates the possibility of failure. Achieving this model-level target supports the prediction-quality requirement, but does not by itself establish \(SC_F\). As a reminder, failure does not necessarily imply a hazard, and a hazard does not necessarily result in harm.

What happens when we use a proxy \(M\) for selection? There are two gaps that we will distinguish. First, its finite-sample measurement may inaccurately estimate the population value of that criterion. This is a generalization problem. In this note we want to distinguish this gap from a second: the criterion may not fully capture the intended construct even when measured accurately.

Shortcut learning

Shortcut learning is about what solutions the criterion allows and learning favors [1]. Geirhos et al. articulate this, writing, "Shortcuts are decision rules that perform well on standard benchmarks, but fail to transfer to more challenging testing conditions, such as real-world scenarios." In the context of artificial intelligence (AI) safety, shortcuts are predictive rules that succeed under development conditions but do not support the intended generalization.

Returning to the food preference example: freshness might predict recorded choices, so a predictor can perform well without learning the food-type preference needed for matched-freshness comparisons. Detection requires evaluations that distinguish the suspected shortcut from the intended behavior. In the previous note three strategies were presented: stratified, shifted, or contrastive evaluations.

The shortcut phenomenon makes the difference between generalizing to new samples from the development distribution and generalizing under the required conditions of the application explicit.

Goodhart effects.

A shortcut shows that success on the development criterion can coexist with failure of the intended generalization. What happens to the relationship between measured success and the target when we increasingly select or optimize for that criterion?

Manheim and Garrabrant categorize variants of this phenomenon by which the relationship between the intended target and the proxy can become less reliable under selection pressure [2]. Goodhart effects are about how proxy-based selection can separate measured success from the target. Target performance need not decline absolutely; measured improvements may simply overstate genuine improvements.

From the paper, there are several ways this can happen: regressional, extremal, causal, and adversarial. Not every shortcut leads to a Goodhart effect, not every Goodhart effect comes from a shortcut, and the taxonomy is neither mutually exclusive nor exhaustive.

Regressional effects can exploit favorable measurement errors to select extreme proxy values. This selection goes beyond genuine target performance. Suppose the target and proxy are related by \(M = G + \eta\) where \(\eta\) is measurement noise. A high measured score could reflect target performance improvement or "lucky" measurement error.

Extremal effects can occur when selection operates over regions of the model space beyond just input space where the proxy-target relationship differs from oridinary conditions. This might happen when an inference relies on extrapolation beyond the supporting region of the development data.

Causal effects show up when an intervention changes the relationship on which the proxy's interpretation depends. Just because there is an observational relationship between proxy and target does not mean intervening to improve the proxy has the intended effect on the target. For example, simply displaying a shorter wait time on a rideshare app does not actually make a driver arrive faster.

Adversarial effects happen when another optimizer with different goals exploits the criterion. This can increase measured success without the corresponding improvement the evaluator intended.

Doing a Goodhart classification does not replace specifying a harm scenario. Specifying the harm scenario would additionally identify what is selected, the criterion, and how the resulting behavior could contribute to the unsafe control action.

Example: Model-selection and leaderboard pressure.

Leaderboards and benchmarks estimate performance with respect to a specified task distribution. Usually, a fresh held-out benchmark estimates this target. However, in practice, performance on a finite and reusable evaluation set is often used as a proxy.

One selection pressure in this setting is that researchers or automated searches may repeatedly choose and revise models using reported scores. Importantly, the examples or item labels themselves need not be public. This can result in a failure mode where selecting the best model can implicitly select favorable estimation noise rather than better population performance. This means the evaluation set is no longer independent of the selected predictor, so the simple held-out argument for generalization no longer applies without an additional argument accounting for selection.

In the ordering example, an overoptimistic score can favor a predictor whose error under the ordering conditions is higher than reported. Score inflation here is upstream of the harm scenario. The harm scenario itself is recreated if an order is followed by automatic submission and nonrefundable payment before customer feedback.

One way to address this failure mode is to update the reported best-so-far score only for sufficiently large improvements over the previous report, and round the released value. This is the "Ladder" mechanism proposed in Blum and Hardt's paper [3]. The mechanism limits informative feedback and then can bound the leaderboard estimation error under the method's statistical assumptions.

Here is an extension of the requirements/evidence table:

Example: Length in reinforcement learning from human feedback (RLHF)

Reinforcement learning from human feedback trains a large language model (LLM) using a learned preference model. Reinforcement learning will be discussed in more depth in the next module. For this example, replace the food alternatives with a pair of LLM responses \((R_1, R_2)\) to the same prompt \(c\), and collect recorded preferences between them.

Here the target is response quality under the intended task criterion. The proxy is the learned scalar reward score. From the previous note, this can be formulated as a maximum likelihood estimation of a shared scoring function \(r_\phi(c,R)\) assigning scores to each response \(R\) given prompt \(c\), followed by a logistic map \(\sigma(\cdot)\) that takes the score difference and converts it to a preference probability:

This is preference-model fitting. Downstream of this an optimization procedure tunes the LLM using the resulting preference model. Notice the distinction: predicting recorded comparisons well and providing a useful objective for downstream use are separate claims.

Indeed, one selection pressure in this setting is that policy optimization favors responses receiving higher predicted utility. Notice the shift from which predictor we obtain to which outputs the system actually produces. This can produce a failure mode where, if some superficial feature predicts preference judgements in the training comparisons, the reward model may rely on it. This can result in improvements that increase reward without increasing quality.

Singhal et al. investigate this in practice [4] through observational comparisons and interventions on the RLHF pipeline. Specifically, in their studied settings, increasing response length accounts for much of the reward improvement, and a length-based reward reproduces much of the measured preference improvement. While length correlation alone neither establishes a shortcut nor a Goodhart effect, it also does not establish that longer responses are worse or that harm occurs.

One way to investigate this failure mode is to intervene on preference data, reward modeling, and optimization. In the paper, these interventions provide empirical evidence about where length-related behavior originates and which changes reduce it, not a general guarantee on response quality. No intervention works uniformly across their settings.

Another approach is to cap the response length, which directly limits length. Again, imposing only a soft penalty can discourage behavior but doesn't necessarily produce a hard bound. Measuring the effectiveness of a proposed constraint requires additional supporting evidence.

The main takeaway is that a particular failure mode may originate as one part of a larger development pipeline. This example transfers the proxy analysis beyond food ordering; a separate harm scenario still requires specifying the system using the responses, its relevant control actions, and the conditions under which those actions could lead to harm.

From selection to behavior under changed conditions.

Up to this point, it has been established that selection can affect the predictor and which outputs from the pipeline are obtained. The examples demonstrate how optimization pressure can exploit estimation error or the gap between the intended target and proxy. In the ordering system, this can affect which predictor enters the controller and why its development evidence may be insufficient. However, we have not yet a characterization of which changes in the inputs or deployment conditions the pipeline's predictor can tolerate.

Testing for consequential underspecification and understanding how shortcut learning and Goodhart effects can exploit such gaps is a foundation for understanding how to address such failure modes. In the next note, we hold the predictor fixed and ask which changed conditions it can tolerate. This does not remove optimization: an adversary may still optimize the inputs presented to the fixed predictor. We revisit the shared assurance question: which conditions support the evidence, and does the intended claim extend beyond them?

References.

[1] Geirhos, Robert, et al. "Shortcut learning in deep neural networks." Nature Machine Intelligence 2.11 (2020): 665-673.

[2] Manheim, David, and Scott Garrabrant. "Categorizing variants of Goodhart's Law." arXiv preprint arXiv:1803.04585 (2018).

[3] Blum, Avrim, and Moritz Hardt. "The ladder: A reliable leaderboard for machine learning competitions." International Conference on Machine Learning. Proceedings of Machine Learning Research (PMLR), 2015.

[4] Singhal, Prasann, et al. "A long way to go: Investigating length correlations in RLHF." arXiv preprint arXiv:2310.03716 (2023).