Corrigibility

Background

Corrigibility

As we build increasingly advanced AI systems, we want to make sure they don’t pursue undesired goals. This is one of the primary concerns of the AI alignment community.

However, it is very difficult to get an AI to learn the intended goal on your first try. (For more, see Goal Misgeneralization.)

To deal with the problem of AIs initially learning imperfect goals, we ideally want to design AIs that are corrigible: open to being modified or shut down.

Unfortunately, by default advanced AIs are incentivized to prevent modification and shutdown because being modified or shutdown prevents the AI from achieving its goals. This is an instance of the general phenomenon of convergent instrumental goals: goals that are helpful for a wide range of long-term goals.

There might be creative ways to design AIs to be corrigible. This contest is meant to promote progress on this problem.

Existing Approaches

There are some existing approaches that may be sufficient for training a corrigible AI. Examples include using reinforcement learning from human feedback (RLHF), debate, and adversarial training (note that these techniques are separate, but they can form an ensemble).

We expect these approaches will help us increase the chance that we develop a corrigible agent. However, they have serious limitations, and we’re not confident that they’ll work. Given enough optimization pressure, an agent trained with RLHF, for instance, might still learn instrumental goals such as seeking power, acquiring resources, deceiving operators, and avoiding modification and shutdown (and this problem applies to processes other than RLHF). Once tasks get difficult enough, debates between AIs may no longer be possible for humans to evaluate or coordinate. Adversarial training could help, but it is empirically extremely difficult to robustly achieve. 

It is unclear if these failure modes will arise or how soon they will arise. However, given the uncertainty and given the enormous stakes, we think the problem of corrigibility is worth taking seriously. Ideally, we would be more confident in our methods before training or deploying dangerously-advanced AI systems. 

We’re excited for researchers to try to (a) fully solve corrigibility, (b) propose partial solutions that may lead to progress in the future, or (c) improve upon existing approaches. We are also interested in submissions that make conceptual progress on corrigibility. See more below.