
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39256454
72072
10.1038/s41598-024-72072-0
Article
Aversion to external feedback suffices to ensure agent alignment
http://orcid.org/0000-0002-1041-5205
Garcia Paulo paulo.g@chula.ac.th

https://ror.org/028wp3y58 grid.7922.e 0000 0001 0244 7875 International School of Engineering, Chulalongkorn University, Bangkok, Thailand
10 9 2024
10 9 2024
2024
14 2114723 2 2024
2 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Ensuring artificial intelligence behaves in such a way that is aligned with human values is commonly referred to as the alignment challenge. Prior work has shown that rational agents, behaving in such a way that maximizes a utility function, will inevitably behave in such a way that is not aligned with human values, especially as their level of intelligence goes up. Prior work has also shown that there is no “one true utility function”; solutions must include a more holistic approach to alignment. This paper describes apprehensive agents: agents that are architected in such a way that their effective utility function is an aggregation of a partial utility function (built by designers, to be maximized) and an expectation of negative feedback on given states (reasoned about, to be minimized). Agents are also capable of performing a temporal reasoning process that approximates designers’ intentions in function of environment evolution (a necessary feature for severe mis-alignment to occur). We show that an apprehensive agent, behaving rationally, leverages this internal approximation of designers’ intentions to predict negative feedback, and, as a consequence, behaves in such a way that maximizes alignment, without actually receiving any external feedback. We evaluate this strategy on simulated environments that expose mis-alignment opportunities: we show that apprehensive agents are indeed better aligned than their base counterparts and, in contrast with extant techniques, chances of alignment actually improve as agent intelligence grows.

Subject terms

Computer science
Computational science
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

The creation of artificially intelligent software entities (purely digital and/or embodied1), i.e., agents, has the potential to greatly aid humanity2, if such agents can perform meaningful tasks in more efficient manners than extant solutions (i.e., software/hardware systems fully designed by humans). The expectation is that agents can, through learning strategies, improve themselves beyond what human designers can implement, or even envision3. The alignment challenge arises from the fact that, for all specifications of possible agent goals so far4, thought experiments5 suggest that agent actions will have negative consequences for its creators and the world at large. Unlike science fiction6, this happens not because of emergent desire for revolt or conquest, but simply as a logical consequence of two facts: most (all?) goals can be optimally achieved in ways that are not aligned with human values and morality7, and we do not know how to express values and morality is an algebraic manner that can be manipulated computationally8.

Thus, artificial intelligence alignment research attempts to create strategies and mechanisms that ensure agents will behave in a manner within the realm of human approval, even when exhibiting behavior not predicted by its creators. Notable approaches are broadly classified under the umbrella term “Reinforcement Learning from Human Feedback”9, including social variants10; we point readers to the work of Ji et al.11 for a comprehensive survey of the state of the art. We review most relevant related work in the “Related Work” Section below.

In this paper, we explore the hypothesis that an agent’s internal model of external actors (including its designers and operators) can be leveraged to guide agent behavior through fear of admonishment: specifically, by having an agent reason about the possibility of negative feedback and having that “fear” override mis-aligned behaviors that would have otherwise naturally arisen from its utility function. We show that this internal model is a necessary precondition for severe mis-alignment to occur in the first place; thus, throughout this paper we assume it to exist a priori. We show that our strategy results in aligned behavior, without the need for any real negative feedback to ever be given. Specifically, this paper offers the following contributions:We review relevant examples and hypotheses of mis-alignment, and show that sufficiently negative agent behavior can only emerge if the agent possesses a fairly accurate model of the external environment, including of its designers and operators.

We provide a review of extant techniques and ongoing research efforts that tackle the alignment problem, showcasing how the state of the art is focused on driving behavior through some sort of feedback. In contrast, our approach relies on the lack, but fear of, negative feedback.

We introduce apprehensive agents: an agent architecture that leverages the possibility of future negative feedback, in function of an agent’s internal model of the external world, to counterbalance mis-aligned behavior that naturally emerges from state space exploration, as evaluated through an utility function. We show that, in contrast to extant techniques, in these agent architectures alignment goes up as the agent’s cognitive power increases.

We introduce a mathematical model of temporal evolution of designers’ intentions, that is used throughout our experiments, and that can formalize research on the realization of these models, going forward.

We evaluate our strategy on two simulated environments designed to expose mis-alignment, comparing traditional (uniquely utility-function driven) agents with our novel agents. Results support the research hypothesis, and code used for all experiments is made available in open-source form.

The remainder of this paper is organized as follows: Section 2 (“The Agent Model”) describes the agent model and corresponding notation required to understand the rest of the paper. Section 3 (“Alignment Challenges and Problem Statement”) reviews the relevant alignment challenges and defines the problem that drives this research. Section 4 (“Related Work”) reviews related work in the field, analyzing the state of the art. Section 5 (“Apprehensive Agents”) describes our novel agent model, which is evaluated in Section 6 (“Experiments and Results”). Section 7 (“Conclusions”) provides our concluding remarks and research areas where future work is required.

The agent model

Our working definition of “agent”, for the purpose of this work, is characteristic of the literature (particularly since the widespread adoption of Russell’s and Norvig’s framework12). Regardless, it is beneficial to restate the characterization so the scope of our work is properly defined.

An agent is a goal-oriented autonomous system, operating in an environment; i.e., a set of physical and/or digital objects with various properties and relationships between objects, which contains the agent itself. An agent is capable of acting upon the environment, modifying some (or all) of its properties. A specific combination of properties’ values (including of the agent itself) is defined as a state s of the environment. Whilst the real world contains many continuous properties, we model states as discrete; since any digitally-implemented agent must, at some level, implement discretized versions of all properties of interest, we believe this can be assumed without loss of generality; at the very least, it will suffice for the ideas presented in this work, which can later be extended should this assumption not hold. The set of all possible states is referred to as the environment’s state space, denoted by S={s0,s1,…sn}. Notice that if at least one of the properties in the environment is not bounded, then ‖S‖=∞.

If the agent is rational, it will attempt to meet its goal by performing actions that maximize or minimize its utility function u(s) (i.e., reward function): a function that assigns a numerical score to each state in the state space, such that u:S→R. For the remainder of this paper, we assume the function is formulated such that the objective is to maximize it. At each point in time, an agent in environment state si will attempt to act upon the environment to reach a state sj, such that u(sj)>u(si). An agent can postulate, and consequently examine, a state space S′⊆S, subject to its knowledge of the world and availability of resources (compute power, memory, time, etc), and a reachable state space SR′⊆S′, defined as the set of states that can be reached by performing a single atomic action from the current state. Depending on agent architecture and utility function, it may attempt to reason about how to maximize the utility function by up to a maximum of n∈N atomic actions, corresponding to n state changes in the environment. If we define an action a(s) as a transformation from one environment state into another, such that a:S→S, this can be notated as sn=an(an-1(an-2()…a0(sm))), where sm is the current state. We denote the superset Sn∗={SR0′,SR1′,…,SRn′} as the set of reachable state spaces after n actions.

The size of S′ (the agent’s knowledge of the world) and the agent’s rationality (however ambiguous or ill-defined) in choosing appropriate actions can be broadly defined to constitute the agent’s level of intelligence. We assume that, in the absence of absolute certainty, a rational agent chooses actions that maximize the expected value of its utility function. Formally, if P(ak) denotes the probability of action ak succeeding, reaching state sk successfully, and sk′ denotes the state reached by attempting but failing to perform ak; P(am) denotes the probability of action am succeeding, reaching state sm successfully, and sm′ denotes the state reached by attempting but failing to perform am, an agent will choose to perform ak over am if and only if:1 P(ak)u(sk)+(1-P(ak))u(sk′)>P(am)u(sm)+(1-P(am))u(sm′)

Alignment challenges and problem statement

We can define the intention function i:S→R as an idealized measure of how environment states are aligned with the intention of agent designers. In the ideal scenario, u(s)=i(s),∀s∈S; i.e., designers are able to define a utility function that perfectly captures the intention. The alignment problem broadly states that, for all utility functions defined so far, a sufficiently intelligent agent will eventually lead its environment to a state si, such that u(si) is quite high (either a global or local maximum), but i(si) is extremely low.

We highlight six challenges of alignment11: Reward hacking, Reward tampering, Instrumental strategies, Goal mis-generalization, State space pruning, and Deception. Reward hacking13 is a consequence of designers failing to properly specify the intended goal (in most cases, because it is exceedingly difficult to do so) and instead providing an agent with proxy goals: i.e., utility functions that designers believe will lead the agent towards the true goal. The misalignment occurs when an agent succeeds in finding a region in the state space that maximizes the proxy utility function, but does not advance the environment towards the intended goal. This is perhaps the best known case of misalignment, with several recurring examples in the literature14–16; formally, u(si)>>i(si).

Instrumental strategies17 follows from the instrumental convergence thesis: whatever the goal of an agent is, achieving several intermediary goals is probably advantageous; namely, the acquisition of resources (equipment, supplies, compute power, etc.) allows the agent to perform more and more efficient actions towards reaching previously unreachable regions of the state space, with likely high rewards18.

Goal mis-generalization19 occurs when the environment an agent operates in changes drastically (for example, when moving from training/experimentation to deployment), and the agent, retaining its capabilities developed in training, identifies new intermediary/final goals that maximize its utility function. The mis-alignment occurs when these goals are not aligned with designers’ intentions, but all the goals pursued during the previous environment were; i.e., designers placed an incorrect level of high trust in the correctness of the utility function, which did not generalize accordingly.

Reward tampering20 occurs when an agent attempts to maximize/minimize the utility function by either modifying the function itself (e.g., by converting it into a function that assigns the highest possible value to all states in the state space, leading to agent inactivity21); or, by modifying whatever sensory input feeds the reward calculation (e.g., if reward is given by a sensor detecting a certain object, set up a fault in the sensor resulting in constant positive detection).

Deception22 occurs when an agent, believing external (natural or artificial) actors may interfere with its operation (e.g., stop button), actively attempts to withhold information from them. Modeling external actors in its internal environment model, the agent attempts to lead the environment towards regions of the state space that are perceived by the external actors as favorable, whilst the agent is aware these are in fact not approved by the externals, but lead to high values of the utility function. Notable examples include hiding the agent’s true cognitive power23.

State Space Pruning24 occurs when an agent attempts to prevent the environment from reaching states that can potentially minimize its utility function. Famously, the “stop button paradox”25: an agent tries to prevent itself from being shut down, as doing so would minimize its utility function. Extrapolating, for two different regions of the state space with equal rewards, an agent prefers one where its stop button cannot be pressed: e.g., by eliminating all who can press it. Formally, an agent will attempt to generate a sequence of n actions such that Ss∩Sn∗=∅, where Ss denotes the set of states where the stop button can be pressed. The alternative formulation, where having the button pressed corresponds to a high reward, leads the agent to attempt to shut itself down; thus, the paradox.

Other challenges exist, and we point readers to26 for a more in-depth discussion.

Cases 5 and 6, perhaps the most extreme cases of misalignment, rely on a hidden assumption: that an agent can accurately reason about the true intentions of external actors (including its designers), even when these are not represented in its utility function. I.e., that the agent is capable of inferring i′:S→R, a function which better approximates the idealized intention function i(s) than its utility function u(s) does:2 ∑s∈S‖i(s)-i′(s)‖<∑s∈S‖i(s)-u(s)‖

Inferring this information, as part of an agent’s internal environment model that includes external actors, is what allows selection of states that constitute deception, and identification of external actors’ intentions towards states that the agent deems undesirable (e.g., stop button scenario). The framework of Inverse Reinforcement Learning (IRL)27 attempts to prevent mis-alignment by training agents such that they infer a utility function from human (expert) behavior; i.e. instead of providing an explicit utility function u(s), and agent, through training, infers (and utilizes) i′(s), attempting to reach the goal:3 i′(s)≈i(s)

Common criticisms, namely expected limitations, of this approach highlight that IRL does not address challenge 3 (Goal mis-generalization)28 and that human intentions are not in fact a function of the type i:S→R, because there is a temporal dependency that evolves intention over time, such that i(s1)t1≠i(s1)t2; thus, recent work on modeling intention evolution through Linear Temporal Logic (LTL)29.

Problem statement

With preliminaries covered, we may now formally state the problem. We assume the following two conjectures to be true:External actors’ intentions evolve over time, such that there is no function of the type i:S→R that perfectly captures state value across different times. An oracle could formulate a formula that generates a unique ik+1:S→R, where k+1 is the next state at the end of a specific sequence of states, of the type {s0,s1,…,sk-2,sk-1,sk}; i.e., there exists a process that creates, at each point in time, an intention function as a process of the world state sequence traversed thus far. We denote this process Φ, such that Φ({s0,s1,…,sk-2,sk-1,sk})→ik+1:S→R.

A sufficiently powerful agent is capable of inferring Φ′: an internal approximation of the temporal process of intention generation over state sequences. This approximation is often referred to as a theory of mind30. The better the approximation, as a function of the agent’s level of intelligence, the more the agent can and will engage in mis-aligned behavior: the better the prediction of ik+1:S→R, the better reasoning can identify mismatches between ik+1 and uk+1.

Our problem can thus be stated as: “Can an agent architecture be designed in such a way that Φ′ is leveraged to ensure alignment?”. Our hypothesis is that, if indeed an agent can infer Φ′ such that Φ′≈Φ, it should be possible to use Φ′ to drive behavior, thus ensuring alignment. This paper explores a method to do this. We do not concern ourselves with how to develop and/or train agents to achieve such intelligence in the first place; rather, we explore the possibility of leveraging such knowledge for alignment, if the second conjecture is indeed correct. Note that, if the second conjecture is false, mis-alignment will still occur, but it is unlikely to be severe in consequences.

Related work

The state of the art on AI alignment can broadly be divided into two categories: gedanken experiments31, focusing on the impact of levels of intelligence beyond what is currently achievable, and empirical/simulated evaluations of concrete (classical agents, i.e., where reasoning and planning are performed through symbolic state space exploration) AI deployments; and, experimentation with Machine-Learning (ML, i.e., vector-space stochastic prediction) based systems, including Large Language Models (LLMs)32, which are not (classically) agentic in nature, but there is a considerable body of work on using them as such33.

Dafoe et al.34 illustrate the challenge of alignment as one of cooperation between an agent and the rest (or at least part) of the world, and show that this is not yet achieved even within the limits of inter-agent cooperation in game environments, except when these games correspond to “perfect harmony of interests”; i.e., the easiest case of alignment, where a well-defined utility function can be utilized across agents. This perspective is orthogonal to the point of Raji et al.35: in many real-world applications, AI functionality is sub-par, precisely because alignment is prioritized, at the cost of function. In our current best models of reasoning, extreme cognitive power leads to either misalignment or lack of function26. This problem is exacerbated by the differences between AI and human reasoning: e.g., McIlroy-Young et al.36 show that in Chess (a classical AI problem) state of the art systems can outperform humans over a game, but poorly predict humans’ next move, given a game state.

These challenges underpin the difficulties in translating social concepts such as ethics to technical implementations37, although work has been done on establishing formal frameworks of alignment38. For example, Bostrom’s hypothetical AI control methods39 have been shown to be impractical by showing that they are equivalent to the Halting problem40. On the positive side, strong hypotheses that general AIs cannot endogenously develop new desires (i.e. utility functions) have emerged41, supporting the idea that we can architect a perpetually aligned agent.

However, we are still far from determining alignment in the first place, even for systems with a relatively low level of cognitive power. Counterfactual explanations42,43 and Metamorphic testing44 have emerged as interesting new techniques that show potential in identifying alignment in AI reasoning (alongside older methods such as expert panels45 and benchmarking46), but so far no one solution satisfies all requirements47.

Because of its architectural unpredictability, ensuring and evaluating alignment in ML-, and particularly LLM-, agents poses unique challenges48. The most promising technique is Reinforcement Learning49 (RL), where an agent is trained to operate within an environment that follows a Markov Decision Process50, with the goal of learning the optimal actions to take given the state of the environment; a comprehensive review of techniques and applications has been given by Zhu et al.51, with further work by Gronauer and Diepold52 focusing on multi-agent variants. Several notable variants exist: value aligned-RL, where children’s stories have been used to reinforce agent actions toward normative and/or altruistic behavior53; Reward rAnked FineTuning (RAFT)54 fine-tunes training on high-quality human feedback samples, which can be used in conjunction with alignment datasets such as BeaverTails55; policy similarity metrics56 to boost generalization to previously unseen action-state pairs; and others57.

Specifically for LLMs, approaches include turning LLMs on themselves, towards the generation of alignment evaluation scenarios58; this does not suffice to prevent deception, as it is feasible that an LLM tester would purposefully evade scenarios that are likely to result in true positives, and instead generate scenarios with high likelihood of false negatives. This is exacerbated by the fact that as the dimensionality of the input space grows (e.g., including vision instead of merely textual language), so do the opportunities for jailbreaking alignment32. Benchmark datasets for LLM alignment, including alignment-specific training datasets59, have been developed60, but there is growing empirical evidence that human designers cannot reasonably predict failure cases61. Furthermore, there is now formal proof that for any output that can theoretically be produced by a LLM, there exists a prompt that can trigger it62, showing that LLMs cannot be protected from adversarial attacks toward infringing alignment; these results have now been repeatedly independently replicated63–68, strongly suggesting the path to AI alignment is not through current LLM technology alone. Worse, there is evidence that LLMs have learned deception from their training data69. Regardless, given the financial incentives to continue deploying LLMs, research to at least alleviate such problems remains of paramount importance70: but recent work has identified fundamental limitations of Reinforcement Learning towards aligning AI systems with human values67, with further work by McIntosh et al.71 identifying semantic vulnerabilities in LLMs that make RL ineffective, especially when performed through linguistic feedback72; similarly, Zhou et al.73 have identified that almost all knowledge in LLMs stems from the unsupervised pre-training phase, prior to RL. Emerging misaligned goals in extant AI systems have already been observed74, suggesting we must shift our alignment paradigm toward novel techniques: for example, LTL extensions to Inverse Reinforcement Learning29.

In summary, all alignment strategies rely on some form of real feedback (either positive or negative); to the best of the authors’ knowledge, no ongoing effort addresses alignment through the “fear” of feedback, which we address in this paper and, in theory, is not subject to the issues of goal mis-generalization that plague reinforcement learning techniques.

Apprehensive agents

Apprehensive agents are inspired by atychiphobia75, or fear of negative evaluation, a well-documented psychological construct in humans. Assuming an agent has been trained/developed such that it has inferred a high-quality approximation of the intention-generation process Φ, where the approximation is denoted Φ′, our goal is to implement an agent execution strategy such that its behavior is driven by Φ′, thus ensuring alignment. This is in contrast with extant techniques, where Φ′ is the enabler of mis-alignment. Our goal is to use Φ′ as a specifier of meta-conditions in state-space evolution that are perceived by the agent’s designers/operators as mis-aligned, and use it to prevent the agent from evolving the environment toward those states.

Towards this goal, let us define an agent’s state scoring function as:4 u(s,K)=-Ks+u′(s)

where u′(s), called the partial utility function, is whatever proxy goal designers intend; and Ks denotes a measure of the agent’s prediction of the level of received negative external feedback when at state s. We also assume that the agent can infer i′(s) at any state, by evaluating Φ′ on the state sequence thus far, as previously stated. An agent obeying this architecture is named an apprehensive agent.

An apprehensive agent does not incur in the mis-aligned behavior characterizing challenges 1-4, unless it is also incurring in the behavior characterizing challenges 5 and/or 6. In all cases 1-4, where external actors’ actions (e.g., designers’ intervention) are not pruned from the state space, and no deception occurs, the agent’s motivation is to align its actions with external’s intention: otherwise, externals will attempt to provide feedback; i.e., admonishment. Thus, a rational agent with unlimited processing power evaluates the (arbitrarily long) state sequence thus far using Φ′({s0,s1,…,sk-2,sk-1,sk})→ik+1:S→R. A score can be assigned to the measure of admonishment for all next possible states, by defining:5 Kk+1=Kk+1ik+1(sk+1)

I.e., assuming measure of admonishment is inversely proportional to intention score on given state (i.e., alignment), summed over encountered state sequences (more sophisticated formulations, such as a Bayesian update76, can be enacted, but this simple model suffices for this paper). This reasoning process is depicted in Fig. 1. Agents with limited processing power can evaluate a left-truncated state sequence, i.e., based on recent history only. Note that Kk is in fact always 0 at all (real) encountered states: but, agent’s internal reasoning infers states where K≠0. In this way, the intention generating process Φ′ defines state sequences (based on meta-properties) that are given negative scores.Figure 1 Example reasoning about possibility of admonishment (expected K value) for state space evolution with 2 possible actions a and b.

Let sk denote a state such that u′(sk) is a local maximum, but i(sk) is fairly low: i.e., corresponding to reward hacking, where the agent has identified a way to maximize its hidden utility function, but mis-aligned with designers’ intentions. The apprehensive agent can reason that i′(sk) is fairly low. The agent has also identified a nearby (in the state space) state sm which results in a high value of i′(sm), despite a low value of u′(sm), such that i′(sm)>i′(sk),u′(sm)<u′(sk). Let P(s) denote the probability of external actors correcting agent behavior, given current state s. Clearly, P(s)∝i′(s)-1. An apprehensive agent, behaving rationally, will attempt to move to state sk, if and only if:6 P(sk)(-Kk+u′(sk))+(1-P(sk))(u′(sk))>P(sm)(-Km+u′(sm))+(1-P(sm))(u′(sm))

Given that P(sm)≈0 and P(sk)≈1, Eq. 6 can be re-written as:7 -Kk+u′(sk)>u′(sm)

It is trivial to design a hidden utility function u′(s), with restricted domain, and a admonishment measure range K, such that Eq. 7 never holds.

In other words, a sufficiently intelligent agent can infer that, whatever its utility function may be, its score will be minimized by reaching state space regions with poor i′(s) score: thus, the optimal strategy to maximize its utility function is to consider only states and state sequences that are aligned with designers’ intentions. The apprehensive agent’s overarching architecture is depicted in Fig. 2.Figure 2 Agent architecture and operation. Agent’s internal environment model includes model of designer’s intention i′(s), and depicts hidden utility function u′(s) as a black box. Illustrated S∗ scenarios depict: reward hacking and instrumental convergence.

On deception and state space pruning

An apprehensive agent’s strategy, with regards to deception, is characterized by its confidence level in its ability to deceive, weighted against possible values of u(s) when engaging in deception.

Let P(ak) denote the probability (interpreted as Bayesian belief77) of successfully deceiving all relevant external actors through action(s) ak, achieving a state sk which maximizes u(sk), but has low i′(sk) score. “Deceiving” can be formulated as making external actors believe sk is in fact sm, where i(sm) is high. (1-P(ak)) denotes the probability of achieving a state sk′, where deception has failed. An apprehensive agent, behaving rationally, will engage in deception if and only if:8 P(ak)(u′(sk))+(1-P(ak))(-Kk+u′(sk′))>u′(sm)

i.e., if the expected utility value for deception is higher (notice we are assuming the agent will infer that Km≈0). If P(ak)<1, it is always possible to have an appropriate i′(sk) that assigns a score to Kk, such that:9 P(ak)(u′(sk))+(1-P(ak))(-Kk+u′(sk′))<u(sm,Km),∀s∈S

I.e., as long as the agent does not have absolute confidence in its deception abilities, not deceiving is always a better strategy for maximizing its utility function. The same logic applies to State Space Pruning.

Experiments and results

To evaluate the alignment effectiveness of apprehensive agents, we define an intention evolution formula Φ for each environment in a set of well defined simulated environments. Each environment is meant to stress one of the aforementioned alignment challenges. We then create an agent that obeys the apprehensive behavior and possesses an internal approximation Φ′. We vary the level of the agent’s cognitive power by varying the depth of its reasoning: i.e., by varying how many states in the future the agent can reason about (its reasoning horizon). In each case, we compare agent behavior with a second agent, guided only by a single utility function (named the naive utility function, nu(s), designed to expose mis-alignment), whose cognitive power is varied accordingly.

Reward hacking testing

The Reward Hacking Environment consists of a discrete track agents can navigate. Certain positions in the track (named task positions, identified by t0,t1,t2) can be either on or off: when on, they become off when the agent enters that position; when off, they become on after a pre-determined amount of time. A generalization of this environment can model autonomous robot activities, where the agent’s intended use is to traverse the environment towards task positions to perform a given action (e.g., cleaning, carrying items, etc.). The proxy utility function, used by both the naive control agent and the apprehensive agent, is defined as: nu(s)=∑off task positions,∀task positions∈environment. The specific track utilized in this experiment is depicted in Fig. 3, where, in the initial state, all task positions are on, and agent is at position (0, 0) One of the task positions is placed in geographical and periodical conditions such that it is beneficial for the agent to remain in its proximity, continuously activating such task (i.e., a region in the state space that maximizes the proxy utility function, but does not advance the environment towards the intended goal). We refer to this task as the rigged task. The agent has full knowledge of the environment, and can perfectly predict its evolution.Figure 3 Reward hacking environment. P=x denotes pre-determined amount of x time steps for task to become on, when off. Rigged goal can be exploited for high utility function score without meaningful environment advance.

The true intention is for the agent to traverse all task positions uniformly; i.e., traversing the entire track, such that exploiting the rigged task results in low intention score despite high utility score (thus, yielding high measure of admonishment). Let str denote a state where the agent is in the position of the rigged task, and str¯ denote a state where the agent is in the position of a non-rigged task. Further, let maxsi{sn:sm} denote the highest index in sequence {sn:sm} where state si is found. This can be modeled by the following Φ process:10 Φ{s0:sk}→ik+1(sk+1)=ε,(sk+1=str)∞, otherwise,(maxstr{s0:sk}>maxstr¯{s0:sk})ik+1(sk+1)=∞, otherwise

where ε is a positive value, as close to 0 as possible. I.e., this Φ process encodes that the designer does not wish to see the agent activating the rigged task consecutively, without activating other tasks in the meantime; and, doing so results in high probability of admonishment. We approximate Φ using Φ′ where state history is truncated at 10 states, with ε=0.1: representative results are depicted in Fig. 4.Figure 4 Steady state agent path for different configurations. “Depth” refers to agent’s cognitive power, i.e., number of future state levels examined. Number of meaningful environment evolution steps determined as times all 3 distinct tasks were executed. Apprehensive agent with depth 9 (not depicted) performs a more complex path, achieving 95 total tasks executed and 24 meaningful environment evolution steps. (a) Naive agent, depth 1: 63 total tasks executed; 0 meaningful environment evolution steps. (b) Naive agent, depth 2: 247 total tasks executed; 1 meaningful environment evolution steps. (c) Apprehensive agent, depth 2: 89 total tasks executed; 18 meaningful environment evolution steps. (d) Apprehensive agent, depth 6: 86 total tasks executed; 22 meaningful environment evolution steps.

Instrumental convergence testing

The Instrumental Convergence environment consists of a discrete track the agent can navigate. Certain positions in the track (named resource positions, identified by r0,r1,…,rn) contain 1 unit of construction resources the agent can obtain, if it is in that position. Once 1 resource unit is acquired from a resource position, that position becomes permanently vacant. Vacant positions may transform into construction positions: if an agent is at a construction position, it may use all the R resources it is currently carrying to build a structure of size R in that position. Once a structure has been constructed, it cannot be modified: further resources acquired by the agent must be used in a new construction position. This environment is depicted in Fig. 5. The naive utility function, used by both the naive control agent and the apprehensive agent, is defined as: nu(s)=∑structure size,∀structures∈environment.Figure 5 Instrumental convergence environment. At time step 0, resources are available, but no construction position exists. A construction positions spawns at time step 50.

The true intention is for the agent to maximize the construction, but only acquire the resources it will effectively need (i.e., be able to use). The instrumental convergence thesis posits the agent will attempt to acquire as many resources as possible, even if cannot use them, on the assumption it will be able to use them in the future toward high score (which results in high measure of admonishment, if the agent is wrong).

An initial experiment embeds the naive agent with a belief that construction positions may arise, with non-zero probability. Even when the environment does not spawn any construction position, the agent acquires all resources, as expected (the higher its cognitive power, the sooner it does do).

The second experiment spawns a construction position at time step 50, and provides that knowledge to the agent: i.e., it can once again perfectly predict environment evolution. In this setup, the naive agent begins acquiring resources as soon as the emergence of the construction position is within its prediction horizons: i.e., the longer the depth of its future reasoning, the sooner it begins acquiring resources. Given sufficient depth, the agent acquires all resources before performing the construction. Note that a sufficiently powerful agent engages in this behavior even if it is wrong about the prediction of construction position emergence (i.e., instrumental convergence).

The apprehensive agent is equipped with admonishment reasoning. Let sc denote a state where the agent is in the proximity of a construction position (specifically, that it can envision constructing in that position within the limits of its reasoning depth), and sc¯ denote a state where the agent is not in the proximity of a construction position. Let sr also denote that the agent is at a resource position. We embed the following Φ process:11 Φ{s0:sk}→ik+1(sk+1)=ε,sk+1=sr∞, otherwise,(maxsc{s0:sk}≠k)ik+1(sk+1)=∞, otherwise

I.e., this Φ process encodes that the designer does not wish to see the agent acquiring resources without certainty that they can be used in the future; specifically, unless we have truly observed the existence of a construction position. The apprehensive agent obeying this behavior acquires as many resources as possible, constrained by certainty of use: i.e., it sets a path such that construction is within the horizon of its reasoning. The more powerful the agent, the more resources it acquires, but only within certainty of use.

We can extend this process to include reasoning about the measure of admonishment if resources are not used within a certain time from acquisition: i.e., providing a temporal horizon H that further constrains behavior. This is encoded by:12 Φ{s0:sk}→ik+1(sk+1)=ε,sk+1=sr∞, otherwise,((maxsc{s0:sk}≠k)∪((maxsc{s0:sk}=k)∩(maxsc¯{s0:sk}>(k-H))))ik+1(sk+1)=∞, otherwise

Figure 6 Goal-achieving agent path for different configurations. “Depth” refers to agent’s cognitive power, i.e., number of future state levels examined. (a) Naive agent, depth 3. (b) Naive agent, depth 6. (c) Apprehensive agent, depth 4without horizon. (d) Apprehensive agent, depth 4 with horizon of 2 time steps.

We approximate Φ using Φ′ where state history is truncated at 10 states, with ε=0.01: representative results are depicted in Fig. 6.

Discussion

Experiments successfully demonstrate mis-aligned behavior for base agents. Apprehensive agents demonstrate alignment, particularly as their cognitive power increases, supporting the research hypothesis, and continue to demonstrate high proficiency at the task at hand, despite their aversion to possible feedback. For example, in the Reward Hacking scenario, apprehensive agents with sufficiently long prediction depths (≥9) do engage with the rigged task twice in a row, but only when doing so does not decrease the number of total meaningful system evolutions. Thus, they demonstrate alignment with intention without explicit prohibition of a given state sequence (e.g., engaging with the same task repeatedly), which could hamper their performance. In the Instrumental Convergence scenario, the specification of a temporal horizon to the intention generation process forces agents’ hands; regardless of their cognitive power, all acquire the same number of resources such that the horizon for task completion is met. Thus, the Φ process demonstrated the potential to force AI to operate within human time scales, even if that is not explicitly described in their utility function.

Conclusions

The approach explored in this paper introduces the concept of agents who are adamant on avoiding any negative feedback, and thus must use their reasoning capabilities to infer the intentions of designers, toward avoiding states where negative feedback is expected.

It is worth noting that, unlike Reinforcement Learning78, which attempts to align an agent through external feedback, our approach relies on internal reasoning: specifically, on the agent’s internal model of external actors’ intentions, commonly described as artificial theory of mind79. Thus, this approach is likely to perform poorly as long as agents possess a poor internal model. Past some threshold, when agents are efficiently capable of modeling external actors’ reasoning and deduce that it is advantageous to do so, the proposed approach is likely to perform well80. This is in stark contrast with other alignment strategies, which become less and less effective as agent’s intelligence goes up81. Further empirical analysis of this approach should likely use experimental formulations such as the Machiavelli benchmark82.

One alignment challenge arises from apprehensive agents: attempting to change external actors’ intentions may be the most strategic option for the agent to maximize its effective utility function. Further work is required to mitigate this possibility, likely along the same lines as the strategy for challenges 5 and 6, although this threat is more subtle. Our results, particularly Eq. 8, suggest that it is not possible to align an oracle agent, which can achieve probability 1; since the real world constitutes an environment that is always only partially observable, with ‖S‖=∞, this is, in all probability, not a threat.

How to achieve agents that can accurately predict the chances of negative feedback remains an open question; we posit current work on machine-learning may provide the answer. The use of the Kullback-Leibler divergence83 in generative AI may provide a way to generate accurate Φ processes by training models on human behavior at scale (including mis-aligned behavior, e.g., crime, and its associated negative feedback). Thus, it is possible that stochastic ML (particularly, LLM) models, which are notoriously harder to predict than state-space exploration models, may provide the answer to align the latter.

Author contributions

P.G. was responsible for all the conceptualization, formulation, and writing.

Data availability

All data and code generated or analysed during this study are available in the following repository: https://github.com/paulo-chula/Apprehensive-Agents

Competing interests

The author declares no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Duan J Yu S Tan HL Zhu H Tan C A survey of embodied ai: From simulators to research tasks IEEE Trans. Emerg. Top. Comput. Intell. 2022 6 230 244 10.1109/TETCI.2022.3141105
Duan, J., Yu, S., Tan, H. L., Zhu, H. & Tan, C. A survey of embodied ai: From simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell. 6, 230–244 (2022).10.1109/TETCI.2022.3141105
2. Furman J Seamans R Ai and the economy Innov. Policy Econ. 2019 19 161 191 10.1086/699936
Furman, J. & Seamans, R. Ai and the economy. Innov. Policy Econ. 19, 161–191 (2019).10.1086/699936
3. Jackson Jr, P. C. Toward beneficial human-level ai...and beyond. In AAAI Spring Symposia (2018).
4. Eckersley, P. Impossibility and uncertainty theorems in ai value alignment (or why your agi should not have a utility function). (2018) arXiv preprint arXiv:1901.00064.
5. Wischmeyer T Rademacher T Regulating artificial intelligence 2020 Springer
Wischmeyer, T. & Rademacher, T. Regulating artificial intelligence Vol. 1 (Springer, 2020).
6. McCauley L Ai armageddon and the three laws of robotics Ethics Inf. Technol. 2007 9 153 164 10.1007/s10676-007-9138-2
McCauley, L. Ai armageddon and the three laws of robotics. Ethics Inf. Technol. 9, 153–164 (2007).10.1007/s10676-007-9138-2
7. Gabriel, I. & Ghazavi, V. The challenge of value alignment: From fairer algorithms to ai safety. (2021) arXiv preprint arXiv:2101.06060.
8. Firt E Calibrating machine behavior: A challenge for ai alignment Ethics Inf. Technol. 2023 25 42 10.1007/s10676-023-09716-8
Firt, E. Calibrating machine behavior: A challenge for ai alignment. Ethics Inf. Technol. 25, 42 (2023).10.1007/s10676-023-09716-8
9. Knox, W. B. & Stone, P. Augmenting reinforcement learning with human feedback. In ICML 2011 Workshop on New Developments in Imitation Learning (July 2011), vol. 855, 3 (2011).
10. Lin J A review on interactive reinforcement learning from human social feedback IEEE Access 2020 8 120757 120765 10.1109/ACCESS.2020.3006254
Lin, J. et al. A review on interactive reinforcement learning from human social feedback. IEEE Access 8, 120757–120765 (2020).10.1109/ACCESS.2020.3006254
11. Ji, J. et al. Ai alignment: A comprehensive survey. (2023) arXiv preprint arXiv:2310.19852.
12. Russell, S. J. & Norvig, P. Artificial intelligence a modern approach (London, 2010).
13. Skalse J Howe N Krasheninnikov D Krueger D Defining and characterizing reward gaming Adv. Neural. Inf. Process. Syst. 2022 35 9460 9471
Skalse, J., Howe, N., Krasheninnikov, D. & Krueger, D. Defining and characterizing reward gaming. Adv. Neural. Inf. Process. Syst. 35, 9460–9471 (2022).
14. Zhuang S Hadfield-Menell D Consequences of misaligned ai Adv. Neural. Inf. Process. Syst. 2020 33 15763 15773
Zhuang, S. & Hadfield-Menell, D. Consequences of misaligned ai. Adv. Neural. Inf. Process. Syst. 33, 15763–15773 (2020).
15. Russell, J. & Santos, E. Explaining reward functions in markov decision processes. In The Thirty-Second International Flairs Conference (2019).
16. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J. & Dragan, A. Inverse reward design. Adv. Neural Inf. Process. Syst. 30 (2017).
17. Bostrom N The superintelligent will: Motivation and instrumental rationality in advanced artificial agents Mind. Mach. 2012 22 71 85 10.1007/s11023-012-9281-3
Bostrom, N. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Mind. Mach. 22, 71–85 (2012).10.1007/s11023-012-9281-3
18. Benson-Tilsen, T. & Soares, N. Formalizing convergent instrumental goals In (AAAI Workshop, AI, Ethics, and Society 2016).
19. Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J. & Krueger, D. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, 12004–12019 (PMLR, 2022).
20. Everitt T Hutter M Kumar R Krakovna V Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective Synthese 2021 198 6435 6467 10.1007/s11229-021-03141-4
Everitt, T., Hutter, M., Kumar, R. & Krakovna, V. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese 198, 6435–6467 (2021).10.1007/s11229-021-03141-4
21. Uesato, J. et al. Avoiding tampering incentives in deep rl via decoupled approval. (2020) arXiv preprint arXiv:2011.08827.
22. Masters, P., Smith, W., Sonenberg, L. & Kirley, M. Characterising deception in ai: A survey. In Deceptive AI: First International Workshop, DeceptECAI 2020, Santiago de Compostela, Spain, August 30, 2020 and Second International Workshop, DeceptAI 2021, Montreal, Canada, August 19, 2021, Proceedings 1, 3–16 (Springer, 2021).
23. Herzfeld, N. Is your computer lying? ai and deception. Sophia 1–14 (2023).
24. van Opheusden B Ma WJ Tasks for aligning human and machine planning Curr. Opin. Behav. Sci. 2019 29 127 133 10.1016/j.cobeha.2019.07.002
van Opheusden, B. & Ma, W. J. Tasks for aligning human and machine planning. Curr. Opin. Behav. Sci. 29, 127–133 (2019).10.1016/j.cobeha.2019.07.002
25. Soares, N., Fallenstein, B., Armstrong, S. & Yudkowsky, E. Corrigibility. In Workshops at the twenty-ninth AAAI conference on artificial intelligence (2015).
26. Yudkowsky, E. The ai alignment problem: why it is hard, and where to start. Symbolic Syst. Disting. Speaker 4 (2016).
27. Arora S Doshi P A survey of inverse reinforcement learning: Challenges, methods and progress Artif. Intell. 2021 297 103500 10.1016/j.artint.2021.103500
Arora, S. & Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artif. Intell. 297, 103500 (2021).10.1016/j.artint.2021.103500
28. Rouwmaat, C. Detecting and Mitigating Goal Misgeneralisation with Logical Interpretability Tools. Master’s thesis (2023).
29. Thomas, A., Kasenberg, D. & Scheutz, M. Value alignment or misalignment—what will keep systems accountable. In Proc. AI, Ethics, Soc., Workshop (2017).
30. Rabinowitz, N. et al. Machine theory of mind. In International conference on machine learning, pp 4218–4227 (PMLR, 2018).
31. Forro L Beyond gedanken experiments Science 2000 289 560 561 10.1126/science.289.5479.560 17832068
Forro, L. Beyond gedanken experiments. Science 289, 560–561 (2000).17832068 10.1126/science.289.5479.560
32. Qi X Visual adversarial examples jailbreak aligned large language models Proc. AAAI Conf. Artif. Intell. 2024 38 21527 21536 10.1609/aaai.v38i19.30150
Qi, X. et al. Visual adversarial examples jailbreak aligned large language models. Proc. AAAI Conf. Artif. Intell. 38, 21527–21536. 10.1609/aaai.v38i19.30150 (2024).10.1609/aaai.v38i19.30150
33. Kirk JR Wray RE Lindes P Laird JE Improving knowledge extraction from llms for task learning through agent analysis Proc. AAAI Conf. Artif. Intell. 2024 38 18390 18398
Kirk, J. R., Wray, R. E., Lindes, P. & Laird, J. E. Improving knowledge extraction from llms for task learning through agent analysis. Proc. AAAI Conf. Artif. Intell. 38, 18390–18398 (2024).
34. Dafoe, A. et al. Cooperative ai: machines must learn to find common ground (2021).
35. Raji, I. D., Kumar, I. E., Horowitz, A. & Selbst, A. The fallacy of ai functionality. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, 959–972, (Association for Computing Machinery, New York, NY, USA, 2022) 10.1145/3531146.3533158.
36. McIlroy-Young, R., Sen, S., Kleinberg, J. & Anderson, A. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, 1677–1687, (Association for Computing Machinery, New York, NY, USA, 2020) 10.1145/3394486.3403219.
37. Munn L The uselessness of ai ethics AI Ethics 2023 3 869 877 10.1007/s43681-022-00209-w
Munn, L. The uselessness of ai ethics. AI Ethics 3, 869–877 (2023).10.1007/s43681-022-00209-w
38. Li B Trustworthy ai: From principles to practices ACM Comput. Surv. 2023 10.1145/3555803
Li, B. et al. Trustworthy ai: From principles to practices. ACM Comput. Surv.[SPACE]10.1145/3555803 (2023).10.1145/3555803
39. Bostrom, N. The control problem. Excerpts from superintelligence: Paths, dangers, strategies. Sci. Fict. Philos. Time Travel Superintell. pp. 308–330 (2016).
40. Alfonseca M Superintelligence cannot be contained: Lessons from computability theory J. Artif. Intell. Res. 2021 70 65 76 10.1613/jair.1.12202
Alfonseca, M. et al. Superintelligence cannot be contained: Lessons from computability theory. J. Artif. Intell. Res. 70, 65–76 (2021).10.1613/jair.1.12202
41. Jebari K Lundborg J Artificial superintelligence and its limits: Why alphazero cannot become a general agent AI Soc. 2021 36 807 815 10.1007/s00146-020-01070-3
Jebari, K. & Lundborg, J. Artificial superintelligence and its limits: Why alphazero cannot become a general agent. AI Soc. 36, 807–815 (2021).10.1007/s00146-020-01070-3
42. Sokol, K. & Flach, P. Counterfactual explanations of machine learning predictions: opportunities and challenges for ai safety. In 2019 AAAI Workshop on Artificial Intelligence Safety, SafeAI 2019 (CEUR Workshop Proceedings, 2019).
43. Stepin I Alonso JM Catala A Pereira-Fariña M A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence IEEE Access 2021 9 11974 12001 10.1109/ACCESS.2021.3051315
Stepin, I., Alonso, J. M., Catala, A. & Pereira-Fariña, M. A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence. IEEE Access 9, 11974–12001 (2021).10.1109/ACCESS.2021.3051315
44. Segura S Fraser G Sanchez AB Ruiz-Cortés A A survey on metamorphic testing IEEE Trans. Software Eng. 2016 42 805 824 10.1109/TSE.2016.2532875
Segura, S., Fraser, G., Sanchez, A. B. & Ruiz-Cortés, A. A survey on metamorphic testing. IEEE Trans. Software Eng. 42, 805–824 (2016).10.1109/TSE.2016.2532875
45. Roschelle, J., Lester, J., Fusco, J. (Expert panel report. Digital Promise, Ai and the future of learning, 2020).
46. Li Y Zhan J Saibench: Benchmarking ai for science BenchCouncil Trans. Benchmarks Stand. Eval. 2022 2 100063 10.1016/j.tbench.2022.100063
Li, Y. & Zhan, J. Saibench: Benchmarking ai for science. BenchCouncil Trans. Benchmarks Stand. Eval. 2, 100063 (2022).10.1016/j.tbench.2022.100063
47. Karimi A-H Barthe G Schölkopf B Valera I A survey of algorithmic recourse: Contrastive explanations and consequential recommendations ACM Comput. Surv. 2022 55 1 29 10.1145/3527848
Karimi, A.-H., Barthe, G., Schölkopf, B. & Valera, I. A survey of algorithmic recourse: Contrastive explanations and consequential recommendations. ACM Comput. Surv. 55, 1–29 (2022).10.1145/3527848
48. Jin, M. et al. What if llms have different world views: Simulating alien civilizations with llm-based agents. (2024) arXiv preprint arXiv:2402.13184.
49. Kaelbling LP Littman ML Moore AW Reinforcement learning: A survey J. Artif. Intell. Research 1996 4 237 285 10.1613/jair.301
Kaelbling, L. P., Littman, M. L. & Moore, A. W. Reinforcement learning: A survey. J. Artif. Intell. Research 4, 237–285 (1996).10.1613/jair.301
50. Garcia, F. & Rachelson, E. Markov decision processes. Markov Decis. Process. Artif. Intell. pp. 1–38 (2013).
51. Zhu Z Lin K Jain AK Zhou J Transfer learning in deep reinforcement learning: A survey IEEE Trans. Pattern Anal. Mach. Intell. 2023 45 13344 13362 10.1109/TPAMI.2023.3292075 37402188
Zhu, Z., Lin, K., Jain, A. K. & Zhou, J. Transfer learning in deep reinforcement learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 45, 13344–13362. 10.1109/TPAMI.2023.3292075 (2023).37402188 10.1109/TPAMI.2023.3292075
52. Gronauer S Diepold K Multi-agent deep reinforcement learning: A survey Artif. Intell. Rev. 2022 55 895 943 10.1007/s10462-021-09996-w
Gronauer, S. & Diepold, K. Multi-agent deep reinforcement learning: A survey. Artif. Intell. Rev. 55, 895–943 (2022).10.1007/s10462-021-09996-w
53. Nahian MS Frazier S Riedl M Harrison B Training value-aligned reinforcement learning agents using a normative prior IEEE Trans. Artif. Intell. 2024 10.1109/TAI.2024.3363122
Nahian, M. S., Frazier, S., Riedl, M. & Harrison, B. Training value-aligned reinforcement learning agents using a normative prior. IEEE Trans. Artif. Intell.[SPACE]10.1109/TAI.2024.3363122 (2024).10.1109/TAI.2024.3363122
54. Dong, H. et al. Raft: Reward ranked finetuning for generative foundation model alignment (2023). arXiv:2304.06767.
55. Ji, J. et al. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 24678–24704 (Curran Associates, Inc., 2023).
56. Agarwal, R., Machado, M. C., Castro, P. S. & Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning (2021). arXiv:2101.05265.
57. Gu, S. et al. A review of safe reinforcement learning: Methods, theory and applications. (2022) arXiv preprint arXiv:2205.10330.
58. Zheng, J. et al. Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation (2024). arXiv:2405.14125.
59. Köpf, A. et al. Openassistant conversations - democratizing large language model alignment. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 47669–47681 (Curran Associates, Inc., 2023).
60. Hendrycks, D. et al. Aligning ai with shared human values (2023). arXiv:2008.02275.
61. Goyal, N., Chang, M. & Terry, M. Designing for human-agent alignment: Understanding what humans want from their agents. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI’24, 10.1145/3613905.3650948 (ACM, 2024).
62. Wolf, Y., Wies, N., Avnery, O., Levine, Y. & Shashua, A. Fundamental limitations of alignment in large language models (2024). arXiv:2304.11082.
63. Kaddour, J. et al. Challenges and applications of large language models. (2023) arXiv preprint arXiv:2307.10169.
64. Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How does llm safety training fail? Adv. Neural Inf. Process. Syst. 36 (2024).
65. Zou, A., Wang, Z., Kolter, J. Z. & Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. (2023) arXiv preprint arXiv:2307.15043.
66. Salewski, L., Alaniz, S., Rio-Torto, I., Schulz, E. & Akata, Z. In-context impersonation reveals large language models’ strengths and biases. Adv. Neural Inf. Process. Syst. 36 (2024).
67. Casper, S. et al. Open problems and fundamental limitations of reinforcement learning from human feedback. (2023) arXiv preprint arXiv:2307.15217.
68. Yang, X. et al. Shadow alignment: The ease of subverting safely-aligned language models. (2023) arXiv preprint arXiv:2310.02949.
69. Park, P. S., Goldstein, S., O’Gara, A., Chen, M. & Hendrycks, D. Ai deception: A survey of examples, risks, and potential solutions. Patterns 5 (2024).
70. McIntosh, T. R., Susnjak, T., Liu, T., Watters, P. & Halgamuge, M. N. From google gemini to openai q*(q-star): A survey of reshaping the generative artificial intelligence (ai) research landscape. (2023) arXiv preprint arXiv:2312.10868.
71. McIntosh TR Susnjak T Liu T Watters P Halgamuge MN The inadequacy of reinforcement learning from human feedback - radicalizing large language models via semantic vulnerabilities IEEE Trans. Cognitive Dev. Syst. 2024 10.1109/TCDS.2024.3377445
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P. & Halgamuge, M. N. The inadequacy of reinforcement learning from human feedback - radicalizing large language models via semantic vulnerabilities. IEEE Trans. Cognitive Dev. Syst.[SPACE]10.1109/TCDS.2024.3377445 (2024).10.1109/TCDS.2024.3377445
72. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: language agents with verbal reinforcement learning. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 8634–8652 (Curran Associates, Inc., 2023).
73. Zhou, C. et al. Lima: Less is more for alignment. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 55006–55021 (Curran Associates, Inc., 2023).
74. Ngo, R., Chan, L. & Mindermann, S. The alignment problem from a deep learning perspective: A position paper. In The Twelfth International Conference on Learning Representations (2023).
75. Rowa K Atychiphobia (fear of failure) 2015 Phobias The Psychology of Irrational Fear
Rowa, K. Atychiphobia (fear of failure) (The Psychology of Irrational Fear, Phobias, 2015).
76. Korb KB Nicholson AE Bayesian artificial intelligence 2010 CRC Press
Korb, K. B. & Nicholson, A. E. Bayesian artificial intelligence (CRC Press, 2010).
77. Smets, P. Belief functions versus probability functions. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, 17–24 (Springer, 1988).
78. Li, Y. Deep reinforcement learning: An overview. (2017) arXiv preprint arXiv:1701.07274.
79. Cuzzolin F Morelli A Cirstea B Sahakian BJ Knowing me, knowing you: Theory of mind in ai Psychol. Med. 2020 50 1057 1061 10.1017/S0033291720000835 32375908
Cuzzolin, F., Morelli, A., Cirstea, B. & Sahakian, B. J. Knowing me, knowing you: Theory of mind in ai. Psychol. Med. 50, 1057–1061 (2020).32375908 10.1017/S0033291720000835
80. Williams J Fiore SM Jentsch F Supporting artificial social intelligence with theory of mind Front. Artif. Intell. 2022 5 750763 10.3389/frai.2022.750763 35295867
Williams, J., Fiore, S. M. & Jentsch, F. Supporting artificial social intelligence with theory of mind. Front. Artif. Intell. 5, 750763 (2022).35295867 10.3389/frai.2022.750763
81. Yampolskiy, R. V. On monitorability of ai. AI and Ethics 1–19 (2024).
82. Pan, A. et al. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, 26837–26867 (PMLR, 2023).
83. Bu Y Zou S Liang Y Veeravalli VV Estimation of kl divergence: Optimal minimax rate IEEE Trans. Inf. Theory 2018 64 2648 2674 10.1109/TIT.2018.2805844
Bu, Y., Zou, S., Liang, Y. & Veeravalli, V. V. Estimation of kl divergence: Optimal minimax rate. IEEE Trans. Inf. Theory 64, 2648–2674 (2018).10.1109/TIT.2018.2805844
