Free Energy Principle and Emotional Valence
This is the third post in our series on Free Energy Principle (FEP) as a unified account of perception, learning, and action, with potential applications in emotion synamics. In this post we are going to discuss porposition by Joffily (2013) to understand emotional valence in terms of the negative rate of change of free energy over time, suggesting a dynamic interaction between valence and learning rate. As discussed earlier, according to the free-energy principle, which posits that adaptive agents must minimize their free-energy to resist disorder, biological agents encode probabilistic models of the causes of their sensations. This principle is grounded in the understanding that minimizing free-energy can be interpreted as minimizing the prediction error between actual and predicted sensory inputs, especially under Gaussian assumptions. Adaptive agents achieve this minimization through two main tactics: adjusting their internal states for more accurate predictions and acting on the environment to sample sensations that align with their predictions. Importantly, perceptual inference, perceptual learning, and active inference all rely on the same Bayesian scheme, wherein agents infer the causes of sensory inputs, learn the relationship between inputs and causes, and act on the world to fulfill prior expectations about sensory inputs, respectively.
In the process of inferring and learning the causes of their sensations in a dynamic environment, adaptive agents encounter various forms of uncertainty: estimation uncertainty, volatility, and unexpected uncertainty. Estimation uncertainty is the known estimation variance of states of the world causing sensory inputs and can be mitigated through learning. Volatility refers to slow and continuous changes in states of the world, often modeled by linking estimation uncertainty to a latent stochastic process. Unexpected uncertainty arises from surprising sensory inputs caused by discrete and rapid changes in states of the world, necessitating the resetting of learning from new sensory data. Dealing with these forms of uncertainty is crucial for Bayesian models of learning in non-stationary environments, presenting a significant challenge for dynamically updating beliefs about the world to optimize predictions. Despite the acknowledged influence of emotions on beliefs and their resistance to change, computational models have largely overlooked or failed to integrate emotional aspects. Emotional valence, representing the positive or negative character of emotion covers aspects such as subjective experiences and expressive behaviors, with valence being considered a core dimension of subjective experiences of moods and emotions. Here we use our understanding of FEP and its relation to uncertainty, combined with the recognition of emotional valence as a critical aspect of subjective experiences, sets the stage for integrating emotions into computational models of adaptive agents. Such integration holds promise for a more comprehensive understanding of human behavior and cognition in dynamic environments.
To bridge the gap between the biological principle of minimizing free-energy in agents at equilibrium with their environment and the traditional understanding of human behavior driven by pleasure-seeking and pain-avoidance, here we discuss a framework that links positive and negative emotional experiences to changes in free-energy over time. In this continuous time domain, the rate of change of free-energy $F(t)$ is represented by its first time-derivative $F’(t)$ at time $t$. We formally define the valence of a state visited by an agent at time $t$ as the negative first time-derivative of free-energy at that state, or simply $F’(t)$. Adaptive agents encode a hierarchical generative model of the causes of their sensations, where states of increasing complexity are encoded in higher levels of the hierarchy and sensory data are encoded at the lowest level. Free-energy is minimized independently for each level of the hierarchy, and $F_i(t)$ represents the free-energy associated with the hidden state at the $i$-th level. According to our definition of emotional valence, when $F’_i(t)$ is positive (indicating an increase in free-energy over time at level $i$ of the hierarchy), the valence of the state at this level is negative at time $t$. Conversely, when $F’_i(t)$ is negative (indicating a decrease in free-energy over time at level $i$), the valence of the state is positive at time $t$. Neutral valenced states, where $F’_i(t)$ is zero, may also exhibit low or high levels of surprise, as free-energy serves as an upper bound on surprise.
Cognitive theories of emotion have often relied on beliefs about states of affairs for their analyses. Emotions like happiness, unhappiness, relief, and disappointment are associated with certain (firm) beliefs about states of affairs, while emotions like hope and fear are related to uncertain beliefs. These two classes of emotions have been termed factive and epistemic, respectively. To illustrate the difference between factive and epistemic emotions, consider the example of Kamran waiting for a train. Kamran is happy if he desires the train to be on time and firmly believes it is, unhappy if he doesn’t desire it to be on time and firmly believes it is, hopeful if he desires it to be on time but is uncertain, and fearful if he doesn’t desire it to be on time but is uncertain. Relief and disappointment, on the other hand, are associated with transitions from uncertain to certain beliefs. In this framework, beliefs and desires can be related to bottom-up conditional expectations and top-down predictions, respectively, in a predictive coding scheme of free-energy minimization. Avoiding the assumption of certain beliefs inherent in cognitive theories, we focus solely on the dynamics of free-energy, showing that factive and epistemic emotions are associated with low and high levels of uncertainty, respectively. In the continuous time domain, the rate of change of the first derivative of free-energy $F’_i(t)$ at the $i$-th level is the second time-derivative of free-energy. Analogously to mechanical physics, $F’_i(t)$ and $F’‘_i(t)$ represent the velocity and acceleration of free-energy at time $t$, respectively. We propose that when both $F’_i(t)$ and $F’‘_i(t)$ are negative, indicating a decrease in free-energy ‘faster and faster’ over time, the agent hopes to visit a state of lower free-energy in the near future at level $i$. Conversely, when $F’_i(t)$ is negative and $F’‘_i(t)$ is positive, the agent is happy to be visiting a state of lower free-energy than before at this level. Similarly, when both $F’_i(t)$ and $F’‘_i(t)$ are positive, indicating an increase in free-energy ‘faster and faster’ over time, the agent fears visiting a state of greater free-energy in the near future. However, when $F’_i(t)$ is positive and $F’‘_i(t)$ is negative, the agent is unhappy to be visiting a state of higher free-energy than before.
Transitions between emotional states follow a pattern, with transitions from negative to positive emotions passing through relief and transitions from positive to negative emotions passing through disappointment. In other words, each basic emotion is mapped onto a particular region of a two-dimensional space defined by the first and second time-derivatives of free-energy, illustrating the complex relationship between affective states and free-energy dynamics. In this post we discussed the integration of emotional valence and basic forms of emotion into the FEP framework that is originally developed to study perception, learning, and action. Valence, computed as the negative rate of change of free-energy, serves as a vital indicator for biological agents, informing them about unexpected changes in their environment. A positive valence suggests that sensory inputs align with the agent’s expectations, indicating a low probability of unexpected changes. Conversely, a negative valence signifies that the agent’s expectations are violated, signaling likely unexpected changes in the environment. In dynamic environments where recent information is a better predictor of the world’s states than past information, such as in changing scenarios, recent information should be weighted more heavily. This necessitates a high learning rate to adapt quickly to new information. In contrast, for stationary environments where both past and recent information are equally informative, a low learning rate is more suitable the significance of both past and recent data. In the context of FEP it is possible to formalize the emotional meta-learning, where the estimation uncertainty is not only determined by the surprise per se but also by the rate of change of surprise. Specifically, when the free-energy associated with posterior beliefs about states at a particular level in the agent’s hierarchical model increases, the posterior certainty about these states decreases. This implies that decreasing evidence for the agent’s estimates of states of the world indicates excessive confidence in those states. Emotional regulation of uncertainty is framed as meta-learning to emphasize that learning is influenced by the consequences of this adjustment, particularly the rate of change of variational free-energy. Importantly, the emotional meta-learning is not tied to any specific generative model, as expectations about states are optimized with respect to variational free-energy either at an evolutionary timescale or during experience-dependent learning. The emotional update presented here it aligns with several key heuristics in optimization literature, particularly regularization schemes. These schemes, such as Levenberg Marquardt Regularization, decrease the learning rate or gradient descent when the objective function being optimized does not change as expected. Typically, this regularization adjusts the relative precision of the data, making more cautious updates in response to adverse changes in the objective function, such as the free energy in our scheme. Importantly, in a hierarchical setting, this adaptation in the rate of optimization or learning at various levels of the hierarchical model can lead to adaptive changes in the agent’s behavior and perception.
To this point we have try to link emotional valence to changes in free energy over time, suggesting that decreasing free energy induces positive emotions, while increasing it induces negative ones. In the same vein, dual-process theories propose that cognition comprises implicit and explicit processes. Neuroscientific studies have shown temporally separated phases of emotional processing, with rapid, automatic processing preceding slower, explicit processing. The pleasure-interest model of aesthetic liking (PIA model) incorporates processing fluency, suggesting that fluently processed stimuli induce pleasure, while disfluency reduction induces interest. This fluency-disfluency paradigm explains preferences for simple, typical stimuli over complex, novel ones. It aligns with theories of cognitive ease and the default mode brain network associated with fast thinking. Recent research propose a novel mathematical framework that applies free energy dynamics to the fluency-disfluency paradigm. This framework adopts variational Bayesian inference to model perceptions in the dual process and uses variations in free energy to represent emotions like pleasure, interest, confusion, and boredom. By focusing on Bayesian priors, this framework formalize emotions in the second explicit process based on changes in free energy. We apply this framework to a Gaussian Bayesian model, analyzing the effects of various parameters on emotions.
The pleasure-interest model of aesthetic liking models emotions in the dual process using the concept of processing fluency. The theory of fluency roposes that the experience of processing fluency of stimuli directly feels good on an affective level. Similar effects can also be confirmed as the link between typicality and preferences and have been demonstrated in various visual stimuli: human faces, painting/patterns, and artifacts/natural entities. The PIA model suggests that the processing fluency of stimuli in the first automatic process (e.g., the first impression of an aesthetic object) induces emotions such as pleasure and displeasure, and the disfluency reduction in the second controlled process (e.g., understanding complex and novel object or the resolution of conflict) induces emotions such as interest and confusion. The positive effect of disfluency reduction has also been confirmed as the aesthetic “Aha” effect or impact of the perceptual insight in the field of psychological aesthetics. Such a fluency–disfluency paradigm explains the aesthetic preferences discrepancy between fluently processed objects, such as simple and typical objects, and difficult-to-process (disfluent) objects such as complex novel objects. The idea of processing fluency in dual processes is also consistent with the theory that fast thinking is subject to “cognitive ease” and that individuals tend to think, choose, and act spontaneously according to associative principles that are easy to understand and process. It has also been shown that a network of multiple brain regions, called the default mode brain network, which includes the ventromedial prefrontal cortex and anterior and posterior cingulate cortex, is associated with the neural basis of fast thinking available in a state of high cognitive fluency.
As mentioned above, the reduction in free energy during the automatic process induces positive emotions due to the decrease in uncertainty, this reduction in free energy is associated with pleasure according to the Pleasure-Interest-Affect (PIA) model where we interpret the reduction in free energy as eliciting positive emotions such as pleasure. In the PIA model, three emotional states of interest, confusion, and boredom are proposed during the transition between automatic and controlled processes. Successful reduction of disfluency in the controlled process leads to the experience of “interest.” Conversely, failure to reduce disfluency results in the perception of “confusion” as a negative emotion. When no disfluency is present, individuals may experience “boredom.” Interest” is defined as the occurrence of both a free energy increase (disfluency) and its reduction. As the increase in free energy (disfluency) increases its reduction, the greater the intital increase free energy the higher is the intial level of interest. “Confusion” is persued in the case where free energy increase (disfluency) occurs but its reduction does not occur. “Boredom” is defined as the case where the increase in free energy (disfluency) does not occur; hence, its reduction does not occur.
The integration of the Free Energy Principle with emotional valence provides a promising framework for understanding the interplay between cognition and affective experiences. Moreover, the incorporation of dual-process theories and the Pleasure-Interest-Affect model has deepened our understanding of emotional processing, highlighting the importance of processing fluency in shaping affective responses. The fluency-disfluency paradigm not only explains preferences for certain stimuli but also sheds light on the cognitive mechanisms underlying aesthetic liking and decision-making.