kamran Afzali's Portfolio

Default Mode Network, Psychoanalysis and Active Inference

Thu, 25 Jun 2026 00:00:00 +0000

Default Mode Network, Psychoanalysis and Active Inference

Introduction

As we discussed in the last post, Freud’s ambition was never purely psychological. From the start, he wanted to anchor the mind in biology. His Project for a Scientific Psychology (1895) sketched neurons, energy flows, and thresholds—a remarkable intuition that outpaced the tools of his era. For most of the 20th century, that project lay dormant. Then came neuroimaging, computational psychiatry, and Karl Friston’s Free Energy Principle (FEP)—and the distance between Freud’s couch and the brain scanner felt shorter. This post highlights can the Default Mode Network (DMN)—a large-scale brain network at the heart of self-referential thought—serve as a neurobiological scaffold for psychoanalytic constructs like the ego, the id, and repression? And can active inference, the behavioral complement of the FEP, give us a computational language precise enough to do justice to both traditions?

In our series of posts, we have discussed the Free Energy Principle, as articulated by Friston (2010) that proposes that all biological systems—from single cells to whole brains—resist disorder by minimizing variational free energy: a measure of the gap between what the organism predicts about its sensory environment and what it actually receives. In plain terms, the brain is a prediction machine that is perpetually trying to reduce surprise. Active inference is how this plays out in behavior. Rather than passively waiting for the world to confirm or disconfirm its models, the organism acts to make the world match its predictions—or updates its internal models when action fails. The key regulatory variable is precision: the confidence assigned to a prediction, which determines how strongly incoming sensory evidence can revise a prior belief. This turns out to be an unexpectedly powerful concept for psychiatry. High precision on prior beliefs, with low weighting of new input, maps neatly onto rigidity—the computational signature of conditions like OCD or trauma-related dissociation. What makes the FEP attractive to psychoanalysts is its formal consonance with Freud’s core claims: that the mind is governed by an economy of bound and unbound energy, that unconscious processes shape behavior by constraining what reaches awareness, and that conflict—between drives and constraints—is the engine of psychopathology. As Solms (2022) and colleagues in the neuropsychoanalytic tradition have argued, free energy can be read as a reformulation of what Freud called psychic energy—”unbound” free energy is aversive and functions like mental pain, while its binding is experienced as relief or pleasure.

The DMN and the Ego

The landmark paper by Carhart-Harris and Friston (2010) placed the DMN at the center of a neuropsychoanalytic model, arguing that its hierarchical organization mirrors the ego’s function, which is integrating internal drives with social and environmental constraints. This remains one of the most cited and debated proposals in neuropsychoanalysis. The DMN is a large-scale network anchored in the medial prefrontal cortex (MPFC), posterior cingulate cortex (PCC), and inferior parietal lobule (IPL). It activates during self-referential thought, autobiographical recall, theory of mind, and future simulation—precisely the capacities that psychoanalysis attributes to the ego. Crucially, MPFC-PCC functional connectivity is absent in neonates and consolidates over early development, tracking the emergence of a coherent self-concept in a way that invites comparison to the ego’s ontogenesis in psychoanalytic theory. More recent work by Yeshurun, Nguyen, and Hasson (2021) reframes the DMN not just as an “intrinsic” system for daydreaming, but as an active sense-making network that integrates incoming information with prior intrinsic models to construct meaning over time. This is an upgrade to the earlier model where the DMN is not just where the ego rests—it is where the ego works, continuously modeling the self in relation to others and to the flow of lived experience. Repression, on this account, is not mystical suppression but a computationally tractable process—the downregulation of precision assigned to signals that would disrupt the model’s coherence. Defense mechanisms, broadly, become strategies for minimizing free energy by selectively constraining what information propagates up the cortical hierarchy.

The Id, Primary and Secondary Process

The id, in Freud’s topography, is the original psychic agency: pre-verbal, affect-laden, indifferent to logic or time. Neurobiologically, this maps onto subcortical structures—particularly the amygdala, hypothalamus, and brainstem nuclei—that generate affective states prior to cortical elaboration. In FEP terms, these structures operate with low-precision priors: they generate urgent, high-entropy signals that demand behavioral response without the contextual filtering that higher cortical regions provide. The clinical evidence for this architecture is compelling. States in which top-down DMN regulation fails such as acute psychosis, temporal lobe seizures, REM dreaming, and high-dose psychedelic experiences—are all characterized by the emergence of primary process material: loose associations, hallucinatory imagery, and affective flooding. These are precisely the states that psychoanalysis describes as a collapse of ego boundaries, where the id’s unconstrained energy breaks through. The phenomenological convergence between these neurologically distinct states is not trivial—it suggests a shared mechanism of DMN disengagement. Solms (2013) makes the argument argument that consciousness itself originates in these subcortical affective systems—that the id, rather than being purely unconscious, is the seat of core subjectivity (cf. blog posts on FEP and consciousness). This inverts Freud’s original topography in a productive way, suggesting that the DMN does not generate consciousness but structures it: shaping raw affective experience into the narrative self we recognize as “I.”

Freud’s distinction between primary process (pleasure-seeking, illogical, timeless) and secondary process (reality-bound, sequential, rational) anticipates the computational distinction between fast, automatic processing and slow, deliberate reasoning that cognitive neuroscience has since formalized (cf. Fast and slow thinking). Within the FEP framework, this maps onto the balance between prior-dominated and likelihood-dominated inference. When priors dominate—when the brain’s internal model overrides incoming evidence—we get primary process-like states: hallucinations, wish fulfillment, confabulation. When sensory precision is high and prior beliefs are updated freely, we get secondary process cognition: evidence-based reasoning, flexible behavior, reality testing. Solms (2013) locates the DMN at the secondary process pole, supporting spontaneous, unconstrained thought, while the Central Executive Network (CEN) drives goal-directed, effortful cognition. The therapeutic implications are real. As we mentioned in our post on PTSD traumatic memories behave like overprecise priors, they override present sensory information, dragging the person back into a past that the body treats as now. Trauma creates, in FEP terms, an overwhelming influx of free energy for which no adequate top-down model exists—leading to defensive operations (dissociation, numbing) that protect the generative model at the cost of impoverished contact with experience.

Interoception, Affect, and Unconscious Conflict

Interoception—the brain’s continuous modeling of the body’s internal physiological state—has emerged as a critical interface between psychoanalytic and neuroscientific accounts of emotion. The predictive brain does not merely model the external world; it maintains ongoing predictions about the body itself, and mismatches between predicted and actual interoceptive states generate the affective signals that drive behavior. Hopkins (2012) argues that interoceptive predictions directly underlie Freud’s account of anxiety: what Freud called “signal anxiety”—the ego’s anticipatory response to potential danger—can be understood as the brain generating a prediction error about its own physiological state before a threatening situation fully unfolds. This is not a metaphorical restatement. It is a mechanistic claim: that anxiety is the phenomenal experience of unresolved interoceptive prediction error propagating through the cortical hierarchy. Depression offers another instructive case. Studies consistently show hypoconnectivity between the DMN and other large-scale networks in depression, alongside abnormal self-referential rumination—a state of rigid, low-precision updating in which negative self-models become entrenched and resistant to disconfirmation. This is the computational portrait of what clinicians describe as a loss of the capacity for new experience: the generative model has collapsed inward, treating its own predictions as facts.

Conclusion (or lack of!)

Although this synthesis and rapprochement seems intellectually tempting but scientific honesty requires a clear view of limitations, for instance Conceptual mapping is not causal explanation and the fact that the DMN activates during self-referential thought, and that the ego is involved in self-regulation, is suggestive but not sufficient. Neural correlates are not neural mechanisms, and the Freudian constructs are complex enough—and slippery enough—that they can be mapped onto almost any sufficiently complex neural system. The mappings in the table above should be read as hypotheses, not established facts. Moreover, empirical validation remains thin most of the proposed correspondences lack direct experimental tests. Neuroimaging studies show correlations between DMN activity and self-referential processing; they do not demonstrate that the DMN implements repression in any mechanistically specific sense. The gap between computational metaphor and neurobiological mechanism is still wide. Liewise, psychoanalytic constructs like the ego or the superego are not simply information-processing functions—they carry a phenomenological weight, an irreducibly first-person dimension, that computational models do not obviously capture. A full account will need to bridge the explanatory gap between third-person neural dynamics and first-person experience. That said, the practical and clinical possibilities here are still present. Psychedelics appear to transiently dissolve the hierarchical precision structure of the DMN, temporarily flattening the self-model in ways that may allow rigid predictive priors to be revised—a computational account of why psilocybin shows early promise in depression and PTSD. Mindfulness practices reduce DMN activity and may work partly by increasing precision on present-moment sensory input, loosening the grip of entrenched prior beliefs. Future work should prioritize developing computational assays that can distinguish adaptive from maladaptive precision regulation in clinical populations; (2) leveraging high-resolution neuroimaging to trace the specific circuits through which DMN-limbic interactions implement defense; and engaging seriously with the phenomenological tradition to ensure that the first-person dimension of psychic life is not lost in the formalism.

References

Andrews-Hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, R., & Buckner, R. L. (2010). Functional-anatomic fractionation of the brain’s default network. Neuron, 65(4), 550–562. https://doi.org/10.1016/j.neuron.2010.02.005
Carhart-Harris, R. L., & Friston, K. J. (2010). The default-mode, ego-functions and free-energy: a neurobiological account of Freudian ideas. Brain, 133(4), 1265–1283. https://doi.org/10.1093/brain/awq010
Cieri, F., & Esposito, R. (2019). Psychoanalysis and neuroscience: the bridge between mind and brain. Frontiers in Psychology, 10, 1790. https://doi.org/10.3389/fpsyg.2019.01790
Connolly, P. (2018). Expected free energy formalizes conflict underlying defense in Freudian psychoanalysis. Frontiers in Psychology, 9, 1264. https://doi.org/10.3389/fpsyg.2018.01264
Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127–138. https://doi.org/10.1038/nrn2787
Hopkins, J. (2012). Psychoanalysis, representation and neuroscience: The Freudian unconscious and the Bayesian brain. In A. Fotopoulou, D. Pfaff, & M. Conway (Eds.), From the Couch to the Lab. Oxford University Press.
Parr, T., & Friston, K. J. (2019). Attention or salience? Current Opinion in Psychology, 29, 1–5. https://doi.org/10.1016/j.copsyc.2018.10.006
Solms, M. (2013). The conscious id. Neuropsychoanalysis, 15(1), 5–19. https://doi.org/10.1080/15294145.2013.10773711
Solms, M. (2022). Friston’s free energy principle: New life for psychoanalysis? BJPsych Bulletin. https://pmc.ncbi.nlm.nih.gov/articles/PMC9345684/
Yeshurun, Y., Nguyen, M., & Hasson, U. (2021). The default mode network: where the idiosyncratic self meets the shared social world. Nature Reviews Neuroscience, 22, 181–192. https://doi.org/10.1038/s41583-020-00420-w
Andrews-Hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, R., & Buckner, R. L. (2010). Functional-anatomic fractionation of the brain’s default network. Neuron, 65(4), 550-562.
Carhart-Harris, R. L., & Friston, K. J. (2010). The default-mode, ego-functions and free-energy: a neurobiological account of Freudian ideas. Brain, 133(4), 1265-1283.
Cieri, F., and Esposito, R. (2019). Psychoanalysis and neuroscience: the bridge between mind and brain. Frontiers in Psychology, 10:1790.
Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138.
Hopkins, J. (2012). Psychoanalysis, representation and neuroscience: The Freudian unconscious and the Bayesian brain. In A. Fotopoulu, D. Pfaff, and M. Conway (Eds.), From the Couch to the Lab: Psychoanalysis, Neuroscience and Cognitive Psychology in Dialogue. Oxford University Press.
Parr, T., & Friston, K. J. (2019). Attention or salience? Current Opinion in Psychology, 29, 1-5.
Solms, M. (2013). The conscious id. Neuropsychoanalysis, 15(1), 5-19.

Bayesian Seasonal Decomposition in Stan

Mon, 25 May 2026 00:00:00 +0000

Bayesian Seasonal Decomposition in Stan

Understanding what drives a time series is not straightforward, with most series you encounter in the real world are shaped by several overlapping forces at once: a slow-moving trend, a repeating seasonal rhythm, and a layer of noise on top of of that. Learning to pull these apart cleanly is one of the foundational skills in time series analysis. Classical decomposition methods like STL have been doing this job for decades, and they do it well. But they share a fundamental limitation of being deterministic. You get a single point estimate for each component, with no indication of how much you can trust it. When your data is short, noisy, or irregularly sampled, that hidden uncertainty can be enormous. Ignoring it tends to produce overconfident conclusions downstream — the kind that look precise on paper but quietly fall apart under scrutiny. Bayesian seasonal decomposition is an alternative that rather than treating each component as a fixed quantity to be estimated treats them as latent random variables, each with a full probability distribution. Instead of asking “what is the trend?”, we ask “what does the posterior distribution over plausible trend trajectories look like, given the data?” In this post, we’ll build a Bayesian structural time series model in Stan from scratch, fit it using RStan, and extract posterior estimates for each component. Before fitting anything, though, we need data. We’ll start with a synthetic series where we already know the ground truth — the exact generating process — so we can actually verify whether the model is recovering what it should. The series combines a gentle linear trend, a sinusoidal seasonal pattern with a 12-period cycle, and Gaussian noise. Think of it as a rough analogue to monthly retail sales: slowly growing over time, with a familiar seasonal rhythm and some unexplained variation on top. With 120 observations spanning exactly 10 complete seasonal cycles, the model has enough structure to get a solid grip on both the trend and the seasonal shape. A noise standard deviation of 0.5 — meaningful relative to the signal — means this isn’t a trivially easy decomposition problem. It’s a realistic one.

set.seed(123)
n <- 120
t <- 1:n
trend <- 0.05 * t
seasonal_period <- 12
seasonal <- rep(sin(2 * pi * (1:seasonal_period) / seasonal_period), length.out = n)
noise <- rnorm(n, 0, 0.5)
y <- trend + seasonal + noise

ts.plot(y, main = "Simulated Time Series with Trend and Seasonality")

Seasonal Decomposition

The core idea is an additive decomposition: at each time point (t), the observed value is the sum of a trend component, a seasonal component, and residual noise.

$[ y_t = \mu_t + \gamma_t + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0, \sigma^2) ]$

Each piece has its own dynamics. The trend (\mu_t) follows a local level model — a random walk that allows the trend to drift gradually over time rather than forcing it to follow a rigid parametric shape:

$[ \mu_t = \mu_{t-1} + \eta_t, \quad \eta_t \sim \mathcal{N}(0, \sigma_{\mu}^2) ]$

The parameter $(\sigma_\mu)$ controls how volatile the trend is. A small value keeps the trend smooth; a larger value allows it to change direction more rapidly. The posterior will learn a value that balances flexibility against overfitting. The seasonal component $(\gamma_t)$ is modeled as a set of $(s)$ free parameters — one per season — constrained to sum to zero over a full period. That zero-sum constraint is what makes the decomposition identifiable: without it, you could shift any constant between the trend and the seasonal component and produce an equally valid fit. By centering the seasonal effects, we pin down a unique solution.

stan_model_1 <-'data {
  int<lower=2> N;
  int<lower=2> s;           // seasonal period
  vector[N] y;
}
parameters {
  vector[N] mu;             // trend
  vector[s] season_raw;     // raw seasonal effects
  real<lower=0> sigma;
  real<lower=0> sigma_mu;
  real<lower=0> sigma_season;
}
transformed parameters {
  vector[N] season;
  vector[s] season_clean;
  
  // Center seasonal component to sum to zero
  season_clean = season_raw - mean(season_raw);
  
  for (t in 1:N)
    season[t] = season_clean[1 + ((t - 1) % s)];
}
model {
  mu[2:N] ~ normal(mu[1:(N - 1)], sigma_mu);
  season_raw ~ normal(0, sigma_season);
  y ~ normal(mu + season, sigma);
}'

This model specifies a Bayesian structural time series with additive trend and seasonal components, where the observed series $( y_t )$ is decomposed into a latent level $( \mu_t )$, a periodic seasonal effect, and Gaussian observation noise. The trend component $( \mu_t )$ evolves as a first-order random walk, formalized by the prior $( \mu_t \sim \mathcal{N}(\mu_{t-1}, \sigma_\mu) )$ for $( t = 2, \dots, N )$, which encodes local smoothness while allowing gradual, stochastic shifts over time. Seasonality is introduced through a vector of raw seasonal effects of length $( s )$, corresponding to the known period (e.g., 12 for monthly data), which are subsequently centered to enforce a sum-to-zero constraint; this transformation addresses identifiability by preventing confounding between the overall level and seasonal offsets. The centered seasonal vector is then recycled across time via modular indexing, so that each observation inherits the appropriate seasonal adjustment according to its position within the cycle. Both the observation noise $( \sigma )$, the trend innovation scale $( \sigma_\mu )$, and the seasonal variability $( \sigma_{\text{season}} )$ are assigned implicit priors through their role as scale parameters in normal distributions. Finally, the likelihood is specified as $( y_t \sim \mathcal{N}(\mu_t + \text{season}_t, \sigma) )$, implying that deviations from the combined latent structure are independently and normally distributed. Taken together, the model can be understood as a relatively flexible decomposition that captures smooth underlying dynamics and recurring seasonal patterns, while maintaining identifiability through centering constraints and borrowing strength across time via hierarchical structure. With the model defined, fitting it in R is straightforward. We package the data into a list and pass it to stan(), running 4 chains with 2000 iterations each (the first 1000 are warmup).

Model Fitting and Extracting the Components

library(rstan)

data_list <- list(
  N = length(y),
  s = seasonal_period,
  y = y
)

fit <- stan(
  model_code = stan_model_1,
  data = data_list,
  chains = 4,
  iter = 2000,
  warmup = 1000,
  seed = 123
)

print(fit, pars = c("sigma", "sigma_mu", "sigma_season"))

Once the model fit is done, take note of the $(\hat{R})$ values and effective sample sizes printed by print(fit). For a model of this complexity, you want $(\hat{R} < 1.01)$ for all parameters, which indicates the chains have converged to the same distribution. If the trend parameters are mixing slowly — which can happen when the noise level is high — you may need to increase iterations or consider reparameterizing. The posterior estimates for sigma, sigma_mu, and sigma_season are particularly informative. If sigma_mu comes back near zero, it’s telling you the trend is essentially linear and doesn’t need the flexibility of a random walk. If sigma_season is large relative to sigma, the seasonal component is being estimated with relatively high uncertainty — which might prompt you to question whether your chosen period is correct.

Once the model has run, we extract the posterior samples and summarize them with posterior means. Because we’re working with full distributions, we could just as easily compute credible intervals or a visualization with posterior samples.

posterior <- extract(fit)
mu_hat <- apply(posterior$mu, 2, mean)
season_hat <- apply(posterior$season, 2, mean)

par(mfrow = c(3, 1), mar = c(4, 4, 2, 1))
plot(y, type = 'l', main = "Observed Time Series", ylab = "y")
plot(mu_hat, type = 'l', col = "blue", main = "Estimated Trend (mu)", ylab = "mu_t")
plot(season_hat, type = 'l', col = "darkgreen", main = "Estimated Seasonal Component", ylab = "season_t")

One of the most useful things you can do here is to go beyond point estimates and shade credible intervals around each component. Something like apply(posterior$mu, 2, quantile, probs = c(0.05, 0.95)) gives you the 5th and 95th percentiles of the trend at each time point, which you can plot as a ribbon. In practice, these intervals tend to widen at the edges of the observed data and in periods where the trend is changing direction quickly — exactly where you’d want to know your uncertainty is high.

What comes next?

The model above is simple but it serves as a foundation for a range of more complex models. A natural first extension is to introduce a local linear trend, which adds a time-varying slope $(\nu_t)$ alongside the level:

$[ \mu_t = \mu_{t-1} + \nu_{t-1} + \eta_t, \quad \nu_t = \nu_{t-1} + \zeta_t ]$

Another useful direction is adding regression components — external predictors that explain some of the variation in $(y_t)$. Holiday indicators, weather variables, or economic covariates can all be incorporated by adding a linear predictor $(\mathbf{x}_t^\top \boldsymbol{\beta})$ to the observation equation. The Bayesian framework handles this because the of uncertainty in the regression coefficients being integrated into uncertainty about the decomposed components. Finally, if you’re working with multiple related time series hierarchical seasonal decomposition lets you share information across series. Individual series can have their own trend and seasonal parameters, but those parameters are drawn from a common prior, which regularizes the estimates and borrows strength where data is sparse. Bayesian seasonal decomposition is one of those techniques that has much more work upfront than just running a frequentist stl() you should, write a Stan model, wait for MCMC to run, and diagnose the convergence. But as a payoff you get uncertainty estimates that are statistically coherent, a model structure that can be extended in principled ways, and a decomposition that reflects what the data actually supports rather than what a deterministic algorithm happens to produce. For exploratory work, the posterior means alone are often enough to get a clean visual decomposition. For anything that feeds into a downstream decision — a forecast, an anomaly detection system, a causal analysis the full posterior matters, and the Bayesian approach is the right tool for the job.

References

Computational Modeling of Healthy and Depressed Decision-Making

Sat, 25 Apr 2026 00:00:00 +0000

Computational Modeling of Healthy and Depressed Decision-Making

As we mentioned in our initial post on computational psychiatry, Reinforcement learning (RL) is one of the frameworks in computational psychiatry. RL models have been applied to characterize the cognitive and affective disturbances in disorders like depression, where depressed individuals are slower to learn rewarding associations and make fewer optimal choices than healthy controls. This suggests that formal RL algorithms can capture some aspects of depressive cognition. In this post, we review how a simple Q-learning toy model can show differences between healthy and depressed decision-makers. The post begins by framing reinforcement learning within computational psychiatry as a way to model depressive decision-making, then says it will compare healthy and depressed agents through toy simulations. It first introduces the basics of Q-learning, explaining the Q-update and softmax choice rules and interpreting (\alpha) and (\beta) as learning rate and value sensitivity, before comparing healthy and depressed agents with different parameter settings to show how slower learning and noisier choice can capture depressive behavior. From there, it shifts to a Bayesian bandit model with Beta–Bernoulli updates and Thompson sampling, then shows how pessimistic priors and biased updates can represent pessimism, learned helplessness, and self-sabotage, where failures are overweighted or successes are discounted. The final conceptual section adds a mood variable that evolves over time and shapes exploration versus exploitation, and the post closes with a series of increasingly rich R toy models that simulate these ideas: basic Q-learning, self-defeating bandits, mood-coupled learning, Bayesian learning, and Bayesian learning with mood.

Fundamentals of Q-Learning

At its core, RL describes how agents learn to map situations to actions in order to maximize cumulative reward. In a simple bandit task, an agent repeatedly chooses among several options (arms), each of which delivers a reward (e.g. win or loss) with some unknown probability. Over time, the agent must learn which arms are most rewarding. A classic model-free algorithm for this task is Q-learning, in which the agent maintains an expected value $Q(a)$ for each action $a$. After each choice and observed reward $r$, the chosen action’s Q-value is updated by a prediction error:

\[Q(a) \leftarrow Q(a) + \alpha \bigl(r - Q(a)\bigr).\]

Here $\alpha$ (0≤α≤1) is the learning rate, controlling how much new outcomes influence the learned value. A high $\alpha$ means the agent gives strong weight to recent feedback (rapid learning), whereas a low $\alpha$ yields gradual updating. The agent then uses its Q-values to guide choice, often via a softmax (Boltzmann) rule: it selects action $a$ with probability

\[P(a) = \frac{\exp[\beta\,Q(a)]}{\sum_{b}\exp[\beta\,Q(b)]},\]

where $\beta$ is an inverse temperature parameter. A larger $\beta$ makes the agent more deterministic in choosing the current best option, while a smaller $\beta$ leads to more random (exploratory) choice. In effect, $\alpha$ captures sensitivity to prediction errors, and $\beta$ captures how strongly value differences influence choice. Empirically, these parameters often differ between populations: meta-analyses have found that depressed or anxious patients tend to have lower learning rates and lower $\beta$ than controls, indicating sluggish updating and more stochastic choices.

To make these ideas concrete, consider an R implementation of a Q-learning agent in a 5-armed bandit. We define a constructor q_learning_agent(alpha, beta, n_arms) that initializes the agent’s parameters and a vector of Q-values. The function simulate_agent then runs the agent through a fixed number of trials on a specified bandit environment. In each trial, the agent samples an action probabilistically via softmax and observes a binary reward (drawn from the chosen arm’s true probability). We then update the Q-value of the chosen arm using the standard Q-learning rule. For simplicity, we show just the core logic without extensive bookkeeping or state representation:

q_learning_agent <- function(alpha, beta, n_arms) {
  Q <- rep(0, n_arms)               # Initialize Q-values for each arm
  list(alpha = alpha, beta = beta, Q = Q)
}

simulate_agent <- function(agent, bandit_probs, trials) {
  Q <- agent$Q
  for(t in 1:trials) {
    # Softmax action selection
    exp_Q <- exp(agent$beta * Q)
    probs <- exp_Q / sum(exp_Q)
    action <- sample.int(length(Q), 1, prob = probs)
    # Observe reward (Bernoulli outcome)
    reward <- rbinom(1, 1, bandit_probs[action])
    # Q-learning update
    Q[action] <- Q[action] + agent$alpha * (reward - Q[action])
  }
  return(Q)  # Return final estimated values (for illustration)
}

In this code, alpha and beta are fixed parameters of the agent. The vector bandit_probs holds the true success probabilities of the 5 arms. For example, we might set

bandit_probs <- c(0.2, 0.4, 0.6, 0.8, 0.9)

so that the fifth arm is best (reward prob = 0.9) and the first is worst (0.2). The agent starts with all Q-values at 0 and then learns from experience.

Simulating Healthy vs. Depressed Q-Learning Agents

We can now instantiate two agents with different parameters to reflect “healthy” versus “depressed” learning styles. Suppose the healthy agent has a moderately high learning rate and high value sensitivity (e.g., α=0.3, β=5), while the depressed agent has a lower learning rate and lower β (α=0.1, β=2). These choices mirror empirical findings: depressed participants often show reduced α and β compared to controls. We then simulate each agent for, say, 100 trials on the same bandit:

healthy_agent    <- q_learning_agent(alpha = 0.3, beta = 5, n_arms = 5)
depressed_agent  <- q_learning_agent(alpha = 0.1, beta = 2, n_arms = 5)

Q_healthy   <- simulate_agent(healthy_agent, bandit_probs, trials = 100)
Q_depressed <- simulate_agent(depressed_agent, bandit_probs, trials = 100)

By the end of the run, Q_healthy might, for example, converge close to the true best values (e.g. near 0.9 for the best arm), while Q_depressed will often remain more conservative and noisy. In practice one would track the chosen arms and rewards to quantify performance. For instance, over many simulations one typically finds the healthy agent earns more cumulative reward than the depressed agent. In one set of runs we observed that the healthy agent averaged, say, ~78 successes out of 100, whereas the depressed agent averaged only ~66 (due to slower convergence and more random choices). More importantly, the pattern of choices differs: the healthy agent quickly identifies and repeatedly selects the high-value arm, while the depressed agent continues to sample suboptimal arms at a higher rate.

These differences align with experimental data. For example, Mukherjee et al. (2023) fit Q-learning models to reward and punishment learning tasks and found lower α and lower β in the depressed group relative to controls. In their simulations, depressed individuals made fewer “rich” (optimal) choices overall. This accords with our toy simulation: the depressed agent, having a low learning rate, is sluggish to update and thus often misses the best arm. In cognitive terms, a low α captures anhedonia or “apathetic” learning – the agent is less influenced by new rewards. A low β captures indecisiveness or excessive exploration – the agent’s choices are less firmly guided by learned values. Blanco et al. (2013) likewise reported that participants with depressive symptoms behaved in a more exploratory fashion (i.e. their choices were less value-driven) and were better described by a simple RL model rather than an “ideal” Bayesian planner. Our simulation reflects this: the depressed agent’s softmax is effectively “flatter” (due to low β), making high-Q arms only slightly more likely than others.

Finally, it is instructive to compare the final value estimates qualitatively. For the healthy agent we might see Q_healthy ≈ (0.18, 0.40, 0.62, 0.81, 0.88), closely tracking the true probabilities (0.2,0.4,0.6,0.8,0.9). In contrast, Q_depressed may look like (0.10, 0.30, 0.55, 0.75, 0.80) or even more scrambled, often underestimating the best arm. This difference echoes the empirical observation that MDD involves “blunted” learning: both reward sensitivity and learning rate are reduced. In other words, depressed agents learn more slowly and less accurately from rewards, a computational signature of anhedonia.

Bayesian Learning and Pessimistic Priors

While Q-learning is a useful model-free approach, one can also frame bandit learning in a Bayesian way. A Bayesian agent maintains a probability distribution over the success rate of each arm and updates it after each trial. For a Bernoulli reward (win/lose), a natural choice is a Beta distribution as the conjugate prior. For arm $a$, let the agent’s prior be Beta($\alpha_a,\beta_a$); upon observing a reward $r\in{0,1}$, the posterior is simply Beta($\alpha_a+r,\;\beta_a+1-r$). One can then choose arms by Thompson sampling (randomly sampling a success probability from each posterior and picking the highest) or by greedy selection of the highest posterior mean.

An R implementation of a basic Bayesian agent might look like:

bayesian_agent <- function(n_arms, prior_alpha = 1, prior_beta = 1) {
  list(alpha = rep(prior_alpha, n_arms),
       beta  = rep(prior_beta,  n_arms))
}

simulate_bayesian <- function(agent, bandit_probs, trials) {
  for(t in 1:trials) {
    # Thompson sampling: draw one sample from each arm's posterior
    samples <- rbeta(length(agent$alpha), agent$alpha, agent$beta)
    action <- which.max(samples)
    reward <- rbinom(1, 1, bandit_probs[action])
    # Update the chosen arm's Beta parameters
    agent$alpha[action] <- agent$alpha[action] + reward
    agent$beta[action]  <- agent$beta[action]  + (1 - reward)
  }
  return(agent)  # Return final Beta parameters
}

# Example usage:
bayes_agent  <- bayesian_agent(n_arms = 5, prior_alpha = 1, prior_beta = 1)
bayes_result <- simulate_bayesian(bayes_agent, bandit_probs, trials = 100)

This “ideal learner” will gradually concentrate its Beta posteriors around the true probabilities. Compared to the simple Q-learning agent, the Bayesian approach explicitly represents uncertainty: early on, all arms have wide posteriors (e.g. Beta(1,1)), but as data accrues the posterior of the best arm narrows around 0.9. One could implement either Thompson sampling (as above) or greedy selection of the maximum expected value (i.e. comparing $\alpha/(\alpha+\beta)$ across arms). The Bayesian framework makes it easy to encode pessimistic priors. By choosing asymmetric priors such as Beta($\alpha=0.5,\beta=1.5$) for each arm, the agent begins with a belief that success is unlikely. This mirrors negative prior beliefs often found in depression. Indeed, predictive processing models of depression propose that patients hold more negative and precise priors. In practice, a depressed Bayesian agent might be initialized with prior_alpha = 0.5 and prior_beta = 1.5. In R:

pessimistic_agent <- bayesian_agent(n_arms = 5, prior_alpha = 0.5, prior_beta = 1.5)
pess_result       <- simulate_bayesian(pessimistic_agent, bandit_probs, trials = 100)

With such a prior, even repeated rewards may fail to fully convince the agent of high probabilities; it remains somewhat biased toward failure. In extreme cases, an agent could even give up exploring if its prior is too pessimistic to try an action at all, echoing learned helplessness. Recent work by Sprengeler et al. (2025) shows that learned helplessness can indeed be modeled as acquiring a pessimistic prior over action outcomes. Similarly, Feldmann et al. (2023) found that higher depressive symptoms were associated with more negative belief updates and possibly more precise (though not necessarily more negative) priors. In contrast, a self-sabotaging bias could be implemented by distorting the update itself. For example, a “self-defeating” Bayesian agent might undercount successes or overweight failures when updating. One simple trick is to multiply the reward signal by a factor <1 before updating, so that positive outcomes have less impact. Alternatively, one could add extra pseudo-counts to the beta (failure) side. In code one might do something like:

# Self-sabotaging Bayesian update example
simulate_bayesian_sabotage <- function(agent, bandit_probs, trials) {
  for(t in 1:trials) {
    samples <- rbeta(length(agent$alpha), agent$alpha, agent$beta)
    action <- which.max(samples)
    reward <- rbinom(1, 1, bandit_probs[action])
    # Down-weight positive outcomes
    if(reward == 1) {
      agent$alpha[action] <- agent$alpha[action] + 0.5  # half-weight for successes
      agent$beta[action]  <- agent$beta[action]  + 0
    } else {
      agent$alpha[action] <- agent$alpha[action] + 0
      agent$beta[action]  <- agent$beta[action]  + 1.5  # overweight failures
    }
  }
  return(agent)
}

Here the agent needs two failures to match the impact of one success. The effect is a systematic underestimation of arm quality, reinforcing negativity. Such biases can make the agent behave as if the environment were worse than it truly is, a hallmark of pessimistic attribution styles in depression.

Mood Dynamics and Exploration

An even richer class of models allows the agent’s mood or affective state to fluctuate and feed back into decision-making. In reality, people’s choices are not only based on static traits but also on transient mood: a person in a low mood might act more erratically or avoid taking risks. To capture this, we can augment our agents with an internal mood variable that evolves with reward history and modulates either learning or choice. For example, one can incorporate mood into the Q-learning agent as follows: the agent has a scalar mood that increases when rewards are obtained and decreases when rewards are omitted. This mood could then affect the inverse temperature β, perhaps making the agent more exploratory when mood is low (consistent with Blanco et al.’s finding of increased exploration with depressive affect). Concretely, we might implement:

q_learning_agent_mood <- function(alpha, beta, n_arms, mood_decay = 0.05) {
  list(alpha = alpha, beta = beta, mood = 0, mood_decay = mood_decay, Q = rep(0, n_arms))
}

simulate_agent_mood <- function(agent, bandit_probs, trials) {
  Q <- agent$Q
  for(t in 1:trials) {
    # Effective inverse temperature scales with mood (more mood => more exploitative)
    beta_eff <- agent$beta * (1 + agent$mood)
    exp_Q <- exp(beta_eff * Q)
    probs <- exp_Q / sum(exp_Q)
    action <- sample.int(length(Q), 1, prob = probs)
    reward <- rbinom(1, 1, bandit_probs[action])
    Q[action] <- Q[action] + agent$alpha * (reward - Q[action])
    # Update mood: small decay plus a fraction of the outcome
    agent$mood <- agent$mood * (1 - agent$mood_decay) + (reward - 0.5)*0.1
  }
  return(list(Q = Q, mood = agent$mood))
}

# Example usage:
mood_agent <- q_learning_agent_mood(alpha = 0.2, beta = 3, n_arms = 5)
result_mood <- simulate_agent_mood(mood_agent, bandit_probs, trials = 100)

In this toy model, the agent’s mood drifts upward after rewards and downward after failures. If mood is negative, the effective β (beta_eff) is reduced, making choices more random; as mood improves, β increases, focusing on high-value options. Such a mechanism can mimic phenomena like the “emotion-boost” where a good outcome temporarily makes one more focused. Conversely, persistent failure (low mood) leads to erratic choice, which in an extreme case resembles learned helplessness. This is one way to formally capture the intuition that affect modulates exploration–exploitation. (Various other implementations are possible, e.g. mood influencing α or biasing Q-values; the key point is that a dynamic mood can produce non-stationary decision patterns. However, empirical work on mood and exploration is mixed, some studies suggest that low mood/depression increases exploration (perhaps by diminishing perceived value and making all options seem similarly unrewarding while others emphasize affective biases in learning from reward versus punishment. Our mood-influenced agent could capture both tendencies: when mood is low, it compensates by exploring more. Over time, this can lead to a cycle: repeated failures keep mood depressed, which in turn sustains exploration of suboptimal arms. Such dynamic simulations can potentially shed light on how rumination or hopelessness might emerge in learning tasks.

Interpreting Agent Behaviors

By comparing these agent simulations, we can begin to link computational mechanisms to cognitive dysfunction. In our Q-learning example, the “depressed” agent’s low α means it essentially “ignores” much of the positive feedback – akin to anhedonia where rewards feel muted. The low β makes its choices appear erratic or disengaged, matching patients’ reported indecisiveness. In Bayesian terms, a pessimistic prior (low $\alpha$, high $\beta$) makes the agent expect failure, so even after success it remains doubtful. A self-sabotaging update rule amplifies this: successes have less impact, so the belief never fully updates. The combination of these factors produces hallmark patterns of depressive decision-making: reduced learning of rewards, excessive random choices, and an ongoing expectation of bad outcomes. In laboratory tasks, these correspond to empirical observations that depressed subjects underperform on reward-learning tasks and often require more feedback to adjust their choices.

It is also instructive to consider punishment or loss learning. One might imagine that a depressed agent would be hypersensitive to losses (a classic hypothesis). However, recent meta-analyses suggest that the deficit is general: depressed individuals show blunted sensitivity to both reward and punishment. Our model can capture this by setting both gain and loss learning rates low. For instance, if we extended the bandit to include negative rewards, the depressed agent would adjust Q-values only slowly in either direction. This aligns with Mukherjee et al.’s finding that reduced learning rates occurred in both reward and punishment conditions, challenging the older idea of an overriding negative bias (Eshel & Roiser, 2010). In behavioral terms, these models illustrate why depressed individuals may lack motivation. A low learning rate means it takes many successes in a row before the agent “believes” that a situation is safe or rewarding. Meanwhile, any setback is overweighted (in the self-sabotage variant) or gives extra punishment counts, reinforcing a gloomy outlook. The result is a vicious cycle: the agent underestimates its successes and overestimates failures, so it does not shift its policy toward better actions as strongly as a healthy learner would. In psychological terms, this could manifest as the learned helplessness phenomenon: “I tried taking action and nothing good happened, so why bother trying now?” – exactly the pattern of expecting failure described by Bayesian helplessness models.

Implications for Computational Psychiatry

Using these computational models can offer several insights, first, they provide mechanistic hypotheses that connect symptoms to cognitive processes. For example, depressed individuals’ difficulty with decision-making can be quantitatively linked to specific parameter values (low α, low β, negative priors). These parameters can be estimated from real patient data and potentially used as biomarkers. Indeed, studies have suggested that RL parameters (like learning rates) can predict treatment outcomes or distinguish subgroups (e.g. Reiter et al., 2021). Second, models allow the generation of novel predictions and interventions. If we identify that a patient’s model has an especially pessimistic prior, we might target therapy to challenge those beliefs (consistent with cognitive-behavioral approaches), if exploration is excessive, one might train patients to recognize the value of sticking with good habits (i.e. increasing β). Computational tasks could even serve as objective phenotypes: for instance, a bandit task analyzed through a Q-learning might reveal a patient’s “anhedonic learning rate” and guide personalized treatment. These models, also, link behavior to neural mechanisms. The learning rate α is often associated with phasic dopamine signaling in the striatum. Thus, pharmacological manipulations (like dopamine agonists) that affect learning can be interpreted through the model. In fact, some of the litterature sited below found blunted striatal prediction-error signals in MDD, consistent with reduced α. Likewise, the inverse temperature β has been related to prefrontal control and decision noise. By mapping model parameters to brain circuits, computational psychiatry looks to bridge the gap from brain to behavior. However, these models are simplifications of complex cognition. Real people do not operate with fixed parameters, and learning rates and biases can change over time or across contexts. Our bandit tasks are far removed from the rich social and emotional contexts of real life. Moreover, depressive cognition involves rumination, negative memory biases, and meta-cognitive factors that go beyond what a basic RL model can capture. For instance, an individual’s expectation about future mood or their sense of self-efficacy are not explicitly modeled here. Looking forward, a promising direction is to build more sophisticated agents that incorporate hierarchical or contextual knowledge. Active inference models, which extend Bayesian RL with uncertainty about model parameters, are another route (see Badcock et al., 2019 for an evolutionary perspective). Incorporating social learning (how depressed patients interpret others’ actions) is also crucial, since depression strongly affects interpersonal behavior. Likewise, embedding these agents in realistic multi-task simulations could help us predict how cognitive deficits translate into daily impairments. We should keep in mind that no single model captures the full complexity of mental illness. Our agents lack the emotional richness of real humans. Computational psychiatry is still young, but it holds promise for connecting algorithms to experiences, and ultimately for improving diagnosis and treatment through quantitative modeling.

References

Mukherjee, D., van Geen, C., & Kable, J. W. (2023). Leveraging Decision Science to Characterize Depression. Current Directions in Psychological Science.
Pike, A. C., & Robinson, O. J. (2022). Reinforcement learning in patients with mood and anxiety disorders vs. control individuals: A systematic review and meta-analysis. JAMA Psychiatry, 79(4), 313–322.
Huys, Q. J. M., Pizzagalli, D. A., Bogdan, R., & Dayan, P. (2013). Mapping anhedonia onto reinforcement learning: A behavioral meta-analysis. Biology of Mood & Anxiety Disorders, 3, 12.
Blanco, N. J., Otto, A. R., Maddox, W. T., & Beevers, C. G. (2013). The influence of depression symptoms on exploratory decision-making. Cognition, 129(3), 563–568.
Feldmann, M., Kube, T., Rief, W., & Brakemeier, E.-L. (2023). Testing Bayesian models of belief updating in the context of depressive symptomatology. International Journal of Methods in Psychiatric Research, 32(2), e1946.
Sprengeler, R., Seth, A. K., Badcock, P., et al. (2025). A task-invariant prior explains trial-by-trial active avoidance behaviour across gain and loss tasks. Communications Psychology.
Eshel, N., & Roiser, J. P. (2010). Reward and punishment processing in depression. Biological Psychiatry, 68(2), 118–124.
Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876–879.
Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014). A computational and neural model of momentary subjective well-being. PNAS, 111(33), 12252–12257. Note: In-text citations correspond to the bracketed references above (e.g., Mukherjee et al., 2023).

Code chuncks for experimentations

Toy model 1

This R code simulates and compares the learning behaviors of two agents—designated as “healthy” and “depressed”—in a two-armed bandit environment using a Q-learning framework. Each agent interacts with a probabilistic environment, where one action yields a reward with a probability of 0.8 and the other with 0.2. The Q-learning algorithm updates value estimates based on received rewards, moderated by a learning rate (α) and a softmax temperature parameter (β) which governs action selection. The healthy agent is modeled with a higher learning rate (α = 0.4) and stronger reward sensitivity (β = 5), enabling more adaptive behavior. In contrast, the depressed agent employs a lower learning rate (α = 0.05) and reduced reward sensitivity (β = 2), resulting in slower and less accurate value updates. The script visualizes the cumulative rewards over 2000 trials, illustrating a performance disparity whereby the healthy agent accrues significantly greater rewards over time.

set.seed(42)

# Environment: Two-armed bandit
reward_probs <- c(0.8, 0.2)  # Probabilities of reward for each arm

q_learning_agent <- function(alpha, beta, episodes = 2000) {
  Q <- c(0, 0)  # Initial Q-values for two actions
  actions <- integer(episodes)
  rewards <- numeric(episodes)
  for (i in 1:episodes) {
    # Softmax action selection
    exp_q <- exp(Q * beta)
    probs <- exp_q / sum(exp_q)
    action <- sample(1:2, 1, prob = probs)  # R is 1-indexed
    reward <- as.numeric(runif(1) < reward_probs[action])
    # Q-learning update
    Q[action] <- Q[action] + alpha * (reward - Q[action])
    actions[i] <- action
    rewards[i] <- reward
  }
  list(actions = actions, rewards = rewards)
}

# Healthy agent: normal learning rate and reward sensitivity
healthy <- q_learning_agent(alpha = 0.4, beta = 5)

# Depressed agent: lower learning rate (slower to update beliefs)
depressed <- q_learning_agent(alpha = 0.05, beta = 2)

# Plotting results
library(ggplot2)
df <- data.frame(
  Trial = 1:2000,
  Healthy = cumsum(healthy$rewards),
  Depressed = cumsum(depressed$rewards)
)
df_long <- reshape2::melt(df, id.vars = "Trial", variable.name = "Agent", value.name = "CumulativeReward")

ggplot(df_long, aes(x = Trial, y = CumulativeReward, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Cumulative Reward: Healthy vs. Depressed Agent",
       x = "Trials", y = "Cumulative Reward") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

Toy model 2

This R code models and compares decision-making behaviors of a “healthy” and a “depressed” agent in a five-armed bandit task using a Q-learning algorithm. The environment includes two high-reward options and three self-defeating arms associated with negative or minimal reward probabilities. Both agents employ softmax-based action selection, but the depressed agent exhibits a self-defeating bias, implemented by increasing the logits of the suboptimal arms (arms 3–5), thereby elevating their selection probability. The healthy agent is characterized by a moderate learning rate (α = 0.2), high reward sensitivity (β = 5), and no self-defeating bias. Conversely, the depressed agent employs a low learning rate (α = 0.05) and a significant self-defeating bias (+2 to arms 3–5 logits), simulating maladaptive behavioral tendencies. The resulting plots reveal that the healthy agent accrues higher cumulative rewards and favors optimal actions, whereas the depressed agent frequently selects suboptimal arms and accumulates lower net rewards, modeling cognitive distortions commonly observed in affective disorders.

set.seed(123)
library(ggplot2)
library(reshape2)

# Environment: 5-armed bandit
reward_probs <- c(0.7, 0.6, 0.2, 0.1, 0.05)   # Arm 1 and 2 are "good", 3-5 are "bad/self-defeating"
reward_vals  <- c(1, 1, -1, -1, -2)           # Arms 3-5 yield negative rewards

q_learning_agent <- function(alpha, beta, self_defeat_bias = 0, episodes = 200) {
  K <- length(reward_probs)
  Q <- rep(0, K)  # Initial Q-values for all actions
  actions <- integer(episodes)
  rewards <- numeric(episodes)
  for (i in 1:episodes) {
    # Softmax action selection with self-defeating bias
    # Add bias to the logits of "bad" arms (arms 3,4,5)
    logits <- Q * beta
    logits[3:5] <- logits[3:5] + self_defeat_bias
    probs <- exp(logits) / sum(exp(logits))
    action <- sample(1:K, 1, prob = probs)
    reward <- ifelse(runif(1) < reward_probs[action], reward_vals[action], 0)
    # Q-learning update
    Q[action] <- Q[action] + alpha * (reward - Q[action])
    actions[i] <- action
    rewards[i] <- reward
  }
  list(actions = actions, rewards = rewards)
}

# Healthy agent: normal learning, no self-defeating bias
healthy <- q_learning_agent(alpha = 0.2, beta = 5, self_defeat_bias = 0)

# Depressed agent: lower learning rate, strong self-defeating bias
depressed <- q_learning_agent(alpha = 0.05, beta = 5, self_defeat_bias = 2)

# Plot cumulative reward and action selection frequencies
df <- data.frame(
  Trial = 1:200,
  Healthy = cumsum(healthy$rewards),
  Depressed = cumsum(depressed$rewards)
)
df_long <- melt(df, id.vars = "Trial", variable.name = "Agent", value.name = "CumulativeReward")

# Plot cumulative reward
p1 <- ggplot(df_long, aes(x = Trial, y = CumulativeReward, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Cumulative Reward: Healthy vs. Depressed Agent",
       x = "Trials", y = "Cumulative Reward") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot action selection frequencies
action_freq <- data.frame(
  Action = factor(1:5),
  Healthy = as.numeric(table(factor(healthy$actions, levels = 1:5))),
  Depressed = as.numeric(table(factor(depressed$actions, levels = 1:5)))
)
action_freq_long <- melt(action_freq, id.vars = "Action", variable.name = "Agent", value.name = "Count")

p2 <- ggplot(action_freq_long, aes(x = Action, y = Count, fill = Agent)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Action Selection Frequencies",
       x = "Action (Arm)", y = "Count") +
  theme_minimal() +
  scale_fill_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

print(p1)
print(p2)

Toy model 3

This R code models complex affective-cognitive dynamics in reinforcement learning by simulating “healthy” and “depressed” agents in a five-armed bandit environment using a modified Q-learning framework. The environment distinguishes between rewarding and self-defeating arms, with associated reward probabilities and valences. The agent’s decision policy integrates multiple biases: pessimistic bias (underestimation of Q-values), self-defeating bias (inflated logit values for negative arms), and mood-dependent modulation of exploration. Mood is operationalized as a moving average of recent rewards and influences the agent’s reward sensitivity (β), such that lower mood increases exploratory behavior. The healthy agent is parameterized with neutral affect, no biases, and stable learning dynamics, while the depressed agent exhibits lower learning rates, pessimistic value expectations, self-defeating preferences, and mood-exploration coupling. Visual analyses reveal that the depressed agent accrues fewer cumulative rewards, disproportionately selects suboptimal actions, and experiences persistently lower mood states. This simulation offers a computational perspective on maladaptive decision-making in affective disorders.

set.seed(2025)
library(ggplot2)
library(reshape2)

# Environment: 5-armed bandit
reward_probs <- c(0.7, 0.6, 0.2, 0.1, 0.05)   # Arm 1 and 2 are "good", 3-5 are "bad/self-defeating"
reward_vals  <- c(1, 1, -1, -1, -2)           # Arms 3-5 yield negative rewards
self_defeating_arms <- 3:5

simulate_agent <- function(alpha, beta, pessimistic_bias = 0, self_defeat_bias = 0, mood_weight = 0, episodes = 200) {
  K <- length(reward_probs)
  Q <- rep(0, K)  # Initial Q-values
  actions <- integer(episodes)
  rewards <- numeric(episodes)
  mood <- 0
  mood_history <- numeric(episodes)
  
  for (i in 1:episodes) {
    # Pessimistic bias: underestimates all values
    Q_biased <- Q + pessimistic_bias
    
    # Self-defeating bias: increases logit for "bad" arms
    logits <- Q_biased * beta
    logits[self_defeating_arms] <- logits[self_defeating_arms] + self_defeat_bias
    
    # Mood integration: mood modulates exploration (lower mood = more exploration)
    mood_mod_beta <- beta * (1 + mood_weight * mood)
    logits <- Q_biased * mood_mod_beta
    logits[self_defeating_arms] <- logits[self_defeating_arms] + self_defeat_bias
    
    # Softmax action selection
    probs <- exp(logits) / sum(exp(logits))
    action <- sample(1:K, 1, prob = probs)
    
    # Simulate reward
    reward <- ifelse(runif(1) < reward_probs[action], reward_vals[action], 0)
    
    # Q-learning update
    Q[action] <- Q[action] + alpha * (reward - Q[action])
    
    # Update mood: running average of recent outcomes (last 10 trials)
    if (i == 1) {
      mood <- reward
    } else {
      window <- max(1, i-9):i
      mood <- mean(rewards[window])
    }
    mood_history[i] <- mood
    
    actions[i] <- action
    rewards[i] <- reward
  }
  list(actions = actions, rewards = rewards, mood = mood_history)
}

# Healthy agent: normal learning, no pessimism, no self-defeating bias, neutral mood
healthy <- simulate_agent(
  alpha = 0.2, beta = 5, pessimistic_bias = 0, self_defeat_bias = 0, mood_weight = 0, episodes = 200
)

# Depressed agent: lower learning, pessimism, self-defeating bias, mood-exploration coupling
depressed <- simulate_agent(
  alpha = 0.07, beta = 2.5, pessimistic_bias = -0.5, self_defeat_bias = 2, mood_weight = -0.5, episodes = 200
)

# Plot cumulative reward
df <- data.frame(
  Trial = 1:200,
  Healthy = cumsum(healthy$rewards),
  Depressed = cumsum(depressed$rewards)
)
df_long <- melt(df, id.vars = "Trial", variable.name = "Agent", value.name = "CumulativeReward")

p1 <- ggplot(df_long, aes(x = Trial, y = CumulativeReward, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Cumulative Reward: Healthy vs. Depressed Agent",
       x = "Trials", y = "Cumulative Reward") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot action selection frequencies
action_freq <- data.frame(
  Action = factor(1:5),
  Healthy = as.numeric(table(factor(healthy$actions, levels = 1:5))),
  Depressed = as.numeric(table(factor(depressed$actions, levels = 1:5)))
)
action_freq_long <- melt(action_freq, id.vars = "Action", variable.name = "Agent", value.name = "Count")

p2 <- ggplot(action_freq_long, aes(x = Action, y = Count, fill = Agent)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Action Selection Frequencies",
       x = "Action (Arm)", y = "Count") +
  theme_minimal() +
  scale_fill_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot mood over time
mood_df <- data.frame(
  Trial = 1:200,
  Healthy = healthy$mood,
  Depressed = depressed$mood
)
mood_long <- melt(mood_df, id.vars = "Trial", variable.name = "Agent", value.name = "Mood")

p3 <- ggplot(mood_long, aes(x = Trial, y = Mood, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Mood Over Time",
       x = "Trials", y = "Mood (Recent Reward Average)") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

print(p1)
print(p2)
print(p3)

Toy model 4

This R code implements a Bayesian reinforcement learning model to simulate decision-making behavior in a five-armed bandit task, comparing a healthy agent and a depressed agent. Each agent estimates the value of available actions using Thompson sampling, drawing from posterior distributions over expected rewards. Initially, both agents are assigned Gaussian priors for each arm’s value, with the healthy agent receiving a neutral prior (mean = 0), while the depressed agent begins with a pessimistic prior (mean = –0.5). Additionally, a self-defeating bias is introduced in the depressed agent by artificially increasing the sampled values of the negatively valenced arms (arms 3–5), making these options more likely to be chosen. Bayesian updates are performed iteratively based on observed outcomes, assuming fixed reward variance. The simulation results indicate that the healthy agent consistently favors the optimal arms and accumulates higher cumulative rewards, whereas the depressed agent exhibits maladaptive action selection patterns, demonstrating how biased priors and cognitive distortions can degrade performance.

set.seed(123)
library(ggplot2)
library(reshape2)

# Environment: 5-armed bandit
reward_probs <- c(0.7, 0.6, 0.2, 0.1, 0.05)   # Arms 1,2 are "good"; 3-5 are "bad"
reward_vals  <- c(1, 1, -1, -1, -2)           # Arms 3-5 yield negative rewards
self_defeating_arms <- 3:5

bayesian_agent <- function(
    episodes = 200,
    pessimistic_prior_mean = 0,
    prior_var = 10,
    self_defeat_bias = 0
) {
  K <- length(reward_probs)
  # Priors for Q-value mean and precision (1/variance)
  mu <- rep(pessimistic_prior_mean, K)     # Prior mean for each arm
  tau <- rep(1/prior_var, K)               # Prior precision (inverse variance)
  n <- rep(0, K)                           # Number of pulls per arm
  actions <- integer(episodes)
  rewards <- numeric(episodes)
  
  for (i in 1:episodes) {
    # Thompson sampling: sample Q for each arm from current posterior
    sampled_Q <- rnorm(K, mean = mu, sd = 1/sqrt(tau))
    # Add self-defeating bias to arms 3-5 (depressed agent)
    sampled_Q[self_defeating_arms] <- sampled_Q[self_defeating_arms] + self_defeat_bias
    # Choose arm with highest sampled Q
    action <- which.max(sampled_Q)
    # Get reward
    reward <- ifelse(runif(1) < reward_probs[action], reward_vals[action], 0)
    # Bayesian update for Normal likelihood with known variance (assume variance=1 for simplicity)
    n[action] <- n[action] + 1
    tau[action] <- tau[action] + 1
    mu[action] <- (mu[action] * (tau[action] - 1) + reward) / tau[action]
    actions[i] <- action
    rewards[i] <- reward
  }
  list(actions = actions, rewards = rewards)
}

# Healthy agent: neutral prior, no self-defeating bias
healthy <- bayesian_agent(
  episodes = 200,
  pessimistic_prior_mean = 0,
  prior_var = 10,
  self_defeat_bias = 0
)

# Depressed agent: pessimistic prior, self-defeating bias
depressed <- bayesian_agent(
  episodes = 200,
  pessimistic_prior_mean = -0.5,  # pessimistic prior
  prior_var = 10,
  self_defeat_bias = 2            # bias toward self-defeating arms
)

# Plot cumulative reward
df <- data.frame(
  Trial = 1:200,
  Healthy = cumsum(healthy$rewards),
  Depressed = cumsum(depressed$rewards)
)
df_long <- melt(df, id.vars = "Trial", variable.name = "Agent", value.name = "CumulativeReward")

p1 <- ggplot(df_long, aes(x = Trial, y = CumulativeReward, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Cumulative Reward: Healthy vs. Depressed Agent (Bayesian Q-learning)",
       x = "Trials", y = "Cumulative Reward") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot action selection frequencies
action_freq <- data.frame(
  Action = factor(1:5),
  Healthy = as.numeric(table(factor(healthy$actions, levels = 1:5))),
  Depressed = as.numeric(table(factor(depressed$actions, levels = 1:5)))
)
action_freq_long <- melt(action_freq, id.vars = "Action", variable.name = "Agent", value.name = "Count")

p2 <- ggplot(action_freq_long, aes(x = Action, y = Count, fill = Agent)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Action Selection Frequencies (Bayesian Q-learning)",
       x = "Action (Arm)", y = "Count") +
  theme_minimal() +
  scale_fill_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

print(p1)
print(p2)

Toy model 5

This R code simulates a Bayesian reinforcement learning framework augmented with a dynamic mood component to model healthy and depressed agents in a five-armed bandit task. The environment includes two optimal arms and three maladaptive, negatively-rewarding arms. Agents employ Thompson sampling, drawing Q-values from a posterior distribution defined by evolving estimates of means and precision. Mood is operationalized as an exponentially smoothed average of past rewards and modulates the variance of the sampling distribution, thereby influencing the agent’s exploration-exploitation trade-off. A healthy agent uses neutral priors, no self-defeating bias, and moderate mood sensitivity, while the depressed agent incorporates pessimistic priors, a bias favoring maladaptive actions, and heightened mood-driven exploration. The simulation reveals that mood fluctuations and cognitive distortions jointly impair learning efficiency in the depressed agent, as reflected in lower cumulative rewards, suboptimal action choices, and mood instability. Visualizations illustrate these behavioral and affective divergences across the trial sequence.

set.seed(123)
library(ggplot2)
library(reshape2)

# Environment: 5-armed bandit
reward_probs <- c(0.7, 0.6, 0.2, 0.1, 0.05)   # Arms 1,2 are "good"; 3-5 are "bad"
reward_vals  <- c(1, 1, -1, -1, -2)           # Arms 3-5 yield negative rewards
self_defeating_arms <- 3:5

bayesian_agent_mood <- function(
    episodes = 200,
    pessimistic_prior_mean = 0,
    prior_var = 10,
    self_defeat_bias = 0,
    mood_decay = 0.9,         # Decay factor for mood (exponential smoothing)
    mood_influence = 1.5      # How strongly mood modulates exploration/exploitation
) {
  K <- length(reward_probs)
  mu <- rep(pessimistic_prior_mean, K)     # Prior mean for each arm
  tau <- rep(1/prior_var, K)               # Prior precision (inverse variance)
  n <- rep(0, K)                           # Number of pulls per arm
  actions <- integer(episodes)
  rewards <- numeric(episodes)
  mood <- 0                                # Initial mood
  mood_hist <- numeric(episodes)
  
  for (i in 1:episodes) {
    # Mood-modulated variance: bad mood = more exploration (higher variance)
    # Good mood = more exploitation (lower variance)
    # Clamp mood to [-1,1] for stability
    mood_clamped <- min(max(mood, -1), 1)
    mood_sigma_scale <- exp(-mood_influence * mood_clamped)
    
    sampled_Q <- rnorm(K, mean = mu, sd = mood_sigma_scale / sqrt(tau))
    sampled_Q[self_defeating_arms] <- sampled_Q[self_defeating_arms] + self_defeat_bias
    
    action <- which.max(sampled_Q)
    reward <- ifelse(runif(1) < reward_probs[action], reward_vals[action], 0)
    
    # Bayesian update for Normal likelihood with known variance (assume variance=1)
    n[action] <- n[action] + 1
    tau[action] <- tau[action] + 1
    mu[action] <- (mu[action] * (tau[action] - 1) + reward) / tau[action]
    actions[i] <- action
    rewards[i] <- reward
    
    # Update mood: exponential smoothing of recent rewards
    mood <- mood_decay * mood + (1 - mood_decay) * reward
    mood_hist[i] <- mood
  }
  list(actions = actions, rewards = rewards, mood = mood_hist)
}

# Healthy agent: neutral prior, no self-defeating bias, normal mood influence
healthy <- bayesian_agent_mood(
  episodes = 200,
  pessimistic_prior_mean = 0,
  prior_var = 10,
  self_defeat_bias = 0,
  mood_decay = 0.9,
  mood_influence = 1.5
)

# Depressed agent: pessimistic prior, self-defeating bias, mood has stronger influence on exploration
depressed <- bayesian_agent_mood(
  episodes = 200,
  pessimistic_prior_mean = -0.5,
  prior_var = 10,
  self_defeat_bias = 2,
  mood_decay = 0.9,
  mood_influence = 3.0   # More mood-driven exploration
)

# Plot cumulative reward
df <- data.frame(
  Trial = 1:200,
  Healthy = cumsum(healthy$rewards),
  Depressed = cumsum(depressed$rewards)
)
df_long <- melt(df, id.vars = "Trial", variable.name = "Agent", value.name = "CumulativeReward")

p1 <- ggplot(df_long, aes(x = Trial, y = CumulativeReward, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Cumulative Reward: Healthy vs. Depressed Agent (Bayesian Q-learning + Mood)",
       x = "Trials", y = "Cumulative Reward") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot action selection frequencies
action_freq <- data.frame(
  Action = factor(1:5),
  Healthy = as.numeric(table(factor(healthy$actions, levels = 1:5))),
  Depressed = as.numeric(table(factor(depressed$actions, levels = 1:5)))
)
action_freq_long <- melt(action_freq, id.vars = "Action", variable.name = "Agent", value.name = "Count")

p2 <- ggplot(action_freq_long, aes(x = Action, y = Count, fill = Agent)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Action Selection Frequencies (Bayesian Q-learning + Mood)",
       x = "Action (Arm)", y = "Count") +
  theme_minimal() +
  scale_fill_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

# Plot mood over time
mood_df <- data.frame(
  Trial = 1:200,
  Healthy = healthy$mood,
  Depressed = depressed$mood
)
mood_long <- melt(mood_df, id.vars = "Trial", variable.name = "Agent", value.name = "Mood")

p3 <- ggplot(mood_long, aes(x = Trial, y = Mood, color = Agent)) +
  geom_line(size = 1) +
  labs(title = "Mood Over Time (Bayesian Q-learning + Mood)",
       x = "Trials", y = "Mood (Exp. Avg. Reward)") +
  theme_minimal() +
  scale_color_manual(values = c("Healthy" = "blue", "Depressed" = "red"))

print(p1)
print(p2)
print(p3)

Active Inference and Psychoanalysis

Wed, 25 Mar 2026 00:00:00 +0000

Active Inference and Psychoanalysis

The intersection between the Free Energy Principle (FEP), active inference, and psychoanalysis has become a new domain of research, with some speculate that active inference may offer psychoanalytic theory new conceptual tools drawn from neuroscience to revitalize the field that has struggled for institutional legitimacy in many decades. As we have mentioned in our previous posts the FEP provides a formal framework describing how living organisms maintain their structural integrity by minimizing free energy, which represents prediction error within a probabilistic model. Active inference, which extends from the FEP, proposes that brains function as active agents that constantly generate predictions about their environment rather than passively processing sensory input. This framework not only models biological and psychological processes but attempts to formalize them in ways that might compare to psychoanalytic concepts. However, psychoanalysis operates within an interpretive tradition rooted in personal narratives, clinical encounter, and symbolic meaning, while active inference is computational and mechanistic, formulated within a Bayesian framework. Freud’s early Project for a Scientific Psychology (1895) represented an ambitious, but ultimately abandoned, attempt to ground psychology in the neurophysiology of the the time. Despite its historical limitations, the Project anticipated several ideas that active inference and the FEP now formalize, including concepts of energy flow, psychic conflict, and the nervous system’s drive toward equilibrium. Freud’s notion of the mind as a dynamic energy system, influenced by 19th-century thermodynamics, is somehow similar to the FEP’s characterization of the brain as a system that minimizes uncertainty. In Freud’s model, concepts like binding (Bindung) and unbinding (Entbindung) describes how the nervous system processes excitation—which appears to parallel the FEP’s distinction between prediction error (unbound energy) and coherent generative models (bound energy). Both frameworks suggest the mind strives toward a state of minimal excitation or surprise—a kind of homeostatic equilibrium. His later distinction between primary (id-driven, wish-fulfilling) and secondary (ego-driven, reality-oriented) processes finds potential resonance in the FEP’s hierarchical generative models. In active inference, lower hierarchical levels process immediate sensory prediction errors, while higher levels encode abstract, temporally extended representations that shape both perception and action. However, this analogy is fundamentally metaphorical, that is (as we mentioned in several previoous posts) FEP offers a formal, information-theoretic framework grounded in variational Bayesian inference, while Freud’s psychodynamics rests on speculative neurophysiology and clinical observation. The two operate on fundamentally different epistemological planes, and conflating them risks obscuring rather than clarifying either framework.

Desire is another point of potential conceptual alignment, Freud’s theory of desire is based on tension between instinctual drives and the regulatory functions of the ego and superego, where repression keeps unacceptable desires unconscious, yet these desires persist, shaping symptoms, dreams, and behaviors. For Freud, conflict is the engine of psychic life, and its resolution remains perpetually incomplete. Active inference provides what might be considered a computational analog to this dynamic. Desire can be modeled as a prior over preferred states—probabilistic expectations about outcomes the agent seeks to attain (see the post about FEP and emotional valence). These priors influence perception, cognition, and action through policy selection, shaping how the agent engages with the world. As Kruglanski et al. note in their 2020 paper synthesizing epistemic motivation with active inference, “all thinking is wishful thinking” in a predictive brain—prediction is inherently motivated by expected outcomes. Hoewever, it is important to note that active inference see desire as precision-weighted prediction over policies, but psychoanalysis treats desire as symbolic, relational, and historically embedded. To link these two concepts directly would risk reductionism where in the computational framework it is possible to model drive and conflict in abstract, formal terms, but it struggles to capture the dense symbolic texture of psychic life—the meanings embedded in dreams, fantasies, and transference relationships.

Psychoanalysis regards dreams as their “royal road to the unconscious,” that reveals disguised wishes, unresolved conflicts, and repressed material. For Freud, dreams serve not only an expressive function but also a defensive one they transform unacceptable desires into symbolic narratives through dream work mechanisms such as condensation, displacement, and secondary revision. From an active inference perspective, dreaming may serve a different function. During REM sleep, the brain appears to relax sensory constraints and engage in generative modeling, simulating possible scenarios and integrating emotionally significant experiences. This “offline inference” may allow the brain to explore hypothetical states to refine its predictive models and minimize future surprise. Research suggests that REM sleep facilitates pattern extraction from past experiences and enables the consolidation of implicit, emotionally salient memories. This account has some similariy to Freud’s idea of dreams as a site of psychological work. Both perspectives put dreaming as a process that metabolizes affect, rehearses responses, and processes emotionally loaded material. However, the underlying logic is different between two frameworks, Freud emphasizes symbolic transformation and the censorship imposed by psychic defenses, while active inference treats dreams as structurally constrained simulations aimed at optimizing generative models. Both models describe adaptation to internal conflict and emotional intensity, but they explain this adaptation through fundamentally different mechanisms—one prioritizing symbolic interpretation, the other formal simulation.

Despite some suggestive parallels, there are some methodological and epistemological differences that complicate any integration of psychoanalysis and active inference such as Different Scientific Paradigms, psychoanalysis belongs to the hermeneutic tradition, concerned with meaning-making and narrative coherence. Active inference is embedded in formal mechanistic science, concerned with algorithmic explanation and testable prediction. These frameworks operate within what philosophers like Kuhn might call incommensurable paradigms of inquiry. Conceptual Ambiguity: Terms like “energy,” “conflict,” or “desire” carry different connotations in each system. In psychoanalysis, energy is metaphorical and affective; in the FEP, free energy is a precise mathematical quantity derived from information theory. Overextending these analogies risks conceptual confusion rather than clarification. Empirical Generalizability: While active inference generates somewhat testable predictions (hopefully we do a post about falsifiability of active inferences in near future) in experimental paradigms and psychiatric modeling, psychoanalytic constructs have historically resisted quantification. Clinical data from psychoanalysis is rich and nuanced but not easily formalized within computational frameworks. Critics of psychoanalysis often cite its lack of falsifiability—though this criticism has been challenged as both philosophically flawed and empirically outdated. Critics of the FEP, meanwhile, warn that it risks becoming a “theory of everything” that explains too much while predicting too little. Any integration must avoid collapsing into nonsense (cf. early Wittgenstein) either through naive formalism or interpretive excess.

The dialogue between psychoanalysis and active inference is rich but has conceptual risks. The FEP and active inference offer formal, computational lenses through which to reinterpret some of Freud’s insights that is the mind as a system under tension negotiating between competing demands to maintain internal coherence and adapt to its environment. However, this interpretive pairing of two frameworks needs epistemological care. Active inference provides elegant formalisms, but these cannot substitute for the narrative, symbolic, and relational dimensions that psychoanalysis brings to understanding human experience. The analogy between psychic energy and free energy proves fruitful only when treated as metaphor rather than equation. True integration, if it proves possible at all, lies not in collapsing one framework into the other but in allowing each to illuminate the other’s blind spots. Psychoanalysis can challenge active inference to account more adequately for affect, narrative, and transference. Active inference can offer psychoanalysis new tools for modeling conflict, adaptation, and learning. By embracing both the synergies and the tensions, we may develop a richer, more pluralistic understanding of the psyche—one that honors both its symbolic complexity and its computational structure.

References

Anderson, M. C., Ochsner, K. N., Kuhl, B., Cooper, J., Robertson, E., Gabrieli, S. W., et al. (2004). Neural systems underlying the suppression of unwanted memories. Science, 303, 232.

Carhart-Harris, R. L., & Friston, K. J. (2010). The default-mode, ego-functions and free-energy: a neurobiological account of Freudian ideas. Brain, 133, 1265–1283.

Cittern, D., Nolte, T., Friston, K., & Edalat, A. (2018). Intrinsic and extrinsic motivators of attachment under active inference. PLoS ONE, 13(4), e0193955.

Gray, J. R., Bargh, J. A., & Morsella, E. (2013). Neural correlates of the essence of conscious conflict: fMRI of sustaining incompatible intentions. Experimental Brain Research, 229(3), 453–465.

Connolly, P. (2018). Expected free energy formalizes conflict underlying defense in Freudian psychoanalysis. Frontiers in Psychology, 9, 1264.

Freud, S. (1895). A project for a scientific psychology. In The Standard Edition of the Complete Psychological Works of Sigmund Freud (Vol. 1, pp. 283–397). London: Hogarth Press.

Freud, S. (1900). L’Interprétation des Rêves (2010e éd.). Paris: PUF.

Freud, S. (1911). Formulations on the two principles of mental functioning. In The Standard Edition of the Complete Psychological Works of Sigmund Freud (Vol. 12, pp. 218–286).

Hopkins, J. (2016). Free energy and virtual reality in neuroscience and psychoanalysis: a complexity theory of dreaming and mental disorder. Frontiers in Psychology, 7, 922.

Holmes, J., & Nolte, T. (2019). “Surprise” and the Bayesian brain: Implications for psychotherapy theory and practice. Frontiers in Psychology, 10, 592.

Kapur, S. (2003). Psychosis as a state of aberrant salience: a framework. Brain, 126, 13–27.

Parr, T., Pezzulo, G., & Friston, K. J. (2022). Active inference: The free energy principle in mind, brain, and behavior. Cambridge, MA: MIT Press.

Scarfone, D. (2018). Free association, surprise, trauma, and transference. Psychoanalytic Inquiry, 38, 468–477.

Shedler, J. (2010). The efficacy of psychodynamic psychotherapy. American Psychologist, 65, 98–109.

Solms, M. (2019). The hard problem of consciousness and the free energy principle. Frontiers in Psychology, 9, 2714.

Zepf, S. (2010). Libido and psychic energy – Freud’s concepts reconsidered. International Forum of Psychoanalysis, 19, 3–14.

Beren, J. (2024, July 27). A retrospective on active inference. https://www.beren.io/2024-07-27-A-Retrospective-on-Active-Inference/

Bruineberg, J., Kiverstein, J., & Rietveld, E. (2016) The anticipating brain is not a scientist: The free-energy principle from an ecological-enactive perspective. Frontiers in Human Neuroscience, 10, Article 20. https://doi.org/10.3389/fnhum.2016.00020

Clark, A. (2020). Beyond the “Bayesian brain”: Predictive processing and the extended mind. Trends in Cognitive Sciences, 24(7), 517–529. https://doi.org/10.1016/j.tics.2020.03.010

Colombo, M., & Wright, C. (2023). First-person methods in computational psychiatry and cognitive neuroscience. Physics of Life Reviews, 47, 1–30. https://doi.org/10.1016/j.plrev.2023.09.003

Freud, S. (1950/1895). Project for a scientific psychology. Pennsylvania State University. https://bpb-us-e1.wpmucdn.com/sites.psu.edu/dist/9/19778/files/2021/12/freud-project_for_scientific_psychology.pdf

Hohwy, J. (2013). The predictive mind. Oxford University Press.

LessWrong. (2019). Why I’m not into the free energy principle. https://www.lesswrong.com/posts/MArdnet7pwgALaeKs/why-i-m-not-into-the-free-energy-principle

Parr, T., Pezzulo, G., & Friston, K. (2022). Active inference: The free energy principle in mind, brain, and behavior. MIT Press. https://direct.mit.edu/books/oa-monograph/5299/Active-InferenceThe-Free-Energy-Principle-in-Mind

PsychStudies. (2018). Is psychoanalytic psychotherapy empirically validated? https://www.psychstudies.net/is-psychoanalytic-psychotherapy-empirically-validated/

Smith, R., Badcock, P., & Friston, K. (2022). Recent advances in the application of the free-energy principle to psychiatry. Current Opinion in Neurobiology, 76, 102602. https://pmc.ncbi.nlm.nih.gov/articles/PMC9260223/

Williams, D. (2022). Predictive processing and mental disorder. Philosophy, Psychiatry, & Psychology, 29(4), 267–290. https://pmc.ncbi.nlm.nih.gov/articles/PMC9731232/

Bayesian AR, ARMA, and ARIMA Models

Wed, 25 Feb 2026 00:00:00 +0000

Bayesian AR, ARMA, and ARIMA Models

Bayesian methods for time series modeling offer something frequentist approaches struggle with: full posterior inference that lets us quantify uncertainty and include prior knowledge directly into our analysis. This is the first post in our series on Bayesian time series analysis, where we’ll work through three models—autoregressive (AR), autoregressive moving average (ARMA), and autoregressive integrated moving average (ARIMA). Each model builds on the previous one. As before, we’ll implement everything using Stan and RStan, which gives us an R interface. The goal here isn’t just to show you code that runs, but to walk through how these models are actually constructed and what the Stan syntax is doing in detail.

Bayesian AR(1) Model

The simplest place to start is the AR(1) process. The idea is that today’s value depends linearly on yesterday’s value, plus some random noise. We can write this as:

\[y_t = \alpha + \phi y_{t-1} + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0, \sigma^2)\]

Here, $(\alpha)$ represents a constant drift term, $(\phi)$ controls how much yesterday’s value influences today (often called the autocorrelation parameter), and $(\epsilon_t)$ is our white noise. When $(

\phi

< 1)$, the process is stationary, meaning it won’t wander off to infinity. If $(\phi)$ gets too close to 1, though, the series develops a long memory and small shocks persist for a long time. Let’s simulate an AR(1) process in R to see what this looks like:

set.seed(123)
n <- 100
phi <- 0.7
sigma <- 1
y <- numeric(n)
y <- rnorm(1, 0, sigma / sqrt(1 - phi^2))
for (t in 2:n) {
  y[t] <- phi * y[t - 1] + rnorm(1, 0, sigma)
}
ts.plot(y, main = "Simulated AR(1) Time Series")

Notice that we initialize y using the stationary distribution of the AR(1) process. This isn’t strictly necessary for simulation, but it helps avoid transient startup effects. The denominator sqrt(1 - phi^2) comes from the variance of a stationary AR(1), which you can derive by taking variances on both sides of the model equation. Now for the Stan model. Stan’s syntax may look unfamiliar if you’re coming from BUGS or JAGS, but it’s designed to be more explicit about data types and constraints:

data {
  int<lower=1> N;
  vector[N] y;
}
parameters {
  real alpha;
  real<lower=-1, upper=1> phi;
  real<lower=0> sigma;
}
model {
  y[2:N] ~ normal(alpha + phi * y[1:N-1], sigma);
}

The data block declares what we’re passing in from R. The parameters block defines what Stan will sample: alpha can be any real number, phi is constrained between -1 and 1 to ensure stationarity, and sigma must be positive. The model block specifies the likelihood. Stan uses vectorized notation here—y[2:N] represents all observations from time 2 onward, and y[1:N-1] is the lagged series.

One thing to note: we’re not explicitly setting priors for alpha, phi, or sigma. Stan uses improper flat priors by default, which is fine for simple models but can cause problems with more complex hierarchical structures.

We fit the model in R like this:

library(rstan)
data_list <- list(N = length(y), y = y)
fit <- stan(
  file = "ar1_model.stan",
  data = data_list,
  chains = 4,
  iter = 2000,
  warmup = 1000
)
print(fit)

This runs four parallel chains, each with 2000 iterations (1000 for warmup, which Stan discards). The warmup phase tunes the sampler’s step size and mass matrix—think of it as the sampler learning the geometry of the posterior.

Bayesian ARMA(1,1) Model

The AR(1) model captures persistence, but it assumes shocks decay exponentially at a fixed rate. Real data often shows more complex behavior. Some shocks die out quickly, while the series still has long-term memory. The ARMA(1,1) model adds a moving average term to handle this:

\[y_t = \alpha + \phi y_{t-1} + \epsilon_t + \theta \epsilon_{t-1}, \quad \epsilon_t \sim \mathcal{N}(0, \sigma^2)\]

The new parameter $(\theta)$ controls how much yesterday’s forecast error affects today’s value. This gives the model more flexibility in shaping the autocorrelation function. When $(\phi)$ and $(\theta)$ have opposite signs, you can get patterns that pure AR models can’t replicate. Simulating this in R requires us to track the error terms explicitly:

set.seed(123)
n <- 200
phi <- 0.6
theta <- 0.5
sigma <- 1
y <- numeric(n)
e <- rnorm(n, 0, sigma)
y <- e 
for (t in 2:n) {
  y[t] <- phi * y[t - 1] + e[t] + theta * e[t - 1]
}
ts.plot(y, main = "Simulated ARMA(1,1) Time Series")

Here’s where things get tricky with Stan. We can’t directly write a recursion over parameters (like $`eps[t] = y[t] - mu[t]`$) inside the model block because Stan’s automatic differentiation system needs to know the full dependency graph upfront. Instead, we compute the residuals in a transformed parameters block:

data {
  int<lower=2> N;
  vector[N] y;
}
parameters {
  real alpha;
  real<lower=-1, upper=1> phi;
  real<lower=-1, upper=1> theta;
  real<lower=0> sigma;
}
transformed parameters {
  vector[N] mu;
  vector[N] eps;

  // Initialization
  mu[1] = alpha;
  eps[1] = y[1] - mu[1];

  for (t in 2:N) {
    mu[t] = alpha + phi * y[t - 1] + theta * eps[t - 1];
    eps[t] = y[t] - mu[t];
  }
}
model {
  eps[2:N] ~ normal(0, sigma);
}

The transformed parameters block lets us compute derived quantities that depend on parameters, and these quantities are saved in the posterior samples. The loop explicitly builds up the conditional mean mu[t] and residuals eps[t] at each time step. We then model eps[2:N] as normal with mean zero and standard deviation sigma. We skip eps because its distribution depends on initial conditions, which we’re treating as fixed here.

Fitting this model works the same way:

data_list <- list(N = length(y), y = y)
fit <- stan(
  file = "arma11.stan",
  data = data_list,
  chains = 4,
  iter = 2000,
  warmup = 1000
)
print(fit, pars = c("alpha", "phi", "theta", "sigma"))

You might notice the model runs slower than AR(1). That’s because the loop in transformed parameters can’t be vectorized, and Stan has to evaluate it at every iteration. For longer series, this becomes a bottleneck.

Bayesian ARIMA(1,1,1) Model

Many real-world time series aren’t stationary. If you try to fit an ARMA model to trending data, you’ll get nonsensical parameter estimates because the model assumes the mean is constant. The ARIMA framework addresses this by differencing the series first.

In an ARIMA(1,1,1) model, the middle “1” means we take one difference: $\(\Delta y_t = y_t - y_{t-1}$\). We then fit an ARMA(1,1) to the differenced series:

\[\Delta y_t = \alpha + \phi \Delta y_{t-1} + \epsilon_t + \theta \epsilon_{t-1}\]

This removes linear trends (and sometimes more complex nonstationarity, depending on what’s driving the trend). After differencing, the series may appear stationary, which satisfies the ARMA assumptions.

Let’s simulate an ARIMA(1,1,1) process. We generate the differenced series dy, then integrate it by taking the cumulative sum:

set.seed(42)
n <- 200
phi <- 0.6
theta <- 0.5
sigma <- 1
eps <- rnorm(n, 0, sigma)
dy <- numeric(n)
dy <- eps 
for (t in 2:n) {
  dy[t] <- phi * dy[t - 1] + eps[t] + theta * eps[t - 1]
}
y <- cumsum(dy)
ts.plot(y, main = "Simulated ARIMA(1,1,1) Time Series")

The resulting plot should show something that looks like it’s wandering around—this is the integrated part of ARIMA at work. In Stan, we handle differencing in a transformed data block, which runs once before sampling starts. This is efficient because differencing doesn’t depend on parameters:

data {
  int<lower=2> N;
  vector[N] y;
}
transformed data {
  vector[N - 1] dy;
  for (t in 2:N)
    dy[t - 1] = y[t] - y[t - 1];
}
parameters {
  real alpha;
  real<lower=-1, upper=1> phi;
  real<lower=-1, upper=1> theta;
  real<lower=0> sigma;
}
model {
  vector[N - 1] mu;
  vector[N - 1] eps;

  // initialization
  mu[1] = alpha;
  eps[1] = dy[1] - mu[1];

  for (t in 2:(N - 1)) {
    mu[t] = alpha + phi * dy[t - 1] + theta * eps[t - 1];
    eps[t] = dy[t] - mu[t];
  }

  // likelihood (conditional)
  dy[2:(N - 1)] ~ normal(mu[2:(N - 1)], sigma);
}

The transformed data block creates dy, a differenced version of y with length N - 1. Everything else looks similar to the ARMA(1,1) model, except now we’re working with the differenced series.

Fitting the model in RStan:

data_list <- list(N = length(y), y = y)
fit <- stan(
  file = "arima11.stan",
  data = data_list,
  chains = 4,
  iter = 2000,
  warmup = 1000
)
print(fit, pars = c("alpha", "phi", "theta", "sigma"))

One subtle point: the parameter alpha in this model represents the mean of the differenced series, not the original series. If alpha is positive, it implies a linear upward trend in the original data. Interpreting parameters after differencing takes a bit of care. The three models—AR, ARMA, and ARIMA presented here form the backbone of classical time series analysis, and their Bayesian versions inherit both the strengths and quirks of their frequentist counterparts. The advantage of going Bayesian is that we get full uncertainty quantification without relying on asymptotic approximations. We can also extend these models more naturally: adding hierarchical structure, incorporating external predictors, or letting parameters vary over time all fit comfortably within the Bayesian framework. That said, Bayesian inference isn’t free. These models can be slow, especially for long time series or when loops can’t be vectorized. ARMA and ARIMA models also assume certain invertibility and stationarity conditions, which aren’t always guaranteed just because we put bounds on parameters. And while Stan’s HMC sampler is generally more efficient than Gibbs sampling, it can still struggle with highly correlated posteriors or poorly identified parameters. Future extensions might involve seasonal ARIMA models (SARIMA), state space formulations that handle missing data more gracefully, or time-varying parameter models that relax the assumption of constant phi and theta.

References

Computational Psychiatry An Overview

Mon, 19 Jan 2026 00:00:00 +0000

An overview of computational psychiatry

Computational psychiatry tries to describe mental health problems in terms of “computations” the brain might be carrying out—how it learns from outcomes, updates beliefs, and chooses actions—so that symptoms can be connected to mechanisms rather than just checklists. The hope is not to replace clinical judgment, but to add a layer of formal, testable explanation that can sometimes clarify why two people with the same diagnosis look very different in daily life. In practice, these models link what can be observed (choices in tasks, symptom ratings, neural signals) to latent processes (like reward sensitivity or belief updating). That said, the field sometimes risks overselling “precision”: a clean parameter estimate in a lab task may not map neatly onto messy real-world experiences like breakup-driven insomnia or job-loss stress. This is a first post in series of posts where we discuss different approaches and methodologies used in computational psychiatry.

Reinforcement learning and decision-making

Reinforcement learning (RL) provides a family of models describing how agents learn to select actions based on past rewards and punishments. In these models, the agent updates expectations about outcomes when reality deviates from prediction, producing a reward prediction error signal that has clear analogues in dopaminergic brain activity. In depression, RL models have been used to formalize anhedonia and motivational deficits as changes in reward sensitivity, learning rate, or beliefs about controllability. Some work suggests that depressed individuals may underweight positive feedback and over‑generalize negative experiences, capturing phenomena like learned helplessness as altered value estimates for actions and outcomes.

Addiction research uses RL to characterize the shift from goal‑directed (model‑based) control to habitual (model‑free) responding, where drug cues continue to drive behavior despite mounting negative consequences. Models in this area often focus on how drugs distort dopamine‑encoded prediction errors and how this biases learning toward drug‑related actions and cues. More recent work extends simple model‑free formulations by combining them with latent‑state or latent‑cause inference, allowing the model to represent persistent internal motivational states (such as craving) that shape decisions across contexts. These hybrid models appear better suited to capturing phenomena like context‑dependent relapse or sudden shifts in behavior after periods of abstinence.

POMDPs and decision-making under uncertainty

Partially observable Markov decision processes (POMDPs) model decision‑making when the true state of the environment is uncertain and must be inferred from noisy observations. Formally, a POMDP can be described by a tuple including states, actions, transition probabilities, rewards, observations, and observation probabilities, together with a discount factor that determines the weight of future outcomes. Rather than directly observing the state, the agent maintains a belief distribution over possible states and updates it in light of new evidence. This structure aligns well with psychiatric contexts where people must infer threats, social intentions, or bodily states from ambiguous information. In anxiety disorders, POMDP‑style models can capture persistent overestimation of threat: the belief state remains biased toward danger, supporting hypervigilance and avoidance even when objective risk is low. Depression may involve pessimistic belief updating, where low‑probability negative outcomes get overweighted relative to positive possibilities, reinforcing withdrawal and hopelessness. In psychosis, POMDPs and related Bayesian state‑estimation models have been used to formalize “reality distortion” as an imbalance between prior expectations and sensory evidence. When priors are granted abnormally high precision relative to incoming data, the model can generate delusional interpretations of ambiguous stimuli; when sensory prediction errors are assigned aberrant salience, hallucinations can emerge from noisy internal signals.

Inverse reinforcement learning and Bayesian estimation

Inverse reinforcement learning (IRL) inverts the usual RL problem: given observed behavior, it seeks to infer the reward structure and preferences that would make that behavior rational within a task. This is particularly useful in psychiatry, where patients’ motivations and value functions may differ qualitatively from those of healthy controls, but cannot be observed directly. Bayesian implementations of RL and IRL, often fit with probabilistic programming languages such as Stan, allow researchers to capture individual variability via hierarchical priors and to quantify uncertainty about parameters. This supports richer “computational phenotyping,” where traits like reward sensitivity, risk preferences, or controllability beliefs are treated as latent parameters that vary across the population and may relate to brain measures or treatment outcomes.

Recent semiparametric IRL approaches applied to major depressive disorder (MDD) use hierarchical Bayesian models to estimate both learning dynamics and reward sensitivity across individuals. Guo and colleagues showed that MDD and control groups can display similar learning rates while differing markedly in the shape of their reward sensitivity functions, which were nonlinear and attenuated in patients. This challenges the idea that depression necessarily involves a primary learning deficit, instead highlighting alterations in how rewards are valued.

Agent-based modeling and population dynamics

Agent‑based models (ABMs) simulate many interacting individuals (“agents”) embedded in an environment, each following simple rules, to study how individual‑level processes give rise to population‑level patterns. In computational psychiatry, ABMs are used to explore how symptoms, treatment access, and social context interact over time across communities. Agents can be endowed with internal psychological dynamics (such as symptom trajectories or learning rules) and placed within social networks that transmit information, norms, or support. This allows simulation of how changes in one person’s symptoms—say after starting cognitive behavioral therapy—might ripple through their support network to influence others’ trajectories, or how service redesign affects wait times, adherence, and overall burden on a mental health system. Multi‑scale ABMs can link medical data (e.g., diagnosis rates, treatment capacities) with behavioral rules to test hypothetical policy changes before real‑world implementation. These models are especially powerful for studying questions that are hard to tackle in randomized trials, such as how economic shocks, digital interventions, or stigma might shape the long‑term prevalence of disorders. That said, they require strong assumptions about agents and environments, so careful validation against empirical data is essential.

Predictive coding and the Bayesian brain

As we have seen in our series of posts about the Free Energy Principle and Active inference, the predictive coding and Bayesian brain frameworks propose that the brain constantly generates predictions about sensory inputs and minimizes prediction errors by updating beliefs or acting on the world. In these accounts, perception, action, and even emotion emerge from hierarchical generative models that balance prior expectations against incoming evidence, weighted by their relative precision. In our post about autism we mentioned that predictive coding theories suggest altered precision assignments may lead to overly precise low‑level sensory predictions and insufficient weighting of higher‑level context. This can help explain both hypersensitivity to sensory detail and difficulty using broader context to interpret ambiguous stimuli, as described in work on “precise minds in uncertain worlds.” In schizophrenia and related psychotic disorders, models attribute positive symptoms to mis‑tuned precision across cortical hierarchies, where over‑precise priors can drive delusions and aberrant assignment of salience to neutral events, while under‑ or mis‑precise sensory predictions contribute to hallucinations. Active inference extends predictive coding by treating action selection as another route to minimizing expected prediction error or free energy, often formalized in POMDP‑like terms. This provides a unified setting in which perception, decision‑making, and action are modeled under a single Bayesian principle, and has inspired work on motor symptoms, interoception, and social cognition in psychiatry.

Temporal dynamics and dynamical systems

Most computational models treat parameters as static—a person’s reward sensitivity or learning rate is estimated once and assumed constant. Yet psychiatric symptoms fluctuate from day to day, sometimes hour to hour, and the relationships among symptoms can shift across timescales. Dynamical systems approaches use differential equations to capture how variables evolve continuously in time, allowing researchers to model symptom trajectories, attractor states, and sudden transitions (bifurcations) that occur when environmental stressors or interventions push someone across a threshold. A dynamical model might track interactions among mood, rumination, sleep, and social contact, with each variable influencing the others according to differential equations that specify rates of change. Time-varying network models extend this idea by estimating how the strength and direction of these associations change over weeks or months. In one study of patients with recurrent depression, daily self-report data revealed marked differences between individuals: some showed stable symptom networks over months, while others exhibited rapid shifts in how symptoms influenced each other within weeks. This suggests that the same diagnosis can hide very different underlying dynamics, with implications for when and how to intervene.

These models raise practical questions about timescales. Should we capture mood variations every 15 minutes, once per day, or weekly? Finer temporal resolution risks drowning signal in noise—every minor reaction to a text message—while coarser bins may miss clinically meaningful fluctuations like morning versus evening mood in cyclothymia. There is no universal answer; the right timescale depends on the disorder and the question. Dynamical models also demand longer stretches of data than traditional cross-sectional studies, which can be burdensome for participants. Still, the payoff is substantial: by mapping how symptoms interact and change, these approaches can identify early warning signals of relapse, critical periods for treatment, and personalized intervention targets that static models overlook.

Machine learning and data-driven models

In parallel with mechanism‑driven models, computational psychiatry increasingly uses machine learning (ML) to extract patterns from high‑dimensional data such as neuroimaging, genetics, and digital traces. Approaches include deep learning for brain images, clustering methods to discover symptom subtypes, and natural language processing to detect linguistic markers of risk in speech or social media. ML models often excel at prediction but can be hard to interpret, whereas mechanistic models (RL, POMDPs, predictive coding) sacrifice some predictive power for explanatory insight. Hybrid strategies are emerging: ML might identify subgroups of patients with distinct trajectories, while mechanistic models characterize the computational profiles of each subgroup. Semi‑supervised and multitask learning approaches can leverage limited labeled clinical data alongside large unlabeled digital datasets, improving prediction while retaining some structure for interpretation. Ethical and practical considerations loom large—data quality, algorithmic bias, transparency, and the risk of over‑reliance on black‑box predictions all pose challenges for clinical deployment. Frameworks for ethical decision‑making in AI for mental health emphasize stakeholder involvement, explanation tailored to patients, and early attention to implementation barriers, not just model performance.

Future directions and conclusion

Computational psychiatry now has a rich toolkit spanning reinforcement learning, POMDPs, IRL, agent‑based models, predictive coding, dynamical systems. These approaches promise more precise, mechanism‑informed psychiatry, where diagnoses and treatments are guided not only by symptom counts but by quantified alterations in learning, valuation, belief formation, and temporal dynamics. Realizing this promise requires careful attention to ethics, education, and infrastructure. Key challenges include protecting privacy while analyzing sensitive multimodal data, mitigating bias in training sets, ensuring that models generalize beyond narrow cohorts, and supporting clinicians in interpreting and communicating computational outputs. Progress will depend on collaboration among computational scientists, clinicians, patients, and ethicists, with a focus on tools that are not only accurate but also trustworthy, equitable, and clinically useful. As the field matures, the questions shift from “Can we model this?” to “Should we model this, and for whom?”. The most useful advances will likely come not from ever-more sophisticated algorithms, but from careful attention to context, stakeholder needs, and the messy realities of implementing formal models in healthcare systems that are already stretched thin. Computational psychiatry has opened doors; walking through them will require humility, interdisciplinary dialogue, and a willingness to test ideas in the real world where patients live.

References

Adams, R. A., Huys, Q. J. M., & Roiser, J. P. (2016). Computational psychiatry: Towards a mathematically informed understanding of mental illness. Journal of Neurology, Neurosurgery & Psychiatry, 87(1), 53–63. https://doi.org/10.1136/jnnp-2015-310737
Adams, R. A., Meyer, S., Smith, R., & et al. (2025). Computational psychiatry: A bridge between neuroscience and clinical practice. Psychiatry and Clinical Neurosciences Reports, 4, 1–15. https://doi.org/10.1002/pcn5.41
Adolphs, R., & Montague, P. R. (2024). Neurocomputational underpinnings of suboptimal beliefs in mental illness. Conference on Cognitive Computational Neuroscience (CCN 2025) Proceedings. https://pehlevan.seas.harvard.edu/sites/g/files/omnuum6471/files/2025-07/Kumar_et_al_CCN_Proceedings_2025.pdf
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1998). Reinforcement learning: An introduction. MIT Press. https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf
Fusar-Poli, P., Correll, C. U., Arango, C., & et al. (2022). Ethical considerations for precision psychiatry: A position statement from the European Brain Research Area. European Psychiatry, 65(1), e57. https://www.ebra.eu/wp-content/uploads/2022/12/PSMD_Fusar-Poli-et-al.-2022-Ethical-considerations-for-precision-psychiatry-A.pdf
Huys, Q. J. M. (2021, November 29). Bayesian hierarchical reinforcement learning in psychiatry. In Computational psychiatry (blog). https://bruno.nicenboim.me/2021/11/29/bayesian-h-reinforcement-learning/
Khaleghi, A., & et al. (2022). Computational neuroscience approach to psychiatry: A review on theory-driven approaches. Clinical Psychopharmacology and Neuroscience, 20(1), 26–36. https://doi.org/10.9758/cpn.2022.20.1.26
Maia, T. V., & Frank, M. J. (2011). From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience, 14(2), 154–162. https://pmc.ncbi.nlm.nih.gov/articles/PMC2866366/
Montague, P. R., Dolan, R. J., Friston, K. J., & Dayan, P. (2012). Computational psychiatry. Trends in Cognitive Sciences, 16(1), 72–80. https://doi.org/10.1016/j.tics.2011.11.018
Możaryn, J. F., & et al. (2025). Computational psychiatry: A bridge between neuroscience and clinical practice. Frontiers in Psychiatry, 16, 1181526. https://pmc.ncbi.nlm.nih.gov/articles/PMC11815263/
Samson, R. D., Frank, M. J., & Fellous, J.-M. (2010). Computational models of reinforcement learning: The role of dopamine as a reward signal. Cognitive Neurodynamics, 4(2), 91–105. https://doi.org/10.1007/s11571-010-9109-x
Seriès, P. (n.d.). A primer on computational psychiatry. School of Informatics, University of Edinburgh. https://homepages.inf.ed.ac.uk/pseries/CPPrimer/CPprimer-submitted.pdf
Wang, X.-J., & Krystal, J. H. (2014). Computational psychiatry. Neuron, 84(3), 638–654. https://pmc.ncbi.nlm.nih.gov/articles/PMC4255477/

Mathematical Formalism of FEP and Active Inference

Sat, 20 Dec 2025 00:00:00 +0000

Mathematical Formalism of FEP and Active Inference

We have discussed in several posts that Free Energy Principle (FEP) and its algorithmic implementation in Active Inference represent a unifying theory of cognition, perception, and action. This framework is based on the idea that biological systems maintain their structural and functional integrity by minimizing a variational upper bound on sensory surprise (negative log model evidence). This minimization drives both perceptual inference and decision-making, allowing organisms to navigate uncertainty while maintaining homeostasis. This post provides a rigorous walkthrough of its formal architecture, transitioning from basic probability and variational principles to advanced formulations involving path integrals, gradient flows, and stochastic differential equations. The aim is to show that, far from being metaphysical or speculative, the FEP is grounded in well-established principles of information theory, thermodynamics, and statistical inference.

Generative Models and Bayesian Inference

At the heart of the FEP is a generative model, which encodes how an agent believes sensory data arise from hidden states of the world. Let $s$ denote hidden states and $o$ denote sensory observations. The generative model defines the joint distribution:

\[P(o, s)\]

This joint distribution specifies the agent’s beliefs about how sensory inputs $o$ are generated from latent causes $s$. From this, inference becomes the process of computing the posterior:

\[P(s \mid o) = \frac{P(o, s)}{P(o)}\]

Exact computation is often intractable because it requires evaluating the model evidence $P(o) = \sum_s P(o, s)$ (or an integral in continuous domains). Active Inference therefore employs variational inference, approximating the true posterior with a tractable distribution $Q(s)$ drawn from a restricted family.

Variational Free Energy

The quality of this approximation is measured by the variational free energy $F$, defined as:

\[F = \mathbb{E}_{Q(s)}[\log Q(s) - \log P(o, s)]\]

This is (up to sign) the Evidence Lower Bound (ELBO) familiar from variational inference, and it satisfies:

\[F = D_{\text{KL}}(Q(s) \| P(s \mid o)) - \log P(o)\]

Here, $D_{\text{KL}}$ is the Kullback–Leibler divergence. Because $-\log P(o)$ is constant with respect to $Q$, minimizing $F$ is equivalent to minimizing the divergence between the approximate and true posteriors. Thus, perception can be written as:

\[Q^*(s) = \arg\min_Q F\]

In this formalism, the agent infers the most plausible hidden causes of its observations by minimizing free energy.

Action and Expected Free Energy

While perception corresponds to belief updating about the present, action requires planning over possible futures. To model this, Active Inference introduces the expected free energy $G(\pi)$, which evaluates a policy $\pi$ — a (possibly stochastic) sequence of future actions — based on its expected impact on beliefs and sensory states:

\[G(\pi) = \mathbb{E}_{Q(o, s \mid \pi)}[\log Q(s \mid \pi) - \log P(o, s)]\]

Under standard factorizations of the generative model and recognition density, this expression can be decomposed as:

\[G(\pi) = \mathbb{E}_{Q(o \mid \pi)}\left[ D_{\text{KL}}(Q(s \mid o, \pi) \| P(s)) \right] - \mathbb{E}_{Q(o \mid \pi)}[\log P(o)]\]

This decomposition reveals two fundamental components:

Epistemic value (information gain): the expected divergence between posterior and prior beliefs, encouraging policies that resolve uncertainty.
Extrinsic value (preference fulfilment): the expected log probability of outcomes under prior preferences, guiding goal-directed behavior.

By selecting the policy that minimizes $G(\pi)$, the agent balances exploration and exploitation:

\[\pi^* = \arg\min_\pi G(\pi)\]

This converts action selection into variational inference over policies.

Generalized Free Energy in Continuous Time

In continuous-time formulations, beliefs and observations evolve along trajectories. This is captured by a generalized free energy functional:

\[\mathcal{F}[q] = \int_0^T \mathbb{E}_{q(s_t)}\left[ \log q(s_t) - \log p(o_t, s_t) \right] \, dt\]

Here, $q(s_t)$ is the time-indexed approximate posterior, and $p(o_t, s_t)$ is the generative model at time $t$. This integral defines a path-dependent cost functional, analogous to action functionals in Lagrangian mechanics.

Minimizing this functional yields variational dynamics of the form:

\[\frac{dq(s_t)}{dt} = - \nabla_q \mathcal{F}[q]\]

This defines a gradient flow on the space of probability densities, closely related to Fokker–Planck or continuity equations that govern the time evolution of distributions in stochastic systems.

Generalized Coordinates of Motion

To model perception in continuous environments, the FEP makes use of generalized coordinates of motion:

\[\tilde{s} = \{s, \dot{s}, \ddot{s}, \dots\}\]

These extended state representations encode position, velocity, acceleration, and higher-order temporal derivatives of hidden states. By representing sensory flows over time, the generative model can predict smooth temporal trajectories instead of isolated snapshots.

This is essential for capturing the temporal structure of perception, particularly in vision and proprioception, where higher-order dynamics (e.g., motion and acceleration) carry information that is critical for accurate interpretation and control.

Decomposition of Expected Free Energy

Returning to the discrete-time expected free energy, its decomposition highlights how Active Inference simultaneously reduces uncertainty and pursues preferred states. The formal expression:

\[G(\pi) = \mathbb{E}_{q(o \mid \pi)}\left[ D_{\text{KL}}(q(s \mid o, \pi) \| p(s)) \right] - \mathbb{E}_{q(o \mid \pi)}[\log p(o)]\]

splits into:

The first term, epistemic value, quantifies expected information gain by measuring the divergence between posterior and prior beliefs. It favors policies that generate informative observations.
The second term, extrinsic value, quantifies the expected log probability of outcomes under prior preferences. It favors policies that steer the agent toward preferred or biologically viable states.

Crucially, both terms are consequences of a single generative model and require no separate reward function, which distinguishes Active Inference from many reinforcement learning formulations based on externally specified utilities.

Active Inference as Stochastic Optimal Control

The FEP also admits an interpretation within stochastic control theory. Let the system dynamics be given by:

\[ds_t = f(s_t, a_t)\,dt + \omega_t, \quad o_t = g(s_t) + \nu_t\]

with $\omega_t$ and $\nu_t$ representing process and observation noise, respectively. In place of minimizing an explicit cost function $C(s, a)$, the agent minimizes expected free energy over policies:

\[\pi^* = \arg\min_\pi \mathbb{E}_{q(o, s \mid \pi)}[\mathcal{F}]\]

Under appropriate choices of generative model and preferences, this framing yields a form of risk-sensitive, information-seeking control. It embeds traditional control objectives (e.g., target reaching or set-point regulation) within a broader variational architecture that also encodes uncertainty reduction and predictive consistency.

Information-Theoretic Foundations

At a deeper level, the FEP is an information-theoretic principle. It states that agents minimize the divergence between predicted and actual sensory states, thereby maximizing the mutual information between internal beliefs and observations, subject to model constraints.

In this setting, expected free energy bounds the expected surprise of observations under a given policy:

\[G(\pi) \geq -\log P(o \mid \pi)\]

Minimizing $G(\pi)$ therefore leads agents to select actions that render future observations less surprising and more consistent with their prior preferences and generative model. This information-theoretic view aligns with efficient coding hypotheses in neuroscience and with Shannon’s treatment of entropy and uncertainty.

Thermodynamic Analogy

A compelling interpretation of the FEP is thermodynamic. Variational free energy can be written in a form that mirrors the Helmholtz free energy from statistical physics:

\[F = U - T S\]

Here, $U$ denotes an expected energy or internal energy term (often related to expected negative log likelihood), $S$ is an entropy term, and $T$ plays a role analogous to an effective temperature or precision. In the FEP context, internal energy reflects the agent’s confidence in its generative model, whereas entropy captures uncertainty about hidden causes.

By minimizing variational free energy, the agent counteracts entropy-increasing environmental perturbations and preserves its characteristic states. This explains why biological systems can persist and adapt despite ongoing exposure to stochastic sensory inputs: life can be viewed as the continual suppression of surprise, or more formally, as the sustained minimization of variational free energy over time.

Table: Summary of Key Mathematical Constructs in FEP and Active Inference

Concept	Mathematical Expression	Interpretation
Generative model	$P(o, s)$	Joint probability over observations $o$ and hidden states $s$
Bayesian inference	$P(s \mid o) = \frac{P(o, s)}{P(o)}$	Posterior belief about hidden causes given observations
Variational free energy	$F = \mathbb{E}_{Q(s)}[\log Q(s) - \log P(o, s)]$	Upper bound on surprise; minimized in perceptual inference
Free energy decomposition	$F = D_{\text{KL}}(Q(s) \| P(s \mid o)) - \log P(o)$	Complexity–accuracy trade-off in approximate inference
Perceptual inference	$Q^*(s) = \arg\min_Q F$	Optimal approximate posterior under free energy minimization
Expected free energy	$G(\pi) = \mathbb{E}_{Q(o, s \mid \pi)}[\log Q(s \mid \pi) - \log P(o, s)]$	Evaluates policies in terms of predicted beliefs and outcomes
EFE decomposition	$G(\pi) = \mathbb{E}{Q(o)}[D{\text{KL}}(Q(s \mid o) \| P(s))] - \mathbb{E}_{Q(o)}[\log P(o)]$	Epistemic (information gain) plus extrinsic (preference) value
Policy selection	$\pi^* = \arg\min_\pi G(\pi)$	Select policies that minimize expected free energy
Path integral formulation	$\mathcal{F}[q] = \int_0^T \mathbb{E}_{q(s_t)}[\log q(s_t) - \log p(o_t, s_t)] \, dt$	Generalized free energy over time in continuous systems
Gradient flow on beliefs	$\frac{dq(s_t)}{dt} = - \nabla_q \mathcal{F}[q]$	Belief dynamics follow variational gradient descent
Generalized coordinates	$\tilde{s} = {s, \dot{s}, \ddot{s}, \dots}$	Temporally extended representation of hidden states
Thermodynamic analogy	$F = U - T S$	Internal energy minus entropy (information-theoretic analogue)
Information-theoretic bound	$G(\pi) \geq -\log P(o \mid \pi)$	Expected free energy upper-bounds expected surprise

The Free Energy Principle and Active Inference framework provide a mathematically coherent and unifying theory of adaptive behavior. Through the minimization of variational free energy, agents simultaneously infer the hidden causes of their sensations and select actions that balance uncertainty reduction with goal-directed behavior. This dual role of inference — retrospective (perception) and prospective (action) — is rendered tractable through variational methods, expected free energy, and gradient flows on belief manifolds. From the derivation of the expected free energy functional to its decomposition into epistemic and extrinsic value, the FEP offers not just a metaphor but a precise algorithmic account of perception and action.

Anomaly Detection with Isolation Forests in R

Sun, 23 Nov 2025 00:00:00 +0000

Anomaly Detection with Isolation Forests in R

As we mentioned in a previous post anomaly detection (outlier or novelty detection) is a task in data analysis where the goal is to identify rare items, events, or observations that deviate significantly from the majority of the data. This process is used in various domains such as fraud detection. Following up on an earlier post this post will guide you through the concept, implementation, and application of isolation forests, with code examples in R, with the isotree package.

The Isolation Forest algorithm, first introduced by Liu et al. in 2008, is a tree-based, unsupervised learning algorithm, based on the principle of isolating anomalies which are few and different compared to normal data points. The algorithm achieves this by randomly partitioning the dataset into isolation trees (iTrees). Anomalies are isolated faster due to their uniqueness, requiring fewer splits to separate them from the rest of the data. The shorter the path length in these trees, the higher the anomaly score of a point. This is in contrast, normal points require more splits due to their denser distribution. The method begins by constructing multiple isolation trees (iTrees) using random subsets of the dataset. Each tree is built through recursive partitioning, where a feature is randomly selected, and a split value is chosen randomly within the feature’s range. This process continues until every data point is isolated or a maximum tree depth is reached. The ease of isolating a data point is reflected in its path length—the number of splits required to separate it from the rest of the data, hence, the isolation forests are efficient with high-dimensional data and can handle both large datasets and complex feature interactions. Another advantage is the versatility in handling both numerical and categorical data directly, eliminating the need for extensive preprocessing or transformations. Isolation Forests do not require explicit data standardization or normalization which reduces the preprocessing overhead, making the algorithm more user-friendly and accessible. Moreover, Isolation Forests provide clear and interpretable results through anomaly scores. Each data point is assigned a score that quantifies its degree of anomaly, enabling straightforward decision-making. This transparency allows practitioners to understand the basis of anomaly detection, fostering trust in the model’s predictions and facilitating communication with stakeholders.

Implementing Isolation Forests in R

We generate a dataset with 500 points where most points are sampled from a normal distribution, and a few are sampled as outliers.

set.seed(42)

# Generate normal data
n <- 500
normal_data <- data.frame(
  x = rnorm(n, mean = 0, sd = 1),
  y = rnorm(n, mean = 0, sd = 1)
)

# Add anomalies
outliers <- data.frame(
  x = rnorm(20, mean = 5, sd = 0.5),
  y = rnorm(20, mean = 5, sd = 0.5)
)

# Combine datasets
data <- rbind(normal_data, outliers)

# Visualize the data
plot(data$x, data$y, col = "blue", pch = 20, main = "Synthetic Dataset", xlab = "Feature X", ylab = "Feature Y")
points(outliers$x, outliers$y, col = "red", pch = 20)

Training the Isolation Forest Model

We now train an isolation forest model using the isolation.forest function. This function builds a series of isolation trees, evaluating anomaly scores for each point.

library(isotree)

# Fit isolation forest model
iso_forest <- isolation.forest(
  data,
  ndim = 2,
  ntrees = 100,
  sample_size = 256
)

# Display model summary
summary(iso_forest)

Generating Anomaly Scores

After training the model, we calculate anomaly scores for each point. These scores range from 0 to 1, with higher scores indicating greater anomaly likelihood.

# Compute anomaly scores
scores <- predict(iso_forest, data, type = "anomaly_score")

# Add scores to the dataset
data$score <- scores

# Visualize the distribution of scores
hist(scores, breaks = 30, main = "Distribution of Anomaly Scores", xlab = "Anomaly Score", col = "lightblue")

Visualizing Anomalies

Points with high anomaly scores can be flagged as potential outliers. We highlight these points in the dataset visualization.

# Flag anomalies
threshold <- 0.7
data$is_anomaly <- data$score > threshold

# Visualize anomalies
plot(data$x, data$y, col = ifelse(data$is_anomaly, "red", "blue"), pch = 20,
     main = "Anomaly Detection with Isolation Forests", xlab = "Feature X", ylab = "Feature Y")
legend("topright", legend = c("Normal", "Anomaly"), col = c("blue", "red"), pch = 20)

Customizing the Isolation Forest Model

The isotree package provides several options to customize the isolation forest. These options include:

ndim: Number of dimensions to consider in splits.
ntrees: Number of isolation trees to build.
sample_size: Subset size for building each tree.
scoring_metric: Metric for scoring anomalies.

An extended isolation forest allows non-linear splits by projecting data onto random hyperplanes. This is useful for datasets with complex distributions.

# Extended isolation forest
ext_iso_forest <- isolation.forest(
  data,
  ndim = 3,
  ntrees = 100,
  sample_size = 256,
  scoring_metric = "standard"
)

# Compute anomaly scores
ext_scores <- predict(ext_iso_forest, data, type = "anomaly_score")

# Compare extended and standard isolation forests
data$ext_score <- ext_scores
plot(data$score, data$ext_score, xlab = "Standard IF Score", ylab = "Extended IF Score",
     main = "Comparison of Anomaly Scores")

Evaluating Performance

To evaluate the performance of an isolation forest, consider metrics like the Area Under the ROC Curve (AUC) or the precision-recall curve. These metrics require labeled data, distinguishing normal points from anomalies.

Example: ROC Curve

library(pROC)

# Simulate labels for evaluation
labels <- c(rep(0, n), rep(1, 20)) # 0 = normal, 1 = anomaly

# ROC Curve
roc_curve <- roc(labels, scores)
plot(roc_curve, col = "darkgreen", main = "ROC Curve for Isolation Forest")

Conclusion

Isolation Forests is a useful anomaly detection method with applications across fields such as healthcare. Their ability to isolate anomalies effectively makes them invaluable for identifying rare but critical patterns in medical datasets. In healthcare, Isolation Forests can identify anomalous patient data that may indicate underlying issues requiring immediate attention. For example, they can detect sudden spikes in heart rate, irregularities in glucose levels, or deviations in vital signs that might signal the onset of critical conditions like sepsis or arrhythmias. In hospital settings, they are used to monitor ICU equipment, identifying anomalies in sensor readings that could suggest malfunctions or false alarms, ensuring timely interventions and reducing risks. In public health, Isolation Forests play a role in detecting anomalies in epidemiological data, such as unusual spikes in emergency room visits or reported symptoms. These insights can help identify early signs of disease outbreaks, enabling preventive measures and resource allocation before the situation escalates. Similarly, wearable health devices equipped with anomaly detection capabilities use Isolation Forests to monitor user health, flagging irregularities like abnormal sleep patterns or activity levels, prompting users to seek medical advice. Beyond healthcare, Isolation Forests are applied in other domains like manufacturing and IoT. For instance, in pharmaceutical production, they can identify deviations in equipment performance that may compromise drug quality.

Isolation Forests are an efficient method for anomaly detection, capable of handling diverse datasets with minimal preprocessing. Although the algorithm is robust to varying feature scales, applying transformations such as log scaling to address skewed data can enhance performance in some cases. Fine-tuning parameters like the number of trees (ntrees), sample size, and the number of dimensions (ndim) allows finding the balance between computational efficiency and detection accuracy, adapting the model to specific use cases. The ‘isotree’ package in R simplifies the implementation of isolation forests, offering a customizable framework for users. However, interpreting anomaly scores requires a contextual approach, as these scores serve as relative indicators rather than absolute measures of anomalies. Domain expertise still plays an important role in determining appropriate thresholds, ensuring meaningful insights about the nature of the anomaly.

References

Anomaly Detection Definition - TechTarget provides an overview of anomaly detection, its types, and applications across industries.
Isolation Forest Explanation - Spot Intelligence discusses the Isolation Forest algorithm in detail, highlighting its advantages and use cases.
GitHub Repository for Isolation Forest - A repository showcasing an implementation of Isolation Forest, including example code and documentation.
An Introduction to Isolation Forests - The CRAN vignette for the isotree package, which explains the use of Isolation Forests in R.
Resources on Anomaly Detection - A Reddit thread sharing valuable resources and insights on anomaly detection techniques.
Isolation Forest Variants - An academic paper on advancements and variants of the Isolation Forest algorithm.
Anomaly Detection with Isolation Forest - A GeeksforGeeks article providing an implementation of Isolation Forest for anomaly detection.
DigitalOcean Guide on Isolation Forest - A tutorial on DigitalOcean covering the theory and application of Isolation Forests.
Outlier Detection with Isolation Forest in R - A Kaggle notebook demonstrating the use of Isolation Forest for detecting anomalies in R.
Isolation Forest Paper (Liu et al., 2008) - The foundational paper introducing the Isolation Forest algorithm, authored by Liu et al.
CRAN isotree Package Documentation - Official documentation for the isotree R package, which implements Isolation Forests.
Introduction to Isolation Forests in isotree - A detailed introduction to Isolation Forests using the isotree R package.
TechTarget’s Overview on Anomaly Detection - An in-depth explanation of anomaly detection, including its methods and real-world applications.
Spot Intelligence’s Tutorial on Isolation Forest - A guide to Isolation Forests, covering their implementation and practical examples.
GitHub Repository for Isolation Forest in R - Source code and examples for applying Isolation Forest in R, hosted on GitHub.

Active Inference Account of Substance Use Disorder

Sun, 26 Oct 2025 00:00:00 +0000

Active Inference Account of Substance Use Disorder

The free energy principle (FEP) and active inference framework offer a novel perspective for understanding substance use disorder (SUD), reframing addiction not as a disease of the brain, but as a complex disorder of inference, decision-making, and environmental interaction. This approach conceptualizes addiction as emerging from the brain’s attempts to minimize free energy—a measure of surprise or uncertainty—through maladaptive predictive models that become pathological. Traditional models of addiction have focused primarily on dopaminergic dysfunction and reward learning abnormalities, leading to the widespread adoption of the “disease model.” While this framework explains the compulsive nature of drug-seeking behavior through dysfunctional reward systems, it provides an incomplete picture of the underlying mechanisms. The predictive processing theory within active inference offers a more nuanced understanding: rather than being passive recipients of reward signals, agents actively predict rewarding outcomes and select actions that minimize long-term prediction error. In this framework, dopamine doesn’t simply signal reward prediction error but instead encodes the precision—or confidence—of affordances, which represent opportunities for action available in the environment. This distinction is crucial for understanding addiction. When individuals use substances, they experience what feels like successful error reduction, but this represents false feedback that creates an illusion of minimizing uncertainty while actually increasing it.

Precision & Uncertainty in Addiction

Recent computational models have identified two seemingly contradictory mechanisms underlying addictive behavior: hyper-precision and hypo-precision. The hyper-precision model suggests that addiction involves excessive confidence in drug-related predictions, creating rigid behavioral patterns focused narrowly on substance use. Individuals become locked into what can be described as fixed-point attractors—predictable behavioral cycles that persist despite their detrimental outcomes. Conversely, the hypo-precision model proposes that addiction results from reduced model-based control, where individuals show decreased ability to plan and consider long-term consequences. This manifests as a change from goal-directed behavior to habitual, automatic responses—a transition from thoughtful decision-making to compulsive action. Environmental factors such as stress, poverty, and social instability can exacerbate this by reducing the precision of model-based predictions, pushing individuals toward more immediate, model-free control strategies. Along the same lines, the active inference perspective emphasizes that addiction cannot be understood solely through individual brain dysfunction but must be viewed as a breakdown in the agent-environment system. Environmental unpredictability—whether through inconsistent social support, economic instability, or uncertain drug supply—compromises the brain’s ability to form accurate predictive models. This uncertainty paradoxically doesn’t affect drug use but instead exacerbates harmful patterns by making individuals more reliant on the immediate, albeit false, certainty that substances appear to provide. Research demonstrates that factors like reduced social support and poverty significantly impact addiction and relapse rates, not merely as social determinants but as computational challenges that impair model-based decision-making. When individuals encounter drug-related cues under conditions of high environmental uncertainty, their working memory-intensive processes for reasoning through action-outcome associations become compromised, leading to steep discounting of future outcomes and maintenance of addictive behaviors.

Generative Models And their Clinical Implications

Perhaps most profoundly, the FEP framework reveals how addiction becomes integrated into an individual’s sense of self. Agents maintain their identity through biological autonomy, which involves fulfilling predictions about their environment through internal generative models—hierarchical prediction systems that guide interaction with the world. In addiction, substance use becomes so central to these generative models that changing behavior requires fundamental alterations to one’s sense of identity. This integration helps explain why addiction is particularly resistant to change and why successful treatment often involves not just addressing substance use but reconstructing fundamental aspects of identity and life structure. The predictability and reliability of substance effects can provide stability in otherwise chaotic circumstances, making the drug-related lifestyle feel necessary for maintaining coherent selfhood.

The Bayesian inference framework embedded within the FEP provides crucial insights into why addiction persists despite obvious negative consequences. Rather than viewing addictive behavior as fundamentally irrational, this approach suggests that such behavior may be optimal given the individual’s generative model of the world. The apparent irrationality emerges not from broken inference mechanisms but from maladaptive generative models that have become entrenched through repeated experience. This perspective shifts focus from presumed optimal behavior to understanding the specific parameters and characteristics of individual generative models that lead to pathological outcomes. For instance, an individual’s generative model might accurately predict that drug use will provide immediate relief from withdrawal symptoms, even while failing to adequately weight longer-term negative consequences.

The FEP approach to addiction suggests several novel therapeutic directions. Rather than focusing solely on blocking reward pathways or teaching willpower, treatments might target the precision of different types of predictions. This could involve therapies aimed at reducing the precision assigned to drug-related affordances while simultaneously strengthening model-based control systems and environmental predictability. Interventions might also focus on reconstructing generative models through carefully structured experiences that challenge pathological predictions and support the development of more adaptive behavioral repertoires. This could include creating therapeutic environments that provide the reliability and predictability that substances falsely promised, while gradually expanding the range of behaviors through which individuals can achieve successful error reduction. The framework also suggests the importance of addressing environmental factors that contribute to uncertainty and stress, which compromise model-based decision-making. Policy implications include ensuring access to stable housing, healthcare, and social support—not merely as social goods but as computational necessities for healthy inference and decision-making.

Conclusions

The free energy principle offers a sophisticated and integrative framework for understanding addiction that moves beyond simple disease models toward a comprehensive account of how individuals interact with their environment through prediction and action. This approach recognizes addiction as neither purely biological nor purely social but as an emergent property of complex systems trying to maintain coherence in uncertain environments. While this theoretical framework is promising, significant challenges remain in translating these insights into practical clinical applications. The mathematical complexity of these models and the deeply embedded nature of addictive patterns in identity and social context make intervention challenging. Nevertheless, this approach offers hope for more effective treatments that address the fundamental computational and inferential processes underlying addictive behavior, potentially leading to more personalized and effective therapeutic approaches. The key insight from this computational perspective is that addiction represents not a failure of willpower or moral character, but a predictable outcome of particular generative models operating in specific environmental contexts. Understanding these mechanisms offers pathways toward more compassionate, effective, and scientifically grounded approaches to treatment and prevention.

Aspect	Traditional View	FEP/Active Inference View
Core Problem	Reward system dysfunction	Inference and prediction errors
Dopamine Role	Reward prediction error	Precision of affordances/confidence
Addiction Mechanism	Hijacked reward circuits	Maladaptive generative models
Environmental Factors	External stressors	Computational challenges to inference
Treatment Focus	Block rewards, teach control	Reconstruct predictive models, reduce precision
Identity	Separate from addiction	Integrated into self-model

References

Holmes J. Friston’s free energy principle: new life for psychoanalysis? BJPsych Bull. 2022;46(3):134-139
Smith R, et al. Imprecise Action Selection in Substance Use Disorder. Front Hum Neurosci. 2020;14:366
Miller M, et al. Embodying addiction: A predictive processing account. Brain Cogn. 2020;138:105495
Zhang Z, et al. An Overview of the Free Energy Principle and Related Research. Neural Comput. 2024;36(5):713-757
Schwartenbeck P, et al. Optimal inference with suboptimal models: Addiction and active Bayesian inference. Med Hypotheses. 2015;84(2):109-117
Haarsma J, et al. Precision weighting of cortical unsigned prediction error signals benefits learning and is impaired in psychosis. Mol Psychiatry. 2021;26(9):5320-5333
Li L, et al. Formalizing Lacanian psychoanalysis through the free energy principle. Front Psychol. 2025;16:1574650
Smith R. Active Inference and its Application to Empirical Data. Yale Medicine Psychiatry. 2025
Blum K, et al. Introducing Precision Addiction Management of Reward Deficiency Syndrome. Front Psychiatry. 2018;9:548
Albarracin M. Addiction as a Disruption of the Markov Blanket. Glob J Addict Rehabil Med. 2024;7(4):555717
Takahashi H, et al. Dopamine D1 Receptors and Nonlinear Probability Weighting in Risky Choice. J Neurosci. 2010;30(49):16567-16572
Friston K, et al. Computational psychiatry: from synapses to sentience. Mol Psychiatry. 2023;28(1):256-268
Chakroun K, et al. Dopamine regulates decision thresholds in human subthalamic nucleus. Nat Commun. 2023;14(1):5506
Taylor S, et al. Active learning impairments in substance use disorders. Drug Alcohol Depend. 2023;252:110932
Groman S, et al. Model-free and model-based influences in addiction-related behaviors. Biol Psychiatry. 2019;85(11):936-945
Doyle W, et al. Environmental Uncertainty and Substance Use Disorders. Perspect Behav Sci. 2023;46(1):191-213
Robinson A, et al. Model-based and model-free mechanisms in methamphetamine use disorder. Addiction. 2024;119(3):456-466
Chen H, et al. Model-Based and Model-Free Control Predicts Alcohol Use Trajectories. Biol Psychiatry. 2021;89(10):980-990
FitzGerald T, et al. Dopamine, reward learning, and active inference. Front Neurosci. 2015;9:136
Friston K, et al. Dopamine, Affordance and Active Inference. PLoS Comput Biol. 2012;8(1):e1002327
The anatomy of choice: active inference and agency
Embodying addiction: A predictive processing account
Optimal inference with suboptimal models: Addiction and active Bayesian inference
Pathologies of precision: A Bayesian account of goals, habits, and episodic foresight in addiction
Imprecise Action Selection in Substance Use Disorder: Evidence for Active Learning Impairments When Solving the Explore-Exploit Dilemma
Active inference and learning

Anomaly Detection in R

Thu, 25 Sep 2025 00:00:00 +0000

Anomaly Detection in R, Approaches, Techniques, and Tools

Anomaly detection, also referred to as outlier detection, is an aspect of data analysis that involves identifying patterns, observations, or behaviors that deviate significantly from the norm. As datasets grow larger and more complex, the need for robust and efficient anomaly detection methods has become more important. Such anomalies may point to uncommun events/situations such as health problems, unusual bilogical activities, or novel discoveries in scientific research. The increasing availability of large and complex datasets has amplified the importance of robust anomaly detection methods, and the R sofware provides a ecosystem of packages to address this need. In this post, we explore anomaly detection approaches including into various techniques available in R, and highlight the packages with their implementation. Whether you’re a beginner or an experienced data scientist, this comprehensive guide aims to equip you with the knowledge and tools to tackle anomaly detection effectively.

Understanding Anomaly Detection

Anomalies are rare occurrences that deviate significantly from typical patterns in data, often indicating critical insights or potential issues. Identifying anomalies accurately is challenging, particularly with high-dimensional or unstructured datasets. Misclassifying anomalies as normal, or vice versa, can have serious implications, especially in healthcare settings. Point anomalies represent individual data points that are markedly different from others. In a medical context, this could be a sudden spike in a patient’s heart rate or an unexpected lab test result outside normal ranges. Contextual anomalies are observations that are unusual in a specific context but may appear normal otherwise. For instance, a slight rise in blood pressure might be normal for a healthy adult but not for a child. Collective anomalies involve groups of observations that collectively deviate from the norm. Examples in healthcare include patterns of symptoms in patients indicating an outbreak of a rare disease or a cluster of abnormal readings in ICU monitors suggesting equipment malfunction or systemic health deterioration. Detecting such anomalies requires appropriate approaches depending on the data and anomaly type. The R sofware provides an ecosystem for anomaly detection, offering statistical, machine learning, and deep learning methods. These tools empower healthcare professionals to uncover anomalies, enabling timely interventions and improved patient outcomes.

Statistical Approaches

Statistical methods are some of the earliest techniques used for anomaly detection. These methods assume that “normal” data follows a specific distribution, and deviations from this distribution are considered anomalies. They are particularly useful for small or well-structured datasets. Z-Score Analysis is one of the simplest statistical techniques. Here, anomalies are identified by measuring how far a data point deviates from the mean, scaled by the standard deviation. R’s stats package provides the necessary tools for this approach. The outliers package extends this functionality by offering functions like grubbs.test for detecting single outliers and dixon.test for multiple outliers. For multivariate data, Mahalanobis distance is a popular metric used to detect outliers. The ‘mvoutlier’ package in R provides functions to calculate Mahalanobis distances and identify multivariate outliers.

Clustering-Based and Density-Based Approaches

Clustering-based methods leverage the notion that normal data points form dense clusters, whereas anomalies are far from any cluster. These methods are well-suited for multi-dimensional data. k-Means Clustering, implemented in the stats package, partitions data into clusters. Observations far from the centroids of these clusters can be flagged as anomalies. A more robust variant, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), is implemented in the dbscan package. DBSCAN does not require a predefined number of clusters and can identify anomalies as points that do not belong to any cluster. The clValid package provides tools for assessing clustering validity, which can be useful for determining whether a clustering-based anomaly detection approach is suitable for your data. Density-based methods assess the density of data points and flag points in sparse regions as anomalies. Local Outlier Factor (LOF) is a popular density-based technique available in the DMwR package. LOF compares the density of a point to its neighbors, assigning higher anomaly scores to points in sparsely populated regions. For datasets with continuous features, Gaussian Mixture Models (GMM), available in the mclust package, model the data as a mixture of Gaussian distributions. Observations with low probabilities under the fitted model are identified as anomalies.

Machine Learning Approaches

Machine learning techniques can be broadly classified into supervised, semi-supervised, and unsupervised approaches. Supervised Learning requires labeled data with known anomalies. These labels are used to train a classification model to distinguish anomalies from normal observations. R packages like caret and mlr3 provide a suite of tools for building and evaluating supervised anomaly detection models. In practice, labeled data is often unavailable, making unsupervised learning a more common choice for anomaly detection. Unsupervised methods aim to identify patterns in the data without explicit labels. For example, Isolation Forests are a tree-based ensemble method designed for anomaly detection. It works on the principle that anomalies are ‘few and different’ and thus easier to isolate than normal points. The isotree package in R offers an efficient implementation of isolation forests, allowing for the detection of anomalies by isolating data points through random partitioning. Another popular unsupervised method is Principal Component Analysis (PCA), which reduces the dimensionality of the data to identify variations. The FactoMineR and princomp packages in R enable PCA-based anomaly detection. Observations with high reconstruction errors after dimensionality reduction are flagged as anomalies. Another popular machine learning approach is the Local Outlier Factor (LOF) algorithm. LOF compares the local density of a point to the local densities of its neighbors, identifying points that have a substantially lower density than their neighbors as potential outliers. The ‘Rlof’ package in R implements this algorithm. Ensemble methods, which combine multiple models or algorithms, can often provide more robust and accurate anomaly detection than single methods, basing on the idea is that by aggregating the results of multiple detectors, we can reduce the impact of individual model biases and improve overall detection accuracy. The ‘anomalyDetection’ package in R implements several ensemble methods for anomaly detection. It combines multiple base detectors using various fusion strategies to produce a final anomaly score.

Time Series Anomaly Detection

Time series data anomaly detection is different to other data types because of its temporal nature and potential seasonality. Unlike static data, time series data is ordered chronologically, meaning that each data point is not independent but is influenced by the values preceding it. This introduces complexities such as trends, periodic fluctuations, and temporal dependencies that must be accounted for during anomaly detection. Additionally, anomalies in time series data can manifest in various forms, such as point anomalies (a single observation is abnormal), contextual anomalies (an observation is only abnormal in a specific context, such as time of day or season), or collective anomalies (a sequence of data points is collectively abnormal). Several R packages are specifically designed to handle anomaly detection in time series data. The ‘anomalize’ package is a powerful tool for detecting anomalies in time series data. It implements a tidy workflow, making it easy to use within the ‘tidyverse’ ecosystem. The package offers multiple methods for decomposing time series and detecting anomalies, including STL decomposition and IQR (Interquartile Range) methods. Another notable package is ‘tsoutliers’, which provides functions for detecting and handling outliers in time series data. It implements several methods, including innovative outlier, additive outlier, and level shift detection. For more complex scenarios, time-series analysis can be applied using the forecast package. The forecast package in R is a robust tool for handling time series data. Methods like ARIMA and Exponential Smoothing can be used to predict future values and compare them to observed values, flagging deviations as anomalies. This is particularly useful for identifying anomalies in sequential data.

Challenges in Anomaly Detection

Anomaly detection challenging because of some issues that complicate the process, such as the imbalance of data, as anomalies are by definition, rare occurrences. This imbalance can bias detection methods, which may become overly focused on common patterns, leading to missed anomalies or an excessive number of false positives. More specifically for high-dimensional data, as the dimensionality increases, the data becomes sparser, making it harder to distinguish between normal and anomalous patterns effectively. This “curse of dimensionality” demands advanced techniques that can reduce complexity without losing critical information. Within this context, scalability is also a significant concern, especially when dealing with large datasets or streaming data. However, some methods, like Isolation Forests, are particularly well-suited for large-scale anomaly detection due to their computational efficiency.

Context-dependent anomalies add another layer of difficulty. In these cases, anomalies are only detectable within specific contexts, requiring domain knowledge to identify and interpret effectively. For instance, a medical condition might appear normal in one demographic but highly anomalous in another. Dynamic or streaming data adds real-time constraints to anomaly detection. Unlike batch-processed datasets, streaming data demands immediate analysis and response, often requiring algorithms that balance speed and accuracy without relying on static assumptions about the data.

Another consideration is the interpretability of the results. While some methods (like statistical approaches) provide clear interpretations, others (like deep learning methods) may act as black boxes. In many real-world applications, it’s crucial not just to detect anomalies but also to understand why they were flagged as anomalous. Finally, it is noteworthy that, while numerous methods and tools are available for anomaly detection, it’s important to note that there’s no one-size-fits-all solution. The choice of method depends heavily on the nature of the data, the type of anomalies expected, and the specific requirements of the application. Overcoming these challenges necessitates a holistic approach. Careful preprocessing to address imbalances, feature engineering to manage high dimensionality, and selecting algorithms tailored to the data’s nature are essential. With these strategies, anomaly detection systems can become robust and adaptable to complex real-world scenarios.

Future Directions and Conclusion

Choosing the best anomaly detection approach depends on the nature of the data and the problem we are solving. For small datasets with well-understood distributions, statistical methods are sufficient. But for large or high-dimensional datasets, machine learning techniques, including isolation forests and autoencoders, offer superior performance. Combining multiple methods often yields the best results. Ensemble techniques, such as stacking or voting, can improve robustness by leveraging the strengths of different approaches. As we look to the future, several exciting developments are shaping the field of anomaly detection. One area of active research is the application of deep learning techniques to anomaly detection. Variational Autoencoders (VAEs) and Generative models show promise in learning complex data distributions and identifying anomalies. Another emerging trend is the integration of domain knowledge into anomaly detection systems. This approach, sometimes called “guided” or “informed” anomaly detection, aims to leverage expert knowledge to improve detection accuracy and interpretability. The increasing ampunt of streaming data and the need for real-time anomaly detection is also driving innovation in this field. Techniques that can efficiently process and analyze data in real-time, updating their models on-the-fly, are becoming increasingly important.

Anomaly detection field has applications such as healthcare where identifying rare but critical patterns can save lives. The ecosystem of tools in R offers healthcare analysts powerful methods for detecting anomalies, whether through statistical models or advanced machine learning algorithms. As health datasets grow in size and complexity, with streams of data from electronic health records, wearable devices, and imaging systems, the demand for real-time anomaly detection becomes more apparent. Innovations such as scalable algorithms and the integration of domain-specific knowledge are shaping the future of this field. In a medical context, detecting anomalies could mean identifying early signs of sepsis from subtle deviations in vital signs, flagging irregularities in medication adherence from patient monitoring, or spotting unusual spikes in emergency room admissions that may indicate the onset of an epidemic. By leveraging these cutting-edge tools, healthcare professionals can detect critical events before they escalate, enabling timely interventions.

References

Outliers Package - Tools for detecting and testing outliers in numerical datasets using methods such as Grubbs’ and Dixon’s tests.
mvoutlier Package - Provides robust methods for detecting multivariate outliers, leveraging Mahalanobis distance and robust covariance matrices.
isotree Package - Implements isolation forests and extended isolation forests for efficient anomaly detection in high-dimensional data.
Rlof Package - Offers the Local Outlier Factor (LOF) algorithm for identifying density-based anomalies in datasets.
e1071 Package - Includes machine learning algorithms, such as support vector machines (SVM), that can be used for anomaly detection.
Twitter AnomalyDetection - An open-source package from Twitter for detecting anomalies in time-series data using automated thresholding and seasonal decomposition.
anomalize Package - A tidyverse-compatible tool for detecting and visualizing anomalies in time-series data.
tsoutliers Package - Tools for detecting and adjusting outliers in time-series data, particularly useful for ARIMA models.
forecast Package - Comprehensive tools for analyzing and forecasting time-series data, including anomaly detection with ARIMA and ETS models.
anomalyDetection Package - Focused on unsupervised anomaly detection for time-series data using statistical and algorithmic approaches.
randomForest Package - Implements the Random Forest algorithm, which can be adapted for anomaly detection through proximity measures.
ROSE Package - Provides resampling techniques to handle imbalanced datasets, enhancing anomaly detection.
ROCR Package - A flexible tool for visualizing the performance of binary classifiers, including those used in anomaly detection.
anomaly Package - Detects anomalies in univariate data, with a focus on changepoint detection techniques.
netstat Package - Analyzes network statistics, including detecting anomalies in network traffic data.
CRAN mclust Package - Implements model-based clustering and classification methods that can assist in anomaly detection.
CRAN DMwR Package - Provides tools for data mining with R, including strategies for detecting outliers and handling imbalanced data.
CRAN FactoMineR Package - Offers multivariate exploratory data analysis techniques, aiding in identifying anomalies in complex datasets.