Intrinsic meaning, perception, and matching

How experience relates to the environment in integrated information theory

William G. P. Mayner1, Bjørn Erik Juel1,2,3, Giulio Tononi1

1 Wisconsin Institute for Sleep and Consciousness, University of Wisconsin–Madison, Madison, WI, USA · 2 Brain Signaling Group, University of Oslo, Oslo, Norway · 3 Vestre Viken Kognitiv Nevrovitenskap, Vestre Viken Health Trust, Drammen, Norway

Read the full paper →

Graphical abstract: a stimulus triggers a portion of a complex's intrinsic Φ-structure (the percept); over a sequence of stimuli this yields perceptual differentiation and matching to the environment.

Perception as structured interpretation: the triggering of intrinsic meaning, not information processing

Cast a glance at the scene outside the window: in a blink of the eye, you see the forest with its intricate canopy of trees. How does this come about? A standard account is that a stimulus from the environment impinges on the retina, conveying information to the brain; the information is processed through a hierarchy of sensory areas, aided by top-down signals that try to predict, fill in, or disambiguate noisy bottom-up data; and finally, the meaning of the information is decoded, with the ultimate goal of guiding behavior. The very idea of processing suggests that the information is in the stimulus, ready to be decoded, and that meaning is in an activity pattern or “code” resulting from that processing. Somewhere along this processing chain, some of this information happens to become conscious (“conscious processing”).

IIT offers a different account: IIT starts from an experience—whether dreamt, imagined, or triggered by a stimulus—which it characterizes as a cause–effect structure, or Φ-structure. The Φ-structure is composed by distinctions and relations that define the feeling of the experience in a way that is fully intrinsic, without any reference to anything outside the complex. From the intrinsic perspective of an experiencing subject, the feeling of the experience is also its intrinsic meaning: what any content of the experience feels like—a distant sound, a sense of nausea—is also what that content means for the subject. Thus, IIT views external stimuli as triggers of intrinsic meaning, rather than as sources of information to be processed.

What, then, is the relationship between experiences and the stimuli triggering them? And how does the intrinsic meaning of experiences reflect features of the environment? Here, we extend IIT’s mathematical framework to address these questions.

Illustration of the framework with toy models of visual object perception

Two simple models, B1 and B2, detect different features. Left: Each is a hierarchy of stochastic units with lateral and top-down feedback: level-1 units copy the input, level-2 configuration detectors fire when their feature appears, and a level-3 invariant unit fires whenever any detector is active. Right: The dynamics of each system. B1 detects ‘segments’; B2 detects ‘centered-odd’ patterns. Note the activation of the invariant unit M in response to the tuned pattern.

Architecture and dynamics of model systems B1 and B2.
The substrate models and their dynamics. (A) The model architecture, shared by B1 and B2. Each consists of 13 stochastic binary units (labeled circles): yellow = ‘ON’, white = ‘OFF’, arranged in hierarchical levels in rough analogy to the visual system. (B) The units at each level and their activation functions. Bottom-up (BU) connections are straight arrows; lateral (Lat) connections curved arrows; the self-connection a looping arrow; top-down (TD) connections from the segment unit curved arrows pointing left. Numbers give the connection strength in the sigmoidal module σ of each activation function. B1 and B2 differ only in the bottom-up configurations their L2 detectors are selective for: B1 — 01110, 01100, 00110; B2 — 01110, 00100. (C) Example dynamics: the average response of each unit across 5000 simulations of an arbitrary stimulus sequence presented to B1 (each stimulus clamped for τ = 5 timesteps). Four stimuli contained ‘segments’ (highlighted). As activity percolates upward, level-1 units reproduce the input, level-2 detectors activate for a segment in their receptive field, and the level-3 segment unit M activates for a segment anywhere on the interface. (D) Same as (C) for B2, which responds to ‘centered-odd’ stimuli rather than ‘segments’.

The triggering coefficient measures to what extent the state of a given subset of the system was caused by a stimulus

Let m be the state of a subset of the system, let x be the stimulus, and let \( p=\Pr(m\mid x) \) and \( q=\Pr(m) \). We define the triggering coefficient as:

\[ t(x,m)=\log(p/q)\,/\,\log(1/q)\quad\in[0,1] \]

This expresses the extent to which the stimulus x caused the subset to be in state m. It is 0 when m never occurs in response to the stimulus, and 1 when it always occurs in response. Here we show its values for the subsets of L1, L2, and L3 of B1 when the system is presented with a segment pattern. Note that the active detector unit and the invariant unit have high triggering values, reflecting their selectivity for this stimulus; this indicates the stimulus is causally responsible for their activation.

Connectedness and triggering coefficients across levels of B1.
Connectedness and triggering coefficients. (A) When the system is connected to the sensory interface, the stimulus x = 01110011 percolates through it, triggering the state ABCDEFGH = 01110011, IJKL = 1000, M = 1. (B–D) Values of the connectedness c(x,s) and triggering coefficient t(x,s) for subsets of level 3 (B), level 2 (C), and level 1 (D). Black: the unit for that row is not included; white: included but not active; yellow: included and active. For space, the 7,921 subsets spanning levels are omitted.

A perceptual structure is the portion of a Φ-structure triggered by a stimulus

The Φ-structure is composed of distinctions and relations. We call the subset of those components supported by a particular subset m of the system the φ-fold of m. Its perception value is the sum of the φ-values of those components, weighted by t(m), its triggering coefficient: i.e. the fraction of the φ attributable to the stimulus. The φ-folds together form the perceptual structure: the portion of the Φ-structure that was triggered by the stimulus. The sum of perception values is the perceptual richness:

\[ \mathcal{P}(x)=\sum_c t(x,c)\,\varphi_c \]
Perceptual structures and perceptual richness.
Perception: perceptual structures and perceptual richness. To the extent that intrinsic meanings are triggered by extrinsic stimuli, they can be considered percepts. (A) The Φ-structure with the system connected to the environment (‘awake’). Inset: stimulus and response; level-1 units relay the stimulus to the levels above, which detect a segment (aBCDe = 01110). In this state the system specifies a Φ-structure with lower structure integrated information Φ than in the all-‘OFF’ (dreaming) state. (B) Heatmaps of, respectively, each distinction φ-fold’s Φ-fold value, its triggering coefficient, and its perception value. (C) The perceptual structure triggered by the stimulus: mechanisms (brown, bottom) colored by triggering coefficient t(x,m); causes (red), effects (green), and their relations colored by perception value p(x,d(m)) = t(x,d(m))·φd(m). Grayscale emphasizes that the perceptual structure is not a Φ-structure in its own right, but the fraction of the Φ-structure triggered by the stimulus.

The same stimulus can trigger different perceptual structures in different systems: every perception is an interpretation

The same stimulus can elicit different response states in different systems (left). However, even when the activity pattern is exactly the same, the perceptual structure can be different (right). B1 interprets the stimulus as a segment (and not a centered odd) while B2 interprets it as a centered odd (and not a segment). It is the causal power of the system’s units that matters for perception, not activity patterns as such.

The same stimulus triggers different perceptual structures in B1 and B2.
Perception as interpretation. The same stimulus can trigger different perceptual structures in different systems — even when the activity pattern in both is the same. (A) B1 and B2 receive 00100000, which contains the centered-odd pattern ‘00100’ that B2 detects; the perceptual structure triggered in B1 has lower perceptual richness than in B2. (B) The stimulus contains ‘01110’ — both a segment and a centered odd — so the top-level invariant detector activates in each system and the triggered activity pattern is identical. Yet the perceptual structures, and therefore (by IIT) the intrinsic meanings, differ. The only difference is the counterfactual behavior — the causal powers — of the L2 configuration units. Each system interprets the stimulus according to its causal powers.

Perception is intrinsic—it need not involve reference or representation

Here, 3 different environments produce the same stimulus (a ‘segment’ state). A true 3-segment, a spurious coincidence of 2-segments that appears as a 3-segment, and a 3-segment appearing purely by chance all trigger the same response state and perceptual structure. This illustrates a simple but important point: perception and meaning, being intrinsic, do not necessarily involve reference or representation of causal features of the environment.

The same percept arises across three environments.
Perception and representation. (A) B1 is exposed to the segment environment E1 and detects the 3-segment pattern. (B) In the ‘2-segment’ environment E1b, the 3-segment generator is replaced by a second 2-segment generator; occasionally two 2-segments overlap to form an ‘apparent 3-segment.’ B1 then perceives the same 3-segment, with the same meaning — yet by construction there is no causal process in E1b for that percept to represent: the apparent 3-segments are spurious coincidences. (C) In the pure-noise environment E3, a 3-segment can occur by chance; again B1 responds identically and the percept has the same meaning, though there are no causal processes whatsoever to represent.

Matching measures the degree to which a system has internalized an environment’s causal regularities

Perceptual differentiation is the total perception value of the union of the perceptual structures across a stimulus sequence. It captures the richness and diversity of the perceptual structures triggered—i.e., how meaningful an environment is to the complex:

\[ \mathcal{D}(X)=\sum_c \max_i \mathcal{P}(x_i,c) \]

Matching is then defined as the maximum expected difference between perceptual differentiation for environmental stimuli versus random stimuli:

\[ \mathcal{M}=\max_l \mathbb{E}\,[\mathcal{D}_{\mathrm{env}}-\mathcal{D}_{\mathrm{noise}}] \]

It is high when the environment triggers more diverse and rich intrinsic meanings than would be expected by chance. Here, B1, which detects ‘segment’ stimuli, has a higher matching value when perceiving stimuli from the ‘segment’ environment, where those stimuli occur frequently (highlighted timesteps); likewise, B2 matches better to the ‘centered-odd’ environment. Their connectivity has internalized aspects of the causal processes in their respective environments (here, by design). Well-adapted systems whose intrinsic connectivity was molded by evolution, development, and learning can be expected to allocate a large set of their intrinsic meanings to reflect causal regularities of the environment that are relevant for fitness.

Matching in different environments for B1 and B2.
Matching in different environments. (A) B1 and B2 are exposed to the segment environment E1 (top) and to uniform noise (bottom); n = 32 trials of k = 4 stimuli were sampled from each. (B) Perceptual richness 𝒫 triggered in B1 by each environmental (top) and random (bottom) stimulus; samples containing any segment are highlighted in teal. (C) Same for B2; segment stimuli tend to trigger higher richness in B1 than B2, since B1 detects any segment while B2 detects only 3-segments. (D–F) Same as (A–C) for the centered-odd environment E2; stimuli containing a centered odd are highlighted in orange, and tend to trigger greater richness in B2 than B1. (G) Matching, estimated as the maximum mean difference between perceptual differentiation for environmental versus random sequences (max over contiguous subsequences of length l ≤ k, mean over trials). As expected, B1 matches E1 and B2 matches E2 — each system has, by construction, “internalized” aspects of the stimulus statistics reflecting causal processes in its matching environment. (Two-way ANOVA system × environment interaction: F(1,124) = 20.23, p < 0.001. Error bars: 95% CI.)

Read the full paper →

Acknowledgments

We thank Larissa Albantakis and William Marshall for their contributions during the early stages of this project; Leonardo S. Barbosa, Melanie Boly, Tom Bugnon, Keiko Fujii, Andrew Haun, Armand Mensen, and Shuntaro Sasai for helpful discussions; and Larissa Albantakis, Chiara Cirelli, Francesco Ellia, Graham Findlay, Matteo Grasso, and Alireza Zaeemzadeh for valuable comments on the manuscript. B.E.J. was supported in part by the Research Council of Norway (FRIPRO grant no. 335828). G.T. acknowledges support from Templeton World Charity Foundation (nos. TWCF0216 and TWCF0526). The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of Templeton World Charity Foundation.

Data and code availability

All code to reproduce the analyses and figures is available at github.com/wmayner/matching. The precomputed data are archived on Zenodo at doi.org/10.5281/zenodo.20972063. The substrate_modeler package used to construct the model systems is available at github.com/bjorneju/substrate_modeler.