Search Results

The story of the evolution of episodic memory is also the story of three complementary brain systems, which evolved together to make it easier for the brain to learn. The first system (of the three) to evolve is what I'll call "the Immediate Action Network" (IAN). In humans, this system is characterized by the outer shell of the brain, the neocortex or cerebrum. This layer of the brain includes all the processing areas for input (i.e. the senses), for output (i.e. motor and speech areas), and some decision-making processes that connect the two (e.g. premotor). The evolved consolidation into a central nervous system allowed for speedy and coordinated processing in systems that are widely distributed across the body. Healthy brains with IAN systems can process information about the outside world, make decisions about what to do next, and quickly execute those decisions. There are things, however, that an IAN system cannot do. It is tuned to the organism's immediate environment, but it cannot imagine or remember other environments. It cannot make plans for the future. And it has a very hard time learning from the past. To be clear, most mammals (among others) do start their lives primarily with an IAN-only system, and it learns very well. Humans' first five years are primarily guided by the IAN system, while our other two systems are still developing/being educated. But there is a tension within each of these IANs, when it comes to plasticity. Plasticity is very necessary in the beginning of life, because that is how the newborn brain maps itself to match its environment. But once the major lessons are learned, then plasticity can become a problem, because it can erase earlier important lessons. So the brain starts extremely plastic, learning lessons like "who is my mother?" and "what do my predators look like?" or even "what is the difference between a shadow and an object?" Those kinds of lessons need to be learned early, and not be over-written. So throughout childhood and adulthood, the early lessons becomes 'baked in' to a neuronal structure that is increasingly rigid and resistant to change. But life-long learning is not only helpful, but it's essential for some kinds of lifestyles, like migratory ones. Landmarks need to be learned. Food sources change. Novel threats arise. So some brains developed two additional systems, to help the IAN learn. Together, they comprise the episodic memory system. The best known of these two systems is the hippocampal system, and its relationship to the IAN is relatively straightforward. The IAN processes information about the outside world, including its own behavior towards the world. And then the IAN sends information about that processing to the hippocampal complex, which turns that information into a storable long-term 'movie' which can be learned from later. So, the hippocampal system evolved to turn the IAN's processing into a movie, a movie which reflects what was happening in the outside world, and what was happening with the body, and it records that movie for the brain to learn from later. If the organism has an interaction with a new predator, for example, that one interaction may not make an important change in the plasticity of the neocortex, so that it learns efficiently about this new danger. So the hippocampus plays back the memory of that initial interaction, over and over again, and teaches the IAN from repeated exposure. When the brain sleeps, it uses those memory movies 1. to help consolidate memories, so they become saliency-rich but data-lean, and 2. to help educate the IAN system. The latter is where a lot of our IAN's learning happens, since our IANs are no longer very plastic to the outside world. In REM sleep in particular, memories are played back, again and again, backward and forward, to help the IAN system learn. Landmarks, for example, become more familiar through hippocampal repetition. Skills, like playing a new song on a guitar, are learned through hippocampal repetition. In fact, almost all adult IAN education happens through the hippocampus. Every interaction we have, every life-lesson, almost entirely makes a mark on us, only because of our memories. We know this because patients who have lost their hippocampi in adulthood can essentially no longer learn. Some very rudimentary physical skills can be taught, but the patient doesn't even remember that they've learned the skills. Once the hippocampus is destroyed, the IAN becomes unmoored from time, essentially unable to learn. So we can see that the IAN processes the current moment, and sends only its (modular) final conclusions to the hippocampus, to be combined into one comprehensive piece of information, an experience movie about the self-in-the-world, which is the new episodic memory. That memory movie can later be replayed, to help the IAN learn, during sleep and down-time. But there seems to be a missing element here. Doesn't a memory movie need an observer, someone or something to recall the memory, to watch it and to derive lessons from it? And indeed, that is what the third system evolved to do, to help close the loop of episodic memory, so that experience movies could not only be created and played back, but also could be deciphered for meaning, and to drive lessons for the IAN to learn. That third system was originally called the Default Mode Network (or DMN), but I'll call it by it its currently accepted name, the Core Network. The Core Network evolved to make sense of the memory movie as it's being made, and as it's being recalled. It has a much greater role in our lives now, but I think this is how it began. From its evolutionary beginning, the Core Network was the home of imagination. This began in a very humble way, as the ability to match disparate noisy inputs. For example, if a rat is navigating novel terrain, it may see a visually distinct landmark, and add the sight of that landmark into a new memory, to be matched later, when navigating the same territory. The recall of that landmark will be pattern-matched with the brand new memory of the new navigation, to ascertain whether this is indeed the same landmark. Now imagine that the rat approaches the same landmark, but from a very different angle. The pattern-match is no longer exact (or even close), because the different angle creates different patterns on the rat's visual cortex. Or maybe the rat's approaching the landmark under different lighting conditions. So imagination began as a way of transforming one view into another, so that the two views could be recognized as the same landmark. Imagination helps memory from being too rigid in recognizing familiar stimuli. The Core Network evolved to specialize in these two functions: 1. to be able to derive meaning or lessons from experiences, and 2. to be able to imagine how things could possibly be, when there's not enough immediate evidence to fill in the blanks. This second skill of the Core Network includes imagining what goes on in other people's heads, or visualizing how to pack bags in the trunk of my car. These three systems: 1. Immediate Action, 2. Memory Movie recording + storage, and 3. the Interpreter of Experience / Imaginer of possibilities created an ongoing loop with which the brain could continue to learn, long after the IAN had lost its initial plasticity. The same loop, however, also reifies the perceived self, Me. For the sake of future lesson-learning, new memory-movies are simulations of the self-in-the-world. From the Core Network's perspective, the memory IS the real experience, as opposed to being a construct of the brain. And the self-in-memory is understood by the Core Network to be the real Me.

In our model, Subjective Experience is a brain-wide activation. The Immediate Action Network (please see the 3 brain systems post) of the neocortex receives sensory input, and motivates behavioral output. It then sends signals of its conclusions to the hippocampus, to be encoded in the next moment's new episodic memory. Those signals are organized in the entorhinal cortex, filled-in using pattern-matching abilities of hippocampal field CA3, and encoded within a theta wave output in field CA3, before traveling through the subiculum and entorhinal cortex, back out to the rest of the brain. It is the activation of the rest of the brain, by the hippocampal theta signal, that gives rise to Experience. The perception of smell is the activation of the olfactory bulb, by the hippocampal signal, just as the perception of vision is the activation of the visual cortices, by the hippocampal signal. The parts of the brain that evolved to make sense of certain sensation stimuli from the outside world are the same parts that are employed in creating perceptual Experience. Video cameras make fairly useful technological analogies for the episodic memory system (even if memories can't be as accurate as video). When light rays enter the lens of a camera, they activate pixels in the camera's CCD or other imaging device. Those individual pixels send their signals to the camera's image processor, where they are combined into one signal, which can be recorded, and then played back later. Upon playback, the recorded master signal is interpreted by the image processor, activating a different array of pixels, those on a video monitor. When those pixels are activated to correspond to the input pixels, then what appears on the monitor is a representation of the original scene. In-pixels are activated, leading to encoding and information storage. Information recall (i.e. playback) leads to decoding and representations upon out-pixels. Those representations playing back on the monitor create a simulation of the original scene, and can help educate the viewer about what was happening. Likewise, sensory stimulus activates the brain's version of in-pixels. Not just the cones and rods of the retina, but also all the brain regions activated by vision (plus the input regions of each other sense). Each in-pixel of the brain sends its code to the hippocampus to be combined into a master signal and recorded, for future playback. Upon memory recall, the master memory signal is reconstructed by the hippocampus, and sent back to all the contributing nodes of the Immediate Action network. It is the activation of those nodes/pixels, that creates Experience. The theta wave serves as a timing signal, so that all receiving brain parts can agree on when is the 'now' of experience. Immediate experience is also like a video camera. Even when a camera is not recording, but is otherwise active, it sends a live video feed to the viewfinder, to activate those pixels into a representation of what's happening right now. That viewfinder allows the camera operator to respond to the input, panning and tilting to keep the image in view, for example. Likewise, Subjective Experience is the live feedback of the hippocampal signal. Even if no part of this experience ever gets recalled, ever again, the in-pixels are still sending information to the hippocampus, which is replying with the theta wave memory, causing those same pixels to act as out-pixels, creating Qualia, creating Experience. The wide-spread activation of out-pixels creates a communication challenge for the Core Network (please see the three brain regions post), because it means the Core Network has to 'observe pixels' all over the brain. And indeed, if we look at a wiring map of the Core Network, we see that it has afferents from all over the brain, coming together to be processed in a few local hubs and parts of the prefrontal cortex. The Core Network observes the out-pixel activations caused by the hippocampal signal's impact on the IAN, allowing it to interpret meaning from the immediate moment, and to imagine elements beyond the current input of the senses. I think it's worthwhile mentioning another technological analogy to this process. Microphones and earphones, for example are input and output devices, respectively, that can sometimes take each other's place. If you plug a microphone into an earphone jack, for example, you can hear some of the audio output that you would expect from an earphone. Likewise, an earphone plugged into a microphone jack can actually transmit some audio into the machine, the way a microphone would. These input devices can also act as parallel output devices, creating a representation of the original input.

(from our 2016 published paper) We want to be careful with how we use the term ‘subjective experience,’ because it can have various slightly different meanings. The brain could be said to be experiencing even when it is in deep sleep because neurons are active and processing. However, NSE is a particular kind of experiencing that only takes place during waking and rapid eye movement (REM) sleep. In this article, we use NSE specifically to mean a neurologically intact person’s typical internal unified three-dimensional (3D) virtual-reality movie-like experience of being a mind and body in the world. It is virtual-reality-like in that there is full-surround immersion into the experience, and movie-like in that there is continuity from moment to moment. However, unlike an audio/visual virtual-reality movie, NSE also includes body sensations, motor feedback, emotions, and even qualities of the mind, like thought and the products of our imagination. The mind subjectively feels like a different kind of thing than the body and the world, but that is an assumption we will avoid. We will instead argue in this article that all elements of perception, including the perception of mind, arise from one unified piece of neural information that is output from the HC. The three main constituent elements of experience (mind, body, and world) do not need to be equally represented in all activities. In many nonphysical activities, like reading or watching television, the experience of body tends to fade. In deeply immersive mental tasks, like memory recall, social rehearsal, or daydreaming, the outside world fades from awareness, and in flow states or tasks that require strong focus on interacting with the world, the awareness of mind may fade instead. We will argue that the HC organizes, binds and broadcasts the informational output, which gives rise to NSE, but NSE’s contents are due to structures upstream of the HC, based on the needs of the moment. There are two additional elements that add context to NSE: emotions and what William James called ‘fringe’ senses, which are usually subtle and are shaped by our previous experiences. These are contextual feelings, like familiarity, certainty, and novelty. Along with emotions, fringe senses flavor our other perceptions. We also need to distinguish between the neural information that activates experiencing and the experiencing itself. In this article, the neural-coded information that activates the experiencing will be called the episodic memory engram, whereas we will use NSE to mean the event of experiencing. Thus, the theta wave HC output is the episodic memory engram and that engram’s activation of relevant target brain structures is NSE.

Qualia is a fancy word for the perceived qualities of the self-in-the-world, within Experience. Qualia include all the perceived aspects of all the sensory inputs, as well as the perceived aspects of our imagination, like the sound of our own inner thoughts, or the feeling of being in whatever mood I'm in. Qualia are first and foremost mnemonics. They help the brain remember how things felt, smelled, tasted, sounded like, or appeared. My favorite example is a rat that encounters a novel berry. This rat does not yet have any prior experience/memory with that berry, so it does not yet have a concept for it, nor any heuristics as to whether it's safe. The rat's brain goes into exploratory mode, in which the rat begins to sniff the berry and take small bites, in order to sample the other sensory inputs to the berry. During this time, the rat's hippocampus is at full throttle, encoding these perceptual details for the future. Let's say that this novel berry happens to be poisonous; not enough to kill the rat, but enough to make it sick. New perceptual details arise, of nausea and disorder. The rat's hippocampus links these new nauseous perceptions to the smell and taste perceptions from the exploratory phase, so that in the future, similar berries will trigger the nauseous memories, and drive the rat to avoid those berries. Let's say that the next time the same rat encounters the same kind of berry, that berry is now in a different part of its ripeness, and so looks different than the original poison berry. Once again, the rat approaches with caution, sniffing and tasting, its hippocampus on full, to explore this novel berry. But now, through the various combined qualia (the sight, smell, and taste), the rat pattern-matches this new berry to the nauseating memory of the old berry, and thereafter spits out the new berry and avoids it. The rat uses the memory of these two berry encounters to help create a concept of that berry, so it can be recognized, and immediately avoided, in the future. We can think of qualia, in some ways as a memory language. In the same way that color, texture, tone, volume, movement, etc. are all part of our cinematic language, and allow filmmakers to tell their stories, likewise those attributes are part of our memory language, and allows our brain to tell its future self stories about the past. Qualia are how the brain maps external sensation upon its internal simulation. Every brain is its own island, self-organizing without direct contact with any other brain. Therefore, each brain has its own unique qualia language, even though they are built upon the same rough architecture. Color, for example, is a function that the brain is born with, but then self-organizes that function to its environment, sculpting its internal simulation to map onto the external reality.

My mind appears to be the essence of Me. It seems to be the decision-maker, the source of voluntary action, the part that feels and thinks and indeed, the part that counts. But of course, these appearances are deceiving. The mind is clearly a process of the brain, and is decisively affected by pathology, substance, or fatigue. Dr. Benjamin Libet was the first to show that our seeming 'volition' is an appearance, rather than a choice. The brain acts, without any need for a mind to guide it. Those brain actions often lead to the perception of the will to do something (although in the case of sleep-driving, for example, or other distracted action, it's possible that awareness may never form, as consequence of those brain actions). The fact that the brain actions happen before the 'perceived will' shows that the actions were the causal force, not the perceived will. The perception of will is rather an after-the-fact construct, created for the sake of memory, so that episodic memories can contain in them reminders of 'what I chose', so that future lessons can incorporate my mistakes and successes within them. In this way, we can recognize the perception of 'will' as another form of 'qualia' (as discussed elsewhere in the FAQ). It is a mnemonic of the memory system, so that when I recall the memory, I can remember making choices within it. The same is true for the rest of what we call 'mind'. Mind is an appearance, rather than an engine of behavior. All those aspects of mind, like thinking in words, or imagining in images, are forms of qualia. Even more subtle qualities of mind, like the feeling that my thoughts and my will belong to me, those feelings are qualia (i.e. mnemonic reminders in the form of feelings) to help structure memories for accurate recall. However, minds, especially human minds, seem especially good at working not just for the far future, but also for the near-future, at figuring out what behavior should come next, for example. This seems to imply a causal link between Mind and Behavior, and seems to cast doubt that Mind is just a part of the episodic memory loop. However, we already know that most of what we call 'Mind' functions are actually performed by the Core Network (please see the three systems post), using the hippocampus as its staging area. The hippocampus has the most efficient memory buffers in the brain, and those buffers, along with the hippocampus' experience-generating powers, make it a great organ for staging mental simulations. It is a virtual workspace for the Core Network to play its simulations upon. As discussed in the three-systems post, the hippocampus evolved to simulate reality for the sake of creating a memory movie. The Core Network, which first evolved to make sense of memories, then learned to use the hippocampus' memory and reality-simulating powers to buffer and manipulate concepts in the virtual workspace. This virtual workspace is what we call mind. This is why we humans think in words. Because the hippocampus already evolved to buffer, record, and recall words, and the Core Network just piggy-backed upon those abilities when using the hippocampal experience-generator to create simulations. The Core Network itself does not think in words, of course, but it enlists the language-generating parts of the Immediate Action Network in populating the hippocampal buffer with words. The hippocampal buffer allows the words and concepts within it to be manipulated and rearranged, in order to find new solutions. The same is true of the relationship between imagination and imaging. The hippocampus evolved to buffer, record, and recall views, in order to pattern-match landmarks, etc. And, as discussed in the three systems post, imagination was always a necessary part of the Core Network's pattern-matching noisy incomplete stimuli within memory, by mentally transforming one memory-pattern into another. These functions of the Core Network and Hippocampus together evolved from the ability to pattern-match incomplete stimuli into the ability to pre-visualize things that have never existed before. Likewise, Theory of Mind is an extremely important mental simulation, in which the Core Network uses its mental models of other people, piggy-backing on the hippocampus, which evolved to remember a lot of details about other people, including tone of voice and past behavior. Once again, the Core Network leverages parts of the memory and simulation powers that the hippocampus evolved to have, in order to simulate other minds, within the appearance of our own minds. So the Mind is an appearance, a virtual workspace, in which the Core Network plays out its mental simulations using the hippocampus' experience-generating powers. It is information, buffered within the hippocampus for the sake of assisting abstract problem-solving. As information, it does not have direct causal power on behavior, but of course, it can influence behavior. The Core Network is just one of many networks tugging at the reins of behavior, and it actually has very little power in affecting immediate behavior. But it remains the best system that the brain has, for simulating future outcomes, and for extrapolating beyond the immediate environment. Our experience of Mind is always part of the hippocampal memory loop, so we experience Mind as part of the overall Subjective Experience. Mind is remembered along with the experience of the outside world, and can be an important part of memory recall. It always involves the same three systems that give rise to episodic memory, and thus it is well-represented within memory/perception (whereas data from certain parts of the brain, like the brainstem or cerebellum, are barely represented at all within Experience). I think this is part of why the Mind seems to be such a compelling appearance, and why we are so easily fooled into thinking that Mind = the real Me.

From the published paper: A male epileptic patient from Connecticut, commonly known as H.M., was the most famous patient with adult-onset bilateral complete hippocampal damage (AOCBHD). Most of his HC was removed surgically to save him from the overwhelming epilepsy that was focused there. Patients like H.M. are unable to form detailed or coherent intrinsic simulations, like mental navigation, future projection, vivid imagination, and social rehearsal. These patients are afflicted with anterograde amnesia, the inability to form new episodic memories, and retrograde amnesia, the inability to replay previous episodic memories. In the model we are proposing, such patients cannot have NSE, which may sound implausible. Specialists who work with AOCBHD patients describe the patients’ responsiveness to the outside world, their personality, sense of (short-lived) continuity, an ability to respond to queries about one’s inside state, and even some limited ability to learn procedural skills and semantic information. This is what we mean from the third-person view when we describe someone else as ‘conscious,’ and indeed, H.M. was that, when awake. These are all part of the usual working definition of consciousness, but none of them are part of how we are defining NSE. We are neither saying that H.M. lacked subjectivity nor that he lacked experience. What he was missing was the specific phenomenon that the phrase ‘Subjective Experience’ refers to (among those of us with working hippocampi), which is cortical activation by a specific type of neural information. According to our theory, NSE is caused by inter-network information (the engram), which helps to expedite processing and which may be stored for future recall. It subjectively represents processes like responsiveness, personality, etc., but it should not be confused with them. All of the processes that are represented in the hippocampal simulation are only news reports of the actual processes in the EN and DMN, and patients like H.M. still have access to those original processes (see section 3.3: Memory Generation as Simulation on boundary extension). Unlike the Cartesian model (Figure 2), we believe that NSE is not necessary to engage with the outside world. All immediate interaction with the world is handled by pre-memory EN departments. We could not survive otherwise because the ~500 ms it takes to revise a new memory to account for unpredictable activity would leave us a half-second behind reality. We could never drive, for example, if we were always a half-second behind changing conditions. Figure 2: ‘Driving Mind’ as Insight into H.M.’s Processes The ability of the EN to act independent of the HC representation is probably not uncommon, but we tend to notice it only in extreme cases, like ‘driving mind.’ Most of us who drive have probably experienced the phenomenon of driving someplace (usually on a familiar route) and then upon arrival, realizing that we have no memory, whatsoever, of the drive. In fact, it seems as if I was not involved in the drive at all. The explanation that is usually given for ‘driving mind’ is that the driving is done ‘automatically,’ but we see that as a highly problematic, dualistic explanation. The body and brain are checking the mirrors, braking and accelerating, changing lanes. All the functions of sensation, evaluation, decision, and action are clearly being employed in the drive or the car would crash. The driving functions are performed by the same EN departments as usual, so the behavior is only ‘automatic’ if it is missing oversight from sort of ‘I,’ some ‘self’ that is distinct from the brain. This would imply that the brain is somehow an unconscious vehicle, driven by a non-brain conscious self, which puts us in dualism (i.e., Descartes’ notion that ‘mind’ is a nonmaterial intelligent essence), and back into the Cartesian Theater. But what is startling about ‘driving mind’ is precisely the lack of memory of the drive. So we think it much more likely that the exact same brain departments as usual are conducting the drive; however, there is just no memory formed of it. That lack of memory can be explained by the fact that the HC serves very different purposes for the EN versus the DMN. As we have seen in section 3: The Hippocampal Complex as Experience Simulator, the DMN uses the ‘Holodeck’ features of the HC as a virtual workspace in which to manipulate representations. The DMN runs predictive simulations, imagining a conversation or exploring scenarios before trying them on the world. It examines, and tries to make sense of, previous memories. It is even tasked with daydreaming/fantasy, in which we get to simulate our wish fulfillment. In the case of ‘driving mind,’ we see it likely that the EN is performing all the driving tasks, as it always does. However, the reports from the sensory and motor cortices do not make it into memory because the HC is already fully occupied with a DMN simulation, like daydream, memory recall, or social rehearsal. Because episodic memory only contains what was processed in the HC, the DMN simulation is all that is remembered, leaving the brain with no memory of the drive, and because NSE is due to the broadcast of episodic memory, I had no NSE of the drive, although my EN was fully engaged during it. In the case of driving mind, my ‘inner H.M.’ is doing the driving, and the resultant NSE is only of the DMN/hippocampal simulation. Was H.M. a Zombie? Philosophers have invented a rhetorical being, which they call a zombie, that behaves and self-reports exactly like a conscious human but who somehow lacks any kind of inner awareness or experience. As we are proposing that AOCBHD patients lack the phenomenon that those of us with working hippocampi know as Subjective Experience, it may sound like we are claiming that H.M. and similar patients fit that zombie category. However, as we have seen in section 3.3 (Memory Generation as Simulation on boundary extension), AOCBHD patients are aware of the photo they see and can report based on that awareness. The neural information that they report, however, is from an EN pre-memory buffer that those of us with working hippocampi have no direct access to. Despite the fact that we have the same functional pre-memory buffers as AOCBHD patients, those of us with working hippocampi (according to our theory) only experience the HC output, and that output obscures the pre-memory awareness. In section 7.3: The Causal Power of NSE, we more fully explore the possible reasons why pre-memory processes are left out of awareness. Based on his work with another AOCBHD patient D., Antonio Damasio wrote that the patient D. does not have ‘an elaborate sense of self ... at a point in individual historical time, richly aware of the lived past and of the anticipated future’ (p. 16). Endel Tulving essentially wrote the same about AOCBHD patient N.N. What is missing in patients D. and N.N. is the context and expanded sense of self that is provided by memory. They are NOT zombies, because they have inner lives, albeit transient ones. We hypothesize that their EN experience is probably substantially different than the NSE of those of us with working HCs. This is where we think H.M. lived: he sensed and interacted with the outside world, could rely upon his short-term visual and auditory buffers, and could report body states and a few details about his self-identity. What he could not do was benefit from the HC’s newscast. He could not use the gestalt from a moment ago to expedite his processing. He could not place himself within a broader sense of what came before and what is likely to happen next. Unfortunately, it is impossible to guess, from our perspective, precisely what H.M.’s, N.N.’s, or D.’s non-memory experience is like, as our own normal pre-memory experience is not available to our subjective view. It is like asking ourselves to describe the unremembered act of driving during ‘driving mind’; there is nothing to remember and thus no way to report. Nor can we expect patients with AOCBHD to reliably know how their perceptions have changed post-morbidity because, missing their HC, they can no longer evoke the vivid memories of previous experience to compare to their current way of seeing the world. Some evidence from the literature does exist, however, and suggests that perception does change for such patients. Graham et al. quoted one patient with limited hippocampal damage, who could still remember fairly well, but whose perception lacked qualities that the HC usually provides. She could form and remember snapshot images of her environment but still had a very difficult time navigating through the world because “whichever angle I look, everything looks the same” (p. 832). The patient continued: “I would prefer not to call my experiences ‘memory problems,’ they are not. This is a total misrepresentation of the damage I have. What I experience are ‘orientational problems.’” Her perception of the world is missing some of the context of the allocentric environmental information from the PHC and hippocampus, elements that those of us with fully working hippocampi take for granted. NSE reflects a simplified, but much more contextualized, representation of events than the pre-memory reports in the EN.