Sensory Systems Work Together

"I smell a rat." "That experience left a bad taste in my mouth." Everyday expressions like these hint at crosstalk between the senses. Examples of synesthesia—when people hear colors, taste sounds, or feel numbers—tells us that in some cases the crosstalk can be intense.

Even though we're not always aware of it, the crosstalk between our senses is constant, and it significantly informs our experiences. The examples are nearly endless. If you're served cold soup or dry bread in a restaurant, you may send it back. A loud sound grabs your attention, sending your eyes searching for movement that might help you find its source. At a holiday meal, sights, sounds, smells, flavors, and feelings fuse into a singular experience.

family at holiday meal

Cross-talk happens in the brain

illustration showing different routes in the brain

Multimodal association areas process information from multiple sensory areas in the brain.

Additional Resources

Listening with your eyes from Scientific American (subscription required)

Each sensory system sends signals to the brain. It's up to the brain to interpret these signals and come up with a response. To do this, the brain often combines information from multiple sensory systems—a process called sensory integration.

Sensory signals first enter the brain through the thalamus (except for olfaction, which bypasses this area). From there, it is routed to a sense-specific area of the cortex. Cortical areas, in turn, send signals to the brain's "association areas," which combine information from multiple senses.

Brain imaging tools like functional MRI help scientists see which parts of the brain are most active during certain tasks. These tools can show us what cross-talk looks like in the brain. In one brain imaging experiment, flashes of light were shown alone or in combination with sounds. Flash and sound together caused more activation in the visual cortex than just the flash alone.

In another experiment, silent videos of people talking activated parts of the brain that process sound.

Senses and Memory

Sensory systems interact with memory in a few different ways. The brain can pull in information from memory as it decides how to respond to incoming sensory information.

Sensory information is also stored as part of a memory. When we remember a specific place or scene or picture something in our minds, sensory processing areas in the brain become activated. Sensory inputs can also cause us to remember. Tasting a food, smelling an odor, or seeing a photo can bring back a flood of memories connected to it.

Related Content

Visit Types of Memory to learn morea bout inputs from sensory systems and their relationship to memory.

Visit Basic Neuroscienceto learn more about brain imaging technologies and how signals travel in the brain.

Odd Experiments

Researchers have long been curious about how sensory systems interact with one another and how they influence behavior. To learn more, people have set up all kinds of strange experiments, including some where they try to trick the brain. A few examples are described below.

Sight and sound

The McGurk effect was discovered in the 1970s, when researchers combined a video of someone saying "gah" with audio of someone saying "bah." Strangely, the sound subjects "heard" was "dah."

The Bad Lip Reading meme combines video of someone talking with silly audio that appears to match. The effect can be ridiculous and hilarious. The converse to this effect is that it is easier to understand what someone is saying in a noisy room when you can also see their mouth movements.

woman taking order in noisy bar

Color and mood

In the early 1800s, the German poet philosopher Goethe wrote about connections between color and mood. People commonly refer to color temperature: reds and oranges are warm and stimulating; blues and greens are cool and calming.

Various studies have found that color can influence people's eating experiences (green eggs and ham, anyone?). And a 2005 study found a red advantage in combat sports: male competitors who wore red were more likely to win than those who wore blue—presumably because red is often a "dominance" signal in nature. But the red advantage doesn't apply everywhere. Because the meanings of different colors are also shaped by culture and experiences, it's hard to find effects that might be universal.

olympic boxers, red vs. blue

Taste, smell, and emotional

People feel negative emotions more strongly when they have a bad taste in their mouth. A small 2011 study found that sipping a bitter drink made subjects judge other people more harshly for their moral transgressions, while drinking something sweet or neutral made subjects kinder in their judgment.

In a similar study, researchers showed that "fart sprays" increased feelings of disgust in subjects who were presented with a questionable moral scenario.

woman with judgemental expression


Elliot, AJ & Maier, MA (2014). Color psychology: effects of perceiving color on phychological functioning in humans. Annual Review of Psychology, 65, 95-120. doi: 10.1146/annurev-psych-010213-115035

Eskine, KJ, Kacinik, NA & Prinz, JJ (2017). A bad taste in the mouth: gustatory disgust influences moral judgment. Psychological Science, 22:3, 295-299. doi: 10.1177/0956797611398497

Howes, D. (2006). Cross-talk between the senses. Senses & Society, 1: 3, 381-390.

Kandel, ER, Schwartz, JH, Jessell, TM, Siegelbaum, SA & Hudspeth, AJ. (2013). Principles of Neural Science, fifth edition. McGraw Hill Medical.

Kayser, C. (2007). Listening with your eyes. Scientific American Mind, April/May 2007, 24-29.