Sensory Systems Work Together

"I smell a rat." "That experience left a bad taste in my mouth." Everyday expressions like these hint at crosstalk between the senses. Examples of synesthesia—when people hear colors, taste sounds, or feel numbers—tells us that in some cases the crosstalk can be intense.

Even though we're not always aware of it, the crosstalk between our senses is constant, and it significantly informs our experiences. The examples are nearly endless. If you're served cold soup or dry bread in a restaurant, you may send it back. A loud sound grabs your attention, sending your eyes searching for movement that might help you find its source. At a holiday meal, sights, sounds, smells, flavors, and feelings fuse into a singular experience.

Each sensory system sends signals to the brain. It's up to the brain to interpret these signals and come up with a response. To do this, the brain often combines information from multiple sensory systems—a process called sensory integration.

Sensory signals first enter the brain through the thalamus (except for olfaction, which bypasses this area). From there, it is routed to a sense-specific area of the cortex. Cortical areas, in turn, send signals to the brain's "association areas," which combine information from multiple senses.

Brain imaging tools like functional MRI help scientists see which parts of the brain are most active during certain tasks. These tools can show us what cross-talk looks like in the brain. In one brain imaging experiment, flashes of light were shown alone or in combination with sounds. Flash and sound together caused more activation in the visual cortex than just the flash alone.

In another experiment, silent videos of people talking activated parts of the brain that process sound.

Sensory systems interact with memory in a few different ways. The brain can pull in information from memory as it decides how to respond to incoming sensory information.

Sensory information is also stored as part of a memory. When we remember a specific place or scene or picture something in our minds, sensory processing areas in the brain become activated. Sensory inputs can also cause us to remember. Tasting a food, smelling an odor, or seeing a photo can bring back a flood of memories connected to it.

Researchers have long been curious about how sensory systems interact with one another and how they influence behavior. To learn more, people have set up all kinds of strange experiments, including some where they try to trick the brain. A few examples are described below.

Sight and sound

The McGurk effect was discovered in the 1970s, when researchers combined a video of someone saying "gah" with audio of someone saying "bah." Strangely, the sound subjects "heard" was "dah."

The Bad Lip Reading meme combines video of someone talking with silly audio that appears to match. The effect can be ridiculous and hilarious. The converse to this effect is that it is easier to understand what someone is saying in a noisy room when you can also see their mouth movements.

Color and mood

In the early 1800s, the German poet philosopher Goethe wrote about connections between color and mood. People commonly refer to color temperature: reds and oranges are warm and stimulating; blues and greens are cool and calming.

Various studies have found that color can influence people's eating experiences (green eggs and ham, anyone?). And a 2005 study found a red advantage in combat sports: male competitors who wore red were more likely to win than those who wore blue—presumably because red is often a "dominance" signal in nature. But the red advantage doesn't apply everywhere. Because the meanings of different colors are also shaped by culture and experiences, it's hard to find effects that might be universal.

Taste, smell, and emotional

People feel negative emotions more strongly when they have a bad taste in their mouth. A small 2011 study found that sipping a bitter drink made subjects judge other people more harshly for their moral transgressions, while drinking something sweet or neutral made subjects kinder in their judgment.

In a similar study, researchers showed that "fart sprays" increased feelings of disgust in subjects who were presented with a questionable moral scenario.