What's behind that smile? (2022)


Psychological researchers are delving deeper into what our facial expressions and bodily movements reveal about our emotions

By Kirsten Weir Date created: October 1, 2020 11 min read

Vol. 51, No. 7
Print version: page 46

  • Perception and Attention
  • Emotions

What's behind that smile? (1)

You smile when you’re happy, frown when you’re sad, scowl when you’re angry—at least, some of the time. After decades of research on emotions, evidence suggests that these signals are a far-from-foolproof way to infer someone’s state of mind.

“Facial movements are not a language that you can read the way you read words on a page,” says Lisa Feldman Barrett, PhD, a distinguished professor of psychology at Northeastern University who studies emotion in the brain.

That realization has implications for researchers in affective computing who are working to design artificial intelligence (AI) systems that can identify a user’s emotional state. These emotion-recognition technologies could have a wide range of possible uses, such as cars that can detect a distracted driver, robots that can help autistic children learn social skills and technology to help law enforcement predict the behavior of suspected criminals.

“Human emotion is involved in almost every interaction people have—even with computers. However, computers have not traditionally been programmed with emotional intelligence,” says Rosalind Picard, ScD, founder and director of the Affective Computing Research Group at the MIT Media Lab. “Human-computer interactions can be less frustrating when technology is equipped to show respect for people’s feelings.”

Barrett and Picard, who are speakers at APA’s 2020 Technology, Mind & Society showcase (see sidebar below), agree that by asking the right questions, and harnessing new technologies, researchers can take the science of inferring our emotions to a new level—one that might even revolutionize technology as we know it.

Basic emotions?

In the 1960s, clinical psychologist Paul Ekman, PhD, now professor emeritus at the University of California, San Francisco, launched the modern study of emotional expression with his work on the six “basic” emotions: anger, disgust, fear, happiness, sadness and surprise.

Using posed photographs of stereotypical expressions of those emotion categories, Ekman and other scientists showed that people in cultures around the world could identify these emotions with a high degree of agreement. A generation later, a meta-analysis of more than 100 studies confirmed these emotions were universally recognized at levels better than chance (Elfenbein, H.A., & Ambady, N., Psychological Bulletin, Vol. 128, No. 2, 2002).

That body of work is strong evidence that these emotional expressions are widespread, if not universal, says Jessica Tracy, PhD, a professor of psychology who studies emotion at the University of British Columbia. “They’ve been shown across many cultures, blind people produce these expressions, and in many cases there’s cross-species evidence for these expressions,” she says.

But Barrett suggests that the way in which we express emotions is—unsurprisingly—more complex than recognizing a posed smile or frown. In a review of more than 1,000 published studies of facial movements and emotions, she and her colleagues found that the way people express and perceive these six emotion categories varies considerably across cultures, across situations and even within a single person (Psychological Science in the Public Interest [PSPI], Vol. 20, No. 1, 2019).

Across groups including babies, congenitally blind people and people from remote, small-scale, non-Western cultures, the evidence for consistent basic expressions of emotion was weak, Barrett concluded. “We found that in urban, large-scale cultures, people do scowl in anger more than expected by chance, about 30% of the time. [That] means that 70% of the time they are moving their facial muscles in other meaningful ways when angry,” she says. “They might scowl when they’re concentrating, when they’re confused, when you tell them a bad joke or when they have gas.”

And the findings are comparable for the other emotions, Barrett says. “The findings are similar for the other five emotion categories we examined, and to the best of our knowledge, hold for any emotion category ever studied.”

That’s not to say facial expressions are meaningless. But their predictive value is limited, Barrett says. “Scowling in anger more often than chance gets you a publication in a respected journal, but it doesn’t necessarily mean scowling is a useful piece of knowledge to infer someone is angry and make decisions about that person’s life. Would you want someone to decide whether or not you get a job, whether or not you’re charged with a crime or whether your child has access to educational opportunities based on a system that is accurate 30% of the time?”

(Video) Podcast: In Machines We Trust - What's Behind a Smile

Much of the research on emotional expression focuses on how people perceive others’ expressions, often using photos of posed stereotypes of happiness or anger, for example. Those studies can reveal something about an observer’s perception of emotion, but the method doesn’t reveal anything about the emotions of the person who is smiling or scowling, Barrett suggests.

“If you want to understand how people move their faces when they’re expressing emotion, shouldn’t you be measuring how they actually move their faces, rather than measuring inferences people make about posed expressions?” she asks. “This debate about whether there are universal expressions has gone on for over a century. And I think that as scientists, the first thing we need to be asking ourselves is whether we’re actually asking the right questions.”

Head-to-toe expressions

If a human can’t reliably infer what someone is feeling based on their facial movements alone, then a computer algorithm programmed to detect those same facial movements won’t be up to the task either, points out Stacy Marsella, PhD, a professor and director of the Centre for Social, Cognitive and Affective Neuroscience at the University of Glasgow and a co-author of Barrett’s 2019 PSPI paper.

“But it isn’t very surprising to most researchers in affective technologies that a computer would need more than facial actions to make such inferences,” he says. Facial movements are one of many elements he and other researchers are using to design systems that can express and infer affective states, which include emotions as well as other emotion-related experiences, such as boredom and frustration.

“The goal of affective computing isn’t just to map facial expressions,” agrees Alan Cowen, PhD, a postdoctoral researcher in computational neuroscience at the University of California, Berkeley. “It’s much more nuanced than that, and the field has moved past the idea that you can just match facial expressions to the basic six emotions.”

Even within one emotion type, there is a multitude of variation, he points out. Consider amusement, joy and pleasant surprise: “There isn’t just one category of happiness,” he says.

And while a photo of a smile might make you think “happy,” facial movements aren’t static. A smile isn’t just a fixed state, Marsella notes. “You would also want to know the dynamics: How fast is it happening? How long is it held? Is it symmetric? What other facial actions are involved? These elements, along with the larger context in which the behavior occurs, tell you a lot.”

In their PSPI paper, he and Barrett champion a more comprehensive study of emotion, which would sample individuals more deeply, with higher-dimension measurements, across many different situations. Other researchers, including Cowen and Tracy, have also called for a more “comprehensive atlas” of human emotions, which would look beyond the face to consider other clues to a person’s feelings, such as the sounds they’re making and the way they’re moving their bodies (Cowen, A., et al., Psychological Science in the Public Interest, Vol. 20, No. 1, 2019). Tracy, for instance, has found that head movements play a significant role in communicating emotions, especially anger. In a review with her student Zachary Witkower, she described how various body movements can telegraph pride, joy, sadness, shame, disgust and other feelings (Emotion Review, Vol. 11, No. 2, 2019). Picture a person hanging their head in shame, or puffing up their chest in pride. “A lot of emotions are expressed and recognized through those behaviors,” she says.

Context matters

Affective computing also considers the context of the situation, and what’s happening in a person’s environment, says Picard, who coined the term “affective computing.” With a background in computer science and engineering, she sees the field of emotion research a little differently than psychologists might.

“For a long time, emotion researchers have argued about whether emotions are just clean and ‘basic,’ like happy and sad and angry, or whether they are continually changing dimensions,” she says.

Years ago, when she first began reading up on emotion theory, she was reminded of the debate in physics about the duality of light. “For a long time, physicists argued about whether light was a particle or a wave. They finally decided it’s both, and it just depends on what you’re trying to explain,” she says. “Emotion researchers could benefit from thinking about emotion this way, and allowing it to be context dependent.”

That fluid way of thinking about emotion might be one factor in Picard’s success in creating sensors and systems that can perceive affective information. She was a co-founder of the startup Affectiva, which develops emotion-sensing AI for use in gaming, robotics, education, marketing and health care. While she is no longer affiliated with Affectiva, the company has developed sensors to detect driver drowsiness and distraction and a computer game that helps kids with special needs develop social-emotional skills by imitating the expressions of characters they see on screen.

(Video) Nat King Cole "Smile" (1954)

After leaving Affectiva, Picard co-founded Empatica with the goal of creating sensors and algorithms to improve health. One such device is an FDA-cleared smartwatch that monitors for seizures, a device born out of work to understand emotional responses in the amygdala. In the process of stimulating the brain to study emotion, researchers found physiological patterns that could be used by AI to detect and alert to grand mal seizures.

In other work, she’s partnered with psychiatrists to study devices to monitor moods in people being treated for depression. “With their informed consent, we’re taking information from wearables and their smartphones and using AI to see if we can tell whether they’re getting better or worse,” she says.

In an initial study of 12 people, she and colleagues found that these machine-learning techniques could predict depression ratings measured by the Hamilton Depression Rating Scale (HDRS) (Ghandeharioun, A., et al., Seventh International Conference on Affective Computing and Intelligent Interaction, 2017). Now she’s testing the technology in a larger group of participants.

The project is a good example of the importance of context, she adds. “We know the person has a depression diagnosis. We have context from their phones: how much they’re getting out and about, how often they’re calling or texting others, how much they’re moving, the local weather, their sleep rhythm and physiology,” she says. “This isn’t just about counting smiles or reading one bio signal. But with all of this information together, we’re getting highly accurate tracking of their scores on the HDRS.”

New tech for emotion research

These projects provide good reason to be hopeful about the future of emotion-recognition technology. But so far, Picard cautions, the systems are reliable only in limited contexts. These technologies may be able to make useful inferences about a person’s affective state, but only in well-defined situations where the computer has a rich sample of all the possible inputs. In new contexts and completely unexpected situations, with people the computer has never “met” before, “then it is likely to run into all kinds of confusion and misunderstanding,” she says.

There are also important ethical issues to consider as the technology moves forward, such as collecting affective data without a person’s direct consent. “I’m in favor of regulations. We want to find the right balance of benefiting people while protecting their privacy and protecting them from being exploited,” she says.

With the right ethical safeguards, emotion-sensing AI could provide benefits, particularly as a research tool that can help plumb the fundamental questions about human emotion, such as whether emotions are truly universal, and how they are represented in the brain.

“If [emotion researchers] want to make real advances that will improve people’s mental health, physical health and behavior change challenges, there are ways to work together to advance both the theory and the practice,” she says. “There’s a lot to learn by teaming up with engineers and working together across the field to address some of the big theoretical questions in emotion.”

With enough data points, Barrett agrees that such systems have promise, especially in research domains. “There is a whole universe of questions [about emotional expression] that haven’t been asked properly. Sometimes in science, progress isn’t about answering questions. It’s about learning to ask better ones.”

(Video) What's Behind That Smile

A virtual exploration of the human-tech connection

This year, APA’s annual Technology, Mind & Society conference is going virtual. TMS 2020 will present curated content to showcase the critical role psychology plays in the design, use, adoption and impact of technology and the artificial intelligence (AI) that powers it. Speakers include:

  • Jeremy Bailenson, PhD, Stanford University. Bailenson studies the psychology of virtual and augmented reality. His recent research focuses on how virtual experiences can transform education, environmental conservation, empathy and health.
  • Lisa Feldman Barrett, PhD, Northeastern University. Barrett’s research focuses on how the human brain, in continual conversation with the human body and the world, creates emotional events. She uses experiential, behavioral, psychophysiological and brain-imaging techniques to study what emotions are and how they work.
  • Maja Matarić, PhD, University of Southern California. Matarić is a pioneer in the field of socially assistive robotics. Her research has developed robot-assisted therapies for children with autism spectrum disorder, survivors of stroke and brain injury, and individuals with dementia.
  • Rosalind Picard, ScD, Massachusetts Institute of Technology. Picard leads research on AI/machine-learning algorithms, analytics and sensors that advance the basic scientific understanding of emotion, stress and arousal. She is the co-founder of Empatica Inc., which creates software and sensors to advance understanding of human behavior and improve health.

The virtual showcase will be held Nov. 12–13, 2020. To learn more, visit TMS.

(Video) Alan Walker x salem ilese - Fake A Smile (Lyrics)

Further reading

Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements
Barrett, L.F., et al., Psychological Science in the Public Interest, 2019

Mapping the Passions: Toward a High-Dimensional Taxonomy of Emotional Experience and Expression
Cowen, A., et al., Psychological Science in the Public Interest, 2019

The Theory of Constructed Emotion: An Active Inference Account of Interoception and Categorization
Barrett, L.F., Social Cognitive and Affective Neuroscience, 2017

Emotion Research by the People, for the People
Picard, R.W., Emotion Review, 2010

Related and recent

(Video) What's Behind a Smile-Offset

Weir, K. (2020, October). What's behind that smile? Monitor on Psychology, 51(7). https://www.apa.org/monitor/2020/10/behind-smile

`; if (document.getElementById('mainwrap') != null) { document.getElementById('mainwrap').insertAdjacentHTML('beforeend', popup); } }


What is behind a smile? ›

When our smiling muscles contract, they fire a signal back to the brain, stimulating our reward system, and further increasing our level of happy hormones, or endorphins. In short, when our brain feels happy, we smile; when we smile, our brain feels happier. Fake it till you make it!

What's behind the power of your smile? ›

Smiling increases mood-enhancing hormones while decreasing stress-enhancing hormones, including cortisol, and adrenaline. It also reduces overall blood pressure. And because you typically smile when you're happy, the muscles used trigger your brain to produce more endorphins—the chemical that relieves pain and stress.

What do smiles hide? ›

However, there's often a plethora of feelings and emotion that accompany the action. One may appear happy, cheerful and content, but what lies behind that smiling face is a whole other story completely. You may be hiding pain, sorry, heartache, or fear.

Who is Duchenne smile Duchenne? ›

The Duchenne smile is named after Guillaume Duchenne, a French anatomist who studied many different expressions of emotion, focusing on the smile of pure enjoyment. He identified thefacial movements that make this smile different from other types of smiles.

What is a smile quote? ›

"Beauty is power; a smile is its sword." "The greatest self is a peaceful smile, that always sees the world smiling back. "A smile is happiness you'll find right under your nose." "Colors are the smiles of nature."

Does fake smiling make you happy? ›

According to a recent study published in the journal Experimental Psychology, researchers found smiling — even a fake smile — can have a positive impact on mood. Essentially, triggering certain facial muscles by smiling can “trick” your brain into thinking you're happy.

What happens if you smile for too long? ›

The researchers found that smiling frequently may actually make people feel worse if they're sort of faking it — grinning even though they feel down. When people force themselves to smile because they hope to feel better or they do it just to hide their negative emotions, this strategy may backfire.

Why do I smile when I'm sad? ›

Typically, smiling depression occurs when individuals who are experiencing depression mask their symptoms. They hide behind a smile to convince other people that they are happy.

What happens when someone smiles at you? ›

Several scientific studies have noticed that when someone smiles at us, the part of our brain that controls feelings of reward is activated. Yes, you've read it right: if we receive a smile, we may feel like we have been given a beautiful prize.

What is one thing that makes you smile? ›

Hearing a bit of good news, getting a thank-you from someone and doing something to make someone else laugh or smile also feature in the list. Eating comfort food, jumping into a bed with freshly washed sheets, finding a ten pound note in an old pair of trousers and seeing a funny video online completed the top ten.

Why do men like a woman's smile? ›

Some men may see a woman smiling at them as a sign that they are available, which ignites attraction, according to Science of People. So maybe you are smiling to appear more attractive, trying to make a connection, flirt, or simply because that blazer really is on sale.

What is a person who hides their pain behind a smile? ›

Typically, smiling depression occurs when individuals who are experiencing depression mask their symptoms. They hide behind a smile to convince other people that they are happy.

What happens when someone smiles at you? ›

Several scientific studies have noticed that when someone smiles at us, the part of our brain that controls feelings of reward is activated. Yes, you've read it right: if we receive a smile, we may feel like we have been given a beautiful prize.


1. What's behind that SMILE? — Cody Wanner interview - Josh Miles
(Josh Miles)
2. No matter what, keep your smile ~ Sad songs for broken hearts that will make you cry (sad music mix)
(Sad Space)
3. What EMOTION Do You Hide Behind Your Smile?
(What's It For?)
4. What's Behind That Smile
(Ludious Productions)
5. Kirk Franklin - Smile Music Video featuring Steve Harvey
(The Official Steve Harvey)
6. •Karasuno react to hinata "behind that smile"•

You might also like

Latest Posts

Article information

Author: Trent Wehner

Last Updated: 11/16/2022

Views: 6304

Rating: 4.6 / 5 (56 voted)

Reviews: 95% of readers found this page helpful

Author information

Name: Trent Wehner

Birthday: 1993-03-14

Address: 872 Kevin Squares, New Codyville, AK 01785-0416

Phone: +18698800304764

Job: Senior Farming Developer

Hobby: Paintball, Calligraphy, Hunting, Flying disc, Lapidary, Rafting, Inline skating

Introduction: My name is Trent Wehner, I am a talented, brainy, zealous, light, funny, gleaming, attractive person who loves writing and wants to share my knowledge and understanding with you.