Do Masks Mask Your Emotions? AI Emotion Recognition in the Era of Face Masks

Caroline Chan

she/her | age 13 | Toronto, ON

2022 Canada Wide Science Fair Gold Medal Junior Division | 2022 Toronto Science Fair Gold Medal Junior Division | Legends of Learning Game Design Challenge: Grand Prize

Edited by Greg Brooks

VIEW PDF

Humans read the emotions of each other to socialize. Machines can be trained to recognize human emotions from facial expressions through artificial intelligence (AI). However, mask-wearing, common due to COVID-19, may affect human and AI’s ability to read emotions. To test this hypothesis, I gathered images of happy or sad faces and added simulated masks to them. I showed the images to 5 volunteers and asked them if each of the images was a happy or a sad face. On average, they correctly recognized 91.6% of the masked images. Next, I trained an AI model with unmasked images and tested it with a different set of unmasked images. The accuracy of the model was 76%. However, when the same model was tested with masked faces, the accuracy dropped to 52%. My study showed that wearing masks makes reading emotions more challenging for humans and AI.

INTRODUCTION

I became interested in artificial intelligence (AI) after learning about it in a STEM summer camp for girls. I wanted to know more about how AI can improve people’s lives. AI allows machines to learn from experience by processing lots of data and recognizing patterns in the data. I wanted to find out if AI can be taught to read human emotions from facial expressions as humans do.

Experiencing different emotions is a part of being human. Humans need to correctly read emotions of each other to socialize appropriately. When you see a person with a sad facial expression, it is important to show concern and empathy for them. AI is increasingly being used in our lives and can also be trained to recognize human emotions. For example, AI can be used to identify patients experiencing pain and help children with autism to better interpret the emotions of others (Fulmer, 2021).

However, since the beginning of the COVID-19 pandemic, students have been required to wear face masks in schools. There are concerns that face masks may affect their ability to recognize emotions and interact with each other. I hypothesized that humans would perform better than AI at recognizing the emotions of faces with and without a mask, as humans have been reading the emotions of each other since they were born. During COVID-19, we have had two years to train ourselves to read emotions with a mask on. However, I think the masks would make it more challenging for AI and humans to read emotions correctly. The findings of my project will provide a better understanding of how masks affect the ability of humans and AI to read emotions.

METHODS & MATERIALS

I started my project by collecting images of human faces showing happy or sad emotions on the internet. I cropped them to a square shape, with the face in the center of the image. I found 175 images total, with 87 showing happiness and 88 showing sadness. Then I edited the images by adding a blue mask-like shape to cover the bottom half of the face to simulate a mask we regularly wear. Two sets of experiments were performed.

1) Human Experiment
I recruited five children between the ages of 8 to 14 to test how well they could read emotions. The experiment was conducted in-person or virtually by zoom, depending on the participant's location. When performed in person, participants outside of my household wore masks during the experiment. To begin, I showed them 50 masked faces (25 sad and 25 happy) in random order. I asked each participant whether they thought the face was happy or sad after each picture, and they had 30 seconds to answer. I then tallied how many they got right and how many they got wrong. Then, I repeated the test with another batch of 50 images of unmasked faces (25 sad and 25 happy) in random order.

2) AI Experiment
For the AI portion of the test, I used a machine-learning app called Lobe (Lobe, 2020). I performed the complete AI testing using both training and testing images with each website. In the end, Lobe had the highest accuracy. I took the 175 images and divided them into two groups: training (N = 125) and testing (N = 50). I first trained the model using the 125 unmasked training images. Then, I tested the model using the 50 unmasked test images and tallied how many the AI model got correct. Next, I tried the model again using the same 50 test images but with masks. For my last experiment, I trained another AI model using the same 125 training images, but this time with masks on. Then, I tested the new model using the same 50 masked and unmasked test images.

RESULTS

After gathering my data, I calculated the percentages of images that the volunteers and AI model correctly recognized as happy or sad faces. Here are my results:

1) Human Experiment
After calculating the results of my experiment, I found that all the participants correctly recognized the emotion of 100% of the unmasked faces. However, when a mask was covering the face, the average percentage that the participants got correct dropped to 87.2% for happy faces and 96% for sad faces. This represents an average of 91.6% correct for masked faces for human participants.

2) AI Experiment
a) AI model trained with unmasked faces:
For the first part of my AI Experiment, I found that the model trained on unmasked faces was 72% correct for happy unmasked faces and 80% correct for sad unmasked faces, making an average of 76% correct for unmasked faces. After testing the same model with masked faces, the average dropped lower to only 40% correct for happy faces and 64% correct for sad faces. The average for masked faces was only 52% correct.

**Figure 2**: AI experiment - Trained with no mask

b) AI model trained with masked faces:
In my second model trained with masked faces, I tested it with unmasked faces and found an average of 48% correct for happy faces and 52% correct for sad faces. This made an average of 50% when tested with unmasked faces, meaning that the AI was not better than a random guess. However, the results were much better when the same model was tested with masked faces. For happy faces, the AI model got 92% correct, and for sad faces, the AI model got 84% correct, making an average of 88% correct.s

**Figure 3**: AI experiment - Trained with mask

Compared to my human participants, AI didn’t do nearly as well. While my human participants got all unmasked images correct for both emotions, AI only got an average of 76% correct when trained with unmasked faces. However, AI only got about half of the faces correct when tested with masked faces on an unmasked trained model, while humans could detect 91.6% of the correct emotions. When AI was trained with masked faces and then tested with masked faces, it got an average of 88%, slightly lower than the human participants. Generally, both AI and my human volunteers found reading sad emotions easier than reading happy emotions, except for when masked faces were used to test and train.

DISCUSSION

I concluded that reading human emotions is much harder when wearing a face mask for humans and AI. With only being able to see the eyes and forehead, being able to read emotions is more challenging. In my machine learning model, humans are better than AI at reading emotions, both happy and sad. However, with training using only masked faces, AI’s emotion reading accuracy was significantly improved and only slightly less than that of humans. Similarly, after spending two years seeing others with face masks covering half their face, we may have trained ourselves to read emotions while only seeing someone’s eyes. I learned that training AI with the most appropriate information will give you the best results.

Generally, both AI and my human volunteers found reading sad emotions easier than reading happy emotions, except for when masked faces were used to test and train. There were more differences in accuracy between detecting sadness and happiness when the faces were covered with a mask. After doing research, I learned that studies have found that the mouth plays a crucial role in showing happiness, while the upper half of the face is more important in showing sad expressions (Grenville & Dwyer, 2022). This might explain why AI and humans had a more challenging time reading happiness and easier reading sadness, especially with a mask on.

An application of my results can be to help kids with autism to read emotions. Autism, or autism spectrum disorder, is characterized by challenges with social skills, communication, and repetitive behaviours. It is quite common, and it is estimated that 1 in 44 children have autism spectrum disorder (Maenner et al., 2021). Autism has been associated with difficulty in recognizing other people’s emotions. With a mask on, it would be even harder for autistic people. My model can help autistic children recognize the emotions of others and, in doing so, improve their ability to read emotions themselves.

Future Steps
Here are some next steps that I could take to improve my project. My current model only has two emotions: happiness and sadness. However, there are six basic human emotions: happiness, sadness, fear, anger, surprise, and disgust. Adding these new emotions could lead to new discoveries. Next, to improve the accuracy of my AI model, I can input more data by adding more images to train the model. Finally, I recruited only five children for the human portion of my study, so finding more kids would give more accurate results and lead to new questions and discoveries. I could also find people of different ages and see how the results differ between other age groups.

Conclusion
I found that humans and AI were less accurate in reading human emotions with masks, but humans still did better. Interestingly, when the AI model was trained on masked images, its accuracy in reading masked faces' emotions improved. Similarly, after spending two years seeing others with face masks covering half their face, we may have trained ourselves to read emotions while only seeing someone's eyes. Generally, both AI and my human volunteers found reading sad emotions easier than reading happy emotions, especially with a mask on. After doing research, I learned that studies have found that the mouth plays a crucial role in showing happiness, while the upper half of the face is more important in showing sad expressions. This might explain why AI and humans had a more challenging time reading happiness and had an easier time reading sadness. In conclusion, masks make it harder for humans and AI to read emotions. However, the accuracy of AI can be improved by proper training.

REFERENCES

Fulmer, J. (2021, September 27). The value of emotion recognition technology. The Value of Emotion Recognition Technology. Retrieved March 21, 2022, from https://www.itbusinessedge.com/business-intelligence/value-emotion-recognition-technology/

Grenville, E., & Dwyer, D. M. (2022, February 14). Face masks have emotion-dependent dissociable effects on accuracy and confidence in identifying facial expressions of emotion - cognitive research: Principles and implications. Retrieved March 20, 2022, from https://cognitiveresearchjournal.springeropen.com/articles/10.1186/s41235-022-00366-w

Lobe. (2020, October 26). Lobe Blog: Machine learning made easy. Lobe Blog | Machine Learning Made Easy. [Computer Software]. Retrieved March 21, 2022, from https://www.lobe.ai/blog/machine-learning-made-easy.

Maenner MJ, Shaw KA, Bakian AV, et al.(2021, December 3) Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years — Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2018. MMWR Surveill Summ 2021;70(No. SS-11):1–16. DOI: http://dx.doi.org/10.15585/mmwr.ss7011a1external icon