Affectiva, one in a series of companies to come out of MIT’s Media Lab whose work revolves around affective computing, used to be best known for sensing emotion in videos. It recently expanded into emotion detection in audio with the Speech API for companies making robots and AI assistants.
Affectiva, one in a series of companies to come out of MIT’s Media Lab whose work revolves around affective computing, used to be best known for sensing emotion in videos. It recently expanded into emotion detection in audio with the Speech API for companies making robots and AI assistants.
Affective computing, the use of machines to understand and respond to human emotion, has many practical uses. In addition to Affectiva, Media Lab nurtured Koko, a bot that detects words used on chat apps like Kik to recognize people who need emotional support, and Cogito, whose AI is used by the U. S. Department of Veteran Affairs to analyze the voices of military veterans with PTSD to determine if they need immediate help. Then there’s Jibo, a home robot that mimics human emotion on its five-inch LED face that Time magazine recently declared one of the best inventions of 2017.
Instead of natural language processing, the Speech API private beta a handful of social robots and device makers is currently using voice to recognize things like laughing, anger, and various forms of arousal, alongside voice volume, tone, speed, and pauses.
The combination of sentiment analysis of voice and face cues, Affectiva CEO Rana el Kaliouby said, make it possible for technology to respond to human moods and emotions and be part of more humanlike interactions that improve lives. Her favorite example of this comes from the movie Her, in which Joaquin Phoenix’s character falls in love with his AI assistant Samantha, played by Scarlett Johansson.
“I think it’s very powerful in that this operating system, because they knew that guy really, really well,” el Kaliouby said. “[He] kind of had this very negative outlook on life, and she was able to turn that around because she knew him so well, so she was able to… persuade him and motivate him to change his behavior, and I think that’s the true power of these conversational interfaces.”
Ask Alexa or a home robot today for a joke and they may tell you one, but they don’t know when or not you found that joke funny. They haven’t learned how to react, and it’s the continued interpretation of those reactions that will remake human-machine interaction.
In essence, El Kaliouby argues, computers need empathy to recognize and respond in a natural way when they see human emotions demonstrated. Empathy, she says, is an intelligence that will lead to a future in which robots can enhance our humanity rather than take away from it.
“There’s a lot of ways these things [conversational agents] can persuade you to lead more productive, healthier, happier lives, but in my opinion they can’t get there unless they have empathy, and unless they can factor in the considerations of your social, emotional, and cognitive state. And you can’t do that without affective computing, or what we call artificial emotional intelligence,” el Kaliouby said.
“We need to build EQ in our AI systems because otherwise they’re not going to be as effective as they were designed to be,” she said.
VentureBeat spoke with el Kaliouby last month, shortly before the World Economic Forum’s global council for robotics and AI, where she joined other members of the business and AI community to discuss how to build ethics and morality into robots and AI systems.
El Kaliouby moved to the United States from Egypt in 2006 to take a postdoctoral position at MIT, where she was part of a project to give people on the autism spectrum real-time feedback on the emotions and expressions people demonstrate in conversations.
In our discussion, we talked about how interpretation and reaction to human emotion may fundamentally change the way humans and machines interact with one another, how voice analytics apply to health care, and what companies mean when they say they want to democratize AI.
This interview was edited for brevity and clarity.
VentureBeat: Affectiva is able to detect, correct me if I’m wrong, seven emotions in videos today?
El Kaliouby: The way we think about it is like facial expressions are the building blocks of different emotional states, so we can read over different tiny facial expressions, then combine these in different ways to represent seven different emotional states plus age, gender, and ethnicity. The key thing with the underlying facial expressions is that sometimes you’ll see somebody squint. That may not have an emotion associated, but it’s a very important facial expression and has a lot of meaning. Or maybe somebody smirks, and if they’re smirking, they’re kind of saying “Yeah, hmmm, I’m not persuaded,” and again that may not map into one of these seven emotional spaces, but it’s still a very representative expression.
VentureBeat: How does emotion detection in video translate to voice? Are those the same emotions you’re detecting?
El Kaliouby: They’re not, though there is some overlap. So the face is very good at positive and negative expressions. The voice, however, is very good about the intensity of the emotions — we call it the arousal level — so we can identify arousal from your voice. We can detect smiles through your facial expression, but then we can identify specifically when you’re laughing through voice. Another example is anger. People communicate anger of course through facial expressions, but in voice there’s a wider spectrum, like cold anger and hot anger and frustration and annoyance, and that entire spectrum is a lot clearer in the voice channel. So they overlap, but they kind of complement each other.
VentureBeat: Everybody’s emotional state is kind of different, so how do you form a baseline of each individual’s emotional state?
El Kaliouby: We factor that into the algorithm. So the clearest example of this is in the face world: Like, some people have wrinkles between their eyebrows, things you can fix with Botox, like “resting bitch face,” basically, and so we developed algorithms that subtract that.
Basically the algorithm first learns, “Oh, this is your neutral face,” and that’s your baseline mode. So if it sees enough of it, and then if it sees a deviation from that baseline, it can subtract that out, and you can do that using neural nets. Eventually, with enough data — like if Alexa has interacted with you every day for the past year — it should have enough information to build a very personalized model of you. We don’t do that yet at Affectiva, but I think that’s where the world will eventually go: superpersonalized models.
VentureBeat: Especially with first-time users, I’ve noticed that people can get really angry at AI assistants at times, and they can get pretty rude at times. What do you think of the idea of making emotional AI that sometimes gets offended and shuts off if you get too rude?
El Kaliouby: That’s interesting; in my mind, that kind of rebels. I don’t know if you want that, but it might, you know. I’m thinking of especially kids, like kids will say, “Alexa, you’re stupid!”
VentureBeat: Right, exactly.
El Kaliouby: So maybe Alexa should kind of rebel for a day.
VentureBeat: I guess it’s rebellion, but in another sense, it’s reinforcement of the social norm that you shouldn’t be mean to somebody who is being servile to you or helping you.
El Kaliouby: Yeah, I absolutely agree. So one reason I think we’re now dehumanizing each other is because we communicate primarily through digital.
A lot of our communication has now become digital, and it does not mimic the natural way we have evolved to communicate with each other, so it’s almost like we have this muscle, these social-emotional skills, and they’re atrophying, right? You look at young kids — you know how there’s all these articles about kids being in an accident and instead of jumping in to help, they’ll just stand and shoot video on their phone — you’ve got to wonder whatever happened to good old empathy right?
And I really think it’s disappearing because we’re not practicing these skills. And so, arguably, you need to be kind to your social robot, and you need to say please and thank you and all these good things. I think that maybe that brings back our humanity in a weird way.
VentureBeat: What are your thoughts on the use of emotional AI to analyze the use of a person’s mental health?
El Kaliouby: I’m very excited about that. I got my start in this area by working on a National Science Foundation-funded project for autism. We built Google-like glasses that had a camera embedded in it, and kids on the spectrum would wear these glasses and it would give them real-time feedback on the emotions and social expressions of people they were talking to.
I actually like this example because it’s an example of where AI can broaden inclusion, because if you take the case of individuals on the spectrum, they usually don’t have equal access to job opportunities because they lack all these social intelligence skills, and that’s really key in the office or on any job.
