IME - Building an Emotion Classification System on Videos

Name:
Mohamed AbdElAziz Khamis Omar

Title:
Senior Data Scientist

Country:
Egypt

Organization:
IME

Description:
We are providing data management solutions including Data integration, Data Quality, Master Data Management, Data Archiving, Data Masking, Big Data, Test Data Management, Data Governance, Operational Data Store, Data Warehouse and Business Intelligence.
IME has customers in Saudi Arabia, United Arab Emirates, Qatar, Kuwait, Oman, Egypt, Iraq, Algeria, Tunisia, Morocco, Nigeria, Pakistan, Bangladesh, and the Maldives. For more information about IME please visit us at infme.com

Awards Categories:

  • Alan Tuning


Challenge:


Our data science challenges revolve around:

  • Maintaining the code
  • Building end-to-end data science projects
  • Using state-of-the-art pre-trained models


Solution:


This project is the developer version of the "Dev Day - Dataiku + phData Workbook".
 
We built an emotion classification system on videos using the Dataiku deep learning for images plugin, which allows us to download pre-trained deep learning networks and provides recipes such as image classification retraining and scoring to classify emotions for the videos. This plugin uses Tensorflow and Keras on Python for image classification.   
 
The emotion classification dataset is the speech video data from RAVDESS, and an example of one of the videos can be found on YouTube
 
The approach that we will take for video classification is to break each emotion video into a fixed number of frames and then use these images to train a deep residual neural network (known as resnet) to classify emotions within each image.
 
This resnet network has been previously trained on the ImageNet dataset, so we do not have to start it from scratch.
 
Finally, we’ll evaluate the predicted emotion for a video through taking a majority vote on labels predicted across all its frames.
 
 

Image Classification RecipeImage Classification RecipeProject WorkflowProject WorkflowPrepare the Emotion Classification DatasetPrepare the Emotion Classification DatasetFrames sampled from each video in Emotion imagesFrames sampled from each video in Emotion imagesFilename identifiersFilename identifiers

Train the Deep Learning ModelTrain the Deep Learning Model

Score Frames and Evaluate Labels for Emotion VideosScore Frames and Evaluate Labels for Emotion Videos

Build the Production PipelineBuild the Production Pipeline

Avg. of prediction_happy_avg by video_path.PNGAvg. of prediction_calm by frame1Avg. of prediction_calm by frame1Emotion Classification EngineEmotion Classification Engine

Impact:

  • Code maintainability: Dataiku provides a high degree of code maintainability through visual recipes and visual workflows.
  • Ease-of-use: The platform made it very easy to build the end-to-end data science project. 
  • Upskilling: We were able to leverage state-of-the-art pre-trained models through a wide range of macros and plugins.
Attachments
Share:
Version history
Publication date:
07-07-2022 07:24 AM
Version history
Last update:
‎07-19-2021 11:28 AM
Updated by:
Contributors