IME - Building an Emotion Classification System on Videos

Options
mohamed-khamis
mohamed-khamis Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Snowflake Advanced, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant Posts: 13 Partner

Name:
Mohamed AbdElAziz Khamis Omar

Title:
Senior Data Scientist

Country:
Egypt

Organization:
IME

Description:
We are providing data management solutions including Data integration, Data Quality, Master Data Management, Data Archiving, Data Masking, Big Data, Test Data Management, Data Governance, Operational Data Store, Data Warehouse and Business Intelligence.
IME has customers in Saudi Arabia, United Arab Emirates, Qatar, Kuwait, Oman, Egypt, Iraq, Algeria, Tunisia, Morocco, Nigeria, Pakistan, Bangladesh, and the Maldives. For more information about IME please visit us at infme.com

Awards Categories:

  • Alan Tuning


Challenge:


Our data science challenges revolve around:

  • Maintaining the code
  • Building end-to-end data science projects
  • Using state-of-the-art pre-trained models


Solution:


This project is the developer version of the "Dev Day - Dataiku + phData Workbook".
We built an emotion classification system on videos using the Dataiku deep learning for images plugin, which allows us to download pre-trained deep learning networks and provides recipes such as image classification retraining and scoring to classify emotions for the videos. This plugin uses Tensorflow and Keras on Python for image classification.
The emotion classification dataset is the speech video data from RAVDESS, and an example of one of the videos can be found on YouTube.
The approach that we will take for video classification is to break each emotion video into a fixed number of frames and then use these images to train a deep residual neural network (known as resnet) to classify emotions within each image.
This resnet network has been previously trained on the ImageNet dataset, so we do not have to start it from scratch.
Finally, we’ll evaluate the predicted emotion for a video through taking a majority vote on labels predicted across all its frames.

Image Classification Recipe.pngProject Workflow.pngPrepare the Emotion Classification Dataset.pngFrames sampled from each video in Emotion images.pngFilename identifiers.png

Train the Deep Learning Model.png

Score Frames and Evaluate Labels for Emotion Videos.png

Build the Production Pipeline.png

Avg. of prediction_happy_avg by video_path.PNGAvg. of prediction_calm by frame1.pngEmotion Classification Engine.png

Impact:

  • Code maintainability: Dataiku provides a high degree of code maintainability through visual recipes and visual workflows.
  • Ease-of-use: The platform made it very easy to build the end-to-end data science project.
  • Upskilling: We were able to leverage state-of-the-art pre-trained models through a wide range of macros and plugins.
Setup Info
    Tags
      Help me…