Error in Deep learning in images plugin

Swapnali

Hi Team.

I am following below link to achieve emotion analysis - https://www.phdata.io/dataiku-deep-learning-tutorial-emotion-classification-in-videos/

I performed 10 epochs with 10 steps in every epochs while using 'Retraining image classification model'

I am getting following error while using 'Image classification'. kindly help us to solve this error

arnaudde

Hello,

It looks like the images your are trying to score with your model are not of the right format.

What images do you use as input for the image classification recipe ?

Could you share one for further investigation ?

Arnaud

Swapnali

Thanks Arnaud for your quick reply,

For my experiment I am using the dataset mentioned in that link.

and following each step explained in that link.

Swapnali

Hi Arnaud,

I am waiting for your reply. Kindly provide you help

arnaudde

Hello Swapnali,
Something has probably gone wrong while doing the tutorial. Can you share the input image that you feed to the image classification recipe ?

Without more information on the input of the plugin I am afraid I can't help.

Best

Swapnali

Following is the flow we have created

In this Emotion videos is the Input folder in which we have multiple mp4 input files.

The link we used to download the input files is - https://zenodo.org/record/1188976/files/Video_Song_Actor_01.zip?download=1

arnaudde

Hello,
Thanks for the additional information.

Could you show what is in your "Testing images files" folder ? Can you share one of the images inside it ?

Swapnali

This is what we have in Testing Images files

following is the image

arnaudde

Thanks for the information
The problem is probably that your images are not at the root of the folder but in the sub directory "/Actor"

Could you try the Image classification recipe again with a folder with images at the root ?

Swapnali

Thanks Arnaud,

It worked!!

However now I am preparing ScoredImages with following python code

def process(row):
max_val = 0
max_label = None

emotions = ['calm', 'sad', 'surprised', 'neutral', 'fearful', 'angry', 'happy', 'disgust']

for e in emotions:
p = float(row['prediction_{}'.format(e)])
if p > max_val:
max_val = p
max_label = e

row['max_prediction'] = max_val
row['max_label'] = max_label
row['label'] = row['images'].split('_')[1]
row['correct'] = 1 if row['label'] == row['max_label'] else 0
row['video_path'] = '/{}.mp4'.format(row['images'].split('_')[0])

return row

and I am getting following error. kindly help to solve

arnaudde

Hello,

You don't have a column named prediction_calm, probably because your network never predicted an image as "calm".

Best

Swapnali

some images are labeled as Calm. you can see

arnaudde

The error you are encountering is because there is no "prediction_calm" column in your dataset.

What are the columns of the dataset you are applying this step on ?

Swapnali

There are Images, Prediction and error columns are in ScoredImages dataset which is input dataset.

and according to the steps in link , ScoredImagesPrepared (Output dataset)should have a new column max_label

arnaudde

You need to use the unnest processor on the prediction column first.

It will create all the prediction_ columns and therefore the custom python processor will be able to find the "prediction_calm" key of row.

You can learn more about python processors in the doc

robertcoop

Thanks for helping out, arnaudde!

Swapnali

Hi Arnaud,

We tried to understand the document you sent .But not able to understand .

Could you help me out with an appropriate processor for our use case?

Thanks a lot in advance

arnaudde

Hello Swapnali,

I think this code will achieve what you want.

Replace the code suggested in the phdata tutorial at step 3.C: Extract Predicted Labels for Images with the following one:

import json
def process(row):
         max_val = 0
         max_label = None

         emotions = ['calm', 'sad', 'surprised', 'neutral',  'fearful', 'angry', 'happy', 'disgust']

         for e in emotions:
                p = float(json.loads(row['prediction'])['prediction_{}'.format(e)])
                if p > max_val:
                      max_val = p
                      max_label = e

        row['max_prediction'] = max_val
        row['max_label'] = max_label
        row['label'] = row['images'].split('_')[1]
        row['correct'] = 1 if row['label'] == row['max_label'] else 0
        row['video_path'] = '/{}.mp4'.format(row['images'].split('_')[0])

        return row

Swapnali

Hi Arnaud,

I am still getting same kind of error. I have attached error in text file

arnaudde

My bad the key is "calm" not "prediction_calm" below code should work

import json
def process(row):
         max_val = 0
         max_label = None

         emotions = ['calm', 'sad', 'surprised', 'neutral',  'fearful', 'angry', 'happy', 'disgust']

         for e in emotions:
                p = float(json.loads(row['prediction'])[e])
                if p > max_val:
                      max_val = p
                      max_label = e

        row['max_prediction'] = max_val
        row['max_label'] = max_label
        row['label'] = row['images'].split('_')[1]
        row['correct'] = 1 if row['label'] == row['max_label'] else 0
        row['video_path'] = '/{}.mp4'.format(row['images'].split('_')[0])

        return row

Swapnali

thanks Arnaud,

Now we are facing error like code is taking keys Only from first row it seems. In first row there is no surprised, angry and disgust keys available so it is throwing errors like: same for angry and disgust

row is like:

arnaudde

The processor is iterating on each row but your predictor does not output probas for every emotion on every line.
I added a if statement to check if the emotion is in the prediction

import json
def process(row):
         max_val = 0
         max_label = None

         emotions = ['calm', 'sad', 'surprised', 'neutral',  'fearful', 'angry', 'happy', 'disgust']

         for e in emotions:
            if e in json.loads(row['prediction']):
                p = float(json.loads(row['prediction'])[e])
                if p > max_val:
                      max_val = p
                      max_label = e

        row['max_prediction'] = max_val
        row['max_label'] = max_label
        row['label'] = row['images'].split('_')[1]
        row['correct'] = 1 if row['label'] == row['max_label'] else 0
        row['video_path'] = '/{}.mp4'.format(row['images'].split('_')[0])

        return row

robertcoop

Alternatively, you can adjust the scoring recipe to provide all 8 classification labels as outputs.

Swapnali

Thanks a lot Arnaud for your extended help. I completed the entire pipeline