Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Using BERT with keras in VisualML

Solved!
Antal
Level 3
Using BERT with keras in VisualML

HI everyone,

 

I'd like to use BERT embeddings as input to a text classification model in VisualML.

I have a code env set up that has the pretrained model in it and I'm able to load it as expected.

 

Does anyone have any experience/tips on how to go about this?

 

I'm not really sure on where to put the preprocessing code (tokenization and embedding) and how to then pick up these inputs in the keras architecture.

Do I do the former in the Feature handling section? But how do I then configure the Input layer in the architecture part? Or do I "pass through" the text feature and do everything in the architecture section?

 

Any examples are welcomed.

0 Kudos
1 Solution
Muennighoff
Dataiker
Dataiker

Hey Antal,

 

In DSS 10.0.4, the Sentence Embedding preprocessing method was introduced for exactly that purpose. You can just replace the models in the link with a BERT model, e.g.

MODELS_REPO_AND_REVISION = [
    ("bert-base-uncased", "b96743c503420c0858ad23fca994e670844c6c05")
]  

By selecting the Sentence Embedding preprocessing method in the right env, you can then use the downloaded model.

 

Let me know if this helps or if you still wish to write the preprocessing code yourself.

View solution in original post

2 Replies
Muennighoff
Dataiker
Dataiker

Hey Antal,

 

In DSS 10.0.4, the Sentence Embedding preprocessing method was introduced for exactly that purpose. You can just replace the models in the link with a BERT model, e.g.

MODELS_REPO_AND_REVISION = [
    ("bert-base-uncased", "b96743c503420c0858ad23fca994e670844c6c05")
]  

By selecting the Sentence Embedding preprocessing method in the right env, you can then use the downloaded model.

 

Let me know if this helps or if you still wish to write the preprocessing code yourself.

Antal
Level 3
Author

Hey, thanks for the quick reply!

Yes, I found that functionality and have a code env set up with the necessary packages and pretrained model included (a Dutch BERT model).

 

EDIT: I fixed my own leftover error. It was due to me using tensorflow.keras.layers with a tensorflow 1.15 backend (as supported by VisualML), instead of keras.layers.

All good!