Using BERT with keras in VisualML

Antal

HI everyone,

I'd like to use BERT embeddings as input to a text classification model in VisualML.

I have a code env set up that has the pretrained model in it and I'm able to load it as expected.

Does anyone have any experience/tips on how to go about this?

I'm not really sure on where to put the preprocessing code (tokenization and embedding) and how to then pick up these inputs in the keras architecture.

Do I do the former in the Feature handling section? But how do I then configure the Input layer in the architecture part? Or do I "pass through" the text feature and do everything in the architecture section?

Any examples are welcomed.

Muennighoff

Hey Antal,

In DSS 10.0.4, the Sentence Embedding preprocessing method was introduced for exactly that purpose. You can just replace the models in the link with a BERT model, e.g.

MODELS_REPO_AND_REVISION = [
    ("bert-base-uncased", "b96743c503420c0858ad23fca994e670844c6c05")
]  

By selecting the Sentence Embedding preprocessing method in the right env, you can then use the downloaded model.

Let me know if this helps or if you still wish to write the preprocessing code yourself.

Antal

Hey, thanks for the quick reply!

Yes, I found that functionality and have a code env set up with the necessary packages and pretrained model included (a Dutch BERT model).

EDIT: I fixed my own leftover error. It was due to me using tensorflow.keras.layers with a tensorflow 1.15 backend (as supported by VisualML), instead of keras.layers.

All good!

GayleFer1

I'm trying to do the same thing with the sentence-transformers 'all-miniLM-L6-v2' model but I can't find the equivalent of this string anywhere. What is this called and where do I find it?

"b96743c503420c0858ad23fca994e670844c6c05"

Antal

Go to the model's model card on huggingface and click on the Files tab. The revision string is mentioned there in the top right.

https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2