Using BERT with keras in VisualML

Antal
Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 90 Neuron

HI everyone,

I'd like to use BERT embeddings as input to a text classification model in VisualML.

I have a code env set up that has the pretrained model in it and I'm able to load it as expected.

Does anyone have any experience/tips on how to go about this?

I'm not really sure on where to put the preprocessing code (tokenization and embedding) and how to then pick up these inputs in the keras architecture.

Do I do the former in the Feature handling section? But how do I then configure the Input layer in the architecture part? Or do I "pass through" the text feature and do everything in the architecture section?

Any examples are welcomed.

Tagged:

Best Answer

  • Muennighoff
    Muennighoff Dataiker, Registered Posts: 3 Dataiker
    edited July 17 Answer ✓

    Hey Antal,

    In DSS 10.0.4, the Sentence Embedding preprocessing method was introduced for exactly that purpose. You can just replace the models in the link with a BERT model, e.g.

    MODELS_REPO_AND_REVISION = [
        ("bert-base-uncased", "b96743c503420c0858ad23fca994e670844c6c05")
    ]  

    By selecting the Sentence Embedding preprocessing method in the right env, you can then use the downloaded model.

    Let me know if this helps or if you still wish to write the preprocessing code yourself.

Answers

  • Antal
    Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 90 Neuron

    Hey, thanks for the quick reply!

    Yes, I found that functionality and have a code env set up with the necessary packages and pretrained model included (a Dutch BERT model).

    EDIT: I fixed my own leftover error. It was due to me using tensorflow.keras.layers with a tensorflow 1.15 backend (as supported by VisualML), instead of keras.layers.

    All good!

  • GayleFer1
    GayleFer1 Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1
    edited July 17

    I'm trying to do the same thing with the sentence-transformers 'all-miniLM-L6-v2' model but I can't find the equivalent of this string anywhere. What is this called and where do I find it?

    "b96743c503420c0858ad23fca994e670844c6c05"

  • Antal
    Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 90 Neuron

    Go to the model's model card on huggingface and click on the Files tab. The revision string is mentioned there in the top right.

    https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

Setup Info
    Tags
      Help me…