Run Stability diffusion model on dataiku

mdsa3d
mdsa3d Registered Posts: 3

Hi,

I would like to know if it is possible to run stability diffusion model on dataiku platform?

The goal is to host the model (already created) on the platform and test it.

Thanks, would highly appreciate any response and suggestion.

Regards,

Answers

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @mdsa3d

    Welcome to the Dataiku Community.

    As a digital artist https://www.41xRT.com/art. I too would be interested in running Stable Diffusion in a Dataiku environment. I'm not aware of anyone that has done this. It is certainly not a built in feature.

    That said, one would need TensorFlow setup with a GPU and enough memory. One could very likely setup Python Code environment with the needed Software. If the computer Dataiku was running on either had needed resources or had access to spin up an environment with the needed resources. One might find some interesting possibilities.

    Is there anyone out there who has done this?

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker

    Hi @mdsa3d
    ,

    I'm currently working on a plugin for DSS that allows you to run a Stable Diffusion model as a recipe.

    It will likely be released soon, although I don't have an ETA at this time. I'll post an update once I have more information.

    Thanks,

    Zach

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @ZachM

    Very interesting...

    I'm interested in your intended use cases for the plug-in?

    How will the plugin interaction work with Dataiku?

    • Are you planning to allow for Stable Diffusion parameters and prompt to be included as a dataset?
    • Will one be able to retrain Stable Diffusion final layers to do "transfer learning"?

    What GPU architecture(s) will you be targeting?

    • For example only Nvidia
    • Or might you also provide Apple Silicon support as well.

    Thanks for letting us know. As a long time Dataiku user and Artist, this is of particularly interesting to me.

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker

    Hi @tgb417
    ,

    The plugin will accept a pretrained model as input, and it can generate images via txt2img or img2img.

    The parameters and prompt are part of the recipe config settings. They currently aren't planned to be included as a dataset input.

    Transfer learning / training isn't supported. The plugin uses a pretrained model. The user can download it from Hugging Face or use their own.

    Only CUDA GPUs are supported, but you can also run it on the CPU (slow).

    Thanks for your interest in the plugin!

    Zach

  • mdsa3d
    mdsa3d Registered Posts: 3

    Hi @ZachM
    ,

    Thanks for your response, I look forward to your implementation.

    If you need any collaboration, please do let me know.

  • mdsa3d
    mdsa3d Registered Posts: 3

    Hi @tgb417
    ,

    Thanks for warm welcome.

    Indeed, without GPU support it is quite slow. I have been running it locally on my PC even with 14 cores with low guidance scale each inference takes about 40s, now you can just imagine the slowness.

    I have also noticed that Dataiku has inbuilt support only for tensorflow and keras, while pytorch is as you mentioned through notebook. I am trying to figure out a way to setup PyTorch also, then running huggingface pretrained models will be easier.
    However, if someone has done this, please do advise!

Setup Info
    Tags
      Help me…