Run Stability diffusion model on dataiku

mdsa3d

Hi,

I would like to know if it is possible to run stability diffusion model on dataiku platform?

The goal is to host the model (already created) on the platform and test it.

Thanks, would highly appreciate any response and suggestion.

Regards,

tgb417

@mdsa3d

Welcome to the Dataiku Community.

As a digital artist https://www.41xRT.com/art. I too would be interested in running Stable Diffusion in a Dataiku environment. I'm not aware of anyone that has done this. It is certainly not a built in feature.

That said, one would need TensorFlow setup with a GPU and enough memory. One could very likely setup Python Code environment with the needed Software. If the computer Dataiku was running on either had needed resources or had access to spin up an environment with the needed resources. One might find some interesting possibilities.

Is there anyone out there who has done this?

Zach

Hi @mdsa3d
,

I'm currently working on a plugin for DSS that allows you to run a Stable Diffusion model as a recipe.

It will likely be released soon, although I don't have an ETA at this time. I'll post an update once I have more information.

Thanks,

Zach

tgb417

@ZachM

Very interesting...

I'm interested in your intended use cases for the plug-in?

How will the plugin interaction work with Dataiku?

• Are you planning to allow for Stable Diffusion parameters and prompt to be included as a dataset?
• Will one be able to retrain Stable Diffusion final layers to do "transfer learning"?

What GPU architecture(s) will you be targeting?

• For example only Nvidia
• Or might you also provide Apple Silicon support as well.

Thanks for letting us know. As a long time Dataiku user and Artist, this is of particularly interesting to me.

Zach

Hi @tgb417
,

The plugin will accept a pretrained model as input, and it can generate images via txt2img or img2img.

The parameters and prompt are part of the recipe config settings. They currently aren't planned to be included as a dataset input.

Transfer learning / training isn't supported. The plugin uses a pretrained model. The user can download it from Hugging Face or use their own.

Only CUDA GPUs are supported, but you can also run it on the CPU (slow).

Thanks for your interest in the plugin!

Zach

mdsa3d

Hi @ZachM
,

Thanks for your response, I look forward to your implementation.

If you need any collaboration, please do let me know.

mdsa3d

Hi @tgb417
,

Thanks for warm welcome.

Indeed, without GPU support it is quite slow. I have been running it locally on my PC even with 14 cores with low guidance scale each inference takes about 40s, now you can just imagine the slowness.

I have also noticed that Dataiku has inbuilt support only for tensorflow and keras, while pytorch is as you mentioned through notebook. I am trying to figure out a way to setup PyTorch also, then running huggingface pretrained models will be easier.
However, if someone has done this, please do advise!