How can I save a fine-tuned HuggingFace model to a Dataiku Managed Folder?
Hello,
I am having an issue with saving a HuggingFace model that I am fine-tuning.
The default implementation for saving HF models is to use:
model.save_pretrained("/path/to/directory")
However, as far as I can tell, this does not work in dataiku. Even after I use the following code to get the path:
folder = dataiku.Folder("xxxxxxx")
d = folder.get_info()
path = d['accessInfo']['root']
I get an error saying that my permission to the path is denied.
In the past I have used pickle objects and saved those, but the specific model I am using this time is not pickleable.
Any help would be much appreciated!
Operating system used: Windows
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,162 Neuron
Where is the managed folder stored on? (What kind of technology)
-
Under the storage settings we have:
Type: Azure Blob Storage
Azure connection: fastpath_conn
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,162 Neuron
Some folder methods only work on local storage. For cloud storage folders you need to use the corresponding read and write API methods.