Fully Port DSS to ARM / Apple Silicon

tgb417

This may be too obvious :

User Story: As a Macintosh user who has enjoyed using DSS on my local computer, it would be great to use Dataiku DSS on a New Apple Silicon Computer.

Value Proposition:

1. Having DSS working on Macintosh Computers has been an excellent entry point for folks interested in using DSS. Installing DSS has been easy. In two or three years Dataiku will lose this advantage of the ease of entry if it only supports Intel Computers. (With Version 11.0.1 this seems to be the case.)
2. Given the significant CPU & GPU performance gains in the first generation of the Apple M1 chip, it is likely that Apple’s Macintosh will be a performance leader in the laptop and desktop space for several years to come. However, to get those performance improvements DSS will need to support native Apple silicon (ARM64) instruction sets.
3. Give the custom Neural Network Processor in the Apple Silicon Chips it is likely that there will be development that can uniquely be done within the M1 Macintosh environment.

tgb417

Here is a link to an eager person asking about DSS on Apple Silicon.

https://community.dataiku.com/t5/Setup-Configuration/DSS-on-ARM-or-AMD-Ryzan-9-Processors/m-p/9795/highlight/true#M1041

tgb417

Here is a second query about this topic.

https://community.dataiku.com/t5/Setup-Configuration/DSS-fail-to-install-on-Mac-M1/m-p/12582/emcs_t/

fsergot

Hello,

this support is indeed in our radar since the announcement from Apple, As said in the link, this is far from trivial but we know the importance of the Mac users community around DSS

DMKnemeijer

I support this! Not only would this enable Mac users to install Dataiku on their machines, but this will also allow us to migrate our Cloud Dataiku installation over to EC2 instances that are using the newer, ARM-based Graviton processors, which would lead to a much better price/performance

natejgardner

It would be great to be able to install Dataiku on ARM devices without Intel emulation.

Marvell and Ampere servers are rising in popularity in enterprise. Microsoft is also rumored to be working on ARM server hardware. ARM laptops are also becoming very popular, where users may like to install Dataiku locally for offline/local convenient data processing ability (I'm in that camp). This would be especially nice for mobility, especially when using limited mobile data plans. Running on locally on a Surface device or a Chromebook would be awesome for doing quick data exploration, validation, and processing on the go without needing to upload a large dataset to a server (e.g. field service type data analysis use-cases).

Baseltov

Unfortunately still waiting on this. I have a new MacBook Pro with M1 Pro chip, but DSS does not run natively.

tgb417

@Baseltov

+1 here. I'm still hopeful.

There are some discussions here in the Dataiku Community of folks making attempts at this.

https://community.dataiku.com/t5/Setup-Configuration/Dataiku-support-for-mac-m1/td-p/20772

https://community.dataiku.com/t5/Setup-Configuration/DSS-fail-to-install-on-Mac-M1/m-p/12295/highlight/true#M1299

Baseltov

@tgb417

Nice to see the community working on this, but there should be an official release compatibel with Apple Silicon.

Turribeach

What's the latest on this? The market is quickly moving into this direction, PyTorch just announced support for Metal for native Apple Silicon GPU training of ML models. Also note this stat from Apple: "50% of Customers Purchasing a Mac in Q2 2022 Were New Mac Users".

https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/

https://www.macrumors.com/2022/04/28/mac-users-q2-2022-new-to-mac/

Baseltov

The community page with the name ‘DSS on Mac with M1 chip’ has been deleted. On that topic, @CoreyS
gave a solution, that apple silicon was supported in DSS 11. But now that topic has been removed and I see no mention of Apple Silicon M1 chip support in the release notes of DSS 11.0.
@CoreyS
can you confirm this solution? Could you also make sure why there then is no mention in the release notes? Seems like a huge thing to announce there!

CoreyS

Hi @Baseltov
I'm not really sure what you mean, the page hasn't been deleted. Here is the solution you are referencing:

https://community.dataiku.com/t5/Setup-Configuration/DSS-installed-on-Mac-M1/m-p/27513/highlight/true#M2505

Baseltov

@CoreyS
That’s weird, I can’t access that page. If I click your link, or any link to that topic, I get the following Exception ID: 1938E724. And it does not open. Also if I manually search that topic name, it doesn’t show up in the search results either.

CoreyS

Hi @Baseltov
sorry for the inconvenience you are facing. Despite testing across multiple accounts, unfortunately, I cannot replicate that behavior so my guess is that this may be a local issue. Can you try refreshing your browser page? Additional steps you can take include logging out and logging back in again, and finally clearing your cache on your browser. Hopefully, that does the trick.

In the meantime here is the solution:

Hi there,
The Dataiku DSS Launcher works out of the box for Intel and M1 based Macs. No extra step is required for m1.

Best,
Alex

tgb417

@CoreyS
,

It is very cool that the new Macintosh launcher runs on an Apple silicon Mac. I’ve found that 11.0.1 works better than 11.0.0.

The next step in this process is to move away from the Rosetta compatibility mode and to native ARM64 code.

abdagon

I just installed DSS on my MacBook Pro with M1 and Ventura, it worked flawlessly so far .

However installing on ARM based Ubuntu did not (yet) work.

As DSS is coming to ARM architecture, what are the plans to make DSS available on ARM based Linux?

Supporting the statement of @natejgardner
in this thread.

tgb417

@abdagon

Welcome to the Dataiku community. We are so glad you have joined us.

I've been using DSS on Apple Silicon a lot recently. In most areas I agree it performs well. (Thank you Dataiku Team.)

That said you may already know that the reason that DSS worked on Apple Silicon is due to Rosetta 2 Translation. This is a technology that comes on Mac that allows Apple Silicon ARM computers to run Intel code.

Although the Data Science Studio (DSS) code works. I find in activity monitor that the architecture of the underlying code is still Intel not ARM. Although Rosetta 2 is amazing technology, the use of it, tends to slow down performance by 20% or so. One can also not take advantage of some of the performance improvements that Apple Silicon provides in the area of GPU and other custom hardware.

I agree it would be great to see native support for Linux on ARM and Apple Silicon (based on ARM) with some support for the extended Architecture that Apple is providing. This would allow data Scientists to work on individual workstation training models on local hardware and then promote their work with fewer hassles into production.

Now that it has been more than two years since this initial product idea. How are others using Apple Silicon as part of your teams workflows?

abdagon

Thanks for the kind welcome message, @tgb417
!

With a PhD in data management in the 80s, I am very keen to use the DSS features .

I am well aware of the emulations (my first Apple Computer was a Mac II based on M680X0, then a PowerPC Mac, then Intel based notebooks and now a M1 notebook - I even wrote emulators for Z80 microprocessors a long time back ).

My request goes towards an ARM64 based hosting without an emulator such as Rosetta for an inexpensive «ever on» platform for prototyping, with a full license once going productive.

Reading through a few threads of this community I understand that the limitation is not so much about DSS, but more about underlying libraries such as Python, which must run natively. Based on Python information, 3.9 should be ARM64 native, with 3.11 being the most recent official version now. So it looks like it's mostly a question of porting DSS to a recent Python version.

Any dataiku statement would be much appreciated.

BTW - quiz question, who knows what ARM means?

Turribeach

While I would love to see Dataiku ported to ARM personally I think Dataiku will never port it to run natively on ARM. While there is interest by folks like us which use Apple Silicon Macs as computer and ML enthusiast there is very little penetration of Apple Silicon Macs in the enterprise. And we should understand that the majority of Dataiku licenses are bought by enterprises hence it is expected that these enterprise customers will drive the product roadmap. Though not officially supported DSS does currently run on Apple Silicon Macs in emulation in the same way that it runs on Linux VMs under Windows. I think this capability is enough for consultants, enthusiasts, etc to be able to use DSS for basic capabilities, demoes, etc. Any serious Dataiku user will probably want a more serious platform like a VM in the cloud or an on-prem server.

tgb417

@abdagon
,

I to when through the Motorola 68000 to Power PC and Power PC to Intel conversions over the years.

Regarding ARM.

If I understand correctly. Back in the day it meant:

Acorn RISC Machine

Because the British maker of the BBC Micro and the Electron Computers Acorn Computer created the first ARM processor.

Today it seems to be morphed into.

Advanced RISC Machine

And Apparently the folks at ARM don't even use it much as an acronym.

The value proposition for enterprise is really around the problems that Intel is having at making innovative performant but low power chips. Electricity and cooling cost real money. However, @Turribeach
. I take your point.