Fully Port DSS to ARM / Apple Silicon

Options
tgb417
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron

This may be too obvious :

User Story: As a Macintosh user who has enjoyed using DSS on my local computer, it would be great to use Dataiku DSS on a New Apple Silicon Computer.

Value Proposition:

  1. Having DSS working on Macintosh Computers has been an excellent entry point for folks interested in using DSS. Installing DSS has been easy. In two or three years Dataiku will lose this advantage of the ease of entry if it only supports Intel Computers. (With Version 11.0.1 this seems to be the case.)
  2. Given the significant CPU & GPU performance gains in the first generation of the Apple M1 chip, it is likely that Apple’s Macintosh will be a performance leader in the laptop and desktop space for several years to come. However, to get those performance improvements DSS will need to support native Apple silicon (ARM64) instruction sets.
  3. Give the custom Neural Network Processor in the Apple Silicon Chips it is likely that there will be development that can uniquely be done within the M1 Macintosh environment.

5
5 votes

In the Backlog · Last Updated

Comments

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options
  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options
  • fsergot
    fsergot Dataiker, Registered, Product Ideas Manager Posts: 117 Dataiker
    Options

    Hello,

    this support is indeed in our radar since the announcement from Apple, As said in the link, this is far from trivial but we know the importance of the Mac users community around DSS

  • DMKnemeijer
    DMKnemeijer Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 2 Partner
    Options

    I support this! Not only would this enable Mac users to install Dataiku on their machines, but this will also allow us to migrate our Cloud Dataiku installation over to EC2 instances that are using the newer, ARM-based Graviton processors, which would lead to a much better price/performance

  • natejgardner
    natejgardner Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 151 Neuron
    Options

    It would be great to be able to install Dataiku on ARM devices without Intel emulation.

    Marvell and Ampere servers are rising in popularity in enterprise. Microsoft is also rumored to be working on ARM server hardware. ARM laptops are also becoming very popular, where users may like to install Dataiku locally for offline/local convenient data processing ability (I'm in that camp). This would be especially nice for mobility, especially when using limited mobile data plans. Running on locally on a Surface device or a Chromebook would be awesome for doing quick data exploration, validation, and processing on the go without needing to upload a large dataset to a server (e.g. field service type data analysis use-cases).

  • Baseltov
    Baseltov Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 6 Partner
    Options

    Unfortunately still waiting on this. I have a new MacBook Pro with M1 Pro chip, but DSS does not run natively.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options
  • Baseltov
    Baseltov Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 6 Partner
    Options

    @tgb417

    Nice to see the community working on this, but there should be an official release compatibel with Apple Silicon.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,727 Neuron
    Options

    What's the latest on this? The market is quickly moving into this direction, PyTorch just announced support for Metal for native Apple Silicon GPU training of ML models. Also note this stat from Apple: "50% of Customers Purchasing a Mac in Q2 2022 Were New Mac Users".

    https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/

    https://www.macrumors.com/2022/04/28/mac-users-q2-2022-new-to-mac/

  • Baseltov
    Baseltov Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 6 Partner
    Options

    The community page with the name ‘DSS on Mac with M1 chip’ has been deleted. On that topic, @CoreyS
    gave a solution, that apple silicon was supported in DSS 11. But now that topic has been removed and I see no mention of Apple Silicon M1 chip support in the release notes of DSS 11.0.
    @CoreyS
    can you confirm this solution? Could you also make sure why there then is no mention in the release notes? Seems like a huge thing to announce there!

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭
    Options

    Hi @Baseltov
    I'm not really sure what you mean, the page hasn't been deleted. Here is the solution you are referencing:

    https://community.dataiku.com/t5/Setup-Configuration/DSS-installed-on-Mac-M1/m-p/27513/highlight/true#M2505

  • Baseltov
    Baseltov Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 6 Partner
    Options

    @CoreyS
    That’s weird, I can’t access that page. If I click your link, or any link to that topic, I get the following Exception ID: 1938E724. And it does not open. Also if I manually search that topic name, it doesn’t show up in the search results either.

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭
    Options

    Hi @Baseltov
    sorry for the inconvenience you are facing. Despite testing across multiple accounts, unfortunately, I cannot replicate that behavior so my guess is that this may be a local issue. Can you try refreshing your browser page? Additional steps you can take include logging out and logging back in again, and finally clearing your cache on your browser. Hopefully, that does the trick.

    In the meantime here is the solution:

    Hi there,
    The Dataiku DSS Launcher works out of the box for Intel and M1 based Macs. No extra step is required for m1.

    Best,
    Alex

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @CoreyS
    ,

    It is very cool that the new Macintosh launcher runs on an Apple silicon Mac. I’ve found that 11.0.1 works better than 11.0.0.

    The next step in this process is to move away from the Rosetta compatibility mode and to native ARM64 code.

  • abdagon
    abdagon Registered Posts: 2
    Options

    I just installed DSS on my MacBook Pro with M1 and Ventura, it worked flawlessly so far .

    However installing on ARM based Ubuntu did not (yet) work.

    As DSS is coming to ARM architecture, what are the plans to make DSS available on ARM based Linux?

    Supporting the statement of @natejgardner
    in this thread.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @abdagon

    Welcome to the Dataiku community. We are so glad you have joined us.

    I've been using DSS on Apple Silicon a lot recently. In most areas I agree it performs well. (Thank you Dataiku Team.)

    That said you may already know that the reason that DSS worked on Apple Silicon is due to Rosetta 2 Translation. This is a technology that comes on Mac that allows Apple Silicon ARM computers to run Intel code.

    Although the Data Science Studio (DSS) code works. I find in activity monitor that the architecture of the underlying code is still Intel not ARM. Although Rosetta 2 is amazing technology, the use of it, tends to slow down performance by 20% or so. One can also not take advantage of some of the performance improvements that Apple Silicon provides in the area of GPU and other custom hardware.

    I agree it would be great to see native support for Linux on ARM and Apple Silicon (based on ARM) with some support for the extended Architecture that Apple is providing. This would allow data Scientists to work on individual workstation training models on local hardware and then promote their work with fewer hassles into production.

    Now that it has been more than two years since this initial product idea. How are others using Apple Silicon as part of your teams workflows?

  • abdagon
    abdagon Registered Posts: 2
    Options

    Thanks for the kind welcome message, @tgb417
    !

    With a PhD in data management in the 80s, I am very keen to use the DSS features .

    I am well aware of the emulations (my first Apple Computer was a Mac II based on M680X0, then a PowerPC Mac, then Intel based notebooks and now a M1 notebook - I even wrote emulators for Z80 microprocessors a long time back ).

    My request goes towards an ARM64 based hosting without an emulator such as Rosetta for an inexpensive «ever on» platform for prototyping, with a full license once going productive.

    Reading through a few threads of this community I understand that the limitation is not so much about DSS, but more about underlying libraries such as Python, which must run natively. Based on Python information, 3.9 should be ARM64 native, with 3.11 being the most recent official version now. So it looks like it's mostly a question of porting DSS to a recent Python version.

    Any dataiku statement would be much appreciated.

    BTW - quiz question, who knows what ARM means?

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,727 Neuron
    Options

    While I would love to see Dataiku ported to ARM personally I think Dataiku will never port it to run natively on ARM. While there is interest by folks like us which use Apple Silicon Macs as computer and ML enthusiast there is very little penetration of Apple Silicon Macs in the enterprise. And we should understand that the majority of Dataiku licenses are bought by enterprises hence it is expected that these enterprise customers will drive the product roadmap. Though not officially supported DSS does currently run on Apple Silicon Macs in emulation in the same way that it runs on Linux VMs under Windows. I think this capability is enough for consultants, enthusiasts, etc to be able to use DSS for basic capabilities, demoes, etc. Any serious Dataiku user will probably want a more serious platform like a VM in the cloud or an on-prem server.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @abdagon
    ,

    I to when through the Motorola 68000 to Power PC and Power PC to Intel conversions over the years.

    Regarding ARM.

    If I understand correctly. Back in the day it meant:

    Acorn RISC Machine

    Because the British maker of the BBC Micro and the Electron Computers Acorn Computer created the first ARM processor.

    Today it seems to be morphed into.

    Advanced RISC Machine

    And Apparently the folks at ARM don't even use it much as an acronym.


    The value proposition for enterprise is really around the problems that Intel is having at making innovative performant but low power chips. Electricity and cooling cost real money. However, @Turribeach
    . I take your point.

Setup Info
    Tags
      Help me…