It's time for a new Laptop. (What would you do?)

tgb417 · May 2020

It's time for me to get a new Laptop.

I've been using a Macintosh laptop for a number of years now and generally like the system. The installation of DSS community edition is fairly straightforward on a Mac.

That said I'm open to a new configuration. And ready to go back and even look at the basic question Mac vs PC (or Linux?) I know that in the past the suggestion from Dataiku for DSS when it came to a PC was to use Linux on a virtual box. (This did not work for me when I tried it last back in 2017.)

I just found this post from @Alex_Reutter
about using Windows 10 WSL (Windows Sub System for Linux). It's been a while since this post was originally made. If you are around @Alex_Reutter
I would love to hear what your current experience is. Is WLS a viable alternative to a Macintosh for DSS on a MS Windows PC? Hows the performance?

Now to the question of what I'm looking for from my new computer:

Needs to work with DSS
It needs to be a laptop
I want to start to work with GPU accelerated Neural Networks on or at least from the new coputer.
I'm currently using online services to make my artwork. This work uses Neural Networks that require GPU power. I would like to move to my own resources in a laptop form factor.

It appears that I don't have the ability to get a Cuda / Pytorch supported GPU on a Macintosh. (Thanks Apple & Nvidia.) Where something like the new Surface Book 3 provides an Nvidia GPU built-in. However, it is generally designed to run Windows.

I'm wondering if anyone has any experience with WSL 2 and DSS on MS Windows 10. I've also noted a presentation at MS Build 2020 about the forthcoming GPU support for WSL 2.

Or do folks have any ideas about how to get generally unlimited access to GPU without lots of incremental costs from a Macintosh?

Thanks for any thoughts you might share.

Omar · May 2020

Hi Tom,

this is somehow a very peculiar question, and we as Dataikers can't really stand for one faction or another, so we'll leave this to the users community. It will be fun to see what users have to say.

From a very personal point of view, I'm sad to see you don't mention linux option in your post. Is there any particular reason ? If it's because you're not familiar, fear not: nowadays linux desktop distributions are very user-friendly (I'm a mac-borrowed linux user myself).

The main issues with linux laptop has always been:

Portability: Linux laptops are usually thicker and heavier than windows laptops, or you have to compromise on something;
Compatibility: Sometimes it's harder if not impossible to find the same tool you've been using since years in Windows, packaged in a linux version. However, the vast majority of times you can find an alternative that might turn out to be better. It's good to see that nowadays even Microsoft is embracing linux. At last, for the things you can't tame, you can always use a vm (or Wine if you're brave enough);
Ease of use: I can really say this is in the past now: look for instance to the new versions of Ubuntu and all the derivates, ElementaryOS in particular. Furthermore, KDE provides a very good experience, and it's so customisable. It's one of my preferred desktop environments.

When it comes to the kind of user community DSS is relevant to, Ubuntu generally is a very good move because you will sure find a python package compiled for it, with all the libraries required. From this point of view, linux really is many steps ahead of Windows. Did I mention that the majority of times, if your machine fails you can take your HDD and mount it to a new linux laptop with similar characteristics and it will just work, or at least let you retrieve your data (restrictions applies, ofc)? Try that with Windows.

Finally, in your specific case (willing to use GPU), the hardest part is to find a laptop with a decent graphics card supported by the OS. There are a lot of super cool and slick laptops with amazing hardware out there, but the vendor just doesn't invest in linux support, so you're on your own there. It might work just fine, it might not (I have a lot of experience on this).

Luckily enough, there are vendors like System76 that build interesting machines from the hardware point of view (they are just not cool and slick as other big vendors, but they work just fine) along with an OS that is tailored for them and provides support to the GPU they carry onboard. As with the very best open-source mentality, you can actually use their OS (for free) on another laptop you might prefer, you'll still leverage the GPU card (provided it's the same kind, ofc). I haven't tried System76 machines myself yet, but the internet is full of happy users.

Take care,

Omar
Architect @ Dataiku

tgb417 · May 2020

@Omar

Excellent point Linux.

In many ways, I’m actually OS agnostic. At various times I’ve had three or more computers at my desk. A PC, a Macintosh, and often a SUN SPARC Stations. For many years I ran Ubuntu Linux with PC VMs running on top because I need to support PC, but want to have my usual terminal commands when I need or want them. I’ve used KDE, gnome and I’m sure a few other window managers over the years. Based on my experience with Linux host at my desk. Running a PC VM I found that I spent most of my time in Windows. In general, I found the window managers and applications for Linux less refined. And my primary work was in support the mainline PC Applications.

I’ve not tried ElementryOS. I may take a look. How smoothly does DSS install in that environment?

Yes at least historically Windows has been a challenge with anything Unix like. They do over the last few years have this Terminal that will run at least a Microsoft version of a bash or csh shell and WSL. However, I suspect that there may be compromises there.

One of the reasons I’ve liked Macintosh in the past is because the Mach kernel and Unix like features of the os. Historically for me, the Macintosh computers have allowed me to run MS Windows in a VM or on the base hardware if the full resources of the computer are needed and I was willing to waste a reboot. Linux could also be run in a VM but with a significant performance hit.

Unfortunately, for many years one could not run a Mac OS on a PC. I might consider a Hackintosh as a VM. But again I hear there are a bunch of compromises with less than fully functional results on the Macintosh side.

VMs yes they are useful to me. The time I have tried to use WINE I found it to be mostly sour grapes.

I’ll take a look at System76. Thanks for the pointer.

tgb417 · May 2020

what are others using?

As DSS users you have each likely had to think through these multiple OS challenges. What are your current personal laptop computer conclusions: PC, Macintosh, Linux,

If it is not Linux on the laptop where and how are you running DSS? desktop computer or server at home or work or in the cloud, raspberry pi cluster?

If DSS is running in Linux on the laptop is Linux the base os of the computer? if so which computer do you like? If not is DSS running on a VM if so set up in what way?

thanks to everyone for your thoughts.

Alex_Reutter · May 2020

Hi @tgb417
, I was mostly playing with WSL to see if I could get a basic install of Dataiku up and running on a Windows machine b/c I was having difficulty with Virtualbox. I didn't do any real work on that instance, so I can't comment on its viability.

Right now, I have DSS running on my MacBook (using the osx tar, not the dmg) as well as on some Linux machines. The experience is very similar, but if I were choosing a new laptop primarily for working with DSS, I would be hesitant to choose anything that wasn't running Linux b/c while my experience with DSS on MacOs has been very good, DSS is not native to it.

tgb417 · May 2020

@Alex_Reutter
what additional benefits are you getting from using the tarball version of DSS on your Macintosh? If I get a new mac. I'm inclined to re-install from scratch. Also on a related point how do you choose to install your python. Anaconda, HomeBrew, something else?

Finally, In reading and thinking about this thread, If I were to pick a laptop primarily for DSS. Then, I think that one might look at one of the high end "gaming laptop" with an Nvidia GPU, However, the size of most of these remind me of the laptops from the late 1990s or very early 2000s. Today these laptops are really more of a "desktop replacement" computer than a "walk around to the coffee shop everyday laptop.

I'm really looking for something smaller, and I'm coming to believe that I'm going to for now have to rent appropriate hardware from one of the hardware/platform as a service providers like Amazon, Azure, PaperSpace, maybe others.

Clément_Stenac · May 2020

Hi,

Slick design + NVidia GPU + Linux-friendly (because that's the only real way to leverage GPUs from Python+DSS) is indeed a complicated equation to solve.

If travel-friendliness isn't an absolute requirement, renting GPUs and settling for a lighter/slicker laptop is probably your best bet.

GCase · May 2020

Windows User / VMWare Linux user here

A couple of notes: the GPU is hard requirement as DSS is native to Linux and at this time there is no way to do GPU passthrough from Windows to a Linux host. If you are comfortable in Linux and are OK with not having all the power-saving features, you could go with something like what I'm using a Lenovo P53 with either Ubuntu, CentOS or Fedora as your primary OS and perhaps run a VM with anything else.

For overall ease, its tough to beat the MacOS, but you get lower specs at a higher price. That said, your GPU setup should be much easier.

There is one other potential option, but it's some time away. Microsoft has been working with Ubuntu on WSL 2 and recently committed to bringing GPUs to WSL through a kernel driver allowing a host Windows system to have a client Linux instance instantiate and use the host Windows GPU. To me, this is a game-changer. Microsoft has said this will be at least "a few months." I wouldn't get my hopes too high and likely the first iterations of this will likely need some kinks ironed out. That said, the #1 WSL request has been GPU support so Microsoft definitely has reasons to get this done as this has been a major stumbling block for more people to use Windows as their primary development platform.

Finally, you could take Clement's advice. If your GPU needs are fairly inconsistent and transitory then you could grab something fun and sleek and just rent GPU time as needed. Hope this helps!

https://ubuntu.com/blog/new-gpu-and-gui-features-announced-for-wsl-at-build

Alex_Reutter · May 2020

Hi @tgb417
, the tarball gives me more flexibility for setting up a DSS instance on my mac.

It's been a little while since I installed python. IIRC, 2.7 came installed on the macbook. For a little while, I used Anaconda to manage a python3 installation on my macbook, but eventually ran into a situation I couldn't resolve, and ended up tearing it out and reinstalling python3 with homebrew...

tgb417 · May 2020

@Alex_Reutter
, Interesting... Yeah, I think that I'm now getting to the same place with the Anaconda Navigator software on my Mac. It has been a lovely set of "training wheels" for me. It resolves many of the library dependencies, compiling, and sourcing challenges. But there are some configurations that Anaconda just does not seem to be able to resolve. There are recent libraries that I want to try that it does not have. And if I go and install those libraries with PIP or install.package(), then the Anaconda dependency resolver seems to get in trouble. Anaconda does provide me some other packages. For example, things like QGIS can be installed. But in my experience so far it is not a replacement for an apt-get type Linux package installer. I'm hearing and have experienced Anaconda and homebrew not playing nicely together. These are part of the reason that if I go the Macintosh direction I'm considering reinstalling from scratch and dropping Anaconda and for HomeBrew.

What non-dmg flexibilities are you actually using with the Tarball? Are you moving the location of the DSS_HOME out of ~/Library/DataSienceStudio/dss_home? If so where? Are there other flexibilities that you value?

tgb417 · May 2020

@GCase

Thanks for your response. Your comments are super helpful.

I too heard at Microsoft Build (5/19/2020) that the folks working on WSL 2 are working on a GPU pass-through "WSL will support GPU Compute workflows" I believe that they showed a working demo. This looks promising for the kind of work I'm considering. I agree that it would be a game-changer.

Good suggestion, the Lenovo P53 has a bunch of things to be liked. I see that the 8th Generation Intel® Core™ i7 Processors is a 10-25 watt chip. Are you aware of any significant thermal throttling or crazy loud fans with style ML loads? Sounds like you have not been able to find drivers to support all of the power management features that the hardware seems to provide.

You suggested that the Macbook GPU setup should be much easier. I agree that Mac OS apps this will be true.

I seem to have found a way in principle to get Neural Network loads to work on the Macintosh GPU.

During my research, I have found PlaidML also discussed in this post for Macintosh and installation instruction here. In the install instruction, it seems to indicate that support for OpenCL 1.2 is required. I'm also seeing Apple going away from OpenCL in favor of their own Metal. And on their list of OpenCL compliment computers, I'm not seeing the latest generation of devices.

Am I going down the right path with PlaidML as a way to get GPU access for ML load on Mac? Is there an easier way to do this I'd be very interested to know anyone's thoughts?

--Tom

GreaseMonkey · June 2020

Late to the party here. I've been running LinuxMint as my primary desk/laptop systems since Mint 18. Prior to that I ran Mandrake and/or PCLinuxOS. All worked pretty well, but LinuxMint has been rock solid since 2017 for me.

YMMV.

tgb417 · June 2020

@GreaseMonkey
Thanks for the reply.

On what type of hardware have you been using LinuxMint and DSS?

I'm thinking about what to replace my 2013 13-inch Macbook Air. (1.7Ghz. i7, 8GB RAM, 512GB SSD) Recently it has been working hard to keep up.

GreaseMonkey · June 2020

I run a desktop system at home that I built: ASRock X99 Extreme4 mobo w/ a cheapo NVidia Card.

I also used to have an HP Omen laptop that ran really well. It only had 12G RAM, but it did great. Touchpad, 4k screen, wireless, sound and everything on it worked out of the box.

System 76 has some decent looking hardware and their own Linux distro.

good luck.

tgb417 · June 2020

Lenovo has just announced Linux® Certification to ThinkPad and ThinkStation Workstation

https://news.lenovo.com/pressroom/press-releases/lenovo-brings-linux-certification-to-thinkpad-and-thinkstation-workstation-portfolio-easing-deployment-for-developers-data-scientists/

tim-wright · June 2020

Hey Tom.

Did you wind up figuring this out?

I had been a Windows user for years (primarily because that is what my IT team supports) and had been using WSL for a while. Pretty straightforward to get DSS set up on WSL (that's where I started). I've also had some experience with OSX and Linux (ubuntu specifically) a few years back.

I would also encourage you to try a linux distro. For the past few months I have switched back to Ubuntu 18.04 on my personal desktop and have become quite fond of it . I have also read recently that Ubuntu 20.04 LTS is a big upgrade over 18.04. I will say, I do miss the some of the integration with the Microsoft ecosystem (Teams/OneDrive/etc/) that are required for collaborating with my company - but most of them are available through web clients anyways. Other than that I'm having a blast.

The primary reason I decided to hop back to linux is that I like to tinker with a lot of open source tools and in my experience running them on WSL or Docker for Windows was that they would "generally" work but every now and then I'd run into environment/setup issues that become a pain. When self-learning those tools there is also vastly amount more content on the interwebs for Linux than Windows.

This was also the case as I was doing some GPU accelerated deep learning with Tensorflow locally. If your GPU learning is specific to DSS it sounds like Linux may be the way to go. If you are interested in GPU accelerated learning just for fun (outside of DSS) Google Colab offers free notebook enviroments with access to a GPU. If it

Best of luck in your search. Let us know what you decide. (and at least the crypto craze has passed and you can actually get a GPU for a reasonably normal price)

tgb417 · June 2020

@tim-wright
,

I'm leaning toward a 13-inch 2020 Macbook Pro (4 USB C port model). Because it just works as a daily driver and as a family, we are invested in the Apple ecosystem.

I hear you about the library issues not working well on Windows or Mac OS. I'm starting to run into this on my older MacBook Air.

That said I will almost certainly make a new Macbook Pro at least dual boot Windows with Bootcamp, and use that partition with either Parallels or VMware Fusion. (I've been doing this for years. That will give me either emulated hardware or non emulated direct hardware access to MS Windows 10 with WSL2 and Terminal.)

I've also considered trying to set up a triple boot with Linux as the third OS. (That would increase the disk space requirements for the overall configuration.) I'm not clear if Parallels or VMware Fusion can book a Linux Partition on a Mac in the same way that they can boot a BootCamp Partition. (I don't really want to use one of the Virtualization Tools disk images, because I can't boot the computer just to the Linux OS from one of those.)

This setup would allow me to "hop onto either Windows or Mac OS" and maybe even one of the Linux Distros.

The real problem with that Mac as a choice is that I do not get Nvidia GPU, and of course the Apple Tax. Apple only seems to want to support AMD right now. (There are some hacks that "Might" work with eGPUs or PlaidML I'm not seeing very clear evidence that folks are having good success.) Best I can tell the state of the Art DL is only being done with NVidea GPUs. AMD and Apple don't see to have good answers in this space for developers.

Going with a Non-Apple Intel-based laptop gives me at best an "it sort of works" Hackintosh type solution for the Double Boot or Triple Boot Gaming laptop. That said I'd like the daily driver part of the computer to be Solid and well-integrated into the rest of my life.

So, If I go with the Macbook Pro, I may go to someone like PaperSpace and rent ML / DL optimized access to NVidea GPUs. I'm aware of Google Colab as well. One of these will work with fast.ai.

I've also considered doing a single board computer from NVidea called the NVIDIA® Jetson Xavier NX™ Developer Kit. Use Docker or Kubernetes to load jobs to that GPU from my laptop for overnight type runs on NVIDIA hardware. If this works well. Then create a bit of a cluster out of several devices like this.

Finally, I'm planning to attend/listen to WWDC 2020 coming up in a few weeks before making a final decision. There are some rumors that Apple will be switching from Intel Processors to ARM platform in the coming years. I'm also interested in their solutions when it comes to DL compute. If I'm going to hear anything useful about this in the Apple ecosystem, WWDC will be the place.

Thanks, I'll reach out when I get this all resolved.

--Tom

tgb417 · June 2020

With Apple's announcements today. (They are going away from Intel Processors. To their own ARM-based processors.) Hmmmmm. Now, where do I go?...

I feel myself less inclined to drop major bucks on a high-end Macbook Pro 16. Because in 6 months the architecture will be different. And in 2.5 years the current computer will be moving on toward obsolescence.

There is a piece of me thinking about jumping to the bleeding edge and get a copy of their developers Mac Mini. I have to see what strings are attached and the price.

--

tgb417 · November 2020

I know that DSS does not yet support Apple ARM architecture. However the new Neural Network apparently supports the ML cores of the M1 chip. Including Tensor flow. I find that an interesting outcome.

tgb417 · February 2021

A quick update.

For now, I've postponed getting a new laptop. For now, I'll use my old i7 Core Duo Macbook Air. It's Ok-ish for mobile activity.

Although M1 Mac's looks super interesting, most of the Data Science Software is still getting its act together regarding compatibility with Apple ARM chips.

For now, I've gotten one of the older Intel Mac Minis Refurbished from Apple. This meets my short term needs. And over the long run will make a good home server.

Today I discovered the Kubuntu Focus M2 it appears to be an interesting Data Science Laptop with NVidea RTX 2060, 2070, or RTX 2080S. Looks sort of interesting. Is there anyone else out there using this computer?

https://kfocus.org/wf/deep.html

tgb417 · April 2021

@Alex_Reutter
,

So I have my new Mac Mini. I'm happy with Homebrew as the App Installer.

However, I'm not clear about the best way to get Python 3.6 installed in a way that DSS can use in a clean way.

Homebrew seems to be installing 3.9.x right now. And there seem to be questions about how to get an older version of Python 3.6 with Homebrew.

Right now I can run Python 3.9 from the terminal as python3.

But I can not install any code environments in DSS that are version 3.x. With or without conda. The latest version that seems to be supported in the drop-down menu below is 3.6. I've got a feeling I'm missing something. When I try to set up a v 3.6 python in the following manner.

Install Code Environment.jpg

I get the following error

Install Code Environment Error.jpg

I'd agree with this error message I've not installed python 3.6 on this computer.

Can DSS use the Python 3.9 already installed on the computer and being maintained by Homebrew? It does not show up in the dropdown menu. If not what would be the best way to install a supported 3.6 python. Should I run one of the "magic" Dataiku installation scripts?

I currently have the standard Mac Installer for DSS 8.0.5 installed on this computer as a Homebrew Cask. I did notice that the Mac Installer for 9.0.0 installed a version of Python 3.6 as part of its bundle. Is it best to just upgrade to DSS 9.0 manually rather than waiting for the Homebrew Cask upgrade? I could try to install the latest version of Python 3.6 available on python.org.

Lots of different options here. Just wondering what experience folks have and what is likely the cleanest way to set up a newish computer.

Thoughts?

Alex_Reutter · April 2021

Hi @tgb417
, DSS supports up to Python 3.7 now. I use pyenv to manage multiple versions of Python on my mac.

tgb417 · April 2021

@Alex_Reutter
,

Thanks for pointing me toward pyenv. I ended up using this description here to set up pyenv with homebrew. We will see if this get me setup OK with DSS.

More to follow.

tgb417 · April 2021

@Alex_Reutter
,

After completing what looks like a successful pyenv install. And getting the latest version of python 3.6 installed (python 3.6.13)

When I go to the Terminal and enter the following commands. I get the results below

Mac-mini ~ % which python
/Users/[user_name]/.pyenv/shims/python

Mac-mini ~ % python --version
Python 3.6.13

Note: the above is altered [user_name] is a replacement in the above post of my actual user name.

So Python looks to be installed ok. And things like qgis installed by homebrew which use python 3.9 seem to be still running OK.

However, DSS even after a re-start is still not able to find Python. I'm getting the same error as before.

Python 3.6 not found.jpg

Do I have to run a re-install script or something for DSS to find the python versions I've installed after I installed DSS?

Alex_Reutter · April 2021

Hi @tgb417
, I think what you want is https://doc.dataiku.com/dss/latest/installation/custom/python.html#rebuilding-the-builtin-python-environment , in order to change the underlying Python subsystem.

tgb417 · April 2021

@Alex_Reutter
,

having to go through:

# Stop DSS
DATA_DIR/bin/dss stop
# Save the list of locally-installed packages
DATA_DIR/bin/pip freeze -l >dss-local-packages.txt
# Remove the virtualenv, keeping backup
mv DATA_DIR/pyenv DATA_DIR/pyenv.backup
# Reinstall DSS (upgrade mode), choosing the underlying base Python to use
dataiku-dss-VERSION/installer.sh -d DATA_DIR -u [-P BASE_PYTHON]
# Review and possibly edit the list of locally-installed packages
vi dss-local-packages.txt
# Reinstall local packages
DATA_DIR/bin/pip install -r dss-local-packages.txt
# Start DSS
DATA_DIR/bin/dss start
# When everything is considered stable, remove the backup
rm -rf DATA_DIR/pyenv.backup

Feels like I did something wrong. I’m not clear where I might have messed up. Any idea how to do a home brew python and DSS install that just works from the start?

It’s getting late here so I not going to do anything more on this tonight.

tgb417 · May 2021

@Alex_Reutter
,

It looks like I've successfully re-installed my python.

Details here.

Turribeach · May 2021

Check this post out:

https://community.dataiku.com/t5/Setup-Configuration/DSS-installed-on-Mac-M1/m-p/15852#M1549

Doesn't solve your GPU requirement since nothing on ML it;s going to run on Metal but could be a good option for general Dataiku use.

tgb417 · October 2021

A Further update.

My new organization gave me a windows laptop. It, fortunately, has a moderate amount of RAM at 24 GB. As a design node, I've installed the community (free) edition of DSS on Windows 11 in Windows Subsystem for Linux VS (WSL2). With the latest patches on Ubunto 18. Although the laptop is underpowered with poor thermal management and a CPU with a very low Total Maximum power. DSS does run surprisingly well. (Note this is not an officially supported configuration and it is not a configuration for a first-time DSS user or someone with little or no Linux experience). However, it does run.

Still looking forward to seeing if Dataiku can support DSS on the new M1 Pro or Max. Particularly if I can get Python to support those GPUs so that I can do my own AI based art.

Turribeach · October 2021

Thanks for the update. Would you mind creating another community thread on how you got Dataiku running under WSL2? We will be moving to new machines at work too and I will be interested to run Dataiku locally in WSL2. Also the latest Windows 10 version (19044.1288) now has GPU support under WSL2 (see here).

I ordered a new MacBook Pro with an M1 Max so I am also interested in running Dataiku under that, specially since I maxed the CPU and RAM (I want to do some video editing too). I doubt Dataiku will be interested in porting their code to ARM, too much work for very little reward since no one in the enterprise it's going to move to ARM any time soon.

The enthusiast market is a different story now. What are geeks going to buy now when they want a good workhorse laptop that can run Windows apps? With Intel Macs you had Boot Camp and Parallels so Windows apps were covered. Parallels can now run on Apple Silicon for both Windows and Linux VMs and Windows 10 ARM edition even emulates x86 64bits apps but not everything works under this emulation. Maybe Parallels could build an x86 emulation layer to have a native Linux x86 VM running on Apple Silicon? Who knows...

tgb417 · October 2021

@Turribeach
,

I'm not clear that I have the time to reproduce, test and write up a whole post on the installation of DSS under WSL2.

However some of the key points.

Based on this set of posts I used WSL2 and Ubuntu 18.
- The Ubuntu was from the MS App Store
This is a Linux install. So DSS instructions like this are helpful.
- Or these are more complete

As with all Linux making sure that you are patched is helpful

 sudo apt update        # Fetches the list of available updates
sudo apt upgrade       # Installs some updates; does not remove packages

Limitations so far:

I have not worked out auto startup when wsl is booted. I'm manually issuing the PostgreSQL startup commands and the DSS startup commands.

I have not yet attempted the R integration or GPU Integration.

I'd like to invite you to start a new thread about your adv

It's time for a new Laptop. (What would you do?)

Answers

Categories

Setup Info

Tags