19/06/2022 Tensorflow Environments in conda

I've talked about how to set up a Tensorflow CUDA environment with pip on a Windows machine here. It was not the most straight forward thing. 

Long ago I have suffered the messiness of my dev environment. I do everything on my windows 10 home computer (and an EC2 on Ubuntu if anything needs to run for a while) that over the past 5 years (old computer still strong), I've accumulated 3 versions of python (2.7, 3.5 and a conda3.8) and each of them has countless random packages. Not only they're taling a ton of space (two pytorch plus things I don't even remember) but I have also struggled to pin down what packages were used in each project. What's worse, sometimes a new package doesn't work and the whole dev env fucks up and you'll have to delete everything and install again, and of course, you'll always miss something and jupyter notebook throws you an error and you'll have to fuck off and pip it and come back and, when you're back, you forgot what you were doing.

I have came across conda before, it left me a rather unpleasant impression. I installed the whole Anaconda with the graphic UI (The instinct that always comes to haunt you) and was amazed how packages can be managed with a mouse click. The amazement didn't last long, mostly because my system drive ran out of space, with all my python distros. And it was terribly slow. It really starts to bother me when I had to switch back and forth between vim and Anaconda (I write in vim for small things, not a good habbit but not particularly important to motivate changes), the contrast betweent the snappyness of command line windows and the chunky GUI of Anaconda was annoying me. So I uninstalled it.

And here I am, installing it back because I'll need to install on lab computer and I don't want to mess with base environment...

Process:

conda create -n tf-gpu tensorflow-gpu

   problem:

enter the env and use jupyter notebook within the env, fails to import tensorflow (couldn't find package)

solution:

install nb_conda_kernels and ipykernel, and choose the env from jupyter UI

Problem:

import tensorflow successfully but error 'No module named 'keras''. This was not a problem in the pip installation

thoughts: keras used to be a separate package at times of the original pensorflow, from Tensorflow2 it became a part of tf and you can used tf.keras

solution:

someone mentioned updating tensorflow, the virsion conda installed is 2.6, and the newest is 2.9.1. Althought 2.6 should have the tf.keras i thought updating it might help. Conda doesn't have a newer package and someone on stack overflow suggested doing it on pip. It was a bad choice because your dependencies fucks up again...

I found some other post talking about the versions might have different syntaxes for importing keras (don't really understand why tf.keras wouldn't work) but on that hope I'll install 2.2.0. But no! conda default channel doesn't have it!

Oh well... back to the start... Lemme uninstall all the tensorflow packages and reinstall them again... conda remove tensorflow 

conda install tensorflow-gpu

from tensorflow import keras

And now somehow it works......

Back to the notebook

Now there's a different error... 

ImportError: cannot import name 'dtensor' from 'tensorflow.compat.v2.experimental'

The problem goes on but I need to sleep......