View All Pages

Computer Tools for Learning and Knowledge

Installing some new tools

We will describe how to install the tools needed for this module in the virtual machine (Ubuntu 16.04) we used in the course WASP Autonomous Systems 1. You find the description of this virtual machine here.

We will install the following software tools

If you have good computer knowledge and prefer not to use the virtual machine and instead want to run things on your native os, feel free to do so. But you are then on your own. Use the discussion forums if you run into problems.

Be mentally prepared to run into some software issues that might require some work to resolve. We have tried to prepared things for you, but . If a toolbox is missing, it can probably be found and installed easily

Installation instructions:

Start VirtualBox and start the virtual machine WASP Ubuntu 1604.

We recommend that you install your packages instead a virtualenv. If you took the AS1 course you probably already did this

virtualenv -p python2 ~/tensorflowenv
source ~/tensorflowenv/bin/activate

You need to activate the environment using the command below. If you forget this you will get error messages about missing files/programs.

source ~/tensorflowenv/bin/activate

Now install the packages that we will use

# check what versions of toolboxes you have with the command 
# pip list
pip install --upgrade keras # tested with 2.2.2 
pip install scikit-learn # tested with 0.19.2
pip install pandas
pip install --upgrade tensorflow  # tested with 1.10.1
pip install matplotlib
pip install seaborn 
pip install jupyter


We will use jupyter notebooks also in this module. The following might resolve some issue with starting some versions of jupyter

pip uninstall tornado
pip install tornado==4.5.3

Jupyter Notebooks

We will use some of the python notebooks and data that go with the book of Geron, Machine Learning and Deep Learning in python using Scikit-Learn and TensorFlow. Links to an external site.  On github you find these data and notebooks Links to an external site. which you can clone by 

cd ~/tensorflowenv/
git clone https://github.com/ageron/handson-ml.git

We will also make use of the notebooks available on github Links to an external site. that goes with the book Deep Learning with Python, by Chollet Links to an external site. 

cd ~/tensorflowenv/
git clone https://github.com/fchollet/deep-learning-with-python-notebooks.git

For the second assignment you need the following files

cd ~/tensorflowenv/
git clone https://github.com/pjensfelt/wasp_as2_m1a2.git
    

Links to an external site.

GPU resources

If you have access to a machine with GPU(s), it is highly recommended that you try to make use of it. It will speed things up quite a lot and could come in handy later in your research. You need an NVidia GPU and you need to install the graphics drivers for the card, and the Cuda and cuDNN libraries from NVidia. Note that you want version 9.0 of Cuda (and not 9.2 as it will not work with the current version of tensorflow). To download cuDNN you need to register as an NVidia developer.

https://developer.nvidia.com/cuda-90-download-archive Links to an external site.

https://developer.nvidia.com/rdp/cudnn-download Links to an external site.

 

Setup on an Ubuntu 16.04 machine

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda-9.0

Now download the cuDNN libraries from from https://developer.nvidia.com/cudnn Links to an external site.

You want to get the files

Then install these with

sudo dpkg -I libcudnn7*

You should be all set now

 

Running Jupyter on a remote machine

If you, for example, work on a laptop but would like to run your code on a desktop with some nice GPUs you can do this by remote access over ssh.

This page describes it well https://www.blopig.com/blog/2018/03/running-jupyter-notebook-on-a-remote-server-via-ssh/ Links to an external site.

Make sure that you have access over ssh to the machine you want to use and setup the environment there, ie all the NVidia stuff to be able to use the GPU and all the jupyter, tensorflow, keras, an what not that you want to work with.

In this example I will assume that you want to work with the assignment code from the second assignment.

On the remote machine

Either open a terminal directly or access it remotely (that was after all why we did this)

cd ~/tensorflowenv/wasp_as2_m1a2
jupyter notebook password
jupyter notebook --port=9000 --no-browser &

In the second step you will be promoted to select a password that you will have to give when you connect from your local machine. The third step will start jupyter on a different port (9000) than the detault (8888) to allow access from outside.

On the local machine

ssh -N -f -L 8888:localhost:9000 your_login@your_remote_machine_address

This created an ssh tunnel from port 8888 on our local machine to port 9000 on the remote machine where jupyter is ready to serve if you started it. Now open a browser and direct it to http://localhost:8888  Links to an external site.as you would if you had ran it locally. Note that the files resides on the remote server side.