CPU and GPU

If you aren’t careful, training can bottleneck on reading data and transferring to GPU! Solutions:

• - Read all data into RAM
• - Use SSD instead of HDD
• - Use multiple CPU threads to prefetch data

The point of deep learning frameworks

• Easily build big computational graphs
• Easily compute gradients in computational graphs
• Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc)

DL frameworks

Pytorch大法好

TensorFlow

First define the graph, and then run it many times.

TOO UGLY!!! Introduces a lot of terms that doesn‘t seem to be important if it is designed right. And the api is not pythonic at all!

use tensorboard to make life easier!

PyTorch

Pytorch大法好

Tensor: ndarray that can do computations on GPU

Variable: node in a computational graph that supports Autograd.

• x.data. Tensor
• x.grad. Variable of gradients with the same size of x.data
• x.grad.data. the Tensor of gradients

nice and clean!

torch.nn package

• already defined layers
• build model on layers

torch.optim

update automatically with various optimization algorithms.

torchvision

pretrained models

Visdom

visualization.

CS231n筆記 Lecture 8, Deep Learning Software