PyTorch Tensors

Tensor Basics

You will be expected to have good understanding of deep learning frameworks such as PyTorch to excel in the course. You need to edit or write PyTorch code for your assignment 1 and also you need to write code for the group project. Although we won't be teaching PyTorch, as we believe you can quickly grasp it based on your understanding of the Python programming language. However, we will be refreshing some important aspects of PyTorch and giving demos wherever possible throughout the term.

What is PyTorch?

PyTorch is a Python-based scientific computing package serving two broad purposes:

A replacement for Numpy to use the power of GPUs and other accelerators
An automatic differentiation library that is useful for implementing neural networks

Instructions on how to Install and Get Started with PyTorch can be found at: PyTorch.org

Tensors

Tensors are a specialised data structure that are very similar to arrays and matrices. In PyTorch, tensors encode the inputs and outputs of a model, as well as the model's parameters. Tensors are similar to Numpy's ndarrays, except that tensors can run on GPUs or other specialised hardware to accelerate computing. Neural network computations are just a bunch of linear algebra operations on tensors, a generalisation of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (or other deep learning frameworks such as TensorFlow, etc. ) is built around tensors.

Tensor Initialization

Tensors can be created directly from data. The data type is automatically inferred.

import torch
import numpy as np
data = [[1,2], [3,4]]
x_data = torch.tensor(data)
print(x_data)

Tensors can be created from Numpy arrays (and vice versa)

import numpy as np
np_array = np.array(data)
print(np_array)
x_data = torch.from_numpy(np_array)
print(x_data)

We can also call torch.tensor() with the optional dtype parameter, which will set the data type. Some useful datatypes are: torch.bool, torch.float, and torch.long

import torch
data = [[1,2], [3,4]]
x_data = torch.tensor(data, dtype=torch.float)
print(x_data)

import torch
data = [[1,2], [3,4]]
x_data = torch.tensor(data, dtype=torch.bool)
print(x_data)

Tensor from a Numpy Array

We can also initialise a tensor from a Numpy array.

import numpy as np
import torch
data = [[1,2], [3,4]]
ndarray = np.array(data)
x_tensor = torch.from_numpy(ndarray)
print(x_tensor)

From existing Tensor

import torch

initial_tensor = torch.tensor([[4., 5.], [6., 7.]])
print(initial_tensor)

# Initialise a new tensor of 1s
new_tensor_ones = torch.ones_like(initial_tensor)
print(new_tensor_ones)

# Initialise a new tensor of 0s
new_tensor_zeros = torch.zeros_like(initial_tensor)
print(new_tensor_zeros)

# tensor elements sampled from uniform distribution between 0 and 1
new_tensor_rand = torch.rand_like(initial_tensor)
print(new_tensor_rand)

# tensor elements sampled from a standard normal distribution
new_tensor_randn = torch.randn_like(initial_tensor)
print(new_tensor_randn)

Instantiating tensors by specifying their shapes

There are many ready methods where we can instantiate tensors by specifying their shapes, namely, torch.zeros(), torch.ones(), torch.rand(), and torch.randn()

import torch
shape = (2, 3, 4)

x_zeros = torch.zeros(shape)
print(x_zeros)

x_ones = torch.ones(shape)
print(x_ones)

torch.manual_seed(142)
random = torch.rand(shape)
print(random)

You can also create a tensor with torch.arange(start, end, step, *), which returns a 1-D tensor with elements from 0 to end-1.

import torch
x = torch.arange(start=0, end=10, step=2)
print(x)

Tensor Shape

You can use dtype to check the data type of a tensor.

import torch
x = torch.rand(3,2)
print(x.dtype)

The shape property can be used to check the shape of a tensor. It helps to identify dimensions of a tensor.

import torch 
x = torch.Tensor([[11, 12], [13, 14], [15, 16]])
print(x.shape)

We can also check the size of a particular dimension of a tensor using size() method.

import torch 
x = torch.Tensor([[11, 12], [13, 14], [15, 16]])
print(x.shape)
print(x.size(0))

We can also change the shape of a tensor with the view() method.

import torch 
x = torch.Tensor([[11, 12], [13, 14], [15, 16]])
print(x.shape)

# we can change the shape from (3,2) to (2,3)
x_view = x.view(2,3)
print(x_view)

# You can also use torch.reshape() for this
x_reshaped = torch.reshape(x, (2,3))
print(x_reshaped)

Device

Torch uses device() property to store tensors. Using the device() property,
we can determine which device a tensor is stored on: CPU or GPU.

import torch
x = torch.Tensor([[7, 8], [9, 10], [15, 17]])
print(x)

# Determine on which device tensor is stored
print(x.device)

# Check if a GPU is available; if so, move the tensor to the GPU
print(torch.cuda.is_available())

if torch.cuda.is_available():
    x.to('cuda')

Tensor Operations

We can perform many operations with tensors, similar to Numpy.

import torch

# creating a tensor
x = torch.ones(2, 3, 4)
print(x)

# Element-wise addition
print(x + 3)

# Element-wise subtraction
print(x - 0.5)

# Element-wise multiplication
import torch

# creating a tensor
x = torch.ones(2, 3, 4)
print(x)

print(x * 2)

# Element-wise division
import torch

# creating a tensor
x = torch.ones(2, 3, 4)
print(x)

print(x / 2)

We can use torch.cat() to concatenate tensors.

import torch

# creating a tensor
x = torch.ones(2, 2, 4)
print(x)

y = torch.ones(2, 2, 4)
print(y)

z_cat = torch.cat([x, y], dim=0)
print(z_cat)

import torch

# creating a tensor
x = torch.ones(2, 2, 4)
print(x)

y = torch.ones(2, 2, 4)
print(y)

z_cat = torch.cat([x, y], dim=1)
print(z_cat)

Arithmetic Operations

We can apply various arithmetic operations on tensors, similar to Numpy operations.

import torch
x = torch.tensor([1, 2, 3], dtype=torch.float)
y = torch.tensor([4, 5, 6], dtype=torch.float)
print(x+y)

# you can also use torch.add()
print(torch.add(x,y))

# You can also do complex operations on tensors
import torch
x = torch.tensor([1, 2, 3], dtype=torch.float)
y = torch.tensor([4, 5, 6], dtype=torch.float)

print(torch.add(x,y))
print(torch.add(x,y).sum())

Dot Product and Matrix Multiplication

import torch
x = torch.tensor([1, 2, 3], dtype=torch.float)
y = torch.tensor([4, 5, 6], dtype=torch.float)
# Dot product
print(x.dot(y))

import torch
x = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float)
y = torch.tensor([[1, 2], [3, 4], [5, 6]], dtype=torch.float)

# Matrix Multiplication
print(torch.matmul(x, y))

# you can also use torch.mm() 
print(torch.mm(x, y))

# or you can use x@y
print(x@y)