TENSORS

Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model's parameters.

Tensors are similar to NumPy's ndarrays, except that (除了) tensors can run on GPUs or other specialized hardware to accelerate computing. If you're familiar with (熟悉) ndarrays, you'll be right at home with the Tensor API. If not, follow along in this quick API walkthrogh (演練).

import torch
import numpy as np

Tensor Initialization

Tensors can be initialized in various ways. Take a look at the following examples:

Directly from data
Tensors can be created directly from data. The data type is automatically inferred (推斷).

data = [[1, 2],[3, 4]]    # 列表list
x_data = torch.tensor(data)
 

From a NumPy array
Tensors can be created from NumPy arrays (and vice versa (反之亦然), see Bridge with NumPy)

np_array = np.array(data)    # numpy.ndarray
x_np = torch.from_numpy(np_array)

From another tensor
The new tensor retains (保留) the properties (shape, datatype) of the argument (引數) tensor, unless explicitly (明確地，顯式) overridden (重寫).

x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor:\n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")

Out:

Ones Tensor:
 tensor([[1, 1],
        [1, 1]])

Random Tensor:
 tensor([[0.6639, 0.3487],
        [0.4164, 0.6755]])

With random or constant (常數) values
shape is a tuple of tensor dimensions. In the functions below, it determines (決定) the dimensionality of the output tensor.

shape = (2, 3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")

Out:

Random Tensor:
 tensor([[0.8345, 0.4769, 0.7677],
        [0.3828, 0.8929, 0.3772]])

Ones Tensor:
 tensor([[1., 1., 1.],
        [1., 1., 1.]])
Zeros Tensor:
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

Tensor Attributes

Tensor attributes describe their shape, datatype, and the device on which they are stored.

tensor = torch.rand(3, 4)

print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")

Out:

Shape of tensor: torch.Size([3, 4])
Datatype of tensor: torch.float32
Device tensor is tored on: cpu

Tensor Operations (運算)

Over 100 tensor operations, including transposing (轉置), indexing (索引), slicing (切片), mathematical operations, linear algebra (線性代數), random sampling, and more are comprehensively (全面地) described here.
Each of them can be run on the GPU (at typically (通常) higher speeds on a CPU).

# We move our tensor to the GPU if available
if torch.cuda.is_available():
  tensor =  tensor.to('cuda')
  print(f"Device tensor is stored on: {tensor.device}")

Out:

Device tensor is stored on: cuda:0

Try out some of the operations from the list. If you're familiar with the NumPy API, you'll find the Tensor API a breeze (微風) to use (輕而易舉地使用).

Standard numpy-like indexing and slicing

tensor = torch.ones(4, 4)
tensor[:, 1] = 0
print(tensor)

Out:

tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

Join tensors You can use to concatenate a sequence of tensors along a given dimension. See also torch.stack, another tensor joining op that is subtly different form torch.cat.

t1 = torch.cat([tensor, tensor], dim=1)
print(t1)

Out:

tensor([[1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1.]])

Multiplying (乘法) tensors

# This compute the element-wise (逐元素) product
print(f"tensor.mul(tensor) \n {tensor.mul(tensor)} \n")
# Alternative syntax (替代語法)
print(f"tensor * tensor \n {tensor * tensor}")

Out:

tensor.mul(tensor)
 tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

tensor * tensor
 tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

This compute the matrix multiplication (矩陣乘法) between two tensors

print(f"tensor.matmul(tensor.T) \n {tensor.matmul(tensor.T)} \n")
# Alternative syntax:
print(f"tensor @ tensor.T \n {tensor @ tensor.T}")

Out:

tensor.matmul(tensor.T)
 tensor([[3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.]])

tensor @ tensor.T
 tensor([[3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.]])

In-place operations Operations that have a suffix (字尾) are in-place.

print(tensor, "\n")
tensor.add_(5)
print(tensor)

Out:

tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

tensor([[6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.]])

Note: In-place operations save some memory, but can be problematic when computing derivatives (導數) because of an immediate loss of history. Hence, their use is discouraged.

Bridge with NumPy

Tensors on the CPU and NumPy arrays can share their underlying (底層，潛在的) memory locations, and changing one will change the other.

Tensor to Numpy arrary

t = torch.ones(5)
print(f"t: {t}")
n = t.numpy()
print(f"n: {n}")

Out:

t: tensor([1., 1., 1., 1., 1.])
n: [1. 1. 1. 1. 1.]

A change in the tensor reflects (反映) in the NumPy array.

t.add_(1)
print(f"t: {t}")
print(f"n: {n}")

Out:

t: tensor([2., 2., 2., 2., 2.])
n: [2., 2., 2., 2., 2]

NumPy array to Tensor

n = np.ones(5)
t = torch.from_numpy(n)

Changes in the NumPy array reflects in the tensor.

np.add(n, 1, out=n)
print(f"t: {t}")
print(f"n: {n}")

Out:

t: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
n: [2. 2. 2. 2. 2.]