From PyTorch 1.4 tutorial

Outline

• Tensor
• torch.autograd.backward
  • If the result node is scalar
  • If the result node is a vector

[PyTorch] Note 02: Autograd auto derivation

In PyTorch, the core of all neural networks is autograd package

1 Tensor

• torch.Tensor is the core class of this autograd

• A Tensor tensor usually records the following properties:

  • Data: stored data information
  • requires_grad: if it is set to True, it means that the Tensor needs derivation
  • grad: the gradient value of this Tensor needs to be reset to zero every time when calculating the backward, otherwise the gradient value will be accumulated all the time
  • grad_fn: the leaf node (i.e. independent variable) is usually None, and only the grad of the result node (i.e. dependent variable)_ FN is valid to indicate which type of gradient function is.
  • is_leaf: used to indicate whether the Tensor is a leaf node.

See Detailed explanation of Python autograd, backward

• requires_grad=True
  • Track all operations of the corresponding tensor
  • Default requirements_ Grad is flame

    a = torch.randn(2, 2)
    a = ((a * 3) / (a - 1))
    print(a.requires_grad)
    a.requires_grad_(True)
    print(a.requires_grad)

    #output
    False
    True

• . detach() method
  • Prevent the tensor from being tracked
  • Call The detach() method separates it from the calculation history and prevents its future calculation records from being tracked
• with torch.no_grad():
  • To prevent tracing history (and memory usage), code blocks can be wrapped in with torch no_ Grad (): medium
• grad_fn
  • Only dependent variables have properties

    x=torch.tensor(1.0,requires_grad=True)
    y=torch.tensor(2.0,requires_grad=True)
    z=x+y
    print(x,y,z)

    #output
    tensor(1., requires_grad=True) tensor(2., requires_grad=True) tensor(3., grad_fn=<AddBackward0>)

2 torch.autograd.backward

Source code interface

    torch.autograd.backward(
                      tensors, 
                      grad_tensors=None, 
                      retain_graph=None, 
                      create_graph=False)

• Parameter meaning
  • Tensor: the tensor used to calculate the gradient
    torch.autograd.backward(tensor) and tensor Backward() is written equivalently
  • grad_tensors: used when calculating the gradient of the matrix. In fact, it is also a tensor. Generally, the shape needs to be consistent with the previous tensor
  • retain_graph: usually, after calling backward once, pytorch will automatically destroy the calculation graph. Therefore, if you want to call backward repeatedly for a variable, you need to set this parameter to True
  • create_graph: when set to True, it can be used to calculate higher-order gradients

2.1 if the result node is scalar

• Scalar can be understood as one-dimensional
• Just apply backward directly
• Reference examples are as follows:

    x=torch.tensor(3.0,requires_grad=True)
    y=torch.tensor(7.0,requires_grad=True)
    z=x+y
    z.backward()
    #Returns the gradient of x,y, and the value of z
    print(x.grad,y.grad,z)

    #output
    tensor(1.) tensor(1.) tensor(10., grad_fn=<AddBackward0>)

2.2 if the result is vector

• Vector can be understood as high-dimensional and multi-dimensional
• Referring to the Chinese documents of pytorch and Zhihu's articles, I feel that Zhihu's articles are better understood, but they are essentially the same, that is, a tensor is introduced that is the same as the previous tensor tensor
• Reference examples are as follows:

    x=torch.ones(4,requires_grad=True)   #x=[x1,x2,x3,x4]
    #print(x.type())      torch.FloatTensor
    z=x+2   #z=[x1+2,x2+2,x3+2,x4+2]
    #If all incoming are 1
    z.backward(torch.ones_like(z))
    #If the incoming is set by yourself
    #z.backward(torch.Tensor([1,2,3,4]))  #z=[x1+2,2(x2+2),3(x3+2),4(x4+2)] note the matching of types. Tensor defaults to torch Floattensor type
    #z.backward(torch.tensor([1.,2.,3.,4.]))  #z=[x1+2,2(x2+2),3(x3+2),4(x4+2)]
    print(x.grad)

    #output
    tensor([1., 1., 1., 1.])
    #tensor([1., 2., 3., 4.])

In the process of writing, I found that tensor and tensor were different

• torch.Tensor
  • Short for default tensor type (torch.FlaotTensor)
• torch.tensor
  • Tensor is created according to the following data, and the tensor type is inferred according to the data.

See [PyTorch] the difference between tensor and tensor

The next section describes how to build neural networks

Welcome criticism and correction, learn and make progress together!!!