[PyTorch] Note 02: Autograd auto derivation

From PyTorch 1.4 tutorial


  • Tensor
  • torch.autograd.backward
    • If the result node is scalar
    • If the result node is a vector

[PyTorch] Note 02: Autograd auto derivation

In PyTorch, the core of all neural networks is autograd package

1 Tensor

  • torch.Tensor is the core class of this autograd

  • A Tensor tensor usually records the following properties:

    • Data: stored data information
    • requires_grad: if it is set to True, it means that the Tensor needs derivation
    • grad: the gradient value of this Tensor needs to be reset to zero every time when calculating the backward, otherwise the gradient value will be accumulated all the time
    • grad_fn: the leaf node (i.e. independent variable) is usually None, and only the grad of the result node (i.e. dependent variable)_ FN is valid to indicate which type of gradient function is.
    • is_leaf: used to indicate whether the Tensor is a leaf node.

See Detailed explanation of Python autograd, backward

  • requires_grad=True
    • Track all operations of the corresponding tensor
    • Default requirements_ Grad is flame
    a = torch.randn(2, 2)
    a = ((a * 3) / (a - 1))

  • . detach() method
    • Prevent the tensor from being tracked
    • Call The detach() method separates it from the calculation history and prevents its future calculation records from being tracked
  • with torch.no_grad():
    • To prevent tracing history (and memory usage), code blocks can be wrapped in with torch no_ Grad (): medium
  • grad_fn
    • Only dependent variables have properties

    tensor(1., requires_grad=True) tensor(2., requires_grad=True) tensor(3., grad_fn=<AddBackward0>)

2 torch.autograd.backward

Source code interface

  • Parameter meaning
    • Tensor: the tensor used to calculate the gradient
      torch.autograd.backward(tensor) and tensor Backward() is written equivalently
    • grad_tensors: used when calculating the gradient of the matrix. In fact, it is also a tensor. Generally, the shape needs to be consistent with the previous tensor
    • retain_graph: usually, after calling backward once, pytorch will automatically destroy the calculation graph. Therefore, if you want to call backward repeatedly for a variable, you need to set this parameter to True
    • create_graph: when set to True, it can be used to calculate higher-order gradients

2.1 if the result node is scalar

  • Scalar can be understood as one-dimensional
  • Just apply backward directly
  • Reference examples are as follows:
    #Returns the gradient of x,y, and the value of z

    tensor(1.) tensor(1.) tensor(10., grad_fn=<AddBackward0>)

2.2 if the result is vector

  • Vector can be understood as high-dimensional and multi-dimensional
  • Referring to the Chinese documents of pytorch and Zhihu's articles, I feel that Zhihu's articles are better understood, but they are essentially the same, that is, a tensor is introduced that is the same as the previous tensor tensor
  • Reference examples are as follows:
    x=torch.ones(4,requires_grad=True)   #x=[x1,x2,x3,x4]
    #print(x.type())      torch.FloatTensor
    z=x+2   #z=[x1+2,x2+2,x3+2,x4+2]
    #If all incoming are 1
    #If the incoming is set by yourself
    #z.backward(torch.Tensor([1,2,3,4]))  #z=[x1+2,2(x2+2),3(x3+2),4(x4+2)] note the matching of types. Tensor defaults to torch Floattensor type
    #z.backward(torch.tensor([1.,2.,3.,4.]))  #z=[x1+2,2(x2+2),3(x3+2),4(x4+2)]

    tensor([1., 1., 1., 1.])
    #tensor([1., 2., 3., 4.])

In the process of writing, I found that tensor and tensor were different

  • torch.Tensor
    • Short for default tensor type (torch.FlaotTensor)
  • torch.tensor
    • Tensor is created according to the following data, and the tensor type is inferred according to the data.

See [PyTorch] the difference between tensor and tensor

The next section describes how to build neural networks

Welcome criticism and correction, learn and make progress together!!!

Tags: neural networks Pytorch Deep Learning

Posted by rusbb on Thu, 19 May 2022 21:17:09 +0300