backpropagation - How does PyTorch autograd backpropogate successfully through non-tensor elements in the computational graph fo

admin管理员组

文章数量:1305166

I am trying to understand the example REINFORCE PyTorch implementation on PyTorch GitHub: .py

One particular point is a sticking point I am unable to understand at line 75.

policy_loss.backward()

There are many non-tensor variables from state input to policy all the way to policy_loss.backward() which would stop autograd from back propagating, based on my understanding.

Eg, policy_loss.backward() is called on policy_loss, derived from policy.saved_log_probs and returns

    for log_prob, R in zip(policy.saved_log_probs, returns):
        policy_loss.append(-log_prob * R)

policy.saved_log_probs is a non-tensor

self.saved_log_probs = []

And so is returns, which in turn is calculated from policy.rewards (which is a non-tensor).

        self.rewards = []

    for r in policy.rewards[::-1]:
        R = r + args.gamma * R
        returns.appendleft(R)

So how would autograd back prop past these all the way to affine1 linear layer’s weights?

class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 128)

I am trying to understand the example REINFORCE PyTorch implementation on PyTorch GitHub: https://github/pytorch/examples/blob/main/reinforcement_learning/reinforce.py

One particular point is a sticking point I am unable to understand at line 75.

policy_loss.backward()

There are many non-tensor variables from state input to policy all the way to policy_loss.backward() which would stop autograd from back propagating, based on my understanding.

Eg, policy_loss.backward() is called on policy_loss, derived from policy.saved_log_probs and returns

    for log_prob, R in zip(policy.saved_log_probs, returns):
        policy_loss.append(-log_prob * R)

policy.saved_log_probs is a non-tensor

self.saved_log_probs = []

And so is returns, which in turn is calculated from policy.rewards (which is a non-tensor).

        self.rewards = []

    for r in policy.rewards[::-1]:
        R = r + args.gamma * R
        returns.appendleft(R)

So how would autograd back prop past these all the way to affine1 linear layer’s weights?

class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 128)

• pytorch
• backpropagation
• autograd
• policy-gradient-descent

Share Improve this question Follow asked Feb 4 at 7:07 TalkArtFunDayTalkArtFunDay 1111 bronze badge

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

As you suspected, the list is indeed not part of the computational graph. The fact that you hold the input or output tensor of an arithmatic operation in a list, dict or any other data structure is irrelevant. Every time a tensor is involved in a derivable operation (e.g. multiplication, addition, or even concatination), the result has a reference to the location in the computational graph that is built by the operation.

In the examples you provided, note that later the tensors inside the list are used in the arithmatic ops, not the list that contains it.

For background, you may find it interesting to read a bit about how computational graphs are built.

本文标签：