Slow response from Python pushing data to BigQuery - Stack Overflow

admin管理员组

文章数量:1405347

We have a Python process which reads a JSON file and pushes the data held therein to a BigQuery table. The files are small and normally would process very quickly. All of a sudden the process has really slowed down. The python resides in a VM which is within Big Query. What we have noticed in the BigQuery logs is a lot of errors similar to the following:

{
  "insertId": "s=3bc18ce95c384588adb5aea97d1ab8fc;i=c645e;b=8f0641e52f4d431db01b9ea986ded58e;m=4ac5bf55c5b;t=62fc1655c06a6;x=91a9a97343928f96",
  "jsonPayload": {
    "message": "2025-03-07T14:25:35.9871Z OSConfigAgent Error main.go:88: code: \"Unauthenticated\", message: \"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See .\", details: []"
  },

We're not sure if this is related.

I should clarify that the script runs and does eventually push the data but it is very slow do to so, it doesn't actually fail.

We have tried restarting the VM and the process itself. What can we try next?

We have a Python process which reads a JSON file and pushes the data held therein to a BigQuery table. The files are small and normally would process very quickly. All of a sudden the process has really slowed down. The python resides in a VM which is within Big Query. What we have noticed in the BigQuery logs is a lot of errors similar to the following:

{
  "insertId": "s=3bc18ce95c384588adb5aea97d1ab8fc;i=c645e;b=8f0641e52f4d431db01b9ea986ded58e;m=4ac5bf55c5b;t=62fc1655c06a6;x=91a9a97343928f96",
  "jsonPayload": {
    "message": "2025-03-07T14:25:35.9871Z OSConfigAgent Error main.go:88: code: \"Unauthenticated\", message: \"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google/identity/sign-in/web/devconsole-project.\", details: []"
  },

We're not sure if this is related.

I should clarify that the script runs and does eventually push the data but it is very slow do to so, it doesn't actually fail.

We have tried restarting the VM and the process itself. What can we try next?

• python

Share Improve this question Follow edited Mar 8 at 17:20 halfer 20.4k1919 gold badges109109 silver badges202202 bronze badges asked Mar 8 at 17:08 KenKen 1

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

It looks like the issue might be related to authentication. The logs show an 'Unauthenticated' error, which could be slowing things down. Try checking authentication with gcloud auth and make sure the VM's service account has the right permissions. Also, have you tested the network to see if there are any delays?

本文标签： Slow response from Python pushing data to BigQueryStack Overflow