airflow - CloudRunExecuteJobOperator fail to trigger job run on GCP - Stack Overflow

admin管理员组

文章数量:1355658

I have a Cloud Composer environment that is mainly made of DAGs that have to trigger jobs on GCP using the CloudRunExecuteJobOperator.

Without any changes to my instance, the tasks begin to stay stuck in the "running" state until they timeout and fail. When I look at the logs in the airflow interface here is what I get:

DEBUG - Retrying <unknown> in 0.11023210674676037 seconds as it raised OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (::1), port 3306 failed: server closed the connection unexpectedly\n This probably means the server terminated abnormally\n before or while processing the request.\n\n(Background on this error at: ).\nTraceback (most recent call last):\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect\n return fn()\n ^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 327, in connect\n return _ConnectionFairy._checkout(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 493, in checkout\n rec = pool._do_get()\n ^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get\n return self._create_connection()\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 273, in _create_connection\n return _ConnectionRecord(self)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 388, in __init__\n self.__connect()\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 690, in __connect\n with util.safe_reraise():\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 686, in __connect\n self.dbapi_connection = connection = pool._invoke_creator(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 574, in connect\n return dialect.connect(*cargs, **cparams)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 598, in connect\n return self.dbapi.connect(*cargs, **cparams)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect\n conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\npsycopg2.OperationalError: connection to server at "localhost" (::1), port 3306 failed: server closed the connection unexpectedly\n This probably means the server terminated abnormally\n before or while processing the request.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/opt/python3.11/lib/python3.11/site-packages/airflow/utils/retries.py", line 102, in wrapped_function\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/airflow/jobs/job.py", line 335, in _fetch_from_db\n session.merge(job)\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 3056, in merge\n return self._merge(\n ^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 3136, in _merge\n merged = self.get(\n ^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2853, in get\n return self._get_impl(\n ^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl\n return db_load_fn(\n ^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity\n session.execute(\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1716, in execute\n conn = self._connection_for_bind(bind)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1555, in _connection_for_bind\n return self._transaction._connection_for_bind(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 750, in _connection_for_bind\n conn = bind.connect()\n ^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/future/engine.py", line 412, in connect\n return super(Engine, self).connect()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3325, in connect\n return self._connection_cls(self, close_with_result=close_with_result)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 96, in __init__\n else engine.raw_connection()\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection\n return self._wrap_pool_connect(self.pool.connect, _connection)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3374, in _wrap_pool_connect\n Connection._handle_dbapi_exception_noconnection(\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2208, in _handle_dbapi_exception_noconnection\n util.raise_(\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect\n return fn()\n ^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 327, in connect\n return _ConnectionFairy._checkout(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 493, in checkout\n rec = pool._do_get()\n ^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get\n return self._create_connection()\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 273, in _create_connection\n return _ConnectionRecord(self)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 388, in __init__\n self.__connect()\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 690, in __connect\n with util.safe_reraise():\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n raise exception\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 686, in __connect\n self.dbapi_connection = connection = pool._invoke_creator(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 574, in connect\n return dialect.connect(*cargs, **cparams)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 598, in connect\n return self.dbapi.connect(*cargs, **cparams)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/python3.11/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect\n conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (::1), port 3306 failed: server closed the connection unexpectedly\n This probably means the server terminated abnormally\n before or while processing the request.\n\n(Background on this error at: )

I already tried to increase my cluster configuration but the problem stays the same. Only a few task go through

本文标签： airflowCloudRunExecuteJobOperator fail to trigger job run on GCPStack Overflow