airflow - Some Dags not visible in Airlfow Web-UI (8 visible, 2 not visible, no errors) - Stack Overflow

admin管理员组

文章数量:1320782

After not very successful restoring backup of virtual private server I got some problems with running containers with Clickhouse, Superset and Airflow. After I renamed some files:

mv /opt/beget/superset/postgres/pg_logical/replorigin_checkpoint \
   /opt/beget/superset/postgres/pg_logical/replorigin_checkpoint.bak

mv /opt/beget/superset/redis/dump.rdb \
   /opt/beget/superset/redis/dump.rdb.bak

I successfully run containers. Next I saw only 7 of 9 dags in Web-Ui. All this dags 100% working and got no mistakes (I using them many times before).

So I start checking params like hide paused dag and so on. All this params is False. I use admin user role, so I can see all dags.

Next I checked dag list from containers (tasks also visible):

root@kkoxelxfwf:/opt/beget/airflow/dags# docker compose exec airflow-scheduler airflow dags list
WARN[0000] /opt/beget/airflow/docker-compose.yml: `version` is obsolete
Error: Failed to load all files. For details, run `airflow dags list-import-errors`
dag_id                                | filepath                                 | owner       | paused
======================================+==========================================+=============+=======
clean_airflow_database                | cleaning_airflow_database.py             | d-chernovol | False
clean_airflow_logs                    | cleaning_airflow_logs.py                 | d-chernovol | False
dag_add_client_loyalty                | dag_updade_clients_loyalty.py            | d-chernovol | False
dag_calculate_staff_salarys           | dag_calculate_staff_salarys_test.py      | d-chernovol | False
dag_check_dag_run                     | check_last_update_time.py                | d-chernovol | False
dag_get_bonus_and_penaltys_for_staff  | dag_get_bonus_and_penaltys_for_staff.py  | d-chernovol | False
dag_retention_rate_for_masters        | dag_retention_rate_for_masters.py        | d-chernovol | False
dag_send_rfm_clients_data_to_telegram | dag_send_rfm_clients_data_to_telegram.py | d-chernovol | False
dag_update_database                   | dag_update_database.py                   | d-chernovol | False

root@kkoxelxfwf:/opt/beget/airflow/dags# docker compose exec airflow-webserver airflow dags list
WARN[0000] /opt/beget/airflow/docker-compose.yml: `version` is obsolete
Error: Failed to load all files. For details, run `airflow dags list-import-errors`
dag_id                                | filepath                                 | owner       | paused
======================================+==========================================+=============+=======
clean_airflow_database                | cleaning_airflow_database.py             | d-chernovol | False
clean_airflow_logs                    | cleaning_airflow_logs.py                 | d-chernovol | False
dag_add_client_loyalty                | dag_updade_clients_loyalty.py            | d-chernovol | False
dag_calculate_staff_salarys           | dag_calculate_staff_salarys_test.py      | d-chernovol | False
dag_check_dag_run                     | check_last_update_time.py                | d-chernovol | False
dag_get_bonus_and_penaltys_for_staff  | dag_get_bonus_and_penaltys_for_staff.py  | d-chernovol | False
dag_retention_rate_for_masters        | dag_retention_rate_for_masters.py        | d-chernovol | False
dag_send_rfm_clients_data_to_telegram | dag_send_rfm_clients_data_to_telegram.py | d-chernovol | False
dag_update_database                   | dag_update_database.py                   | d-chernovol | False

All dags is visible (except one, I add one test dag with error, so I can see error in Web-UI).

So new dags is visible and I can see errors in Web-UI.

I also download Google Chrome and reset all setting, I still see only 7 dags. So this is not browser error. I also restart all containers, all got "Up ** minutes (healthy)" status. I don't know that I should check next. I see no error in all airflow related containers, but still don't see this dags.

Edit: after posting dag, that I can't see in web-ui runs 1 time. I know it, because I send messages with errors and important data to telegram chat, so I sure, that dags was run in 3 a. m.

Update: I discover that if I make new dag_id for 'invisible" dags, I can see them. And it's not about role permissions, cause I tried add permission: "can read on DAG: dag_update_database".

I also see, that some dags successfully running.

Any ideas, that I should check next? Maybe there's more junk files after bad restoring backup?

Many thanks for reading!

OS: Ubuntu 22.04.4 LTS Docker: version 26.1.3, build b72abbb Docker Compose: version v2.27.0 Airflow: 2.8.2

本文标签：