This schedule means the DAG will execute five minutes after midnight each day. As it happens, we set the schedule to " 5 * * * *", expecting the DAG to execute every five minutes. So, I changed the DAG schedule to execute every 5 minutes. After this change, once again, we watched Airflow, but nothing happened - the DAG did not execute.Īfter giving it some thought, I realized that the time at which we wanted to execute the DAG in the production schedule was not important for the demonstration. (Remember that Airflow executes a DAG only after the scheduled period for that DAG has passed/completed) So we set the clock to Jun 15, 2023. When the DAG did not execute, I realized that we had to set the date to at least one day prior to the clock. We set the start date on the DAG to Jun 17 and waited anxiously. We changed the time to Jun 17, 2023, and off we went. We spun up a Windows VM where we had control to change clock settings. We also tried adding a new clock and setting the value but were not successful. ![]() We did not have permission to change the clock. ![]() When we tried to change the clock on our company-issued laptops, we ran into trouble. This was easy to do on a Linux VM using the date -s command. To test daylight savings, we had to change the clocks. Once we demonstrated that the time zone setting was working as expected, we wanted to test that the DAG works when daylight savings was enabled. Confusing? Yes, but this is expected behavior because Paris time is one hour ahead of UTC. Airflow executes the DAG at 10 past 4 am. The DAG schedule means the DAG is set to execute at 10 part 5 am Paris time. As Airflow is still running on the UTC clock, it executes the DAG at 10 minutes past five AM, as expected, but the UI displays the time as 10 minutes past four AM. The only issue was how Airflow displays the information on the screen. Given that we are in the Jan/Feb period of the year, setting the time zone meant we could set the schedule for the DAG and wait for the clock to hit the proper time, and the DAG was off. Here is how we went about testing the DAG. Carrying this analogy further, the problem we faced was how to get the liquid butter on the bread using a knife :-) Specifying the time zone parameter for a DAG was as easy as passing a hot knife through butter. The problem we ran into was while demonstrating that the DAG works - more so when daylight savings are enabled, given that daylight saving is yet to kick in for this year. Once we set this information, we were set. In this example, we are using 'Paris' as our time zone, which is one hour ahead of UTC in the winter period and two hours ahead when daylight savings are enabled/active. ![]() Local_tz = pendulum.timezone("Europe/Paris") ![]() To make an Airflow DAG understand daylight savings, we have to use the pendulum library, create a time zone object and add it to the start_time parameter of the DAG object as below It is possible to update the Airflow config to make it work as per the required time zone, but the recommendation is to let Airflow work on UTC while configuring DAGs to recognize the required time zone.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |