Skip to content

Conversation

@pateash
Copy link
Contributor

closes: #14563
related: #14640

Add meaningful description above
Hi All,

This PR proposes to allow users to provide external_task_group_id in ExternalTaskSensor to make tasks on a DAG wait for another task_group on a different DAG for a specific execution_date.
we can also provide options to set if the Task on a remote DAG succeeded or failed via allowed_states and failed_states parameters.

@pateashpateash changed the title allowing task_group Sensor in ExternalTaskSensorAllowing task_group Sensor in ExternalTaskSensorJul 7, 2022
@pateash
Copy link
ContributorAuthor

pateash commented Jul 7, 2022

cases

  1. group1_sensor -> wait for task_group from external dag which exists and succeeded ( as per allowed_states )
  2. group2_sensor- > wait for task_group from external dag which doesn't exists ( wait till timeout )
  3. group2_sensor_check_enabled -> wait for task_group from external dag which doesn't exists and with check_enabled=True ( fail with error )

image

@pateashpateashforce-pushed the airflow-14563 branch 2 times, most recently from 1425c95 to eb29793CompareJuly 9, 2022 22:20
@pateash
Copy link
ContributorAuthor

Hi @eladkal, could you please take a look into this PR.

@uranusjr
Copy link
Member

Since we are already allowing multiple task IDs, I wonder if we should just go directly with external_group_ids instead of just one single group. Someone’s going to ask for that sooner or later.

@pateash
Copy link
ContributorAuthor

@uranusjr, we already allow multiple taskIds which I personally don't think a good idea, because it is not very transparent.

Although For TaskGroup, I think if someone want to depend on multiple of them, they must create different task sensors.

@eladkaleladkal added this to the Airflow 2.4.0 milestone Aug 3, 2022
Copy link
Contributor

@eladkaleladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eladkaleladkal merged commit bc04c5f into apache:mainAug 22, 2022
@pateashpateash deleted the airflow-14563 branch August 23, 2022 07:42
@ephraimbuddyephraimbuddy added the type:new-feature Changelog: New Features label Sep 13, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:documentationtype:new-featureChangelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TaskGroup Sensor

4 participants

@pateash@uranusjr@eladkal@ephraimbuddy