Long running tasks killed before completion about aws-airflow-stack HOT 2 OPEN

Doing a bit of research : seems like the issue is that the graceful shutdown Lifecyle Hook is a 3 minute countdown for the EC2 Instance defined in the template

  GracefulShutdownLifecycleHook:
    Type: AWS::AutoScaling::LifecycleHook
    Properties:
      AutoScalingGroupName: !Ref AutoScalingGroup
      DefaultResult: CONTINUE
      HeartbeatTimeout: 180
      LifecycleTransition: autoscaling:EC2_INSTANCE_TERMINATING

A possible solution is to change that timeout so that it is as long as the longest task (not ideal) or potentially running a lambda function that records a heartbeat
https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html#lifecycle-hook-considerations.

from aws-airflow-stack.

Hm, I believe there's probably a template substitution bug on the lifecycle hook helper daemon, which would normally extend the timeout until the service is done. But you're right, if the service is not working, all you've got are those 3 minutes. I'm gonna go ahead and convert this issue to a bug report about the hearbeat service.

from aws-airflow-stack.