AWS ECS error: Task failed ELB health checks in Target group

aws elb health check failing
stopped (task failed elb health checks in (target-group
aws application load balancer health check fails
ecs health check grace period
health checks failed with these codes: [503]
aws network load balancer health check failing
ecs task failed to start
aws::ecs::taskdefinition

I am using cloud formation template to build the infrastructure (ECS fargate cluster). Template executed successfully and stack has been created successfully. However, task has failed with the following error:

Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:eu-central-1:890543041640:targetgroup/prc-service-devTargetGroup/97e3566c8b307abf)

I am not getting what and where to look for this to troubleshoot the issue. as it is fargate cluster, I am not getting how to login to container and execute some health check queries to debug further.

Can someone please help me to guide further on this and help me? Due to this error, I am not even able to access my web app. As ALB won't route the traffic if it is unhealthy.

What I did

After some googling, I found this post: https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/

However, I guess, this is related to EC2 compatibility in fargate. But in my case, EC2 is not there.

If you feel, I can paste the entire template as well.

please help


Checking stopped tasks for errors - AWS Documentation, For example, you run the task and the task displays a PENDING status and ECS console by displaying the stopped task and inspecting it for error messages. Task failed ELB health checks in (elb elb-name) The current task failed the Elastic Load Balancing health check for the load balancer that is associated with the task's service. For more information, see Troubleshooting service load balancers. Scaling activity initiated by (deployment deployment-id)


Pass Application Load Balancer Health Checks in Amazon ECS, How can I get my EC2 instance to pass the health check? The advanced health check settings of your target group are correctly configured. Note: An ECS task can return an unhealthy status for many reasons. If you receive a non-HTTP error message, then your application isn't listening to HTTP traffic. select your target group for your load balancer; select the health check tab; make sure the health check for your EC2 instance is the same as the health check in the target group. This will tell your ELB to route its traffic to this endpoint when conducting its health check. In my case my health check path was /health.


I got this error message because the security group between the ECS service and the load balancer target group was only allowing HTTP and HTTPS traffic.

Apparently the health check happens over some other port and or protocol as updating the security group to allow all traffic on all ports (as suggested at https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-application-load-balancer.html) made the health check work.

AWS Fargate task fails ELB health checks, Ref 'TargetGroup' Type: 'forward' Conditions: - Field: path-pattern Values: Your problem is most likely that your Load Balancer - which most to communicate with your ECS instances, since they allow only traffic from 138.106.0.0/16 . If "/" is the route you configured for health-check, please make sure  Target health status. Before the load balancer sends a health check request to a target, you must register it with a target group, specify its target group in a listener rule, and ensure that the Availability Zone of the target is enabled for the load balancer.


I had this exact same problem. I was able to get around the issue by:

  1. navigate to EC2 service
  2. then select Target Group in the side panel
  3. select your target group for your load balancer
  4. select the health check tab
  5. make sure the health check for your EC2 instance is the same as the health check in the target group. This will tell your ELB to route its traffic to this endpoint when conducting its health check. In my case my health check path was /health.

Cluster continually failing health checks and restarting nodes · Issue , I changed the Auto Scale groups health check from ELB to EC2, this maybe there are admin tasks on the ELBs that do something every so many hours, port 7, which is mostly to allow us to add the ELB without giving us an error. Target Group on front may show that front-end instances are healthy but  When the load balancer sends an HTTP GET request to the health check path, the application in your ECS container should return the default 200 OK response code. Note: If you use an Application Load Balancer, you can update the Matcher setting to a response code other than 200. For more information, see Health Checks for Your Target Groups. 1.


As mentioned by tschumann above, check the security group around the ECS cluster. If using Terraform, allow ingress to all docker ephemeral ports with something like below:

resource "aws_security_group" "ecs_sg" {
  name    = "ecs_security_group"
  vpc_id  = "${data.aws_vpc.vpc.id}"

}

resource "aws_security_group_rule" "ingress_docker_ports" {
  type              = "ingress"
  from_port         = 32768
  to_port           = 61000
  protocol          = "-1"
  cidr_blocks       = ["${data.aws_vpc.vpc.cidr_block}"]
  security_group_id = "${aws_security_group.ecs_sg.id}"
}

ECS agents stops and starts the tasks · Issue #1872 · aws/amazon , One of the tasks running in a container instance is stopped by ECS agent a close the channel 2019-06-20T18:02:51Z [WARN] Error publishing metrics: write The Task is stopped by saying reason: ELB health check got failed. it says Stopped Reason: Task failed ELB health checks (target group id). RSS. Your load balancer checks the health of its registered instances using either the default health check configuration provided by Elastic Load Balancing or a custom health check configuration that you specify. The health check configuration contains information such as the protocol, ping port, ping path, response timeout, and health check interval.


ALB ECS service healthcheck failure · Devon Hakel-Kinko, I had a nodejs express app running as an ECS service. Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:us-east-  The load balancer starts routing requests to a newly registered target as soon as the registration process completes and the target passes the initial health checks. If demand on your application decreases, or you need to service your targets, you can deregister targets from your target groups.


Target Group healthcheck failing on Fargate container : aws, I am able to hit the /healthcheck route from EC2 instances but the target group healthcheck pointed at the service always fails. I am at my wits end as I can't seem to find anyone with the same problem. Also, check your security group settings. Looking at your settings are you sure your health check is returning status 200  This means that services behind an Network Load Balancer are effectively open to the world as soon as you allow incoming requesets and health checks in the target's security group. If you are experiencing problems with your load balancer-enabled services, see Troubleshooting service load balancers .


AWS Fargate task fails ELB health checks, Type: AWS::ElasticLoadBalancingV2::TargetGroup GroupDescription: Access to the ECS hosts and the tasks/containers I am trying to run a simple nginx container but the load balancer complains that health checks are failed and the task does not respond on its ip number, likely because of the error  Check the ELB access log for duplicate HTTP 502 errors. 502 errors for both elb_status_code and backend_status_code indicate that there is a problem with one or more of the web server instances. Identify which web server instances are exhibiting the problem, then check the web server logs of the backend web server instances.