Here are some common reasons why Amazon EC2 instances time out in Data Pipeline.

Software updates after launch

If you don't specify the optional imageId field for the Ec2Resource object, Data Pipeline defaults to an older Amazon Machine Image (AMI). Instances launched from older AMIs automatically install software updates after they launch, and they might time out in Data Pipeline.

Use one of the following options to resolve this issue:

• Specify the latest AMI in the imageId field for the Ec2Resource object. This significantly reduces the amount of time required for software updates.
• Use an existing resource instead of the Ec2Resource
  object. Install Task Runner on an existing EC2 instance, and then add a workerGroup field to connect the object to the pipeline activities that it should process. For more information, see Executing Work on Existing Resources Using Task Runner.

Network connectivity

Task Runner must be able to access the AWS Data Pipeline endpoint and Amazon Simple Storage Service (Amazon S3). This means that the virtual private cloud (VPC) that the EC2 instance is in must have a subnet and an internet gateway.

Keep the following subnet requirements in mind:

• If the EC2Resource object is launched in a public subnet, the subnet's route table must point to an internet gateway.
• If the EC2Resource object is launched in a private subnet, the subnet's route table must point to a NAT instance or NAT gateway that has a route to an internet gateway. You can use an Amazon Virtual Private Cloud (Amazon VPC) endpoint for Amazon S3 to access an S3 bucket in the same region as the VPC.

Task Runner installation failure

When you use the EC2Resource object, cloud-init installs Task Runner. If installation fails, the Amazon EC2 instance can't communicate with the Data Pipeline endpoint. For example, if the pipelineLogUri field contains unexpected characters, such as quotation marks, the Task Runner installation script fails. Check the cloud-init logs for the EC2 instance to find out why the installation failed.

Note: After you change a pipeline definition, you must reactivate the pipeline for your changes to take effect. For more information, see Editing Your Pipeline.

Missing AWS Identity and Access Management (IAM) permission

The IAM instance profile used by the EC2 instance must give Task Runner permission for the APIs that Task Runner attempts to call, such as PollForTask. Connect to the EC2 instance and then check the Task Runner logs to find out if a missing permission is causing the error.