The goal of this document is submitting a job to AWS Batch and confirming the result in CloudWatch Logs.
Since I've worked on ap-northeast-1
region, The following examples includes this region name.
You need to create some AWS resources as the document explained.
- Create IAM Roles for your Compute Environments and Container Instances
- Create a Key Pair
- Create a Virtual Private Cloud
- Create a Security Group
When I set up everything and submitted a job to AWS Batch, the submitted job stuck in runnable.
- Ref. https://docs.aws.amazon.com/batch/latest/userguide/troubleshooting.html#job_stuck_in_runnable
- Ref. https://forums.aws.amazon.com/thread.jspa?threadID=247401
The problems was a wrong configuration of MapPublicIpOnLaunch
. It have to be true
as shown below.
Managed compute environments launch Amazon ECS container instances into the VPC and subnets that you specify when you create the compute environment. Amazon ECS container instances need external network access to communicate with the Amazon ECS service endpoint. If your container instances do not have public IP addresses (because the subnets you've chosen do not provide them by default), then they must use network address translation (NAT) to provide this access. Ref. https://docs.aws.amazon.com/batch/latest/userguide/compute_environments.html#managed_compute_environments
See also https://docs.aws.amazon.com/batch/latest/userguide/create-public-private-vpc.html
Select Enable auto-assign public IPv4 address and choose Save, Close.
If you create VPC and Public Subnet from VPC with a Single Public Subnet
in Start VPC Wizard
, the creaated public subnet has MapPublicIpOnLaunch
as false
by default. So you have to enable it.
$ aws ec2 describe-subnets
{
"Subnets": [
{
"AvailabilityZone": "ap-northeast-1a",
"Tags": [
{
"Value": "Public subnet",
"Key": "Name"
}
],
"AvailableIpAddressCount": 251,
"DefaultForAz": false,
"Ipv6CidrBlockAssociationSet": [],
"VpcId": "[your_vpc_id]",
"State": "available",
"MapPublicIpOnLaunch": true,
"SubnetId": "[your_subnet_id]",
"CidrBlock": "10.0.0.0/24",
"AssignIpv6AddressOnCreation": false
}
]
}
Another problem I had was EnableDnsHostnames: false
. It have to be true
as well.
$ aws ec2 describe-vpc-attribute --vpc-id [your_vpc_id] --attribute enableDnsSupport
{
"VpcId": "[your_vpc_id]",
"EnableDnsSupport": {
"Value": true
}
}
$ aws ec2 describe-vpc-attribute --vpc-id [your_vpc_id] --attribute enableDnsHostnames
{
"VpcId": "[your_vpc_id]",
"EnableDnsHostnames": {
"Value": true
}
}
$ aws batch create-compute-environment --generate-cli-skeleton
$ aws batch create-compute-environment --cli-input-json file://compute-environments.json
$ aws batch describe-compute-environments
$ cat compute-environments.json
{
"computeEnvironmentName": "on-demand-compute-environment",
"type": "MANAGED",
"state": "ENABLED",
"computeResources": {
"type": "EC2",
"minvCpus": 0,
"maxvCpus": 1,
"desiredvCpus": 0,
"imageId": "",
"instanceTypes": [
"m3.medium"
],
"subnets": [
"[your_subnet_id]"
],
"securityGroupIds": [
"[your_security_group_id]"
],
"ec2KeyPair": "[your_key_pair_name]",
"instanceRole": "arn:aws:iam::[aws_account_id]:instance-profile/ecsInstanceRole",
"tags": {
"Name": "ex-aws-batch"
},
"bidPercentage": 0,
"spotIamFleetRole": ""
},
"serviceRole": "arn:aws:iam::[aws_account_id]:role/service-role/AWSBatchServiceRole"
}
$ aws batch create-job-queue --generate-cli-skeleton
$ aws batch create-job-queue --cli-input-json file://job-queue.json
$ aws batch describe-job-queues
$ cat job-queue.json
{
"jobQueueName": "ex-aws-batch",
"state": "ENABLED",
"priority": 1,
"computeEnvironmentOrder": [
{
"order": 1, "computeEnvironment": "arn:aws:batch:ap-northeast-1:[aws_account_id]:compute-environment/on-demand-compute-environment"
}
]
}
$ aws batch register-job-definition --generate-cli-skeleton
$ aws batch register-job-definition --cli-input-json file://job-definition.json
$ aws batch describe-job-definitions
$ cat job-definition.json
{
"jobDefinitionName": "my-test-job",
"type": "container",
"parameters": {
"Name": "alice"
},
"containerProperties": {
"image": "busybox",
"vcpus": 1,
"memory": 500,
"command": [
"echo",
"Ref::name"
],
"jobRoleArn": "",
"volumes": [
],
"environment": [],
"mountPoints": [],
"readonlyRootFilesystem": true,
"privileged": true,
"ulimits": [],
"user": ""
},
"retryStrategy": {
"attempts": 1
}
}
$ aws batch submit-job \
--job-name test \
--job-queue arn:aws:batch:ap-northeast-1:[aws_account_id]:job-queue/ex-aws-batch \
--job-definition arn:aws:batch:ap-northeast-1:[aws_account_id]:job-definition/my-test-job:1 \
--parameters name=bob
- Ref. https://github.com/jorgebastida/awslogs
After submitting a job to AWS Batch, you have to be able to see a log in
jobDefinitionName/default/ecs_task_id
log stream which belongs to/aws/batch/job
.
$ awslogs get /aws/batch/job ALL --watch
/aws/batch/job my-test-job/default/dbee4196-0d1c-4d0b-bc13-73642c759b9f bob