<p>You didn't say what version of the AWS CLI you're using, but I suspect it's version 1. It looks like the <code>evaluateOnExit</code> feature was added in version 2 but was not backported. Installing AWS CLI version 2 resolves the issue:</p>
<p><a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html</a></p>
<p>Note that <a href="https://www.nextflow.io/" rel="nofollow noreferrer">Nextflow</a> also provides built-in support for <a href="https://www.nextflow.io/docs/latest/aws.html#aws-batch" rel="nofollow noreferrer">AWS Batch</a>. You'd still need to configure some AWS infrastructure using Nextflow, but it should be a lot less painful than writing/maintaining your own script(s). It's definitely worth checking out if you haven't already.</p>

CC BY-SA 4.0
2022-10-31T01:09:41.197Z
2022-10-31T01:09:41.197Z
Steve
<p>I am in the process of creating a bioinformatics pipeline and I am currently trying to setup the AWS infrastructure for which it will run.</p>
<p>The script I am running to setup the AWS batch job looks like the following:</p>
<pre><code>#! /bin/bash

# Setup the AWS Batch architecture
# Copyright 2022, SolareaBio

set -e


# ***** BEGIN custom lines

container=&quot;BGC_bacteria&quot;
vCPUs_req=12
memory_MiB_req=40000
# ASK ABOUT CUSTOM CPU AND MEMORY SETTINGS

# ***** END custom lines



container_lc=<span class="math-container">$(echo $</span>container | tr [A-Z] [a-z])

subnet_useast1a=&quot;subnet-42371d0a&quot;
subnet_useast1b=&quot;subnet-b03c54ea&quot;
subnet_useast1c=&quot;subnet-00ee1364&quot;

usage() {
  cat &lt;&lt; EOF
Usage $0 [-h]

Builds the <span class="math-container">$container docker container, pushes it to the AWS ECR, and sets up the $</span>container AWS Batch environment
EOF
}

while getopts &quot;h&quot; option; do
  case $option in
    h)   usage;exit;;
    [?]) usage; exit;;
  esac
done

script_dir=<span class="math-container">$(dirname "$</span>0&quot;)

repo=&quot;990850896279.dkr.ecr.us-east-1.amazonaws.com&quot;

echo &quot;Logging into ECR&quot;
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${repo}

image_and_tag=&quot;<span class="math-container">${container_lc}:latest"
echo "Building the docker container: $</span>image_and_tag&quot;

# If you get errors from the docker build command, try adding --no-cache
docker build --no-cache -f <span class="math-container">$script_dir/$</span>{container}.dockerfile -t <span class="math-container">$image_and_tag $</span>script_dir

echo &quot;Tagging Image&quot;
docker tag <span class="math-container">${image_and_tag} $</span>{repo}/${image_and_tag}

# Create repository in AWS ECR
repo_container_info=<span class="math-container">$(aws ecr describe-repositories --repository-names $</span>{container_lc})
if [[ &quot;<span class="math-container">$repo_container_info" == "" ]]; then
  aws ecr create-repository --repository-name $</span>{container_lc}
fi

echo &quot;Pushing Image up to AWS&quot;
docker push <span class="math-container">${repo}/$</span>{image_and_tag}

# Create compute environment
EC2_type=ON_DEMAND
ce_info=<span class="math-container">$(aws batch describe-compute-environments --compute-environments $</span>{container}_<span class="math-container">${EC2_type}_CE)
ce_status=$</span>(echo <span class="math-container">$ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
if [[ $</span>ce_status != &quot;VALID&quot; ]]; then
  compute_resources='{&quot;type&quot;:&quot;EC2&quot;,&quot;allocationStrategy&quot;:&quot;BEST_FIT_PROGRESSIVE&quot;,&quot;minvCpus&quot;:0,&quot;maxvCpus&quot;:10000,&quot;tags&quot;:{&quot;Name&quot;:&quot;'<span class="math-container">${container}_Batch'"},"instanceTypes":["c5a","m5a","c5","m5"],"subnets":["'$</span>subnet_useast1a'&quot;,&quot;'<span class="math-container">$subnet_useast1b'","'$</span>subnet_useast1c'&quot;],&quot;securityGroupIds&quot;:[&quot;sg-ff2fd48f&quot;,&quot;sg-65ff5315&quot;],&quot;ec2KeyPair&quot;:&quot;SBTools&quot;,&quot;instanceRole&quot;:&quot;ecsInstanceRole&quot;}'
  aws batch create-compute-environment \
    --compute-environment-name <span class="math-container">${container}_$</span>{EC2_type}_CE \
    --type MANAGED \
    --state ENABLED \
    --compute-resources $compute_resources \
    --service-role AWSBatchServiceRole

  printf &quot;Waiting for Compute Environment to be created&quot;
  while [[ <span class="math-container">$ce_status != "VALID" ]]; do
sleep 5
printf "."
ce_info=$</span>(aws batch describe-compute-environments --compute-environments <span class="math-container">${container}_$</span>{EC2_type}_CE)
    ce_status=<span class="math-container">$(echo $</span>ce_info | jq '.computeEnvironments[0] | .status // &quot;&quot;' | tr -d '&quot;')
  done
  echo &quot;&quot;
fi

# Create the job queue
jq_info=<span class="math-container">$(aws batch describe-job-queues --job-queues $</span>{container}_<span class="math-container">${EC2_type}_JQ)
jq_status=$</span>(echo <span class="math-container">$jq_info | jq '.jobQueues[0] | .status // ""' | tr -d '"')
if [[ $</span>jq_status != &quot;VALID&quot; ]]; then
  aws batch create-job-queue \
    --job-queue-name <span class="math-container">${container}_$</span>{EC2_type}_JQ \
    --state ENABLED \
    --priority 5 \
    --compute-environment-order '{&quot;order&quot;:1,&quot;computeEnvironment&quot;:&quot;'<span class="math-container">${container}_$</span>{EC2_type}_CE'&quot;}'
fi

# Register job definition
job_definition=&quot;<span class="math-container">${container}_JD"
jd_info=$</span>(aws batch describe-job-definitions --status ACTIVE --job-definition-name <span class="math-container">$job_definition)
jd_status=$</span>(echo <span class="math-container">$jd_info | jq '.jobDefinitions[0] | .status // ""' | tr -d '"')
if [[ $</span>jd_status != &quot;ACTIVE&quot; ]]; then
  aws batch register-job-definition \
    --job-definition-name <span class="math-container">$job_definition \
    --type container \
    --container-properties '{"privileged":true,"image":"'$repo/$image_and_tag'","vcpus":'${vCPUs_req}',"memory":'$</span>{memory_MiB_req}',&quot;mountPoints&quot;:[{&quot;sourceVolume&quot;:&quot;SB-EFS&quot;,&quot;containerPath&quot;:&quot;/EFS&quot;}],&quot;volumes&quot;:[{&quot;host&quot;:{&quot;sourcePath&quot;:&quot;/&quot;},&quot;name&quot;:&quot;SB-EFS&quot;}],&quot;command&quot;:[&quot;Ref::container&quot;,&quot;Ref::run_id&quot;,&quot;Ref::input_file&quot;]}' \
    --retry-strategy '{&quot;attempts&quot;: 10,&quot;evaluateOnExit&quot;:[{&quot;onStatusReason&quot; :&quot;Host EC2*&quot;,&quot;action&quot;: &quot;RETRY&quot;},{&quot;onReason&quot; : &quot;*&quot;,&quot;action&quot;: &quot;EXIT&quot;}]}'
fi

# Make sure the EFS has an access point set up
efs_id=<span class="math-container">$(aws efs describe-file-systems | jq -r '.FileSystems[] | select (.Name=="SB-EFS") | .FileSystemId')
if [[ "$</span>efs_id&quot; == &quot;&quot; ]]; then
  echo &quot;ERROR: SB-EFS does not exist&quot;
  exit
fi
ap_id=<span class="math-container">$(aws efs describe-access-points | jq -r '.AccessPoints[] | select (.Name=="EFS_AP") | .AccessPointId')
if [[ "$</span>ap_id&quot; == &quot;&quot; ]]; then
  # 1001:1001 is user sb in group sb
  ap_id=<span class="math-container">$(aws efs create-access-point --file-system-id "$</span>efs_id&quot; --posix-user '{&quot;Uid&quot;:1001,&quot;Gid&quot;:1001}' --root-directory '{&quot;Path&quot;:&quot;/&quot;}' --tags '[{&quot;Key&quot;:&quot;Name&quot;,&quot;Value&quot;:&quot;EFS_AP&quot;}]' | jq -r '.AccessPointId')
fi  
</code></pre>
<p>Now when I run it, everything works as it should until getting to the --retry-strategy command line in the &quot;register job definition&quot; section of the script. The following error is subsequently thrown:</p>
<pre><code>Parameter validation failed:
Unknown parameter in retryStrategy: &quot;evaluateOnExit&quot;, must be one of: attempts
</code></pre>
<p>Not sure what the issue is with the &quot;attempts&quot; parameter that is set. Any potential solutions or guidance to solving this error is much appreciated. Thanks!</p>

CC BY-SA 4.0
2022-10-28T21:08:57.347Z
Hi @Joe_G, could you please provide a bit more information about what you're trying to do with this script? Stories help a *lot* with getting good, helpful answers. I can see something suggesting metagenomics, but it's not clear to me what a correct script would do.
gringer
CC BY-SA 4.0
2022-10-30T00:00:36.100Z
My personal approach would be to use the NextFlow AWS pipeline. It would simplify everything, e.g. NextFlow automatically 'Dockerises'
M__
CC BY-SA 4.0
2024-11-19T19:13:30.047Z
Just a suggestion, but in my experience it's much more readable to use a Python library for such things, try out boto3. I've never tried submitting batch jobs, but I believe it could help in debugging the issue !
h3ab74
<bash><shell><aws>
Error Registering Job Definition AWS
# Error Registering Job Definition AWS

(Asked by: username_0 on 2022-10-27)

I am in the process of creating a bioinformatics pipeline and I am currently trying to setup the AWS infrastructure for which it will run.
The script I am running to setup the AWS batch job looks like the following:
#! /bin/bash

# Setup the AWS Batch architecture
# Copyright 2022, SolareaBio

set -e


# ***** BEGIN custom lines

container="BGC_bacteria"
vCPUs_req=12
memory_MiB_req=40000
# ASK ABOUT CUSTOM CPU AND MEMORY SETTINGS

# ***** END custom lines



container_lc=$(echo $container | tr [A-Z] [a-z])

subnet_useast1a="subnet-42371d0a"
subnet_useast1b="subnet-b03c54ea"
subnet_useast1c="subnet-00ee1364"

usage() {
  cat << EOF
Usage $0 [-h]

Builds the $container docker container, pushes it to the AWS ECR, and sets up the $container AWS Batch environment
EOF
}

while getopts "h" option; do
  case $option in
    h)   usage;exit;;
    [?]) usage; exit;;
  esac
done

script_dir=$(dirname "$0")

repo="990850896279.dkr.ecr.us-east-1.amazonaws.com"

echo "Logging into ECR"
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${repo}

image_and_tag="${container_lc}:latest"
echo "Building the docker container: $image_and_tag"

# If you get errors from the docker build command, try adding --no-cache
docker build --no-cache -f $script_dir/${container}.dockerfile -t $image_and_tag $script_dir

echo "Tagging Image"
docker tag ${image_and_tag} ${repo}/${image_and_tag}

# Create repository in AWS ECR
repo_container_info=$(aws ecr describe-repositories --repository-names ${container_lc})
if [[ "$repo_container_info" == "" ]]; then
  aws ecr create-repository --repository-name ${container_lc}
fi

echo "Pushing Image up to AWS"
docker push ${repo}/${image_and_tag}

# Create compute environment
EC2_type=ON_DEMAND
ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
if [[ $ce_status != "VALID" ]]; then
  compute_resources='{"type":"EC2","allocationStrategy":"BEST_FIT_PROGRESSIVE","minvCpus":0,"maxvCpus":10000,"tags":{"Name":"'${container}_Batch'"},"instanceTypes":["c5a","m5a","c5","m5"],"subnets":["'$subnet_useast1a'","'$subnet_useast1b'","'$subnet_useast1c'"],"securityGroupIds":["sg-ff2fd48f","sg-65ff5315"],"ec2KeyPair":"SBTools","instanceRole":"ecsInstanceRole"}'
  aws batch create-compute-environment \
    --compute-environment-name ${container}_${EC2_type}_CE \
    --type MANAGED \
    --state ENABLED \
    --compute-resources $compute_resources \
    --service-role AWSBatchServiceRole

  printf "Waiting for Compute Environment to be created"
  while [[ $ce_status != "VALID" ]]; do
sleep 5
printf "."
ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
    ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
  done
  echo ""
fi

# Create the job queue
jq_info=$(aws batch describe-job-queues --job-queues ${container}_${EC2_type}_JQ)
jq_status=$(echo $jq_info | jq '.jobQueues[0] | .status // ""' | tr -d '"')
if [[ $jq_status != "VALID" ]]; then
  aws batch create-job-queue \
    --job-queue-name ${container}_${EC2_type}_JQ \
    --state ENABLED \
    --priority 5 \
    --compute-environment-order '{"order":1,"computeEnvironment":"'${container}_${EC2_type}_CE'"}'
fi

# Register job definition
job_definition="${container}_JD"
jd_info=$(aws batch describe-job-definitions --status ACTIVE --job-definition-name $job_definition)
jd_status=$(echo $jd_info | jq '.jobDefinitions[0] | .status // ""' | tr -d '"')
if [[ $jd_status != "ACTIVE" ]]; then
  aws batch register-job-definition \
    --job-definition-name $job_definition \
    --type container \
    --container-properties '{"privileged":true,"image":"'$repo/$image_and_tag'","vcpus":'${vCPUs_req}',"memory":'${memory_MiB_req}',"mountPoints":[{"sourceVolume":"SB-EFS","containerPath":"/EFS"}],"volumes":[{"host":{"sourcePath":"/"},"name":"SB-EFS"}],"command":["Ref::container","Ref::run_id","Ref::input_file"]}' \
    --retry-strategy '{"attempts": 10,"evaluateOnExit":[{"onStatusReason" :"Host EC2*","action": "RETRY"},{"onReason" : "*","action": "EXIT"}]}'
fi

# Make sure the EFS has an access point set up
efs_id=$(aws efs describe-file-systems | jq -r '.FileSystems[] | select (.Name=="SB-EFS") | .FileSystemId')
if [[ "$efs_id" == "" ]]; then
  echo "ERROR: SB-EFS does not exist"
  exit
fi
ap_id=$(aws efs describe-access-points | jq -r '.AccessPoints[] | select (.Name=="EFS_AP") | .AccessPointId')
if [[ "$ap_id" == "" ]]; then
  # 1001:1001 is user sb in group sb
  ap_id=$(aws efs create-access-point --file-system-id "$efs_id" --posix-user '{"Uid":1001,"Gid":1001}' --root-directory '{"Path":"/"}' --tags '[{"Key":"Name","Value":"EFS_AP"}]' | jq -r '.AccessPointId')
fi  

Now when I run it, everything works as it should until getting to the --retry-strategy command line in the "register job definition" section of the script. The following error is subsequently thrown:
Parameter validation failed:
Unknown parameter in retryStrategy: "evaluateOnExit", must be one of: attempts

Not sure what the issue is with the "attempts" parameter that is set. Any potential solutions or guidance to solving this error is much appreciated. Thanks!

**Tags:** bash, shell, aws
> **Comments:**
> **username_1**: Hi @Joe_G, could you please provide a bit more information about what you're trying to do with this script? Stories help a *lot* with getting good, helpful answers. I can see something suggesting metagenomics, but it's not clear to me what a correct script would do.  
> **username_2**: My personal approach would be to use the NextFlow AWS pipeline. It would simplify everything, e.g. NextFlow automatically 'Dockerises'  
> **username_3**: Just a suggestion, but in my experience it's much more readable to use a Python library for such things, try out boto3. I've never tried submitting batch jobs, but I believe it could help in debugging the issue !  

---

## 1 answers

### Answer by username_4 on 2022-10-31. Score: 0

You didn't say what version of the AWS CLI you're using, but I suspect it's version 1. It looks like the `evaluateOnExit` feature was added in version 2 but was not backported. Installing AWS CLI version 2 resolves the issue:

<https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html>

Note that [Nextflow](https://www.nextflow.io/) also provides built-in support for [AWS Batch](https://www.nextflow.io/docs/latest/aws.html#aws-batch). You'd still need to configure some AWS infrastructure using Nextflow, but it should be a lot less painful than writing/maintaining your own script(s). It's definitely worth checking out if you haven't already.


---

