<p>Bash functions (even when exported) are not preserved across <code>commands</code> steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:</p>
<ol>
<li><p>Combine steps which need the function in a single step. E.g.</p>
<pre class="lang-yaml prettyprint-override"><code>commands:
  - |
    replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
    replaceConfig DB_USERNAME &amp;&amp;
      replaceConfig DB_PASSWORD &amp;&amp;
      replaceConfig DB_HOST
</code></pre>
</li>
<li><p>Source the file with the functions you need in every step. This may sound laborious, but can be automated via the <code>BASH_ENV</code> environment variable. E.g. let's say you put all the functions you need in a <code>codbuild-helper.bash</code> file in your project root, then you can do:</p>
<pre class="lang-yaml prettyprint-override"><code>env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
    BASH_ENV: &quot;$CODEBUILD_SRC_DIR/codebuild-helper.bash&quot;
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....
</code></pre>
<p>But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.</p>
</li>
</ol>

CC BY-SA 4.0
2022-08-26T02:07:11.107Z
2022-08-26T02:07:11.107Z
David Ongaro
<p>In cases where you have a complex bash script with functions for AWS CodeBuild that may not translate well into YAML syntax, encoding the script with base64 provides a robust solution. It preserves the script's integrity, particularly when it contains special characters or constructs that are problematic to escape directly in YAML.</p>
<p>Here's a streamlined approach</p>
<pre><code>import base64

# Read the script content from a file
with open('path/to/your/script.sh', 'r') as file:
    script_content = file.read()

# Base64 encode the script to avoid issues with special characters
encoded_script = base64.b64encode(script_content.encode()).decode()

# Generate the commands to decode and execute the script within the build environment
commands = [
    &quot;echo 'Starting the pre-build process'&quot;,
    f&quot;echo {encoded_script} | base64 --decode &gt; setup_env.sh&quot;,
    &quot;chmod +x setup_env.sh&quot;,
    &quot;./setup_env.sh&quot;
]

# Integrate these commands into your buildspec configuration
build_spec_yaml = {
    &quot;version&quot;: &quot;0.2&quot;,
    &quot;env&quot;: {
        &quot;parameter-store&quot;: {
            &quot;ModelPackageName&quot;: config['SsmModelPackageNameParameter'],
        }
    },
    &quot;phases&quot;: {
        &quot;pre_build&quot;: {
            &quot;commands&quot;: commands
        }
        # Other phases as needed...
    }
    # Additional buildspec configurations...
}

# If using AWS CDK, you can then convert this dictionary to YAML format
build_spec = BuildSpec.from_object_to_yaml(build_spec_yaml)
</code></pre>
<p>This method is advantageous for those utilizing AWS CDK for their infrastructure as code, allowing the dynamic generation and modification of buildspec configurations directly from Python scripts. It leverages the BuildSpec.from_object_to_yaml method for seamless translation to YAML, which can then be used by CodeBuild for provisioning through AWS CloudFormation.</p>
<p>Hope this solution aids anyone facing similar challenges and enhances the flexibility of managing build specifications in complex environments.</p>
<p>Feel free to share your insights or improvements to this approach!</p>

CC BY-SA 4.0
2024-04-13T17:22:13.250Z
2024-04-13T17:22:13.250Z
cjoun16
<p>I thought I could slightly improve readability of repetitive <code>sed</code> calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:</p>
<pre><code>replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
</code></pre>
<p>This can be called like so:</p>
<pre><code>replaceConfig DB_PASSWORD
</code></pre>
<p>And this will replace all instances of the text <code>{DB_PASSWORD}</code> in config.json with the value held in secret environment variable <code>DB_PASSWORD</code>. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:</p>
<pre><code>[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
*codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found
</code></pre>
<p>I get the same error defining the bash function with the alternative syntax:</p>
<pre><code>function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }
</code></pre>
<p>I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.</p>
<p>Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?</p>
<pre class="lang-yaml prettyprint-override"><code>version: 0.2

run-as: root

env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

</code></pre>

CC BY-SA 4.0
2020-08-03T22:14:28.783Z
Welcome to DevOps.SE, Kyle.  This seems like a good idea to me.  I'm not sure why it doesn't work.  Have you tried contacting AWS support?
chicks
CC BY-SA 4.0
2020-08-15T15:35:13.670Z
Thank you for commenting. I use free tier, which as far as I know does not allow me to contact AWS support for this kind of issue. I will try posting in the AWS Developer Forums though.
Kyle
CC BY-SA 4.0
2021-08-04T18:21:14.477Z
Sidenote: using envsubst is easier than sed almost every time. `envsubst < config.json.tpl > config.json`. It substitutes env vars into a template file, simple one liner. We can also tell envsubst to only replace a set of specific var names if desired.
`envsubst` is installed by default since codebuild standard 2.0 iirc
OnethingSimple
CC BY-SA 4.0
2022-01-17T14:57:01.053Z
@OnethingSimple Thanks for this. Although it's a sidenote to the "XY Problem" I asked about, it does solve my original problem. :-)
Kyle
<amazon-web-services><bash><aws-codepipeline>
Defining and using bash functions in AWS CodeBuild
Kyle
# Defining and using bash functions in AWS CodeBuild

(Asked by: username_0 on 2020-07-31)

I thought I could slightly improve readability of repetitive sed calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:
replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }

This can be called like so:
replaceConfig DB_PASSWORD

And this will replace all instances of the text {DB_PASSWORD} in config.json with the value held in secret environment variable DB_PASSWORD. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:
[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
*codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found

I get the same error defining the bash function with the alternative syntax:
function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }

I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.
Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?
version: 0.2

run-as: root

env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

**Tags:** amazon-web-services, bash, aws-codepipeline

> **Comments:**
> **username_1**: Welcome to DevOps.SE, username_0.  This seems like a good idea to me.  I'm not sure why it doesn't work.  Have you tried contacting AWS support?  
> **username_0**: Thank you for commenting. I use free tier, which as far as I know does not allow me to contact AWS support for this kind of issue. I will try posting in the AWS Developer Forums though.  
> **username_2**: Sidenote: using envsubst is easier than sed almost every time. `envsubst < config.json.tpl > config.json`. It substitutes env vars into a template file, simple one liner. We can also tell envsubst to only replace a set of specific var names if desired. `envsubst` is installed by default since codebuild standard 2.0 iirc  
> **username_0**: @username_2 Thanks for this. Although it's a sidenote to the "XY Problem" I asked about, it does solve my original problem. :-)  

---

## 2 answers

### Answer by username_3 on 2022-08-26. Score: 1

Bash functions (even when exported) are not preserved across commands steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:

Combine steps which need the function in a single step. E.g.
commands:
  - |
    replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
    replaceConfig DB_USERNAME &&
      replaceConfig DB_PASSWORD &&
      replaceConfig DB_HOST


Source the file with the functions you need in every step. This may sound laborious, but can be automated via the BASH_ENV environment variable. E.g. let's say you put all the functions you need in a codbuild-helper.bash file in your project root, then you can do:
env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
    BASH_ENV: "$CODEBUILD_SRC_DIR/codebuild-helper.bash"
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.


---

### Answer by username_4 on 2024-04-13. Score: 0

In cases where you have a complex bash script with functions for AWS CodeBuild that may not translate well into YAML syntax, encoding the script with base64 provides a robust solution. It preserves the script's integrity, particularly when it contains special characters or constructs that are problematic to escape directly in YAML.
Here's a streamlined approach
import base64

# Read the script content from a file
with open('path/to/your/script.sh', 'r') as file:
    script_content = file.read()

# Base64 encode the script to avoid issues with special characters
encoded_script = base64.b64encode(script_content.encode()).decode()

# Generate the commands to decode and execute the script within the build environment
commands = [
    "echo 'Starting the pre-build process'",
    f"echo {encoded_script} | base64 --decode > setup_env.sh",
    "chmod +x setup_env.sh",
    "./setup_env.sh"
]

# Integrate these commands into your buildspec configuration
build_spec_yaml = {
    "version": "0.2",
    "env": {
        "parameter-store": {
            "ModelPackageName": config['SsmModelPackageNameParameter'],
        }
    },
    "phases": {
        "pre_build": {
            "commands": commands
        }
        # Other phases as needed...
    }
    # Additional buildspec configurations...
}

# If using AWS CDK, you can then convert this dictionary to YAML format
build_spec = BuildSpec.from_object_to_yaml(build_spec_yaml)

This method is advantageous for those utilizing AWS CDK for their infrastructure as code, allowing the dynamic generation and modification of buildspec configurations directly from Python scripts. It leverages the BuildSpec.from_object_to_yaml method for seamless translation to YAML, which can then be used by CodeBuild for provisioning through AWS CloudFormation.
Hope this solution aids anyone facing similar challenges and enhances the flexibility of managing build specifications in complex environments.
Feel free to share your insights or improvements to this approach!


---

