<blockquote>
  <p>What are the general ways to tackle this problem?</p>
</blockquote>

<p>The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.</p>

<p>Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:</p>

<pre><code>python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&amp;\&amp; pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*
</code></pre>

CC BY-SA 3.0
2017-09-17T08:38:56.583Z
2018-01-22T05:33:34.667Z
2018-01-22T05:33:34.667Z
user2640621
<p>Currently, we build our application on a build server where we create a virtual environment using <a href="https://virtualenv.pypa.io/en/stable/" rel="nofollow noreferrer"><code>virtualenv</code> command</a>, install all the Python dependencies into it, then "patch" it using the following bash commands to make it <em>relocatable</em>:</p>

<pre><code># Adopt virtualenv to the target system
echo "* Patching virtual envirinment for target system"
for f in "${VENV_PATH}/bin"/* ; do
    if [ -n "$(file "${f}" | grep "text")" ] ; then
    echo "  + ${f}"
        sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
    fi
done
rm -rf "${VENV_PATH}/lib64"
ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"
</code></pre>

<p>Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via <code>yum</code>.</p>

<p>The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to <em>avoid installing the same dependencies on all the target servers</em>.</p>

<p>The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).</p>

<p>Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate <code>pip</code> installation commands with, for example, <a href="https://github.com/paylogic/pip-accel" rel="nofollow noreferrer"><code>pip-accel</code></a> or caching) What are the general ways to tackle this problem?</p>

CC BY-SA 3.0
2017-09-10T00:21:52.097Z
unfortunately the `--relocatable` flag for virtualenv never did work well, as documented here

https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable

If the project found it fragile I bet it will be for you too.

deb's have `dh-virtualenv` but I don't know of one for rpm, but how about pip freeze and wheel?

`pip wheel --wheel-dir=./artifacts -r requirements.txt`

Package ./artifacts into your RPM, create and activate the venv then run `pip install ./artifacts/`

We shove our wheels into a gcp bucket and just add the urls for builds via jenkins.
gdahlm
<deployment><python><performance><virtualenv>
Re-using Python virtual environment vs Re-building it
alecxe
# Re-using Python virtual environment vs Re-building it

(Asked by: username_0 on 2017-09-07)

Currently, we build our application on a build server where we create a virtual environment using virtualenv command, install all the Python dependencies into it, then "patch" it using the following bash commands to make it relocatable:
# Adopt virtualenv to the target system
echo "* Patching virtual envirinment for target system"
for f in "${VENV_PATH}/bin"/* ; do
    if [ -n "$(file "${f}" | grep "text")" ] ; then
    echo "  + ${f}"
        sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
    fi
done
rm -rf "${VENV_PATH}/lib64"
ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"

Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via yum.
The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to avoid installing the same dependencies on all the target servers.
The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).
Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate pip installation commands with, for example, pip-accel or caching) What are the general ways to tackle this problem?

**Tags:** deployment, python, performance, virtualenv

> **Comments:**
> **username_1**: unfortunately the `--relocatable` flag for virtualenv never did work well, as documented here  https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable  If the project found it fragile I bet it will be for you too.  deb's have `dh-virtualenv` but I don't know of one for rpm, but how about pip freeze and wheel?  `pip wheel --wheel-dir=./artifacts -r requirements.txt`  Package ./artifacts into your RPM, create and activate the venv then run `pip install ./artifacts/`  We shove our wheels into a gcp bucket and just add the urls for builds via jenkins.  

---

## 1 answers

### Answer by username_2 on 2017-09-17. Score: 4 | Accepted answer

What are the general ways to tackle this problem?

The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.
Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:
python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*


---

