<p>I use <a href="https://www.theforeman.org/" rel="nofollow noreferrer">The Foreman</a> for this. It leverages the <a href="https://github.com/fog/fog-vsphere" rel="nofollow noreferrer">fog vsphere</a> module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.</p>

<p>This software stack is backed by Red Hat and re-branded as <a href="https://access.redhat.com/products/red-hat-satellite" rel="nofollow noreferrer">Red Hat Satellite 6</a> if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.</p>

<p>Upon deployment of your VM, you have the following options for configuring your newly deployed VM:</p>

<ul>
<li>Kickstart script</li>
<li>Puppet</li>
<li>Ansible</li>
<li>Chef</li>
<li>SaltStack</li>
</ul>

<p>As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.</p>

<p>This software comes with a <a href="https://theforeman.org/api/1.17/" rel="nofollow noreferrer">REST API</a> that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).</p>

CC BY-SA 3.0
2018-03-22T15:44:16.940Z
2018-03-22T15:44:16.940Z
James Shewey
<p>Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.</p>

<p>I would further split your two Tasks a bit more differently, such that Step 1 would be to just "<em>Set up the VM</em>", and Step 2 would be to "<em>Provision the VM with required programs and configuration</em>".</p>

<p>My 2-cents would be to use tools such as <a href="https://www.packer.io/" rel="nofollow noreferrer">Packer</a> and <a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a>.</p>

<p>You would use (for example), the <a href="https://www.packer.io/docs/builders/virtualbox-ovf.html" rel="nofollow noreferrer">VirtualBox Packer builder</a>, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the <a href="https://www.packer.io/docs/provisioners/ansible-local.html" rel="nofollow noreferrer">Ansible-local</a> provisioner, or something to this effect, to run a <a href="https://docs.ansible.com/ansible/latest/playbooks.html" rel="nofollow noreferrer">playbook</a>, which would handle the rest of Step 1, and all of Step 2.</p>

<hr>

<p>The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.</p>

<p>There are of course, other tools such as <a href="https://www.vagrantup.com/" rel="nofollow noreferrer">Vagrant</a>, <a href="https://puppet.com/" rel="nofollow noreferrer">Puppet</a>, and <a href="https://www.chef.io/chef/" rel="nofollow noreferrer">Chef</a>, which can handle what you'd like to achieve, so it's up to you find your preferred tool.</p>

<hr>

<p>Mini breakdown of Ansible modules you can use:</p>

<ul>
<li>Open up port: <a href="https://docs.ansible.com/ansible/latest/win_firewall_rule_module.html" rel="nofollow noreferrer">win_firewal_rule_module</a></li>
<li>Installing bitvise and python: <a href="https://docs.ansible.com/ansible/latest/win_chocolatey_module.html" rel="nofollow noreferrer">win_chocolatey</a></li>
<li>Setting up pip and virtual envs: <a href="https://docs.ansible.com/ansible/latest/pip_module.html" rel="nofollow noreferrer">pip</a></li>
<li>Transferring public keys / requirements.txt etc: <a href="https://docs.ansible.com/ansible/latest/win_copy_module.html" rel="nofollow noreferrer">win_copy</a></li>
</ul>

<p>Etc. The full list of Ansible Windows Modules can be found <a href="https://docs.ansible.com/ansible/latest/list_of_windows_modules.html" rel="nofollow noreferrer">here</a>.</p>

CC BY-SA 3.0
2018-03-20T10:50:02.460Z
2018-03-20T10:50:02.460Z
Rekovni
<h2>Automating creating of new VM and pushing code to server</h2>

<p>Hi all,</p>

<p>I am writing a integration test for a client-server application. To simulate
the production server setup, and to update the code, I set up VMs on my
computer. The application runs on a Windows server, so I have to use systems
compatible with Windows. </p>

<h2>STEP 1: Set up VM properly:</h2>

<ol>
<li>Download the OVA file from MS</li>
<li>create a new VM from the OVA file</li>
<li>reset the mac id</li>
<li>set a static dns ip address and a VM ip address</li>
<li>open up the port 22</li>
<li>install bitvise ssh server</li>
<li>transfer public keys to the VM</li>
<li>install python 3.5.2</li>
<li>update <code>pip</code></li>
<li>install <code>virtualenv</code> systemwide</li>
</ol>

<h3>STEP 2: Deploy code to VM:</h3>

<ol>
<li>create a requirements file using pip using: <code>pip freeze -r requirements</code></li>
<li>zip the entire repo using powershell using: 
<code>powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"</code></li>
<li>using <code>pscp</code> I transfer the files to the VM using:
<code>pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop</code> </li>
<li>unzip the files again using powershell using:
<code>powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"</code></li>
<li>delete the <code>virtualenv</code> folder called <code>env</code> in the folder using <code>plink</code></li>
<li>create a fresh <code>env</code> folder using using <code>virtualenv</code>: <code>virtualenv env</code></li>
<li>using the <code>requirements.txt</code> document, I install the requirements again by
direcatly using the <code>pip</code> installed in the virtualenv above using:
<code>/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt</code></li>
<li>run the file with the <code>python</code> distribution installed in the virtualenv
using:
<code>/path/to/python file_name.py</code></li>
</ol>

<p>Because I was trying to execute relatively complex commands, which in turn had
multiple quotes (double and single), spaces etc, I had to spend quite a bit of
time just debbuging how to run these commands over the network.  I have
automated most of the deployment part(<em>STEP 2</em>) using a combination of
<code>subprocess.call</code>, <code>plink</code>, and <code>pscp</code>. However, I haven't yet automated
setting up the VM (<em>STEP 1</em>).</p>

<p>I am guessing, the above set of actions are not just something I am doing, and
probably others have done this as well. </p>

<p>Currently, I am using low level tools to acheive this. While this works for
setting up Windows VM, it wont probably work for Linux based VMs or if I change
python versions use a different language etc etc.</p>

<p>I have also searched a bit and came across this book:
<a href="https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html" rel="nofollow noreferrer">https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html</a></p>

<p>where the author goes into how to deploy to a linux server. However, it seems
that he is essentially doing what I am doing by directly running commands on
the linux server, just through the <code>fabric</code> wrapper. Because <code>ssh</code> doesn't work
as well on Windows, I am guessing, I will have to get back to the nitty-grittys
of deployment with <code>fabric</code> as well.</p>

<h3>Question:</h3>

<ol>
<li><p>Is there a better way to setup the VM, and deploy code to the VMs, while
abstracting away the difference between different operating systems, python
versions, tools etc?</p></li>
<li><p>I have also heard of a <code>git</code> based workflow. Is that better than directly pushing the code to the server? Why so?</p></li>
</ol>

CC BY-SA 3.0
2018-03-20T09:28:44.983Z
When looking for this kind of things I've choosen Chef because the client has native support for windows functionality and there's a cookbook for python supporting windows made by coderanger which is someone you can be confident in the quality of work: https://github.com/poise/poise-python#windows-support
Tensibai
CC BY-SA 3.0
2018-03-20T09:30:46.797Z
BTW I'm afraid this is going to be mainly opinion based, everyone will  have his/her own method working great for him/her. You mainly need a mix of Infrastructure as code and configuration management. I think vagrant+chef could be a good setup for your case, but there's probably other.
Tensibai
<deployment><git><python><virtualenv>
Automating creation of new VM and pushing code to server
alpha_989
# Automating creation of new VM and pushing code to server

(Asked by: username_0 on 2018-03-19)

## Automating creating of new VM and pushing code to server

Hi all,

I am writing a integration test for a client-server application. To simulate
the production server setup, and to update the code, I set up VMs on my
computer. The application runs on a Windows server, so I have to use systems
compatible with Windows.

## STEP 1: Set up VM properly:

1. Download the OVA file from MS
2. create a new VM from the OVA file
3. reset the mac id
4. set a static dns ip address and a VM ip address
5. open up the port 22
6. install bitvise ssh server
7. transfer public keys to the VM
8. install python 3.5.2
9. update `pip`
10. install `virtualenv` systemwide

### STEP 2: Deploy code to VM:

1. create a requirements file using pip using: `pip freeze -r requirements`
2. zip the entire repo using powershell using:
   `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
3. using `pscp` I transfer the files to the VM using:
   `pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop`
4. unzip the files again using powershell using:
   `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
5. delete the `virtualenv` folder called `env` in the folder using `plink`
6. create a fresh `env` folder using using `virtualenv`: `virtualenv env`
7. using the `requirements.txt` document, I install the requirements again by
   direcatly using the `pip` installed in the virtualenv above using:
   `/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt`
8. run the file with the `python` distribution installed in the virtualenv
   using:
   `/path/to/python file_name.py`

Because I was trying to execute relatively complex commands, which in turn had
multiple quotes (double and single), spaces etc, I had to spend quite a bit of
time just debbuging how to run these commands over the network. I have
automated most of the deployment part(*STEP 2*) using a combination of
`subprocess.call`, `plink`, and `pscp`. However, I haven't yet automated
setting up the VM (*STEP 1*).

I am guessing, the above set of actions are not just something I am doing, and
probably others have done this as well.

Currently, I am using low level tools to acheive this. While this works for
setting up Windows VM, it wont probably work for Linux based VMs or if I change
python versions use a different language etc etc.

I have also searched a bit and came across this book:
<https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html>

where the author goes into how to deploy to a linux server. However, it seems
that he is essentially doing what I am doing by directly running commands on
the linux server, just through the `fabric` wrapper. Because `ssh` doesn't work
as well on Windows, I am guessing, I will have to get back to the nitty-grittys
of deployment with `fabric` as well.

### Question:

1. Is there a better way to setup the VM, and deploy code to the VMs, while
   abstracting away the difference between different operating systems, python
   versions, tools etc?
2. I have also heard of a `git` based workflow. Is that better than directly pushing the code to the server? Why so?

**Tags:** deployment, git, python, virtualenv

> **Comments:**
> **username_1**: When looking for this kind of things I've choosen Chef because the client has native support for windows functionality and there's a cookbook for python supporting windows made by coderanger which is someone you can be confident in the quality of work: https://github.com/poise/poise-python#windows-support  
> **username_1**: BTW I'm afraid this is going to be mainly opinion based, everyone will  have his/her own method working great for him/her. You mainly need a mix of Infrastructure as code and configuration management. I think vagrant+chef could be a good setup for your case, but there's probably other.  

---

## 2 answers

### Answer by username_2 on 2018-03-22. Score: 2

I use [The Foreman](https://www.theforeman.org/) for this. It leverages the [fog vsphere](https://github.com/fog/fog-vsphere) module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.

This software stack is backed by Red Hat and re-branded as [Red Hat Satellite 6](https://access.redhat.com/products/red-hat-satellite) if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.

Upon deployment of your VM, you have the following options for configuring your newly deployed VM:

* Kickstart script
* Puppet
* Ansible
* Chef
* SaltStack

As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.

This software comes with a [REST API](https://theforeman.org/api/1.17/) that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).


---

### Answer by username_3 on 2018-03-20. Score: 2

Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what username_1 has suggested in his comments.

I would further split your two Tasks a bit more differently, such that Step 1 would be to just "*Set up the VM*", and Step 2 would be to "*Provision the VM with required programs and configuration*".

My 2-cents would be to use tools such as [Packer](https://www.packer.io/) and [Ansible](https://www.ansible.com/).

You would use (for example), the [VirtualBox Packer builder](https://www.packer.io/docs/builders/virtualbox-ovf.html), to stand up your VM, which would handle points 1-4 of your Step 1, and then use the [Ansible-local](https://www.packer.io/docs/provisioners/ansible-local.html) provisioner, or something to this effect, to run a [playbook](https://docs.ansible.com/ansible/latest/playbooks.html), which would handle the rest of Step 1, and all of Step 2.

---

The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.

There are of course, other tools such as [Vagrant](https://www.vagrantup.com/), [Puppet](https://puppet.com/), and [Chef](https://www.chef.io/chef/), which can handle what you'd like to achieve, so it's up to you find your preferred tool.

---

Mini breakdown of Ansible modules you can use:

* Open up port: [win\_firewal\_rule\_module](https://docs.ansible.com/ansible/latest/win_firewall_rule_module.html)
* Installing bitvise and python: [win\_chocolatey](https://docs.ansible.com/ansible/latest/win_chocolatey_module.html)
* Setting up pip and virtual envs: [pip](https://docs.ansible.com/ansible/latest/pip_module.html)
* Transferring public keys / requirements.txt etc: [win\_copy](https://docs.ansible.com/ansible/latest/win_copy_module.html)

Etc. The full list of Ansible Windows Modules can be found [here](https://docs.ansible.com/ansible/latest/list_of_windows_modules.html).


---

