<p>We have a pre-receive hook script in a bash shell script that enforces commits with messages having the right ticket number in a pre-defined format. We also have a separate python script that can be called in the pre-receive hook bash script to call PMD to statically analyze source code in a branch that developers push to the target remote branch.</p>

<p>Prior to now, we have been using the pre-receive bash script for validating our development process with respect to our Gitlab server issue tickets.</p>

<p>I have recently attempted to call the python script in the bash script in such a way that our ticketing and milestone validations are done before calling up PMD to expose source code that does not meet the standard. This did not work in that order, so I decided to call the python script to run PMD before the ticket validations. This also resulted in only the python script running alone without calling the rest of the bash script code.</p>

<p>I will appreciate any help. Please kindly find the source code of the pre-receive bash script for when I tried calling python/PMD first. Kindly point me to how I can call the python last after the ticket validations.</p>

<pre class="lang-sh prettyprint-override"><code>#!/bin/bash
#
# pre-receive hook for Commit Check
#
COMPANY_EMAIL="mycorp.org"

readonly PROGNAME=$(basename $0)
readonly PROGDIR=$(readlink -m $(dirname $0))
IS_MERGE=0

check_single_commit()
{
  COMMIT_CHECK_STATUS=1
    echo "Repo &gt;&gt; $REPOSITORY_BASENAME"

    if [[ "$COMMIT_MESSAGE" == "Merge branch"* ]] || [[ "$COMMIT_MESSAGE" == "WIP MI"* ]]; then
      COMMIT_CHECK_STATUS=0
      IS_MERGE=1
    else
    workFlowResult=`java -jar -Dspring.config.location=/home/gitlab/gitlab_custom_hooks/application.properties /home/gitlab/gitlab_custom_hooks/gitlab-tool.jar -prercv "$COMMIT_AUTHOR" "$COMMIT_MESSAGE" "$REPOSITORY_BASENAME"`
    echo "COMMIT_AUTHOR=$COMMIT_AUTHOR, COMMIT_MESSAGE=$COMMIT_MESSAGE, REPOSITORY_BASE=$REPOSITORY_BASENAME"

      echo " &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; $workFlowResult &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" &gt;&amp;2
    if [[ "$workFlowResult" == *"PRE_RECEIVE_OK"* ]]; then
      echo " &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; $workFlowResult &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" &gt;&amp;2
      COMMIT_CHECK_STATUS=0
    fi

    fi
}

check_all_commits()
{
  REVISIONS=$(git rev-list $OLD_REVISION..$NEW_REVISION)
  IFS='\n' read -ra LIST_OF_REVISIONS &lt;&lt;&lt; "$REVISIONS"
if [ $(git rev-parse --is-bare-repository) = true ]
then
    REPOSITORY_BASENAME=$(basename "$PWD")
else
    REPOSITORY_BASENAME=$(basename $(readlink -nf "$PWD"/..))
fi
    echo REPOSITORY_BASENAME is $REPOSITORY_BASENAME
    REPOSITORY_BASENAME=$(basename "$PWD")
    REPOSITORY_BASENAME=${REPOSITORY_BASENAME%.git}

  for rid in "${!LIST_OF_REVISIONS[@]}"; do
    REVISION=${LIST_OF_REVISIONS[rid]}
    COMMIT_MESSAGE=$(git cat-file commit $REVISION | sed '1,/^$/d')
    COMMIT_AUTHOR=$(git cat-file commit $REVISION | grep committer | sed 's/^.* \([^@ ]\+@[^ ]\+\) \?.*$/\1/' | sed 's/&lt;//' | sed 's/&gt;//' | sed 's/@$COMPANY_EMAIL//')
    check_single_commit

    if [ "$COMMIT_CHECK_STATUS" != "0" ]; then
      echo "Commit validation failed for commit $REVISION" &gt;&amp;2
      exit 1
    fi

  done
}



echo "Checking the coding style, coding standards and look out for potential issues"

if python /home/gitlab/gitlab_custom_hooks/pmd-policy.py; then
   echo "PMD_CHECK_OK"
   PMD_CHECK_STATUS=0

  # Get custom commit message format
    while read OLD_REVISION NEW_REVISION REFNAME ; do
        check_all_commits
    done



    exit 0

else

  echo "PMD_CHECK_FAILED: Review your Code and fix the issues"
  exit 1

fi
</code></pre>

<p>Below is the code for the python script that calls PMD last commit to a branch at the Gitlab server.</p>

<pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python

import subprocess
import sys
import tempfile
import shutil
import os
import errno


pmd = '/home/gitlab/gitlab_custom_hooks/pmd-bin-5.5.4/bin/run.sh'

# implementing check_output for python &lt; 2.7
if not hasattr(subprocess, 'check_output'):
    def check_output(*popenargs, **kwargs):
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
            er = subprocess.CalledProcessError(retcode, cmd)
            er.output = output
            raise er
        return output
    subprocess.check_output = check_output


# helper for calling executables
def call(*args, **kwargs):
    return subprocess.check_output(*args, **kwargs).strip()


# helper for calling git
def call_git(cmd, *args, **kwargs):
    return call(['git'] + cmd, *args, **kwargs)


# get all new commits from stdin
def get_commits():
    commits = {}
    for line in sys.stdin:
        old, new, ref = line.strip().split(' ')
        if old == '0000000000000000000000000000000000000000':
            old = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'

        if ref not in commits:
            commits[ref] = []
        commits[ref].append({
            'old': old,
            'new': new,
            'files': get_changed_files(old, new)
            })

    return commits


# get a list of changed files between to commits
def get_changed_files(old, new):
    return call_git(['diff', '--name-only', old, new]).split('\n')


# get filemode, object type (blob,tree,commit), hash for the given file at the
# given commit
def get_change_type(commit, filename):
    return call_git(['ls-tree', commit, filename]).split('\t')[0].split(' ')


commits = get_commits()

# use the latest file commit only
print "Cleaning up file list..."

files = {}
count = 0
for ref, data in commits.iteritems():
    files[ref] = {}
    for commit in data:
        for filename in commit['files']:
            if not filename.lower().endswith('.java'): continue
            files[ref][filename] = get_change_type(commit['new'], filename)
    count += len(files[ref])

print "%d Files to check in %d branches" % (count, len(files))

# create temporary dir and save a copy of the new files
tempdir = tempfile.mkdtemp('git_hook')
for ref, files in files.iteritems():
    for filename, data in files.iteritems():
        dname = os.path.dirname(filename)
        bname = os.path.basename(filename)
        try:
            os.makedirs(os.path.join(tempdir, dname))
        except OSError, exc:
            if exc.errno == errno.EEXIST:  # directory exists already
                pass
            else:
                raise

        with open(os.path.join(tempdir, dname, bname), 'w') as fp:
            fp.write(call_git(['cat-file', data[1], data[2]]))

try:
    # call checkstyle and/or pmd and print output
    print call([pmd, 'pmd', '-d', tempdir, '-f', 'text', '-R', 'rulesets/java/basic.xml,rulesets/java/unusedcode.xml,rulesets/java/imports.xml,rulesets/java/strings.xml,rulesets/java/braces.xml,rulesets/java/clone.xml,rulesets/java/design.xml,rulesets/java/clone.xml,rulesets/java/finalizers.xml,rulesets/java/junit.xml,rulesets/java/migrating.xml,rulesets/java/optimizations.xml,rulesets/java/strictexception.xml,rulesets/java/sunsecure.xml,rulesets/java/typeresolution.xml'])
    print "SUCCESS"
    sys.exit(0)
except subprocess.CalledProcessError, ex:
    print ex.output  # print checkstyle and/or pmd messages
    exit(1)
finally:
    # remove temporary directory
    shutil.rmtree(tempdir)
</code></pre>

CC BY-SA 3.0
2017-05-06T02:34:37.820Z
It's hard to correlate your description of the events with the code. Can you show some logs - how were the scripts invoked, what actually happened, what were you expecting?
Dan Cornilescu
CC BY-SA 3.0
2017-05-06T21:34:01.973Z
@DanCornilescu, it is difficult posting the logs from PMD, as it is copious. The git hook pre-receive is called whenever the Git server (Gitlab in this case) receives a push from the client. It is a standard script that is provided for development teams to customize to suit their workflows. You can fail or allow a push depending on pre-defined conditions. The pre-receive hook, in turn, calls the python script.
sage
CC BY-SA 3.0
2017-05-06T22:17:02.633Z
I mean the significant messages from the two scripts, showing how they relate to each other, like "PMD_CHECK_OK" or "PMD_CHECK_FAILED: Review your Code and fix the issues". Or how did the python script exit? Success/failure? etc. You need to be able to debug your scripts and understand what they do to understand how the end-to-end system works and why. Especially when they're not running on your own servers - post mortem logs is all you get - you need to figure out exactly what happened, you messages must tell you that.
Dan Cornilescu
CC BY-SA 3.0
2017-05-09T07:54:09.357Z
Isn't what you're after just moving the while loop before the `if python` ?
Tensibai
CC BY-SA 3.0
2017-05-10T09:28:48.010Z
@Tensibai, thank you very much for your reply. I have moved it before the if python statement and I have also moved it after. In each case, it seems that the git branch files are consumed by the first script that processes it. This might require a deeper understanding of how git works under the hood. The only option I think is viable in this instance is to move the Gitlab ticket validation code from the calling pre-receive bash script to the python script while the pre-receive script will only be a wrapper for invoking the python script. What do you think?
sage
CC BY-SA 3.0
2017-05-10T09:44:27.510Z
I'm unsure of what your scripts does,  there's a bunch of things, including a java call within. both seems to be independent hooks, there's probably something to do to refactor all this into one script
Tensibai
CC BY-SA 3.0
2017-05-10T09:44:49.187Z
Or to refactor them to handle only one commit and loop in a wrapper script, can't really tell.
Tensibai
<git><gitlab><git-hooks><shell-script><python>
Problems with integrating Python script in a Bash script Git pre-receive hook
sage
# Problems with integrating Python script in a Bash script Git pre-receive hook

(Asked by: username_0 on 2017-05-05)

We have a pre-receive hook script in a bash shell script that enforces commits with mesusername_0s having the right ticket number in a pre-defined format. We also have a separate python script that can be called in the pre-receive hook bash script to call PMD to statically analyze source code in a branch that developers push to the target remote branch.
Prior to now, we have been using the pre-receive bash script for validating our development process with respect to our Gitlab server issue tickets.
I have recently attempted to call the python script in the bash script in such a way that our ticketing and milestone validations are done before calling up PMD to expose source code that does not meet the standard. This did not work in that order, so I decided to call the python script to run PMD before the ticket validations. This also resulted in only the python script running alone without calling the rest of the bash script code.
I will appreciate any help. Please kindly find the source code of the pre-receive bash script for when I tried calling python/PMD first. Kindly point me to how I can call the python last after the ticket validations.
#!/bin/bash
#
# pre-receive hook for Commit Check
#
COMPANY_EMAIL="mycorp.org"

readonly PROGNAME=$(basename $0)
readonly PROGDIR=$(readlink -m $(dirname $0))
IS_MERGE=0

check_single_commit()
{
  COMMIT_CHECK_STATUS=1
    echo "Repo >> $REPOSITORY_BASENAME"

    if [[ "$COMMIT_MESSAGE" == "Merge branch"* ]] || [[ "$COMMIT_MESSAGE" == "WIP MI"* ]]; then
      COMMIT_CHECK_STATUS=0
      IS_MERGE=1
    else
    workFlowResult=`java -jar -Dspring.config.location=/home/gitlab/gitlab_custom_hooks/application.properties /home/gitlab/gitlab_custom_hooks/gitlab-tool.jar -prercv "$COMMIT_AUTHOR" "$COMMIT_MESSAGE" "$REPOSITORY_BASENAME"`
    echo "COMMIT_AUTHOR=$COMMIT_AUTHOR, COMMIT_MESSAGE=$COMMIT_MESSAGE, REPOSITORY_BASE=$REPOSITORY_BASENAME"

      echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
    if [[ "$workFlowResult" == *"PRE_RECEIVE_OK"* ]]; then
      echo " >>>>>>>>>>>>>>>>> $workFlowResult >>>>>>>>>>>>>>>>>" >&2
      COMMIT_CHECK_STATUS=0
    fi

    fi
}

check_all_commits()
{
  REVISIONS=$(git rev-list $OLD_REVISION..$NEW_REVISION)
  IFS='\n' read -ra LIST_OF_REVISIONS <<< "$REVISIONS"
if [ $(git rev-parse --is-bare-repository) = true ]
then
    REPOSITORY_BASENAME=$(basename "$PWD")
else
    REPOSITORY_BASENAME=$(basename $(readlink -nf "$PWD"/..))
fi
    echo REPOSITORY_BASENAME is $REPOSITORY_BASENAME
    REPOSITORY_BASENAME=$(basename "$PWD")
    REPOSITORY_BASENAME=${REPOSITORY_BASENAME%.git}

  for rid in "${!LIST_OF_REVISIONS[@]}"; do
    REVISION=${LIST_OF_REVISIONS[rid]}
    COMMIT_MESSAGE=$(git cat-file commit $REVISION | sed '1,/^$/d')
    COMMIT_AUTHOR=$(git cat-file commit $REVISION | grep committer | sed 's/^.* \([^@ ]\+@[^ ]\+\) \?.*$/\1/' | sed 's///' | sed 's/@$COMPANY_EMAIL//')
    check_single_commit

    if [ "$COMMIT_CHECK_STATUS" != "0" ]; then
      echo "Commit validation failed for commit $REVISION" >&2
      exit 1
    fi

  done
}



echo "Checking the coding style, coding standards and look out for potential issues"

if python /home/gitlab/gitlab_custom_hooks/pmd-policy.py; then
   echo "PMD_CHECK_OK"
   PMD_CHECK_STATUS=0

  # Get custom commit mesusername_0 format
    while read OLD_REVISION NEW_REVISION REFNAME ; do
        check_all_commits
    done



    exit 0

else

  echo "PMD_CHECK_FAILED: Review your Code and fix the issues"
  exit 1

fi

Below is the code for the python script that calls PMD last commit to a branch at the Gitlab server.
#!/usr/bin/env python

import subprocess
import sys
import tempfile
import shutil
import os
import errno


pmd = '/home/gitlab/gitlab_custom_hooks/pmd-bin-5.5.4/bin/run.sh'

# implementing check_output for python < 2.7
if not hasattr(subprocess, 'check_output'):
    def check_output(*popenargs, **kwargs):
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
            er = subprocess.CalledProcessError(retcode, cmd)
            er.output = output
            raise er
        return output
    subprocess.check_output = check_output


# helper for calling executables
def call(*args, **kwargs):
    return subprocess.check_output(*args, **kwargs).strip()


# helper for calling git
def call_git(cmd, *args, **kwargs):
    return call(['git'] + cmd, *args, **kwargs)


# get all new commits from stdin
def get_commits():
    commits = {}
    for line in sys.stdin:
        old, new, ref = line.strip().split(' ')
        if old == '0000000000000000000000000000000000000000':
            old = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'

        if ref not in commits:
            commits[ref] = []
        commits[ref].append({
            'old': old,
            'new': new,
            'files': get_changed_files(old, new)
            })

    return commits


# get a list of changed files between to commits
def get_changed_files(old, new):
    return call_git(['diff', '--name-only', old, new]).split('\n')


# get filemode, object type (blob,tree,commit), hash for the given file at the
# given commit
def get_change_type(commit, filename):
    return call_git(['ls-tree', commit, filename]).split('\t')[0].split(' ')


commits = get_commits()

# use the latest file commit only
print "Cleaning up file list..."

files = {}
count = 0
for ref, data in commits.iteritems():
    files[ref] = {}
    for commit in data:
        for filename in commit['files']:
            if not filename.lower().endswith('.java'): continue
            files[ref][filename] = get_change_type(commit['new'], filename)
    count += len(files[ref])

print "%d Files to check in %d branches" % (count, len(files))

# create temporary dir and save a copy of the new files
tempdir = tempfile.mkdtemp('git_hook')
for ref, files in files.iteritems():
    for filename, data in files.iteritems():
        dname = os.path.dirname(filename)
        bname = os.path.basename(filename)
        try:
            os.makedirs(os.path.join(tempdir, dname))
        except OSError, exc:
            if exc.errno == errno.EEXIST:  # directory exists already
                pass
            else:
                raise

        with open(os.path.join(tempdir, dname, bname), 'w') as fp:
            fp.write(call_git(['cat-file', data[1], data[2]]))

try:
    # call checkstyle and/or pmd and print output
    print call([pmd, 'pmd', '-d', tempdir, '-f', 'text', '-R', 'rulesets/java/basic.xml,rulesets/java/unusedcode.xml,rulesets/java/imports.xml,rulesets/java/strings.xml,rulesets/java/braces.xml,rulesets/java/clone.xml,rulesets/java/design.xml,rulesets/java/clone.xml,rulesets/java/finalizers.xml,rulesets/java/junit.xml,rulesets/java/migrating.xml,rulesets/java/optimizations.xml,rulesets/java/strictexception.xml,rulesets/java/sunsecure.xml,rulesets/java/typeresolution.xml'])
    print "SUCCESS"
    sys.exit(0)
except subprocess.CalledProcessError, ex:
    print ex.output  # print checkstyle and/or pmd mesusername_0s
    exit(1)
finally:
    # remove temporary directory
    shutil.rmtree(tempdir)

**Tags:** git, gitlab, git-hooks, shell-script, python

> **Comments:**
> **username_1**: It's hard to correlate your description of the events with the code. Can you show some logs - how were the scripts invoked, what actually happened, what were you expecting?  
> **username_0**: @DanCornilescu, it is difficult posting the logs from PMD, as it is copious. The git hook pre-receive is called whenever the Git server (Gitlab in this case) receives a push from the client. It is a standard script that is provided for development teams to customize to suit their workflows. You can fail or allow a push depending on pre-defined conditions. The pre-receive hook, in turn, calls the python script.  
> **username_1**: I mean the significant mesusername_0s from the two scripts, showing how they relate to each other, like "PMD_CHECK_OK" or "PMD_CHECK_FAILED: Review your Code and fix the issues". Or how did the python script exit? Success/failure? etc. You need to be able to debug your scripts and understand what they do to understand how the end-to-end system works and why. Especially when they're not running on your own servers - post mortem logs is all you get - you need to figure out exactly what happened, you mesusername_0s must tell you that.  
> **username_2**: Isn't what you're after just moving the while loop before the `if python` ?  
> **username_0**: @username_2, thank you very much for your reply. I have moved it before the if python statement and I have also moved it after. In each case, it seems that the git branch files are consumed by the first script that processes it. This might require a deeper understanding of how git works under the hood. The only option I think is viable in this instance is to move the Gitlab ticket validation code from the calling pre-receive bash script to the python script while the pre-receive script will only be a wrapper for invoking the python script. What do you think?  
> **username_2**: I'm unsure of what your scripts does,  there's a bunch of things, including a java call within. both seems to be independent hooks, there's probably something to do to refactor all this into one script  
> **username_2**: Or to refactor them to handle only one commit and loop in a wrapper script, can't really tell.  

---

## 0 answers

