In one of my previous blog posts I described that using pre-commit hooks makes life easier, it helps you writing better code. When you want to commit your changes, you immediately get a result if your code has met the various criteria the owner of the repository has set. When you google around, you will find various scripts that you can make use in your own setup. But what if you can not find that specific one? Then you will need to create your own hook.

I had to do this as well, not because I had a rare case to solve, I just wanted to learn how to write a hook myself so I can easily expand my own pre-commit library. And it seems out to be very simple. 🙂

Use case

My very basic use case: I want to validate a yaml file and make sure that it is properly formatted.

I found a very basic tool that helped me format an yaml file, named “yamlfmt“. I need to use this tool in a bash script, which you can see here:

#!/usr/bin/env bash
# Will pretty print YAML files.

if which yamlfmt &> /dev/null $? != 0 ; then
    echo "yamlfmt must be installed: pip install yamlfmt"
    exit 1
fi

EXIT_CODE=0

for file in $@; do
  yamlfmt ${file} -w
  if [[ $? -ne 0 ]]
    then  EXIT_CODE=1
          echo $file
  fi
done
exit $EXIT_CODE

First we need to make sure that we check that the tool is installed, otherwise we print an statement on how the user can install the tool. We need to have that tool installed, so we immediately exit the script with an exit code of 1. With that, the “git commit” action will also fail and thus the user needs to take action to install the tool before it can try to commit the changes.

The last part of the script is doing a for loop and will run the “yamlfmt” command on each file that is passed as an argument to the script. With each pre-commit hook, all of the files that are part of the commit are passed as an argument to the script (Not entirely, but we will discuss this a bit further on ;-)). We collect the exit code of the “yamlfmt” command and checks if that will not equals to 0. If that happens, then there will be an issue with the yaml file and we print the name of the file to stdout.

But how does the script know that it should only do yaml files? If we have a text file or png file, this will fail!

First, we need to add this script into a repository that contains other pre-commit hooks scripts. If you don’t have one, or don’t have other scripts is also fine. I have a dedicated repository for that, which you can find here https://github.com/dj-wasabi/pre-commit-hooks. This repository contains all the pre-commit hooks that I use in 1 or more (public) available repositories. We just need to have a git repository where we can store this script and most importantly, we need to store a “.pre-commit-hooks.yaml” file. This file will contain some information about the scripts which we can use in the rest of our repositories.

In order to get the above menionted script work, we store this script inside the “bin” directory in the git repository and name the file “verify-yaml.sh“. It doesn’t have to be specific in the “bin” directory, you can also just place it in the “ROOT” or some other directory, whatever you please. But in the “ROOT” of the git repository we will have to create the “.pre-commit-hooks.yaml” file and we will include the following contents:

- id: verify-yaml
  name: Pretty Print YAML files
  description: Checks YAML files and pretty prints them
  entry: bin/verify-yaml.sh
  language: script
  files: \.(ya?ml)$

There are 2 important keys that requires some additional information (I won’t have to tell you that the “entry” key is the location to the script right? Oh wait, I just did. :)) These are the “id” and the “files“.

The “id” is the value that you need to use in your repositories where you want to make use of the pre-commit hooks. Like with the https://github.com/dj-wasabi/dj-wasabi-release/blob/main/.pre-commit-config.yaml you will see the following:

repos:
- repo: https://github.com/dj-wasabi/pre-commit-hooks
  rev: master
  hooks:
  ...
  - id: verify-yaml
  ...

So every time I do a commit in my “dj-wasabi/dj-wasabi-release” repository, it will execute the pre-commit hook with id “verify-yaml“.

The “files” key in the “.pre-commit-hooks.yaml” entry is when the script should be executed. We only want the script to be executed when the file is a yaml file and not with a text or png file. The “\.(ya?ml)$” makes sure that it only affect files with the file extention “.yml” and “.yaml“.

Thats all folks!

It looks very easy right? Yes it sure is and when you start to work on your own hooks, you will create more. Because I am in a writing mood right now, lets take a look at the following script that I wrote:

#!/usr/bin/env bash
# Do not allow commits on provided branches.


EXITCODE=0
while getopts b: flag
do
    case "${flag}" in
        b) BRANCHES=${OPTARG};;
        *) echo "Unsupported option provided."
           exit 1;;
    esac
done

for BRANCH in $(echo ${BRANCHES} | sed 's/,/ /g');
    do
        if git rev-parse --abbrev-ref HEAD | grep -e ${BRANCH} > /dev/null 2>&1
            then  echo "You are not allowed to commit on ${BRANCH} branch."
                  EXITCODE=1
        fi
done
exit ${EXITCODE}

I don’t want to commit my changes on “master” or on “main“. And yes you can configure in most cases on the remote site (Read: the Git Server, like Bitbucket or Github) that it will not allow pushes to “master” or “main“, but I just want to prevent the commiting to actual take place. Everytime when I commit on a branch that the remote server is not accepting, I have to google for “git commit undo” and look for the command to undo my commit (Because for some reason I can not remember the git undo command :)).

Once I have undo my commit, I will create a proper branch and do the commit and push again. So if I can prevent committing to “master” or “main” with a pre-commit hook, my life will get easier because I don’t have to undo my commit etc! (Wait, wasn’t that the title of the blog post that I have written before this blog post? ) 🙂

If you have any question and or remarks, please let me know. If you have written a hook yourself and you want to show it, let me know as well!

May the hooks be with you!

Pull requests, you either love and see that they do provide benefits in the way of working, or you dislike them and see no purpose in them at all. I discussed the Pull Requests process in these “Author“, “Reviewer” and “Process” blogposts, so check these out as well. No matter which side you are on, but I think you want to create quality code and be consistent with each change. And if you are like me, then you would expect this as well from your teammates/co-workers. But how can we make that happen?

Pre-commit hooks can help with that. A pre-commit is one of the several hooks that are available in git that we can use to execute scripts, while running an git command. A full list with hooks can be found on this page https://githooks.com/. 

With a pre-commit hook, we can execute scripts as part of the “git commit” command. If the exit code of all of the scripts result in a zero (successful execution) then the commit will take place. If 1 or more of the scripts fail, the commit will be unsuccessful and fails to complete. Once that is happening we need to resolve the issue and try again.

When you have a git repository somewhere cloned on your host, you are already able to use pre-commit hooks or any of the other hooks that Git has. In the “.git/hook” directory you will find a bunch of “.sample” scripts. You can use these as an example and if you remove the “.sample” in the filename, then the script is active and triggered when the stage is executed.

$  ls -l .git/hooks
total 112
-rwxr-xr-x  1 wdijkerman  staff   478 Jun 14 21:01 applypatch-msg.sample
-rwxr-xr-x  1 wdijkerman  staff   896 Jun 14 21:01 commit-msg.sample
-rwxr-xr-x  1 wdijkerman  staff  3327 Jun 14 21:01 fsmonitor-watchman.sample
-rwxr-xr-x  1 wdijkerman  staff   189 Jun 14 21:01 post-update.sample
-rwxr-xr-x  1 wdijkerman  staff   424 Jun 14 21:01 pre-applypatch.sample
-rwxr-xr-x  1 wdijkerman  staff  1638 Jun 14 21:01 pre-commit.sample
-rwxr-xr-x  1 wdijkerman  staff   416 Jun 14 21:01 pre-merge-commit.sample
-rwxr-xr-x  1 wdijkerman  staff  1348 Jun 14 21:01 pre-push.sample
-rwxr-xr-x  1 wdijkerman  staff  4898 Jun 14 21:01 pre-rebase.sample
-rwxr-xr-x  1 wdijkerman  staff   544 Jun 14 21:01 pre-receive.sample
-rwxr-xr-x  1 wdijkerman  staff  1492 Jun 14 21:01 prepare-commit-msg.sample
-rwxr-xr-x  1 wdijkerman  staff  3610 Jun 14 21:01 update.sample

But we are focussing on to pre-commit hooks and there is an easier way to maintain the pre-commit hooks. Like you already have seen, we can only use 1 pre-commit file, and we want to use a lot more than just one. Sure you can call other scripts, but that makes the maintenance a bit more complicated. Especially when you have more than just a single git repository. There is a package that helps us to have a more maintainable soultion with regarding to pre-commit hooks and using them.

Installation

We have to install it first and we have to install it on our development workstation/laptop. Make sure you have Python and Pip installed before executing the following command:

pip install pre-commit

This will install the pre-commit application on our system.

“That is all nice, but I have no idea what kind of scripts we can execute as part of a pre-commit?”

– you?

No worries, lets use an example. This Github repository https://github.com/dj-wasabi/dj-wasabi-release/ contains some Python scripts that I use for maintaining my Github repositories. We will not discuss these scripts, but there is 1 important file in this repository and is named “.pre-commit-config.yaml“. You will probably find this file also on other repositories that are on my Github account. This file contains all the scripts that will be executed when we do a “git commit”. Lets take a look at the file.

repos:
- repo: https://github.com/dj-wasabi/pre-commit-hooks
  rev: master
  hooks:
  - id: shellcheck
  - id: markdown-toc
  - id: verify-yaml
  - id: no-commit-on-branch
    args: ['-b master,main']

It contains a “repos” key which is a list, this means you can have multiple entries configured in the file (As this is currently the case). Each of these entries needs the following 3 properties to work properly:

1. repo: The git location where the scripts are stored;
2. rev: The version of the repository, can be a tag, branch or git commit;
3. 1 or more hooks, which has 1 or 2 keys: id (and args).

In the code block above, we see that there is a Github repository mentioned, namely https://github.com/dj-wasabi/pre-commit-hooks and on master we have several scripts available. The pre-commit hook will see that in this repository a file named “.pre-commit-hooks.yaml” exist. This file knows exactly what to do when the pre-commit hook is executed with the provided id’s. In the “.pre-commit-hooks.yaml” file, there is a configuration for the “shellcheck” id.

- id: shellcheck
  name: Shellcheck Bash Linter
  description: Performs linting on bash scripts
  entry: bin/shellcheck.sh
  language: script

The pre-commit hook now knows, that with the “shellcheck” id a script “bin/shellcheck.sh” in the https://github.com/dj-wasabi/pre-commit-hooks reposity needs to be executed. But in all fairness, you are not ready yet. 9 out of 10 times you will need to check the README on the mentioned repository to see if you need to install any other tool to make the script work. Because I am working on a Mac, I do need to make sure that shellcheck is installed:

brew install shellcheck

And I need to make sure that the dependencies that the other scripts in the “.pre-commit-hook.yaml” uses are installed on my Mac.

Once I have done that, we need to tell our “.git/hooks” directory that we want to make use of the pre-commit hooks. We will do that by executing the following command in the root of our git repository:

pre-commit install

This will give us the following output:

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

You can do now an “ls -l .git/hooks” and compare the output with before. When you execute a “git commit” from now on in this repository, it will execute the pre-commit hooks. Remember that for each git repository you have cloned on your host, it will contains their own configuration.

But why?

“But I still don’t know what kind of scripts I can execute as part of a pre-commit hook?”

– you?

The problem I see and/or have experienced a lot is that when people gets invited to be a reviewer on a PR, is that they focus on the small nitpicky things that don’t really that much. Things like, wrong identation, to many empty lines, missing space or trailing space (well, I do find this one annoying ;-)). They should be focussing on the actual change like I mentioned in one of my earlier blogposts. And 9/10 times you can prevent things like this by executing a linter or formatter. This linter will provide an overview of warnings and errors on what needs to be fixed before the linter is happy, so an ideal candidate to execute it as a pre-commit hook. So with pre-commit hooks we can sort of prevent “garbage” to be committed into a Git repository.

When you write for example Python scripts, like what I do with the https://github.com/dj-wasabi/dj-wasabi-release/ repository, you can make use of the “flake8” tool and with every “git commit”, the linter will be executed and provide errors. And that can be achieved by adding the following few lines in the “.pre-commit-hooks.yaml“:

- repo: https://gitlab.com/pycqa/flake8
  rev: 3.8.4
  hooks:
  - id: flake8
    additional_dependencies: [flake8-typing-imports==1.7.0]

If I now do a “git commit” (or when you just want to execute it without doing an actual commit: “pre-commit run -a“):

$ git commit -am "Some message that I know that it will not make it to the log"
Shellcheck Bash Linter...................................................Passed
Generate Markdown toc....................................................Passed
Pretty Print YAML files..................................................Passed
No commit on master or main..............................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

lib/djWasabi/git.py:59:1: E302 expected 2 blank lines, found 1
lib/djWasabi/git.py:79:22: E231 missing whitespace after ','
lib/djWasabi/git.py:80:52: W291 trailing whitespace
lib/djWasabi/git.py:83:53: E231 missing whitespace after ','

Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing lib/djWasabi/git.py

Check for merge conflicts................................................Passed
$ echo $?
1

Look, the pre-commit “Failed” (Or succeeded, depends on how you look at it ;-)) and it prevented to do the actual commit in git. You can see that there where 4 lines in the “lib/djWasabi/git.py” file that needs to be fixed. But you will also see a 2nd script has failed, namely the “Trim Trailing Whitespace“.

A pre-commit script can also update files when needed (which isn’t a bad thing!) and not just fail on when certain criteria is met. The “Trim Trailing Whitespace” script has updated a single file “lib/djWasabi/git.py” and removed the trailing space. When a script updates a file, the script should exit with an exit code of 1 and provide info on what file has been updated. If I would run the git commit again, then you will see that the “Trim Trailing Whitespace” is “Passed” and that the “flake8” script only has 3 errors:

Shellcheck Bash Linter...................................................Passed
Generate Markdown toc....................................................Passed
Pretty Print YAML files..................................................Passed
No commit on master or main..............................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

lib/djWasabi/git.py:59:1: E302 expected 2 blank lines, found 1
lib/djWasabi/git.py:79:22: E231 missing whitespace after ','
lib/djWasabi/git.py:83:53: E231 missing whitespace after ','

Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed

Nice right?

Updating files as part of the pre-commit hooks are very common and I do make use of it a lot. When you work with Terraform, you will probably know the “terraform fmt” command as well. And I do too make use of the “terraform fmt” command as part of the pre-commit hook, it will make sure that all my terraform files are formatted in one way. So people that will review my Pull Requests, can not write any comment on formatting issues. 🙂

Before I end this blog post, I want to say one last things that really helps me with pre-commit hooks. I really like writing documentation in Asciidoctor format, i personally thinks it is a bit better that any other “language” and I don’t want to start a flamewar, but just writing on top of the document “:toc: left” and I have my Table of Content on the left side of the page. With Markdown, you’ll have to manually write one and keep the Table of Content up 2 date. Or, you install a tool with this command:

pip install md-toc

Then you can make use of the following line in your markdown file:

<!--TOC-->

And make sure that an id with “markdown-toc” is added to the “.pre-commit-hooks.yaml” file like shown in the beginning of this blog post and you are done! (small note is that you need to commit your changes to see that it is generated :))

“I can now see the benefits of using pre-commit hooks and will use them as well!“

– you?

I hope so? I hope I showed you how awesome pre-commit hooks are that they help you write better code. Not only that, it also helps the reviewers to focus on a PR that matters instead of looking for the nitpicky things. And with this blog post I only mentioned linting and formatting scripts, but you basically can do anything with a pre-commit hook. One small tip with using pre-commit hooks, don’t go wild on it. My commits will take some seconds to complete, but you can even run unit tests as well but that will result in longer duration of doing a commit. And when you do a “git commit” you basically are waiting on it to finish correctly, so don’t add lots and lots of tests because that will increase the duration of a commit.

But if you do have any questions and or remarks please let me know and I will happily help you.

In this post I will describe how to create your own pre-commit hook.

May the commits be with you!