.. title: Port Shell Scripts to Python
.. slug: port-shell-scripts-to-python
.. date: 2016-04-16 15:32:29 UTC
.. updated: 2016-04-16 15:32:29 UTC
.. tags: python
.. category:
.. link:
.. description:
.. type: text
``make`` is a legendary tool to build software and is equally
notorious at the same time. It is really good at what it does
and has been forked many times to fix its many flaws and
papercuts.
There are alternatives implemented with
or without custom domain specific languages (DSL). Some are
implemented in Python, such as
`SCons `_,
`pymake `_,
`Waf `_,
`BuildIt `_,
and
`Invoke `_.
Some are implemented in other languages like
`cmake `_,
`RAKE `_,
`Ninja `_,
and
`Blur `_.
The question to ask is: which would you use to solve what problems?
.. TEASER_END: Read more
My use case had the following properties:
* Build rpm packages through a script
* Use Python instead of shell
I ported a shell script to Python using Invoke in a couple hours.
Most of that time was spent trying to understand the semantics
of the shell code.
During the exercise I learned that Invoke really is a simple and
straightforward way to write a build script. It also emerged that
I could write pretty much anything in Invoke that needed to have
*tasks* or steps; much like a *Makefile*.
Invoke excels at writing a standalone executable that needs to
perform discrete steps individually or chained in a sequence.
It feels very similar to recipes in Chef or playbooks in Ansible.
A Python script that uses the `subprocess` and `argparse`
modules can do pretty much what Invoke does, only Invoke takes
care of the plumbing for you. This nicety is very much appreciated.
Using Invoke means not having to learn a DSL just to do something.
It's all Python code optionally interspersed with some shell commands. The
learning curve is thus almost non-existent. Sure some things that
are one-liners in shell script need more lines to implement but
the batteries-included Python library ecosystem makes up for it.
The end result being code that's more readable and thus
maintainable.
I'm sure at some point in someone's requirements a simpler DSL or
config file will be better than writing Python code but here again
I think they'll miss something important. I'm all for simple config
files to drive software. There's a reason for a plethora of tools
for building software: they all solve a particular set of problems
well enough to get the job done. They're widely used in different scenarios
and have the bug fixes and improvements to show for it.
Recently I have come across a lot of DSLs. Chef, Ansible, Vagrant, Packer,
and Terraform all have their own DSLs. They may or may not share syntax
but they share a lot of semantics: order of execution, data structures,
loops, conditionals, etc. These all integrate well with the problem domain
and have valid reasons for their existence.
Nevertheless, having the power of a
complete programming language at hand is much more useful. Instead
of operating on a blackbox one gets to use libraries to write
exactly what needs to be done. As one keeps building on top of these
libraries the end result may equal that of another build tool that
is driven by config files. The advantage, though, is that it's
not a blackbox but an engine that you fully understand and control.
Your engine has its own inspirations and philosophies that fit your
use case. It's not hampered by assumptions made by someone else who
did not have the exact same problem that you want to solve. For example,
I did not need a *Makefile* that I would have needed to refresh my
knowledge on; I needed to copy and edit some files and run ``rpmbuild``. A
shell script was already doing the job. Invoke just gave more people
in my team an opportunity to modify it or reuse it for their own
use cases.
Invoke is the next generation iteration of Fabric. In fact, Fabric 2
is apparently being built on top of Invoke, thus providing remote
execution capabilities. The current Fabric version already does both.
If you're looking to the future and do not need to use SSH in
your scripts then building on Invoke is a better choice.
Finally, I don't intend to encourage you to write an alternative to
``make``. Quite the contrary. I encourage everyone to use ``make`` or
an alternative blackbox build system when a lot more people will be
using it. Standards are good and preferable to custom solutions even
if those standard tools and practices have flaws.
Invoke -- and tools
like it -- should be used to replace long-maintained shell scripts.
Python has a much more readable and understandable syntax than shell.
In the long term it will be easier to maintain a Python script. For
these use cases Invoke is a great asset. It just
so happened that my use case was in the area of build tools. On
another day it might not be. But I was able to get the job done in
Python, opening up the possibility of using Python and Invoke again
for a future project.