Mads Hartmann
If you want to get into some deeper aspects of Make after reading this I highly recommend the book Managing Projcts with GNU Make 3rd edition. Furthermore you can always dig into a specific topic by consulting the Make documentation – I will link to the specific sections where relevant but I won’t cover all of the features that Make has to offer so feel free to go explore.
Table of contents
Why Make is Still Useful Today
Make is a programming language agnostic tool that provides a simple set
of abstractions for describing the relationships between the source
files in your project and the artifacts you wish to produce. This makes
it a very useful tool for automating the development work-flow as well
as the deployment pipeline of a given project. Ideally getting started
on a project should be a simple matter of executing a single shell
command which installs the required tools, configure the project and
build the required artifacts; nobody likes having to go through a 14
steps README in order to get a project up and running – the same goes
for deploying your code to production.
However, having been created in 1977, Make can at times feel extremely archaic. It’s fair to assume that it must have been surpassed by more modern tools. You are probably using a more recent programming language specific build tool such as SBT for Scala, Leinigen for Clojure, Rebar3 for Erlang, Webpack for web assets, etc. These tools solve the task of turning your source files into an artifact that you can run and deploy. For this specific task I don’t recommend that you use Make, but rather that you should use Make to automate the use of these tools.
In the next section I’ll introduce Make in the way that I like to think of it – as a programming language.
Make as a Programming Language
I like to think of Make as a declarative string-oriented functional programming language with very good support for executing shell commands. It’s a programming language that happens to be very good at creating build systems 😉
You write your Make program in a file called a Makefile. The purpose
of your Makefile is to describe a dependency
graph of your build
artifacts and source files such that Make knows how and when to build a
specific artifact. The dependency graph is specified using rules – this
is what I think of as the declarative part of Make. In order to specify
these rules concisely Make has a string-oriented functional programming
language that is mostly used to compute the value of variables that can
then be used when defining rules.
Once you have your Makefile you invoke Make like so
make my_file
where my_file is the name of the file you want Make to produce. Based
on your Makefile Make computes and traverses the dependency graph of
my_file in order to figure how and if it should produce my_file.
Let’s start by having a look at the language used to specify the rules.
The Rule Language
In your Makefile you declare your
rules
using the following syntax.
target: prerequisite1 ... prerequisiteN
command1
...
commandN
The target is the name of the file you want to produce. The
prerequisites are the files that are required to produce the taget.
The commands (also known as the recipe) are the shell commands Make
should run in order to produce the target.
Image you have a project where you need to convert a
Markdown file to HTML.
The following rule would achieve this (assuming you have a program
called marked on your system).
index.html: index.md
marked --input index.md --output index.html
This tells Make that it can produce a file named index.html if a file
named index.md exists and that it can produce it using the shell
command marked --input index.md --output
index.html. Additionally Make knows that if index.md changes it
should re-produce the index.html file. An important note is that each
command in the recipe (is this case there’s just one) should be prefixed
with a tab – it’s one of the many archaic aspects of Make.
A rule doesn’t necessarily have to contain a recipe, the following is a perfectly valid rule.
build: index.html about-me/index.html posts/hello-world/index.html
This simply tells Make that there’s a target called build and that it
should consider it satisfied if all the prerequisites are satisfied.
Targets like these makes it possible to provide a nice interface to your
Makefile – you can now invoke make build instead of
make index.html about-me/index.html posts/hello-world/index.html.
Implicit Rules
With the rules you’ve seen so far you can imagine that your
Makefile gets a bit long as the number of Markdown files increase
as you have to explicitly describe the relationship between each
Markdown and HTML file.
build: index.html about-me/index.html posts/hello-world/index.html
index.html: index.md
marked --input index.md --output index.html
about-me/index.html: about-me/index.md
marked --input about-me/index.md --output about-me/index.html
posts/hello-world/index.html: posts/hello-world/index.md
marked --input posts/hello-world/index.md --output posts/hello-world/index.html
For cases like these you can define a pattern rule
(one of many types of implicit rule
that Make has) instead and get a nice and succinct Makefile.
build: index.html about-me/index.html posts/hello-world/index.html
%.html: %.md
marked --input $< --output $@
The
pattern rule %.html:
%.md tells Make that it can generate an .html file from a .md
file of the same name using the body of the rule. The symbols $< and
$@
are
automatic variables that
expand to the name of the left-most prerequisite and the name of the
target respectively. More on variables later.
Auto-generating prerequisites
There are cases where managing the lists of prerequisites of a
target can become cumbersome and error-prone. A solution to this
problem is to generate prerequisites automatically for some targets.
This can be achieved using the
include
directive.
The include directive tells Make to read another Makefile – an
example would be include foobar.mk. Before reading the Makefile
foobar.mk it checks if there exists a rule that can produce
foobar.mk. If such a rule exists it will reproduce foobar.mk
before including.
This makes it possible to automatically generate a Makefile that
declares prerequisites and then import it. A tool that produces such
a Makefile has historically been called make-depend.
This is a little out of scope for this blog post so head over to Generating Prerequisites Automatically to read more about this; I only mention it here as it’s something you are likely to need in a more complex build systems.
String-oriented functional programming
So the dependency graph is specified through rules. Let’s have a look at
the language features that Make has that enables you to write nice
concise Makefiles.
Variables
There are two kinds of variables in Make – simple and recursive – they differ in the way they’re expanded when referenced. There are four different operators that can be used when assigning values to variables.
:= assignment operator In the example python :=
python$(PYTHON_VERSION) the := assignment operator will set the
value of python to python2.7 given the value of PYTHON_VERSION
is set to 2.7. That is, the right-hand side is evaluated during
the assignment of the variable.
= assignment operator In the example python =
python$(PYTHON_VERSION) the = assignment operator won’t evaluate
the right-hand side but rather set the value of python to
python$(PYTHON_VERSION) and the final expansion of python
depends on the value of the variable PYTHON_VERSION when the
variable python is referenced. You can think of this as lazy
evaluation as you might be used to from using the lazy keyword in
Scala (or if you’ve used
Haskell). Some good advice is to only
use = if you really have to. Otherwise stick to the other
operators.
?= conditional variable assignment operator In the example
PYTHON_VERSION ?= 2.7 the conditional assignment operator will set
the value of PYTHON_VERSION to 2.7 unless PYTHON_VERSION is
already defined as an environment variable.
+= append operator In the example foo += bar baz Make will
append bar baz to the current value of foo. For simple variables
this is simply a shorthand for foo := $(foo) bar baz but for
recursive variables the operator will still do the right thing without
crashing in an infinite evaluation of foo.
There’s a convention that words in variables should be
separated_by_underscores and that variables that a user might want
to customize should be written using UPPERCASE letters.
When you reference a variable it is expanded in-place – In that
sense a Make variable is similar to variables as you know them from
templating languages such as
jinja2 or
ejs. The syntax for referencing a
variable is $(variable_name).
In adittion to the variables you define Make has a set of variables that are called automatic variables. These are variables that change value based on the rule that is currently being executed.
Functions
Make contains a bunch of built-in functions
and it’s worth reading through them as it will make your Makefile
much more concise, especially the File Name Functions.
The syntax for calling a functions is $(function-name arg1, ..., argN).
It is possbile to define your own functions. A user-defined function
is really just a variable that is defined in such a way that it’s
intended to be used with the built-in
call
function. So when you’re calling a user-defined functions the
pattern is $(call variable-name, param1, ..., paramN)
The following shows how to define a function. The arguments to the
function are available through the variables $1, $2, etc.
# $(call print-rule, variable, extra)
# Used to decorate the output before printing it to stdout.
define print-rule
@echo "[$(shell date +%H:%M:%S)] $(strip $1): $(strip $2)"
endef
There’s a convention that user-defined functions use lowercase-words separated-by-dashes.
Example – A Static Website
Let’s see how all of this fits together. Let’s create a Makefile for a simple statically generated web-site based on Markdown files. You can find this little example on Github. The project has the following structure.
├── Makefile
└── site
├── make.md
└── tools
├── osx
│ └── homebrew.md
└── xargs.md
For each Markdown file we want to generate a HTML file with an
appropriate path and put it in a folder named _build. For the example
above the result of building the site should be the following.
_build
├── make
│ └── index.html
└── tools
├── osx
│ └── homebrew
│ └── index.html
└── xargs
└── index.html
This can be achieved with the following Makefile.
### User-changable variables.
BUILD_DIR := _build
### Variables
markdown_sources := \
$(shell find site -name "*.md")
html_pages := \
$(foreach f, $(markdown_sources), \
$(subst site/,$(BUILD_DIR)/, \
$(dir $(f)))$(basename $(notdir $(f)))/index.html)
# Alternative way to achieve the same thing as html_pages
html_pages_alternative := \
$(patsubst site/%.md,$(BUILD_DIR)/%/index.html,$(markdown_sources))
### Targets
build: setup $(html_pages)
setup: node_modules/.installed
clean: ; rm -rf $(BUILD_DIR)
distclean: ; rm -rf $(BUILD_DIR) node_modules
print-%: ; @echo $* is $($*)
### Rules
$(BUILD_DIR)/%/index.html: site/%.md
@mkdir -p $(dir $@)
node_modules/.bin/marked --input $< --output $@
node_modules/.installed: requirements.txt
rm -rf node_modules
npm install $(shell cat $<)
touch $@
Let’s go through the Makefile step by step. First we define a single variable that the user might be interested in changing (hence the capitalized letters).
### User-changable variables.
BUILD_DIR := _build
We then go on to define a variable that contains the path of all of the
Markdown files in site and the sub-folders of site. This is achieved
by using the very handy shell
function
which allows you to execute a shell command and use the result in your
Makefile. An alternative way to find all Markdown files could’ve been to
use the wildcard built-in
function.
markdown_sources := \
$(shell find site -name "*.md")
Before we move on let’s make sure we got the use of shell right by
inspecting the variable using the print-% target – the print-%
target is something I copy-paste into all of my Makefiles as it’s
extremely useful for debugging Makefiles.
make print-markdown_sources
markdown_sources is site/make.md site/tools/osx/homebrew.md site/tools/xargs.md
Great, so we’ve successfully managed to capture the the filenames of the
Markdown files we want to process. Next up we use the markdown_sources
variable to create a new variable that contains the filenames of the
HTML pages we want to generate.
html_pages := \
$(foreach f, $(markdown_sources), \
$(subst site/,$(BUILD_DIR)/, \
$(dir $(f)))$(basename $(notdir $(f)))/index.html)
Here we’re using some of the most commonly used built-in in functions.
We’re using the
text-function
subst to replace site/ with _build/. We’re using the file name
functions
dir, notdir and basename to get the directory part of a path, the
filename part of a path and finally the filename part of a path without
the extension. We’re using the the foreach
function
to perform the transformation on each path separately.
There’s another very useful
text-function
named patsubst which takes a pattern instead of a string to perform
substitution. Here’s how to use it to provide an alternative
implementation of the variable html_pages.
html_pages_alternative := \
$(patsubst site/%.md,$(BUILD_DIR)/%/index.html,$(markdown_sources))
In this case I think the use of patsubst yields the clearest result
but it depends on the situation so it’s nice to keep both approaches in
your toolbox. Let’s have a look at the value of both variables.
make print-html_pages ; make print-html_pages_alternative
html_pages is _build/make/index.html _build/tools/osx/homebrew/index.html _build/tools/xargs/index.html
html_pages_alternative is _build/make/index.html _build/tools/osx/homebrew/index.html _build/tools/xargs/index.html
With the variables in order we go on to define the targets that we intend the user to call; this is the interface of the Makefile.
build: setup $(html_pages)
setup: node_modules/.installed
clean: ; rm -rf $(BUILD_DIR)
distclean: ; rm -rf $(BUILD_DIR) node_modules
print-%: ; @echo $* is $($*)
The clean, distclean and print-% are using a convenient syntax for
defining the recipe of a rule on the same line as the rule itself.
Notice that build depends on setup as well as $(html_pages) –
this means that if you were to run make build before having run
make setup Make will ensure the the setup target is satisfied before
attempting to satisfy the targets in the html_pages variable; it’s a
small convenience this that means you have to remember fewer targets as
a developer.
Finally let’s have a look at the rules.
$(BUILD_DIR)/%/index.html: site/%.md
@mkdir -p $(dir $@)
node_modules/.bin/marked --input $< --output $@
node_modules/.installed: requirements.txt
rm -rf node_modules
npm install $(shell cat $<)
The first rule $(BUILD_DIR)/%/index.html: site/%.md describes how to
generate a HTML file from a Markdown file using a pattern
rule.
The recipe of the rule makes sure that the directory exists using
mkdir -p before generating the HTML files using the NPM package
marked. The prefix @ of @mkdir
tells Make not to output the command when executing it. Notice we’re
using two automatic variables $@ and $<: The first is the name of
the target, the second is the name of the left-most prerequisite – in
our case there is only one and it’s the name of the Markdown file that
should be used to generate the HTML file.
The last rule node_modules/.installed: requirements.txt describe how
to install the NPM package(s) that we’re using. The packages are
enumerated in a file named requirements.txt. First we remove the
previously installed node_modules, then we install the packages and
finally touch the node_modules/.installed file. The rule is using a
common pattern where we introduce an empty file (in this case
node_modules/.installed) which represents the successful execution of
the rules recipe. Here’s the Makefile manual
section
that describes this pattern.
Here’s a quick demonstration. We’re running make distclean build to
first get a completely clean workspace and then build to show how Make
installs the NPM package before attempting to generate the output.
make distclean build
rm -rf _build node_modules
rm -rf node_modules
npm install marked
/Users/hartmann/dev/mads-hartmann.github.com/examples/markdown-site
└── marked@0.3.6
touch node_modules/.installed
node_modules/.bin/marked --input site/make.md --output _build/make/index.html
node_modules/.bin/marked --input site/tools/osx/homebrew.md --output _build/tools/osx/homebrew/index.html
node_modules/.bin/marked --input site/tools/xargs.md --output _build/tools/xargs/index.html
If we were to run make build again without changing any files then
Make is smart enough to know that nothing needs to be re-built.
make build
make: Nothing to be done for `build'.
If we change a file Make is smart enough to only re-build that single file.
touch site/make.md ; make build
node_modules/.bin/marked --input site/make.md --output _build/make/index.html
And that’s it.