Programming

There comes a point in any project that when it reaches a certain size, a flat directory hierarchy no longer works. There are too many files, some perhaps similarly named, and additional organization becomes needed. One approach is to use recursive Makefiles, with a top-level set of Makefiles controlling the build which call into multiple subdirectories that contain the actual source code.

This article describes one such approach, with the additional requirements that it support cross-compiling for another CPU target and tracks file dependencies. Additional goodies include packaging the generated executables into a filesystem image and digitally signing it with GnuPG.

Much of this work is inspired from various manuals, blog posts, and Stackoverflow questions found on the Internet. Unfortunately this framework is several years old and I did not record information sources at the time it was initially constructed. Many thanks are owed to anonymous coders on the Internet.

This framework uses GNU Make and has not been tested with other implementations.

A complete copy of this example can be found on my GitHub page here.

Table of Contents

Structure Layout
Makefile
common.mk
version.mk
exeA/Makefile
libA/Makefile
exeB/Makefile
Adding New Components

Structure Layout

At the root level, three Makefiles are utilized.

common.mk – common targets and definitions that are global to the project
version.mk – specifies version control information
Makefile – top-level Makefile which ties it all together

Adjacent to these Makefiles, one or more subdirectories are created to hold appropriately grouped code. These are nominally components that will each be compiled into an executable binary or a static .a library. Shared .so libraries was outside the scope of my needs, but likely simple to implement.

The example framework consists of three components.

exeA – executable process “A”, which also depends on libA
exeB – executable process “B”
libA – static .a library pulled in by exeA

The full directory tree thus looks like:

$ ls -l
total 24
-rw-rw-r-- 1 eric eric 1450 Jan 20 22:22 common.mk
drwxrwxr-x 2 eric eric 4096 Jan 20 22:59 exeA
drwxrwxr-x 2 eric eric 4096 Jan 20 22:59 exeB
drwxrwxr-x 2 eric eric 4096 Jan 20 22:59 libA
-rw-rw-r-- 1 eric eric 1067 Jan 20 21:41 Makefile
-rw-rw-r-- 1 eric eric  892 Jan 20 22:05 version.mk

Makefile

The top-level Makefile is used to define components and major build targets.

include version.mk

libA		:= libA
libraries	:= $(libA)
exeA		:= exeA
exeB		:= exeB

imagedir	:= image
imagenamebase	:= $(partNumber)_$(appName)_v$(versionMajor).$(versionMinor).$(versionRevision)
imagefile	:= $(imagenamebase).squashfs

.PHONY: all $(exeA) $(exeB) $(libraries)
all: $(exeA) $(exeB) $(libraries)

install: $(exeA) $(exeB)
	$(RM) -rf $(imagedir) $(imagefile)
	install -D -m 755 -t $(imagedir)/	$(exeA)/exeA
	install -D -m 755 -t $(imagedir)/	$(exeB)/exeB
	mksquashfs $(imagedir) $(imagefile)

sign: install
	gpg --detach-sign $(imagefile)
	mkdir $(imagenamebase)
	mv $(imagefile) $(imagefile).sig $(imagenamebase)
	zip -r $(imagenamebase).zip $(imagenamebase)
	rm -rf $(imagenamebase)
	$(eval versionAutoNum=$(shell echo $$(($(versionAutoNum)+1))))
	@echo $(versionAutoNum) > $(autoNumFile)

clean:
	$(MAKE) -C . TARGET=clean
	rm -rf image *.squashfs *.zip

$(exeA) $(exeB) $(libraries):
	$(MAKE) --directory=$@ $(TARGET)

# Configure the various module dependencies
$(exeA): $(libA)

Next we will break down the major pieces of this Makefile.

include version.mk

Include version.mk, as we will need to know versioning information for constructing the final firmware filename, etc.

libA		:= libA
libraries	:= $(libA)
exeA		:= exeA
exeB		:= exeB

Define various variables that can be reused later, rather than hard-coding specific component names. The example looks rather redundant, but becomes helpful once scaled out a bit. Additionally, all libraries are grouped into the libraries variable.

imagedir	:= image
imagenamebase	:= $(partNumber)_$(appName)_v$(versionMajor).$(versionMinor).$(versionRevision)
imagefile	:= $(imagenamebase).squashfs

Various file and directory names that will be used later for firmware image construction. imagedir is the temporary directory to use to stage the firmware filesystem, imagenamebase is the base filename for the image which takes into account some metadata including software part number and version, and imagefile is the final target filesystem image. As the name implies, this image will be a squashfs image.

.PHONY: all $(exeA) $(exeB) $(libraries)
all: $(exeA) $(exeB) $(libraries)

Set up the all build target. This will compile each executable and library.

install: $(exeA) $(exeB)
	$(RM) -rf $(imagedir) $(imagefile)
	install -D -m 755 -t $(imagedir)/	$(exeA)/exeA
	install -D -m 755 -t $(imagedir)/	$(exeB)/exeB
	mksquashfs $(imagedir) $(imagefile)

Set up the install build target. This will copy various ad-hoc files into $(imagedir), which is them constructed into the filesystem image.

sign: install
	gpg --detach-sign $(imagefile)
	mkdir $(imagenamebase)
	mv $(imagefile) $(imagefile).sig $(imagenamebase)
	zip -r $(imagenamebase).zip $(imagenamebase)
	rm -rf $(imagenamebase)
	$(eval versionAutoNum=$(shell echo $$(($(versionAutoNum)+1))))
	@echo $(versionAutoNum) > $(autoNumFile)

Sign and archive the filesystem image. There is a lot going on here, so we will break down each step.

Create a detached signature for the image. This will use whatever default private key GnuPG selects.
Create a staging directory to archive both the filesystem image and the signature.
Move the filesystem image and the signature into the staging directory.
Archive the staging directory into the firmware image package.
Remove staging directory.
The final two lines track the build “auto-number”. This build number is appended to the firmware image in order to differentiate subsequent builds during the development process, and incremented here.

clean:
	$(MAKE) -C . TARGET=clean
	rm -rf image *.squashfs *.zip

Clean the project, with a slightly convoluted approach. Call this Makefile again, but with the TARGET=clean variable set. This will be picked up by the following build target.

$(exeA) $(exeB) $(libraries):
	$(MAKE) --directory=$@ $(TARGET)

Target for the various executables and libraries. By default this will recurse into those component subdirectories and call make, which will then build the all target (the code). Should make clean have been called above, the recursed target will instead be clean, as defined in common.mk. This will clean up the various executables, libraries, object files, etc.

# Configure the various module dependencies
$(exeA): $(libA)

Defines component dependencies. In this case, exeA depends on libA, such that libA will be checked for being up-to-date and rebuilt automatically if not.

common.mk

This file defines global data and recipes, including how to compile C files, dependency information, libraries, and link executables.

# Set to 1 to enable debugging
# Be sure to clean and rebuild after modifying
DEBUG=0

MV	:= mv -f
RM	:= rm -f
SED	:= sed

# If building for docker/host system, add extra options
# Otherwise, specify the cross compiler
ifeq ($(DOCKER),1)
	# Add any extra options here...
else
	CROSS	:= arm-linux-
endif

CC		:= $(CROSS)gcc
LD		:= $(CROSS)ld
AR		:= $(CROSS)ar

objects		:= $(subst .c,.o,$(sources))
dependencies	:= $(subst .c,.d,$(sources))

# Add any custom global cpp flags you might need here...
CPPFLAGS	+= -DSOMETHING=1

# General flags and includes
include_dirs	:= ../libA
CFLAGS		+= -Wall -Werror

ifeq ($(DEBUG),1)
	CFLAGS	+= -ggdb
else
	CFLAGS	+= -O3
endif

# Combine include dirs for the pre-processor
CPPFLAGS	+= $(addprefix -I,$(include_dirs))

vpath %.h $(include_dirs)

.PHONY: library
library: $(library)

$(library): $(objects)
	$(AR) $(ARFLAGS) $@ $^

.PHONY: program
program: $(program)

$(program): $(objects) $(libraries)
	$(CC) -o $(program) $(LDFLAGS) $(objects) $(libraries) $(LDLIBS)

.PHONY: clean
clean:
	$(RM) $(objects) $(program) $(library) $(dependencies)

ifneq "$(MAKECMDGOALS)" "clean"
  -include $(dependencies)
endif

%.c %.h: %.y
	$(YACC.y) --defines $<
	$(MV) y.tab.c $*.c
	$(MV) y.tab.h $*.h

%.d: %.c
	$(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< |      \
	$(SED) 's,\($*\.o\) *:,\1 $@: ,' > $@.tmp
	$(MV) $@.tmp $@

Next we will break down the major pieces of this Makefile.

# Set to 1 to enable debugging
# Be sure to clean and rebuild after modifying
DEBUG=0

Flag enabling debug builds. It can be set to 1 here, or invoked via make DEBUG=1.

MV	:= mv -f
RM	:= rm -f
SED	:= sed

Various standard utilities.

# If building for docker/host system, add extra options
# Otherwise, specify the cross compiler
ifeq ($(DOCKER),1)
	# Add any extra options here...
else
	CROSS	:= arm-linux-
endif

If DOCKER=1 is defined, run a Docker build. This is also synonymous with a host build. The intent of it is for compiling the application which can be run within a Docker container for localize execution and debugging.

If not defined, specify the prefix of the cross-compiler to use.

CC	:= $(CROSS)gcc
LD	:= $(CROSS)ld
AR	:= $(CROSS)ar

Define our compiler, linker, and archiver. If $(CROSS) is defined, it will expand to our cross-compiler. Otherwise, just use the host versions.

objects		:= $(subst .c,.o,$(sources))
dependencies	:= $(subst .c,.d,$(sources))

Macro expansion of the sources variable, defined in the component subdirectory Makefiles, generating lists of corresponding .o object files and .d dependency files.

# Add any custom global cpp flags you might need here...
CPPFLAGS	+= -DSOMETHING=1

Somewhere to set any global defines.

# General flags and includes
include_dirs	:= ../libA
CFLAGS		+= -Wall -Werror

Configure global project include directories and base C flags. In this case, we warn on everything and consider warnings to be errors.

ifeq ($(DEBUG),1)
	CFLAGS	+= -ggdb
else
	CFLAGS	+= -O3
endif

If this was called as make DEBUG=1, then enable GDB debugging data with no optimizations. Otherwise for a standard build, turn on optimizations.

# Combine include dirs for the pre-processor
CPPFLAGS	+= $(addprefix -I,$(include_dirs))

Macro expansion to turn all include_dirs into -I arguments for the compiler.

vpath %.h $(include_dirs)

Tell the dependency checker where to find project-global include directories for header files.

.PHONY: library
library: $(library)

$(library): $(objects)
	$(AR) $(ARFLAGS) $@ $^

How we build a library. This depends on the specific objects, so all this needs to do is archive them into a .a file.

.PHONY: program
program: $(program)

$(program): $(objects) $(libraries)
	$(CC) -o $(program) $(LDFLAGS) $(objects) $(libraries) $(LDLIBS)

How we build a program executable. This depends on the specific objects, so all this needs to do is link them into the executable.

.PHONY: clean
clean:
	$(RM) $(objects) $(program) $(library) $(dependencies)

How to clean a component directory. This will remove any objects, the executable, library, and dependency (.d) files.

ifneq "$(MAKECMDGOALS)" "clean"
  -include $(dependencies)
endif

If cleaning, do not generate dependencies before cleaning as it would be pointless and wasteful.

%.c %.h: %.y
	$(YACC.y) --defines $<
	$(MV) y.tab.c $*.c
	$(MV) y.tab.h $*.h

Recipe for compiling yacc files. Not sure exactly why this is here, or if it is really necessary.

%.d: %.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -M $< |      \
	$(SED) 's,\($*\.o\) *:,\1 $@: ,' > $@.tmp
	$(MV) $@.tmp $@

How to generate dependency files. This is done by invoking the compiler and converting the output into a .d file corresponding to the .c file. This generates a fair bit of noise at compile-time, which can be suppressed by prefixing $(CC) and $(MV) with @ if desired.

version.mk

This file defines various version control information, exposing it for the build process and also the compiled software. An “auto number file” is utilized to append -devX to the build, for developer builds, or in the case of GitLab CI/CD pipelines, the commit hash.

# Basic application data
partNumber	:= 12345
appName		:= APPNAME

# Version number
versionMajor	:= 1
versionMinor	:= 0
# versionMinor with leading zeroes stripped
versionMinorEx  := $(shell echo $(versionMinor) | sed -E 's/^0+([0-9]+)/\1/')
versionRevision	:= dev
versionAutoNum	:= 0
autoNumFile	:= .autonum

ifeq ($(origin CI_COMMIT_SHORT_SHA), undefined)
# If not building under GitLab, auto-increment the build number
	num := $(shell cat ${autoNumFile})
	ifeq ($(num),)
	else
		versionAutoNum := $(num)
	endif

	versionRevision := $(versionRevision)$(versionAutoNum)
else
# Otherwise, override version revision
	versionRevision := $(CI_COMMIT_SHORT_SHA)
endif

# If debug build, append "-debug" to the string
ifeq ($(DEBUG), 1)
	versionRevision := $(versionRevision)-debug
endif

CPPFLAGS		+= -DVERSIONMAJOR=$(versionMajor) -DVERSIONMINOR=$(versionMinorEx) -D$(appName)
export CPPFLAGS

Next we will break down the major pieces of this Makefile.

# Basic application data
partNumber	:= 12345
appName		:= APPNAME

As the comment says, basic application data. The part number and application name are used for firmware image filename construction, and the application name is passed to the compiler as -D$(appName).

# Version number
versionMajor	:= 1
versionMinor	:= 0
# versionMinor with leading zeroes stripped
versionMinorEx  := $(shell echo $(versionMinor) | sed -E 's/^0+([0-9]+)/\1/')
versionRevision	:= dev
versionAutoNum	:= 0
autoNumFile	:= .autonum

Version number information. The compiler is given -DVERSIONMAJOR=$(versionMajor) and -DVERSIONMINOR=$(versionMinorEx). versionMinorEx is a copy of versionMinor with leading zeroes removed, as once you get to 08 this becomes an invalid octal number.

versionRevision defaults to “dev”, which will have versionAutoNum appended. This will be overridden with the git commit hash in the case of GitLab pipelines.

ifeq ($(origin CI_COMMIT_SHORT_SHA), undefined)
# If not building under GitLab, auto-increment the build number
	num := $(shell cat ${autoNumFile})
	ifeq ($(num),)
	else
		versionAutoNum := $(num)
	endif

	versionRevision := $(versionRevision)$(versionAutoNum)
else
# Otherwise, override version revision
	versionRevision := $(CI_COMMIT_SHORT_SHA)
endif

Configure versionRevision based on developer build or GitLab pipeline.

# If debug build, append "-debug" to the string
ifeq ($(DEBUG), 1)
	versionRevision := $(versionRevision)-debug
endif

If this is a debug build, include that in the firmware filename as well.

CPPFLAGS		+= -DVERSIONMAJOR=$(versionMajor) -DVERSIONMINOR=$(versionMinorEx) -D$(appName)
export CPPFLAGS

Expose version information via C pre-processor flags as described above.

Component Makefiles

Finally, the hard part is over. Next we configure the component-specific Makefiles. For the most part, these simply define the source files, target output name, and perhaps some additional compiler options. All of the heavy lifting is done by including common.mk.

exeA/Makefile

Executable A has a single .c file, and also depends on libA.

sources := \
	exeA.c

libraries	:= ../libA/liblibA.a
program		:= exeA

# Add any other customizations here, such as CFLAGS

all: $(program)

include ../common.mk

First, sources is simply a list of all constituent .c files.

libraries is an optional variable that contains a list of libraries, either as relative .a files or -l options for ld. In this case we need libA.

program is the target executable name that the .c files will link to.

all: $(program) is the recipe to compile the target executable. Probably this could be generalized in common.mk somehow.

Finally, include common.mk for the rest of the build process.

exeB/Makefile

Executable B has a single .c file, with no library dependencies.

sources := \
	exeB.c

libraries	:= 
program		:= exeB

all: $(program)

include ../common.mk

As described for exeA/Makefile, except no library dependencies.

libA/Makefile

This library consists of a single .c file.

library := liblibA.a
sources := \
    libA.c

# Add any other customizations here, such as CFLAGS

include ../common.mk

As described for exeA/Makefile, except here we define library as the target .a file instead of program. No all recipe is required for libraries.

Adding New Components

Adding new components is relatively straightforward using this framework.

Create a new component directory.
Add source files as needed.
Create a Makefile in the component directory by copying an existing executable or library Makefile as appropriate.
Update the new Makefile to define the program, library, and sources variables as needed.
Update the root/top-level Makefile to make it aware of the new component.

This final step is the most complicated, as there is a fair bit of copy/pasting going on as can likely be improved in some way. Start by adding the component to the variable lists at the top of the file. Next, update the .PHONY: all and all targets in the same way the other components are listed. At the bottom of the file, list the components to enable Make recursion and dependency tracking as appropriate. Finally, update the install target to copy any output files into the firmware image.

A Recursive, Cross-Compiling Make System

Structure Layout

Makefile

common.mk

version.mk

Component Makefiles

exeA/Makefile

exeB/Makefile

libA/Makefile

Adding New Components

Recent Posts

Recent Comments