• Dropbox mypy (Python)
• Microsoft pyright (TypeScript)
• Facebook pyrefly (rust)
• Astral ty (rust)

I like to integrate them into my GitLab workflow, which includes generatin a Code Quality report.

On top of this my pipeline also runs Astral ruff as a linter and code formatter and uses Astral uv to build and publish the Python package to GitLabs’s PyPI package repository.

The complete example is available on gitlab.com.

Basic GitLab pipeline

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"  # MR
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH  # Branch,Schedule,Web,CLI
    - if: $CI_COMMIT_TAG  # Tag

My workflow required pipelines for merge-requests, stable/default/protected branches, tags and manually and time triggered pipelines.

stages:
  - lint
  - build
  - publish
  - release

default:
  interruptible: true
  artifacts:
    expire_in: 1 day

variables:
  FF_SCRIPT_SECTIONS: true
  FF_TIMESTAMPS: true
  FF_USE_NEW_BASH_EVAL_STRATEGY: true

.py:
  rules:
    - changes:
        paths:
          - pyproject.toml
          - uv.lock
          - "**/*.py"
  variables:
    GIT_DEPTH: 1

I set several GitLab Runner feature flags to get some better experience.

Prepare uv

.uv:
  variables:
    UV_VERSION: "0.11"
    PYTHON_VERSION: "3.13"
    BASE_LAYER: trixie
    # GitLab CI creates a separate mountpoint for the build directory,
    # so we need to copy instead of using hard links.
    UV_LINK_MODE: copy
    UV_CACHE_DIR: .uv-cache
  image: ghcr.io/astral-sh/uv:$UV_VERSION-python$PYTHON_VERSION-$BASE_LAYER
  cache:
    - key:
        files:
          - uv.lock
      paths:
        - $UV_CACHE_DIR
  after_script:
    - uv cache prune --ci

I’m using uv here with some specific version matching Debian 13 “Trixie”. This sets up caching as documented in uv’s GitLab integration.

Running ruff

.ruff:
  extends: [.uv, .py]
  stage: lint

ruff check:
  extends: [.ruff]
  script:
    - uvx ruff check --output-format=gitlab --output-file=code-quality-report.json
  artifacts:
    reports:
      codequality: $CI_PROJECT_DIR/code-quality-report.json

ruff format:
  extends: [.ruff]
  script:
    - uvx ruff format --diff

This runs ruff twice:

1. Once as a linter to check form common issues
2. Once again as a code formatter to check, if the formatting does not follow the configured style.

Running the type checkers

.lint:
  stage: lint
  extends: [.uv, .py]
  artifacts:
    reports:
      codequality: $CI_PROJECT_DIR/gl-code-quality-report.json

mypy:
  extends: [.lint]
  script:
    - uvx mypy --no-error-summary >mypy-out.txt
  after_script:
    - uvx mypy-gitlab-code-quality <mypy-out.txt >gl-code-quality-report.json
    - !reference [.uv, after_script]

pyright:
  extends: [.lint]
  script:
    - uvx --from=pyright[nodejs] pyright --outputjson >pyright-raw.json
  after_script:
    - uvx pyright-to-gitlab -i pyright-raw.json -o gl-code-quality-report.json
    - !reference [.uv, after_script]

pyrefly:
  extends: [.uv, .py]  # .lint
  stage: lint  # TEMPORARY
  script:
    - uvx pyrefly check  # --output-format CodeQuality --output gl-code-quality-report.json

ty:
  extends: [.lint]
  script:
    - uvx ty check  --output-format gitlab >gl-code-quality-report.json

mypy and pyright do not generate the Code Quality JSON themselves. They require running some converter, which transforms their output format. This is done in after_script to always run them, even when the type checkers abort with an exit code other than 0. This overwrites the after_script from the template job .uv, which calls uv cache prune --ci to maintain its cache. As such we have to restore that functionality and use !reference to do that.

ty already generated the required JSON. For pyrefly there is issue 3049, where I asked to add native support for it.

Build and publish the Python package

build python package:
  stage: build
  extends: [.uv]
  rules:
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG
  variables:
    GIT_DEPTH: 0
    GIT_FETCH_EXTRA_FLAGS: --prune --quiet --tags --filter=tree:0
  script:
    # - |
    #   VERSION=$(git describe --exact-match --tags) &&
    #   uvx --from=toml-cli toml set --toml-path=pyproject.toml project.version "$VERSION"
    - uv build
  artifacts:
    paths:
      - dist/

This builds the Python package.

Alter uv publish will fail when you try to upload a Python package with an already existing version. This happens mostly because I forget to update [project]version in the pyproject.toml. The out-commented code above would poke the version into the file before each build.

My alternative was to switch to setuptools-scm: This will use git desctibe to generate the version from git tags, which requires two things:

1. The image must contain the git binary. Debian’s -slim-images do not. Switch to the non--slim versions.
2. Fetch enough history: GIT_DEPTH: 1 may not be enough to walk the git commits from HEAD to any previous git tag. Therefor I use GIT_DEPTH: 0 to fetch the complete history, but combine it with GIT_FETCH_EXTRA_FLAGS: --filter=tree:0: That way I use git’s partial-clone feature: It fetch all commit object, but only the tree and blob objects required for HEAD.

publish python package:
  stage: publish
  extends: [.uv]
  rules:
    - if: $CI_COMMIT_TAG
  needs:
    - job: build python package
  variables:
    GIT_STRATEGY: none
    UV_PUBLISH_URL: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi
    UV_PUBLISH_USERNAME: gitlab-ci-token
    UV_PUBLISH_PASSWORD: ${CI_JOB_TOKEN}
  script:
    - uv publish dist/*.whl

This published the Python package to GitLab’s PyPI package registry.

Creating a GitLab release

release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/cli:latest
  rules:
    - if: '$CI_COMMIT_TAG =~ /^v?\d+\.\d+\.\d+$/'
  variables:
    GIT_STRATEGY: none
    GLAB_CONFIG_DIR: ${CI_PROJECT_DIR}/.glab-config.${CI_PIPELINE_ID}
    GLAB_ENABLE_CI_AUTOLOGIN: true
    GITLAB_HOST: $CI_SERVER_URL
  dependencies: []
  script:
    - >
      glab changelog generate >changelog.md
      --repo "$CI_PROJECT_PATH"
      --version "$CI_COMMIT_TAG"
      --to "$CI_COMMIT_BRANCH"
  release:
    name: 'Release $CI_COMMIT_TAG'
    description: changelog.md
    tag_name: $CI_COMMIT_TAG
    assets:
      links:
        - name: 'PyPi package $CI_COMMIT_TAG'
          url: $CI_PROJECT_URL/-/packages/
          link_type: package

The final part creates a GitLab release. It uses GitLab’s changelog API to automatically create a changelog in Markdown format from the git commits having a Changelog: trailer.

For my environment I have to tell glab to use configuration file in a writeable directory. Without that it will try to write to /.glab/, which will fail.

That assets:links: part creates a link to the PyPI package registry. GitLab 18.11 just received a feature, where packages are included as release evidence, which might make this optional.

gitGraph
    commit
    branch develop
    checkout develop
    commit
    commit
    commit

There are different ways to run your linters:

1. You may run your linter like ruff just on the last commit cN, which then just checks the state after applying all commits c1…cN on top of branch. This will not only check and find issues introduced by your changes, but will also check all other files and find issues there.
2. You may want to build the diff from main to develop and run your linter on that. That way you check you effective change.
3. You may also want to run the linter on each commit. This is important if you later want to use git bisect to find regressions.

Depending on which strategy you want to use, you have to configure your jobs differently in GitLab.

Check state after last commit

This is the easies one as GitLab already checks out the branch for us. You can optimize that by using Shallow cloning to only clone the last commit. By default GitLab clones the last 20 commits, which might pull too much data.

run linter:
  variables:
    GIT_DEPTH: 1
  script:
    - ruff check

You can reduce the clone depth down to 1 to improve that further, but that may introduce problems later. Shallow cloning has more information on that and contains a warning, that this might not work with GitLab in some cases: GitLab-Runner clones by ref, but might then pick a previous commit for the job to run on. With --depth=1 that might not work as there will be no previous commits to run on.

Disclaimer: I have not checked that behavior myself and doubt, that this is actually correct.

Check diff from fork-point to last commit

To build the diff you must also fetch the fork-point in addition to the branch develop. This is not equivalent to main as other commits might have been pushed there. Luckily GitLab already tells us the fork-point commit: The pre-defined variable CI_MERGE_REQUEST_DIFF_BASE_SHA contains the SHA1 of the merge request diff. We have to fetch this either manually by running git fetch ourself — or we can add it to GIT_FETCH_EXTRA_FLAGS. That has the benefit that the GitLab-Runner will fetch the commit for us when it fetches develop. This is more efficient and also uses the credentials of the Runner.

run linter:
  variables:
    GIT_DEPTH: 1
    GIT_FETCH_EXTRA_FLAGS: "--prune --quiet --no-tags $CI_MERGE_REQUEST_DIFF_BASE_SHA"
  script:
    - git diff "${CI_MERGE_REQUEST_DIFF_BASE_SHA}..HEAD" | checkpatch -

Correction (2026-04-17): Actually this does not work as the GitLab-Runner uses git init followed by git fetch by default. For the later GitLab will add the refspecs +refs/heads/*:refs/origin/heads/* and +refs/tags/*:refs/tags/* to git fetch, which will fetch all heads and tags. Neither the --no-tags disables fetching the tags not is the $CI_MERGE_REQUEST_DIFF_BASE_SHA needed.

Instead of using init+ferch, git clone can be used if

• the feature-flag FF_USE_GIT_NATIVE_CLONE is enabled.
• the version of git is at least 2.49 released 2025-03-14 supporting --branch and the newer --revision.
• GIT_STRATEGY=clone is set as a job variable.

GIT_DEPTH can be used to limit the git clone --depth, which then also adds --single-branch. That option and other options like --no-tags can be passed via GIT_CLONE_EXTRA_FLAGS.

The drawback is, that git clone can only clone a single branch or revision. Fetching CI_MERGE_REQUEST_DIFF_BASE_SHA thus requires a manual call of git fetch.

run linter:
  variables:
    GIT_STRATEGY: "clone"
    GIT_DEPTH: 1
    GIT_CLONE_EXTRA_FLAGS: "--no-tags --single-branch"
  script:
    - git fetch origin "$CI_MERGE_REQUEST_DIFF_BASE_SHA"
    - git diff "${CI_MERGE_REQUEST_DIFF_BASE_SHA}..HEAD" | checkpatch -

Check all commits from fork-point to last commit

To check all commits of a MR individually, you have to do more work. The git protocol only supports fetching refs by name or tags / commits / trees / blobs by SHA1! The problem is that you don’t know how many commits are between the fork-point and HEAD; you do not know which value to use for GIT_DEPTH. By using GIT_DEPTH: 0 you tell the GitLab-Runner to not do a shallow-clone and to clone all commits reachable from the tip of the branch. For large repositories with many commits or large blobs that can become very costly. So reducing the number or type of objects to download can become important.

There are two sub-cases:

Only check all commit messages

If you only need the commit messages (and not the files itself), you can use git clone --filter to limit which type of objects you want to clone initially: commits, trees, blobs. Missing objects are lazy-fetches, which can result in a dramatic performance issue when your initial clone filters too much!

But there is one caveat and you have to be careful to use the right git commands: If any git command requires missing data, git will fetch it on demand. It will connect again to the remote serer and fetch the missing data. As this will result in (many) network connection with lots of round-trips, this is then slower than to download it at once during the initial clone.

Therefore check the commands you run:

• git format-patch and git show require the associated trees and blobs to be present and will trigger delayed fetches.
• git log --no-patch does not and works on commits alone.

run linter:
  variables:
    GIT_DEPTH: 0
    GIT_FETCH_EXTRA_FLAGS: --prune --quiet --no-tags --filter=object:type=commit
  script:
    - git log --no-patch "${CI_MERGE_REQUEST_DIFF_BASE_SHA}..HEAD" | mrcheck

If you also need the latest tree, use --filter=tree:0 instead.

Check all individual commits and their trees (WIP)

Similar to above we first fetch only all commit objects. We then let git fetch the required tree and blob objects on demand.

run linter:
  variables:
    GIT_DEPTH: 0
    GIT_FETCH_EXTRA_FLAGS: --prune --quiet --no-tags --filter=tree:0
  script:
    - git log --patch "${CI_MERGE_REQUEST_DIFF_BASE_SHA}..HEAD" | checkpatch -

The alternative to start first with a limited number of commits and to incrementally deepen that number of commits until the fork-point is reached also works, but requires more work and leads to multiple network round-trips to the repository server. Which strategy is faster may depend on the history size.

TBC…

1. The Original implementation was from Martin von Löwis.
2. Anton Altaparmakov created the 2nd implementation Linux-NTFS from scratch, which replaced the original implementation.
3. Captive was the first user-space based implementation, which used the original Windows driver ntfs.sys from Microsoft and run it under Wine.
4. NTFS-3G also runs in user-space and uses FUSE to talk to the kernel.
5. Paragaon donated an open-source version of if proprietary NTFS3 to the Linux kernel. It was the first read-write implementation in the kernel, but is less documented.
6. NTFS Plus is based on the older Linux-NTFS implementation, adds read-write support and updates the implementation to use modern Linux APIs. It is scheduled to replace NTFS3 again.
7. AVM – now FRITZ! Technology – ported the NTFS-3G to kernel space, which is used in FritzOS only.

gantt
    title NTFS
    dateFormat  YYYY-MM-DD
    axisFormat %Y
    Original    : 1997-01-01, 2002-04-01
    Linux-NTFS  : 2002-04-01, 2024-01-01
    Captive     : 2003-01-01, 2006-01-01
    NTFS-3G     : 2006-01-01, 2030-01-01
    NTFS3       : 2021-11-01, 2026-01-01
    NTFS+       : 2026-03-01, 2030-01-01
    ANTFS       : 2012-01-01, 2030-01-01
    2.0.0       : vert, 1996-06-09, 1m
    2.2.0       : vert, 1999-01-26, 1m
    2.4.0       : vert, 2001-01-04, 1m
    2.6.0       : vert, 2003-12-18, 1m
    2.6.16      : vert, 2006-03-20, 1m
    2.6.27      : vert, 2008-10-09, 1m
    3.0         : vert, 2011-07-21, 1m
    3.8         : vert, 2013-02-18, 1m
    4.4         : vert, 2016-01-10, 1m
    4.19        : vert, 2018-10-22, 1m
    5.10        : vert, 2020-12-13, 1m
    5.15        : vert, 2021-10-31, 1m
    6.1         : vert, 2022-12-11, 1m
    6.6         : vert, 2023-10-29, 1m
    6.12        : vert, 2024-11-17, 1m

Links

• Wikipedia: Linux Kernel version history

In addition to that Makefiles can also be used to track implicit dependencies: If one file is modified, only those commands are re-run which are needed. For large projects that can be a big time-saver if incremental changes are done.

But how to do that properly (for a C project)?

The historical way

• Makefile

    main: main.o
    main.o: main.c
  

• main.c

    #include <stdio.h>
    #include "main.h"
  

• main.h

    #include <stdint.h>
  

In the past many projects implemented that themselves. They used the pre-processor cpp to process all #include statements and then used regular expressions to extract the path of all files, which have been read. These dependencies are then converted into a make fragment, which declares that dependency:

main.o: main.h /usr/include/stdio.h /usr/include/stdint.h

The main Makefiles has to include this fragment using something like -include main.d.

This solution has multiple issues.

Vanishing dependencies

Consider, you refactor your code and remove main.h. In that case your automatically generated dependencies show an issue: As main.o depends on main.h, which no longer is there, make will fail as there is no receipt to remake it.

This fix this your dependency generation tool needs to output empty rules for all dependencies:

main.h:
/usr/include/stdio.h:
/usr/include/stdint.h:

There are three cases:

1. if the file still exists and was not updated — it is older than the target — no remake is triggered by this dependency — but others may still trigger one.
2. if the file still exists and was updates — it is newer than the target - a rebuild is triggered for the target.
3. if the file does no longer exist, make invokes the empty receipt to remake it. The will not really create the file, but make will consider it as newer than the target and continue with the previous case 2 above and remake the target.

Without that any developer would have to invoke make clean to remove all targets and dependency files, resulting in a full rebuild:

.PHONY: clean
clean:
    $(RM) main *.o *.d

Maintaining the dependency tool

First of all you must run the pre-processor a 2nd time to generate the input for you dependency extraction tool. For small projects that cost might be negligible, but for larger projects that might add up.

Second you must maintain yet another tool. While the pre-processed output is relatively easy to parse, newer compiler versions may add new features or change the output slightly, which your tool then must handle also.

Third you must make sure to invoke your pre-process run with exactly the same arguments as your real compilation: Any -Ddefine, -Idirectory, -include, -imacros is important as otherwise you might miss or record wrong dependencies.

You must also decide, when to call your tool: Many projects call it before the actual compilation, but that is unneeded: If the target is missing, make must remake it anyway. If the target exists, but you don’t no longer have the dependency information, you must also remake the target as you cannot guarantee, that any (changed) header might not introduce a significant change.

Generating the dependency information afterwards looks okay. But you might get into situations, where you have stale information, for example if you interrupt make between the compilation and dependency-gathering steps.

Best would be to do it at the same time. Luckily that is possible with gcc and other modern compilers like clang.

The gcc way

Luckily modern GCC has built-in support to generate dependency information in make-syntax itself:

• -M enables generating dependency information instead of compiling the file. The output is written to STDOUT unless -o is used to redirect it to a file.
• -MM similar to the above, but system header files are not mentioned.
• -MD and -MMD are variants of -M and -MM respectively, which generate dependency information in addition to the requested action, e.g. -c to compile the unite.
• -MF file writes the information to the given file instead of STDOUT.
• -MP adds additional .PHONY targets for all dependencies to solve the Vanishing dependencies problem from above.
• -MT target allows to overwrite the target name. By default the base-name of the main input file is used, where the suffix is replaced by .o.
• -MQ target is the variant of the above, which also quotes any make meta-characters to make sure, the name is not mangled by make but reaches the shell command as-given.

So let’s rewrite our Makefile and try this:

main: main.o

%.o %.d &: %.c
	$(CC) $(CPPFLAGS) $(CFLAGS) -MMD -MF $*.d -MP -c -o $*.o $<

-include *.d

1. ‘&: tells make`, that the recipt generated both files at the same time. (grouped targets)
2. -MMD tells gcc to both compile and generate dependency information at the same time. System header files are excluded.
3. -MF $*.d tells gcc to write the dependency information into a file with the file name extension .d.
4. -MP tells gcc to generate .PHONY targets for all included file to make the dependency information future-proof in case one of them gets deleted.
5. -c -o $*.o $< to compile the unit.
6. -include *.d includes the dependency information as far as it already exists

First compilation issue

This does not work as expected: make has a built-in mechanism to Remake Makefiles. All files included via include are considered Makefiles and make tries to update them. If there is no file *.d, make applies our rule and will try to compile *.c to *.d :-( (That is why the above rule already uses $*.o instead of $@ as the later would be *.d, which then is passed to both -MF and -o with catastrophic results.)

We can avoid this by explicitly using $(wildcard ) to include only the existing files:

-include $(wildcard *.d)

Second compilation issue

While the solution looks okay, actually it is not: This way dependency information is optional. If you delete all dependency files *.d, modify main.h and re-run make: Nothing will happen. We lost the information, that main.o depends on main.h. Therefore we must change the rule to always require the associated file $*.d to always exist:

%.o: %.c %.d
	$(CC) $(CPPFLAGS) $(CFLAGS) -MMD -MF $*.d -MP -c -o $@ $<
%.d: ;
.NOTINTERMEDIATE: %.d

• the empty rule for %.d is needed for make to handle the case, when the file is missing. For that case we tell make that it should consider that file as remade, so it newer than the target. That will remake the target to actually generate the real dependency information.

• the .NOTINTERMEDIATE is needed as %.d is never mentioned as a real target. make will search its chain of implicit rules main → main.o → main.d and mark it as intermediate. Because of that the file is not remade and/or will be deleted if it is remade. By marking it as non-intermediate we tell make to handle it as a regular file and to keep it afterwards.

  This is only available since GNU make 4.4!

Final version — make 4.4

#!/usr/bin/make -f
# Disable built-in rules and variables
MAKEFLAGS += --no-builtin-rules

main: main.o

CFLAGS := -g
MYCFLAGS := -Wall -Werror
DEPFLAGS = -MMD -MP -MF $*.d -MT $@

COMPILE.c = $(CC) $(DEPFLAGS) $(CPPFLAGS) $(CFLAGS) $(MYCFLAGS) -c

%.o: %.c %.d
	$(COMPILE.c) $(OUTPUT_OPTION) $<
%.d: ;
.NOTINTERMEDIATE: %.d
%: %.o
	$(LINK.o) $^ $(LOADLIBES) $(LDLIBS) -o $@

-include $(wildcard *.d)

.PHONY: clean
clean:
	$(RM) main *.o *.d

Final version — make 4.3

#!/usr/bin/make -f
# Disable built-in rules and variables
MAKEFLAGS += --no-builtin-rules

SRCS := main.c
OBJS := $(SRCS:%.c=%.o)
DEPS := $(SRCS:%.c=%.d)

main: $(OBJS)

CFLAGS := -g
MYCFLAGS := -Wall -Werror
DEPFLAGS = -MMD -MP -MF $*.d -MT $@

COMPILE.c = $(CC) $(DEPFLAGS) $(CPPFLAGS) $(CFLAGS) $(MYCFLAGS) -c

%.o: %.c %.d
	$(COMPILE.c) $(OUTPUT_OPTION) $<
$(DEPS):

-include $(wildcard $(DEPS))

.PHONY: clean
clean:
	$(RM) main $(OBJS) $(DEPS)

The kbuild way

The Linux kernel uses its own build system called kbuild, which is based on a bunch of make receipts. It has some additional requirements:

1. The Linux is heavily configurable. There is a huge .config file, which lists all options. If that file would be used as a pre-dependency, all such files would get rebuilt each time a single option was changed. Therefore kbuild uses some mechanisms to split that big file into smaller chunks, so that each compilation unit can just depend on those options, it really depends on.
2. The above solution does not track the $(…FLAGS) variables or $(CC). Changing them might a complete rebuild to have a consistent kernel again. As such kbuild logs the final command used to compile the target also in the dependency information file. On the next run the commands are compared and the invocation may only be skipped, if they match.

For that kbuild overwrites most of makes dependency mechanism with its own implementation:

1. Most targets have FORCE as their pre-dependency, so that the receipt will always run.
2. The receipt itself will then use some heavy macro magic to read back its dependency information from a file and compare that to the actual run. The command is only executed if any pre-requisite is changed or any relevant configuration option is changed.
3. If a command cannot determine, if it needs to run, it will run by default but will write its output to a temporary file. That file is then compared to the previous version.
   • if the content differs, the temporary file is renamed over the real output file.
   • if the content did not change, the temporary file is deleted. That way the old time stamp is preserved if no change did happen. This is done to prevent needless downstream rebuilds.

Closing word

Much of this was inspired by the article Auto-Dependency Generation from Paul D. Smith. Thank you very much for writing this in the first place. The main difference is, that he uses a variable $(SRCS), which explicitly lists all source C files. That way he can explicitly name the expected *.o and *.d files, which bypasses the problem with intermediate files from my solution above. That version also works for make 4.3 an earlier as .NOTINTERMEDIATE is only available since make 4.4.

A: gdb --silent --batch -ex 'ptype /o struct my_t' some.o

The past days I was investigating some performance issues with a proprietary SoC: The code consists of closed-source pre-compiled binaries combined with public header files. Some public glue-code added accessors to allocate, copy, and free the data structures.

Padding

We have the requirement to extend the data-structure and add some additional members. As some code was close-sourced, it is important to not change the layout of the existing structure. Luckily C adds padding between members to align the next member according to common hardware constraints:

The start address of a 1,2,4,8,16,32,64,… sized member must align to that size.

Consider this:

#include <stdint.h>
struct my0_t {
    uint8_t foo;
    uint32_t bar;
} var0[3];
static_assert(sizeof(var0) == 8 * 3, "Unexpected sizeof");

bar has size 32 bits or 4 bytes, so var0[0] must be placed into memory so that &(var[0].bar) % 4 == 0 is true. The C-compiler will thus add padding bytes before bar to satisfy that requirement. Compiling the code with -Wpadded shows this:

$ gcc -c -g -Wpadded c-padding.c
c-padding.c:4:14: warning: padding struct to align ‘bar’ [-Wpadded]
    4 |     uint32_t bar;
      |              ^~~

But you don’t know, what the C compiler does here: gcc may either insert padding before foo or after it:

struct my1_t {
    uint8_t foo;
    uint8_t _padding[3];
    uint32_t bar;
} var1[3];
static_assert(sizeof(var1) == 8 * 3, "Unexpected sizeof");

or

struct my2_t {
    uint8_t _padding[3];
    uint8_t foo;
    uint32_t bar;
} var2[3];
static_assert(sizeof(var2) == 8 * 3, "Unexpected sizeof");

Both are valid, but all I have ever seen is padding being inserted after the previous member and before the next member.

But you can use gdbs ptype command to dump the exact layout including offset, size and inserted padding:

$ gdb --silent --batch -ex 'ptype /o struct my0_t' c-padding.o
/* offset      |    size */  type = struct my0_t {
/*      0      |       1 */    uint8_t foo;
/* XXX  3-byte hole      */
/*      4      |       4 */    uint32_t bar;
                               /* total size (bytes):    8 */
                             }

For this to work you need DWARF debugging information. So please make sure you compile your code with -g enabled!

This extra padding increases the size of your struct, which might be undesired: On embedded systems you often have less memory and excessive padding might waste a lot of memory. To minimize this, you have multiple options:

Packing

You can declare the struct as packed:

struct my3_t {
    uint8_t foo;
    uint32_t bar;
} __attribute__((packed)) var3[3];
static_assert(sizeof(var3) == 5 * 3, "Unexpected sizeof");

This makes the struct as compact as possible by not inserting any padding automatically. But you will get into trouble and risk getting a SIGBUS error on some architectures: Accessing a 32 bit variable which is not 4 byte aligned requires additional work:

1. Either the hardware has some extra logic to split non-aligned memory access into multiple accesses and to recombine both parts into the final value,
2. Or the compiler has to generate extra code to not do the unaligned access,
3. Or your program terminates with SIGBUS as the processor raises the unaligned trap

Please do not use -fpack-struct to make every struct packed by default!

Re-ordering descending by size

Most often you can re-order your members descending by size – assuming sizes being a power-of-two. The compiler still adds padding, but only at the end of the structure. That way you do not have holes in the middle.

That changes the layout and breaks any ABI compatibility! So not not do this with structs, which are used to communicate with your hardware or some closed source binary, which assumes the old layout.

Careful re-ordering

If you only need to insert some small data, look for those hole: As the compiler added padding there automatically, there is no guarantee that these bits/bytes are zero initialized. If you need that guarantee, you must manually insert padding bytes!

On the other hand that provides the opportunity, to re-use those undefined bits for additional members. Just look for a hole which is large enough for your data and add your member in between the members bordering that hole.

Just be careful with structures which are used with hardware: If their accessor function does a memset(…, 0, …) to initialize the struct to zero, it might be important that those bits remain cleared. If you then start using those bits, the hardware might get confused.

Alignment

You might have noticed, that sizeof(struct my0_t) == 8 and not 5 == sizeof(uint8_t) + sizeof(uint32_t). gcc also adds padding before or after all members to extend the struct, until its sizeof if a natural multiple of the widest element. This is important for arrays where multiple instances are placed after each other. There each instances start address must be aligned properly, which requires padding in between. The distance between two elements is called “stride size”, which equals the sizeof.

This also applies to nested structs like this:

struct my5_t {
    struct my4_t baz;
    uint8_t bla;
} var5[3];
static_assert(sizeof(var5) == 12 * 3, "Unexpected sizeof");

This might be unexpected as my4_t ends with 3 padding bytes, where bla might fit it. Instead baz gets placed after the padding from baz, after which 3 more padding bytes are required. So in total you get 6 bytes of padding.

Cache line size

Alignment becomes even more important for performance. Modern CPUs have lots of caches and their line size specifies the smallest quantity for data transfer. Even when you only require a single bit, the cache will transfer 32 or 64 or even more bytes from RAM.

• with more tightly packed structs you get more data per cache-line and require fewer cache-lines, leaving more free cache lines for other tasks.
• on the other hand false sharing might become a performance issue with multi-threading, where data with different access patterns are stored in the cache line.

struct my6_t {
    uint8_t foo;
    uint32_t bar;
} __attribute__((aligned(32))) var6[3];
static_assert(sizeof(var6) == 32 * 3, "Unexpected sizeof");

In this case we get 3 bytes of padding between foo and bar. But we also get 24 bytes of padding after bar to make sizeof(struct my6_t) a multiple of 32 as requested by __attribute__((aligned(32))).

This easily becomes worse with nested ``structs where inner struct`s also have alignments:

struct my7_t {
    uint8_t foo;
    struct inner {
        uint8_t foo;
    } __attribute__((aligned(32))) bar[4];
} var7[3];
static_assert(sizeof(var7) == 64 * 3, "Unexpected sizeof");

Runnig gdb shows what happens:

$ gdb --silent --batch -ex 'ptype /o var7' c-padding.o
type = struct my7_t {
/*      0      |       1 */    uint8_t foo;
/* XXX 31-byte hole      */
/*     32      |      32 */    struct inner {
/*     32      |       1 */        uint8_t foo;
/* XXX 31-byte padding   */
                                   /* total size (bytes):   32 */
                               } bar;
                               /* total size (bytes):   64 */
                             } [3]

Summary

• Use gccs -Wpadded to get a warning.
• Use gdbs ptype to print the real layout.
• Verify your assumtions, especially if the same code is compiled for multiple platforms with different alignment requirements.
• Do not trust the comments in the code claiming ancient values for sizeof or proper cache line alignment.
• Explicitly add padding bytes as they are then also initialized; otherwise the compiler may do as it likes.

Further reading

• gcc: -Wpadded
• ESR: The Lost Art of Structure Packing

• the number of arguments must match
• the ordering of the arguments must match
• the data types must match, which includes the structure and layout of all input and output parameters

If any of them changes, the Application Binary Interface (ABI) changes and you risk crashing the kernel. If you’re lucky, recompiling the kernel and the modules is enough for both ends to pick up the new Application Programming Interface (API).

To detect such breaking changes, the Linux kernel can be compiled with CONFIG_MODVERSIONS enabled: This calculates a Cyclic Redundancy Check (CRC) checksum over the function signature and embeds this information with the kernel and the modules. The dynamic linker of the Linux kernel checks, that for each requested symbol its CRC matches the CRC of the Linux kernel or already loaded modules. A module is only loaded, if a match is found for all symbols. Otherwise loading fails.

Rust goes DWARF

The mechanism described here does not work with Rust. As such the Linux kernel learned a new trick and can use the DWARF (Debugging With Arbitrary Record Formats) debugging information to calculate the CRC. When CONFIG_RUST is enabled, gendwarfksyms is used instead of genksyms. Both versions are incompatible as they calculate different CRCs for the same function. But they work similar enough, so I will not go into details here. If you’re interested, look for CONFIG_EXTENDED_MODVERSIONS.

Executable and Linkable Format

Linux Kernel modules object files using the Executable and Linkable Format (ELF). Instead of using the well-known suffix .o, they use the suffix .ko, but are otherwise the same. They are comprised of multiple sections containing executable code, read-only constants, initialized data and other informations required for linking.

$ objdump --section-headers --wide avm-modver.ko
$ LC_ALL=C readelf --wide --section-headers avm-modver.ko
There are 38 section headers, starting at offset 0x25d80:

Sections Header:
  [Nr] Name                              Type     Addresse    Off   Size ES Flg Lk Inf Al  Usage
🔵[ 0]                                   NULL            0      0      0  0      0   0  0  ELF header
🟠[ 1] .note.gnu.build-id                NOTE            0     40     24  0   A  0   0  4  unique build ID bitstring
🟣[ 2] .note.Linux                       NOTE            0     64     30  0   A  0   0  4  Architecture data
🟢[ 3] .text                             PROGBITS        0     a0     1f  0  AX  0   0 16  Code
⚪[ 4] .rela.text                        RELA            0  14e50     30 18   I 35   3  8
🔴[ 5] __ksymtab                         PROGBITS        0     c0      c  0   A  0   0  4  EXPORT_SYMBOL
⚪[ 6] .rela__ksymtab                    RELA            0  14e80     48 18   I 35   5  8
🔴[ 7] __kcrctab                         PROGBITS        0     cc      4  0   A  0   0  4  CRC
🟣[ 8] __mcount_loc                      PROGBITS        0     d0      8  0   A  0   0  1  ftrace()
⚪[ 9] .rela__mcount_loc                 RELA            0  14ec8     18 18   I 35   8  8
🟣[10] .modinfo                          PROGBITS        0     d8     92  0   A  0   0  1  MODULE_INFO
🟣[11] .return_sites                     PROGBITS        0    16a      4  0   A  0   0  1  Live patching
⚪[12] .rela.return_sites                RELA            0  14ee0     18 18   I 35  11  8
🟣[13] .call_sites                       PROGBITS        0    16e      4  0   A  0   0  1  Live patching
⚪[14] .rela.call_sites                  RELA            0  14ef8     18 18   I 35  13  8
🔴[15] __ksymtab_strings                 PROGBITS        0    172      d  1 AMS  0   0  1  EXPORT_SYMBOL
🔴[16] __versions                        PROGBITS        0    180     51  0   A  0   0 32  CRC
🟣[17] __patchable_function_entries      PROGBITS       58    1d8      8  0 WAL  3   0  8  NOPs
⚪[18] .rela__patchable_function_entries RELA            0  14f10     18 18   I 35  17  8
⚫[19] .data                             PROGBITS        0    1e0      0  0  WA  0   0  1  Initialized data
🟣[20] .gnu.linkonce.this_module         PROGBITS        0    200    500  0  WA  0   0 64
⚫[21] .bss                              NOBITS          0    700      0  0  WA  0   0  1  Uninitialized data
🟠[22] .debug_info                       PROGBITS        0    700   b51f  0      0   0  1
⚪[23] .rela.debug_info                  RELA            0  14f28   fc00 18   I 35  22  8
🟠[24] .debug_abbrev                     PROGBITS        0   bc1f    71e  0      0   0  1
🟠[25] .debug_aranges                    PROGBITS        0   c33d     50  0      0   0  1
⚪[26] .rela.debug_aranges               RELA            0  24b28     48 18   I 35  25  8
🟠[27] .debug_line                       PROGBITS        0   c38d    3be  0      0   0  1
⚪[28] .rela.debug_line                  RELA            0  24b70   1050 18   I 35  27  8
🟠[29] .debug_str                        PROGBITS        0   c74b   7792  1  MS  0   0  1
🟠[30] .debug_line_str                   PROGBITS        0  13edd    943  1  MS  0   0  1
🟡[31] .comment                          PROGBITS        0  14820     58  1  MS  0   0  1  Compiler version
🟡[32] .note.GNU-stack                   PROGBITS        0  14878      0  0      0   0  1  Stack hardening flag
🟠[33] .debug_frame                      PROGBITS        0  14878     40  0      0   0  8
⚪[34] .rela.debug_frame                 RELA            0  25bc0     30 18   I 35  33  8
🔵[35] .symtab                           SYMTAB          0  148b8    438 18     36  40  8  Symbols
🔵[36] .strtab                           STRTAB          0  14cf0    15f  0      0   0  1  Symbol names
🔵[37] .shstrtab                         STRTAB          0  25bf0    18b  0      0   0  1  Section names
Key to Flags:
  Write, Alloc, eXecute, Merge, Strings, Info, Link order, extra Os processing required, Group, TLS,
  Compressed, x=unknown, o=OS specific, Exclude, mbinD, large, processor specific

The section names have varying lengths. As such the names are collected in their own section called .shstrtab, which is referenced by index in the ELF file header. All sections are listed in the section header table and their names are referenced by offset. Run readelf -p .shstrtab avm-job.ko to dump those names.

Similar for symbols: There names are collected in the section .strtab and referenced via offset from .symtab. Run readelf -p .strtab avm-job.ko to dump those names.

.symtab contains all symbols (and .strtab) their names. When shared objects (.so) are used, the linker moves those symbols to .dynsym and their names to .dynstr. Already resolved symbols may be removed respectively both tables .symtab and .strtab may be stripped completely.

The remaining dynamic symbols are only resolved by the dynamic linker, when section is loaded. The dynamic linker has to go through the section and substitute the placeholders with the then correct address. For that the ELF file contains the relocation sections, of which there are two types: REL (relocation without addend) and RELA (relocation with addend), which allows to add an additional constant. Either one may be used per section and each table references a symbol table, which gets used.

Not all of them are loaded into memory respectively are freed again, when they are no longer needed by the linker. Only those sections, which contain information that is necessary for runtime execution of the file, are kept. Multiple (similar) sections can be combined and are then called segments. But that is only relevant for fully linked executables: Only they have a program header

References to functions are then resolved by the linker and the place-holders get replaced by the real addresses. This is where versioning kicks in.

Anatomy of a Linux kernel module

When you write and export a function in the Linux kernel or an module, the following happens:

void my_function(void) {
    return;
}

1. The compiler/assembler puts the code into the .text section.
2. The name of the function is added to the .strtab section.
3. An entry is added to the symtab section linking the offset within the .text section to the name via its offset in the strtab section.

Using EXPORT_SYMBOL adds more magic:

#include <linux/module.h>
EXPORT_SYMBOL(my_function);

1. It puts the name of the function into a section called __ksymtab_strings.

    $ LC_ALL=C readelf --wide --string-dump=__ksymtab_strings avm-modver.ko
    String dump of section '__ksymtab_strings':
      [     0]  my_function
   

2. It creates a new section called __ksymtab+my_function with a single struct kernel_symbol linking the address of the function to its name. Later on these sections will be collected by the linker script scripts/module-common.lds and will be put into the section called __ksymtab. Similar happens for EXPORT_SYMBOL_GPL and EXPORT_SYMBOL_GPL_FUTURE and EXPORT_SYMBOL_NS, but with different prefixes.

    $ LC_ALL=C readelf --wide --relocated-dump=__ksymtab --relocs avm-modver.ko | grep -A4 __ksymtab
    Relocation section '.rela__ksymtab' at offset 0x14fc8 contains 3 entries:
        Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
    0000000000000000  0000002e00000002 R_X86_64_PC32          0000000000000010 my_function + 0
    0000000000000004  0000001b00000002 R_X86_64_PC32          0000000000000000 __kstrtab_my_function + 0
    0000000000000008  0000001c00000002 R_X86_64_PC32          000000000000000c __kstrtabns_my_function + 0
    --
    Hex dump of section '__ksymtab':
      0x00000000 10000000 fcffffff 04000000          ............
   

Too see more details, use make avm-modver.i to run the pre-processor and to get the intermediate file, where all macros have been expanded.

With CONFIG_MODVERSIONS enabled even more magic happens. If a module uses EXPORT_SYMBOL, then genksyms is called. The source code of the module is pre-processed again via cpp, but with a different definition for EXPORT_SYMBOLS.

1. For each function exported via EXPORT_SYMBOL a CRC for the function signature is computed by parsing the C function call. A new section called ___kcrctab+my_function with a single long containing the CRC is created. Later on these sections will be collected by the linker script scripts/module-common.lds and will be put into the section called __kcrctab. Similar happens for EXPORT_SYMBOL_GPL and EXPORT_SYMBOL_GPL_FUTURE and EXPORT_SYMBOL_NS, but with different prefixes.
2. For each used symbol the CRC is looked up in the Module.symvers files. They are created as part of the kernel or any module compilation process when CONFIG_MODVERSIONS is enabled. The file collects the CRC and module path for each symbol. The symbol name and its CRC is collected in a const char __versions[] array in section __versions.

Module loading

When a kernel module is loaded, the Linux kernel linker resolves all dynamic symbols of the module. It looks up each unresolved symbol from .symtab and resolves it to all symbols loaded so far. You can view them from user-space in /proc/kallsyms.

In addition to that simple lookup the loader also checks the modules licence from .modinfo: Symbols exported via EXPORT_SYMBOL_GPL can only be resolved if the module has MODULE_LICENCE("GPL") and such.

When CONFIG_MODVERSIONS is enabled, the linker inside the Linux kernel also checks the CRC: For every undefined symbol there is a matching entry for it in section __versions, which contains the CRC of the symbol from compile time.

$ LC_ALL=C readelf --wide -s avm-modver.ko | grep UND
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
    42: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __fentry__
    43: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _printk
    44: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __x86_return_thunk

But there are two different layouts used:

Upstream Linux Kernel

$ LC_ALL=C readelf --wide --hex-dump=__versions avm-modver.ko
Hex dump of section '__versions':
  0x00000000 bb6dfbbd 00000000 5f5f6665 6e747279 .m......__fentry
  0x00000010 5f5f0000 00000000 00000000 00000000 __..............
  0x00000020 00000000 00000000 00000000 00000000 ................
  0x00000030 00000000 00000000 00000000 00000000 ................
  0x00000040 d87e9992 00000000 5f707269 6e746b00 .~......_printk.
  0x00000050 00000000 00000000 00000000 00000000 ................
  0x00000060 00000000 00000000 00000000 00000000 ................
  0x00000070 00000000 00000000 00000000 00000000 ................
  0x00000080 cb8119bf 00000000 6d6f6475 6c655f6c ........module_l
  0x00000090 61796f75 74000000 00000000 00000000 ayout...........
  0x000000a0 00000000 00000000 00000000 00000000 ................
  0x000000b0 00000000 00000000 00000000 00000000 ................

The original Linux kernel uses const struct modversion_info __version[]. The structure has a fixed size of 64 bytes:

• the first 8 bytes contain the CRC.
• the remaining 56 bytes contain the symbol name.

Longer symbol names are not supported and require the use of the extended modversions.

Ubuntu Linux Kernel

$ LC_ALL=C readelf --wide --hex-dump=__versions avm-modver.ko
Hex dump of section '__versions':
  0x00000000 14000000 bb6dfbbd 5f5f6665 6e747279 .....m..__fentry
  0x00000010 5f5f0000 10000000 7e3a2c12 5f707269 __......~:,._pri
  0x00000020 6e746b00 1c000000 ca39825b 5f5f7838 ntk......9.[__x8
  0x00000030 365f7265 7475726e 5f746875 6e6b0000 6_return_thunk..
  0x00000040 18000000 eb7b33e1 6d6f6475 6c655f6c .....{3.module_l
  0x00000050 61796f75 74000000 00000000 00000000 ayout...........
  0x00000060 00

Ubuntu has changed this and uses const char ____versions[]:

• the first 8 bytes contain the CRC.
• next follows the symbol name with a terminating NUL byte.
• more NUL bytes for padding up to the next address dividable by 4.

Ubuntu changed this to support longer symbol names, which Ubuntu claims is required for RUST support. See modpost: support arbitrary symbol length in modversion for details. This has been reverted by 2039010.

…

Further reading

• Linux: Building External Modules - Module Versioning
• LWN: A new version of modversions
• Anatomy of the Linux loadable kernel module
• Linux manual page: Executable and Linking Format

#!/bin/sh
set -e
date && false && true
date

Wie häufig wird date ausgeführt?

Wie üblich muss man die Manual-Page von bash sehr genau lesen:

The ERR trap is not executed if the failed command is … part of a command executed in a && or || list except the command following the final && or ||.

Die korrekte Antwort lautet also: 2

Beim date && false && true Endet die Ausführung nach dem false und der Exit-Code ist 1: Das nachfolgende && … wird nicht mehr ausgeführt. Da das false aber dadurch nicht der letzte Befehl ist, bricht set -e nicht ab und das 2. date wird trotzdem ausgeführt.

Das sollte man bedenken, wenn einem shellcheck folgende Warnungen ausspuckt und einen dazu anregen, mehr && zu benutzen:

• SC2015: Note that A && B || C is not if-then-else. C may run when A is true.
• SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well-defined.

Man baut sich dadurch leicht semantische Unterschiede ein.

Oder um es mit den Worten von G. Aschemann aus 1995 zu sagen:

Jedes gute Shell-Script fängt mit #!/usr/bin/perl an.

Naja, das ist 30 Jahre her und ich würde doch perl durch python ersetzten wollen 😉

I just updated my laptop and servers and stumbled upon some issues:

cyrus-imapd

I’m running my own mail server infrastructure: I have grown up with Unix-to-Unix-Copy-Protocol (UUCP) and was an admin for UUCP Freunde Lahn e.V. for a long time. I’m still using Postfix with Cryus IMAPd and never switched to Dovecot.

After the upgrade I noticed that no mails were delivered: mailq shown a growing list of mails stuck in queue. Postfix was complaining that its lmtp service was no longer able to establish an encrypted connection to lmtpd from Cyrus.

For historic reason my setup is using STARTTLS, which is now deprecated and has been disabled by default in Cyrus IMAPd. You have to explicitly re-enable it in your /etc/imapd.conf by adding some lines:

imap_allowstarttls: yes
lmtp_allowstarttls: yes

saslauthd

saslauthd.service failed to start as I had to move its UNIX socket to /var/spool/postfix/var/run/saslauthd/. This also moves the location of the PID file to that directory, which then no longer matches the information in /usr/lib/systemd/system/saslauthd.servie, which expects the file in /var/run/saslauthd.pid.

A fixed this by creating an override with systemctl edit saslauthd.service:

[Service]
PIDFile=/var/spool/postfix/var/run/saslauthd/saslauthd.pid

Previously I had a shell-hack in /etc/default/saslauthd to replace the old location with a symbolic link to the chroot-location. This no longer works as that file is not sourced by systemd, which does not execute that shell code. Therefore I had to tell Cyrus IMAPd to also use that changed location by putting this into /etc/imapd.conf:

sasl_saslauthd_path: /var/spool/postfix/var/run/saslauthd/mux

PS: On a side node: /var/run/ is deprecated and should be replaced by just /run/; systemd already complains about this every time it sees /var/run/.

docker.io and libvirt

For some unknown reason docker.io and libvirt got removed during the upgrade. Running apt autopurge afterwards was a very bad idea as that purged all images, volumes and containers. 🤦

I have to investigate why that happened. 🔍

PHP-8.4

Debian-13-Trixie has PHP-8.4, while Debian-12-Bookworm had PHP-8.2. My local NextCloud (and Wordpress) setup was unhappy about that as it needs several php8.4-… packages. Luckily just installing the equivalent of the matching packages did fix this.

KDE

In the past I did not install kde-full as it depends on many optional packages like KMail, KOrganizer, DragonPlayer, and such. I don’t use may of those and thus don’t want them to be installed. During the upgrade plasmashell got removed so on the next login I did not get back a working KDE session. Installing kde-standard fixed this. As it only Recommends most other packages, I was able to get rid of those packages I do not want.

And I got Wayland, which has this annoying bug: Konsole no longer stores the open sessions and starts with only one shell in $HOME. 🤔

Out-of-space /usr

My desktop system has many packages. Upgrading all those (KDE-)libraries required too much space on /usr. dpkg failed to unpack a package during upgrade.

After some manual dpkg --configure --pending, apt install --fix-broken, apt autopurge and dpkg -P I was finally able to continue. I would have expected for APT to check for enough disk space, but apparently it does not. So double-check manually before doing an upgrade.

PS: Afterwards systemd complains about usr-not-merged, but that is normal and expected.

KeePassXC

I used a self-compiled version of KeePassXC. Debian now has two packages keepassxc and keepassxc-full – the later has support for browser-integration and more. As some file have been move, the upgrade failed and I had to manually remove by self-compiled version.

Network

Running the upgrade while being logged into KDE is not a good idea: During the upgrade NetworkManager got restarted and killed my local network connection. Afterward even ping did no longer work, as I already had the new version but still the old Linux kernel.

Sadly I still need my r8168-dkms and v4l2loopback-dkms packages.

Prometheus MySQL/MariaDB exporter

v0.15.0 has a breaking change, which is neither mentioned in any NEWS file nor the debian/changelog.

• DATA_SOURCE_NAME is no longer supported and you must pass the credentials via --mysqld.username= and via MYSQLD_EXPORTER_PASSWORD=.
• You also cannot specify the UNIX domain socket /run/mysqld/mysqld.sock

I’m now using --config.my-cnf /var/lib/prometheus/mysql.cnf to configure the credentials via another file.

Mailman3

cron

mailman3-web still runs a CRON job every minute, which imports robot_detection, which spams you with a ton of SyntaxWarnings. See mailman3-web#1082541 and python3-robot-detection#1078661

Edit /etc/cron.d/mailman3-web and add 2>/dev/null to each command.

authentication

Authentication was now broken for me: /var/log/mailman3/web/mailman-web.log complains about Missing column 'socialaccount_socialapp.provider_id'. Run /usr/bin/mailman-web migrate as user root to fix this.

template

/var/log/mailman3/web/mailman-web.log showed another error:

django.template.exceptions.TemplateSyntaxError: ‘humanize’ is not a registered tag library.

Adding django.contrib.humanize TO INSTALLED_APPS in /etc/mailman3/mailman-web.py fixes this.

#!/bin/sh
trap 'echo Cleanup…' EXIT HUP INT TERM
...

Exit and signals

Before we begin: Actually exit codes are mutual exclusive to signal statuses: A process may either exit normally using exit or terminate via a signal.

If you read man:bash you will read this:

The return value of a simple command is its exit status, or 128+n if the command is terminated by signal n.

That might give you the idea, that they are the same, but that is only a (broken) shell convention to map signal statuses to exit codes. Reading man:exit you see this:

The value status & 0xFF is returned to the parent process as the process’s exit status,

So there are 256 exit codes from 0 to 255, which a process can use to exit.

The parent process then uses waitpid() to wait for the childs state change:

That may be the process exited by calling exit() itself or caught a signal(), which might have kill()ed the process or just suspended it.

You then have to first use WIFEXITED() or WIFSIGNALED() to check, if the child exited normally via exit() or caught a signal(). Only after that you should either use WEXITSTATUS() to extract the byte containing the exit code or use WTERMSIG() to extract the signal number.

In a shell script you do not have access to these low-level C functions, but only get the mangled exit status. You cannot distinguish is the called process did exit(130) itself or was terminated by the user pressing Ctrl-C so send SIGINT to it.

Signals and EXIT trap

Here’s a short overview of commonly used signals and traps.

Please not that shells misuse signal 0 here: By default there is not signal numbered 0. Actually it is a no-operation and can be used to check, if process A can send signals to process B or if process B is still alive. bash and other shells re-use that number to give their EXIT handler a number, which is supposed to be called on any exit from shell. But that behaviour is very implementation dependant as you will see later on.

Implementation specific handling of EXIT

Let’s try this with the more informative shell script trap.sh:

#!/bin/bash
cleanup () {
    local rv=$? sig=${1:-0}
    echo "Process $$ received signal $sig after rv=$rv"
    case "$sig" in
    0|'') exit "$rv";;
    *) trap - "$sig"; kill "-$sig" "$$";;
    esac
}
trap 'cleanup 0' EXIT
trap 'cleanup 1' HUP
trap 'cleanup 2' INT
trap 'cleanup 3' QUIT
trap 'cleanup 15' TERM

[ -n "${1:-}" ] && kill "-$1" "$$"

bash

$ bash ./trap.sh 0  # EXIT
Process 499218 received signal 0 after rv=0
$ bash ./trap.sh 1  # SIGHUP
Process 499237 received signal 1 after rv=0
Process 499237 received signal 0 after rv=0
Hangup
$ bash ./trap.sh 2  # SIGINT
Process 499256 received signal 2 after rv=0
Process 499256 received signal 0 after rv=0

$ bash ./trap.sh 3  # SIGQUIT
Process 499275 received signal 3 after rv=0
Process 499275 received signal 0 after rv=0
$ bash ./trap.sh 15  # SIGTERM
Process 499294 received signal 15 after rv=0
Process 499294 received signal 0 after rv=0
Terminated

As you can see bash always calls the trap handler for EXIT!

dash

Let’s repeat this with dash:

$ dash ./trap.sh 0  # EXIT
Process 502873 received signal 0 after rv=1
$ dash ./trap.sh 1  # SIGHUP
Process 501892 received signal 1 after rv=0
Hangup
$ dash ./trap.sh 2  # SIGINT
Process 501912 received signal 2 after rv=0

$ dash ./trap.sh 3  # SIGQUIT
Process 501929 received signal 3 after rv=0
Verlassen (Speicherabzug geschrieben)
$ dash ./trap.sh 15  # SIGQUIT
Process 501971 received signal 15 after rv=0
Terminated

busybox

And once more with busybox:

$ busybox sh ./trap.sh 0  # EXIT
Process 502338 received signal 0 after rv=0
$ busybox sh ./trap.sh 1  # SIGHUP
Process 502366 received signal 1 after rv=0
Hangup
$ busybox sh ./trap.sh 2  # SIGINT
Process 502402 received signal 2 after rv=0

$ busybox sh ./trap.sh 3  # SIGQUIT
Process 502439 received signal 3 after rv=0
Process 502439 received signal 0 after rv=0
$ busybox sh ./trap.sh 15  # SIGTERM
Process 502269 received signal 15 after rv=0
Terminated

There EXIT is almost never called, except by busybox on SIGQUIT.

That is why portable shell scripts setup trap not only for EXIT, but also for other SIGnals.

But if you do that, please make sure to do it right:

1. Reset the trap handler to its default.
2. Afterwards kill the process by re-sending the received signal to the process again.

Why proper trap handling is important

Viacheslav Biriukov wrote a great blog post about Process groups, jobs and sessions explaining why proper exiting is important. A program might setup a signal handler for SIGINT to prevent the program from just terminating, which might loose important data. It might ask the user if terminating is okay or if the data should be saved first before quitting. A surrounding shell script must then decide, if this is an abnormal exit and should terminate itself afterwards, or should continue normally. The UNIX convention is to transfer that detail via exit codes and signal statuses. So be careful and do it right if your shell script starts using trap.

Conclusion

1. Use bash as it has consistent handling of trap EXIT.
2. If you want to or must use other shells: Do not use the same cleanup trap of EXIT and other signals.
3. If you trap signals, make sure to reset the handler and to re-raise the signal to properly propagate them.

#!/bin/bash
TMPDIR=$(mktemp -d)
trap 'rm -r $TMPDIR' EXIT
...

Let’s ask shellcheck:

No issues detected!

Actually there are multiple issues:

TMPDIR

Please do not assign to TMPDIR as that variable in an input parameter to mktemp itself: When you read man:mktemp your will find this for option -p:

if DIR is not specified, use $TMPDIR if set, else /tmp.

The variable is used for example by pam_tmpdir to setup per user temporary directories to improve security on multi-user systems. By using TMPDIR inside your script to store the path of your specific temporary directory, you risk chanhing the behavior of other called child-processes also using mktemp. Other equivalent implementations thereof) also use TEMP and TMP, so better do not use these as well.

So lets use tmp:

tmp=$(mktemp -d)
trap 'rm -r $tmp' EXIT

IFS

By default man:mktemp will only create safe file names, e.g. none containing blanks and characters of $IFS. Remember that $IFS is used by the shell to split every argument — which is not quoted — into multiple arguments. By default it is set to space, tab and newline. But you can redefine or extend it, after which hell breaks loose:

$ bash -c 'IFS="$IFS/."; . my-trap-script"
rm: cannot remove '': No such file or directory
rm: cannot remove 'tmp': No such file or directory
rm: cannot remove 'user': No such file or directory
rm: cannot remove '1000': No such file or directory
rm: cannot remove 'tmp': No such file or directory
rm: cannot remove 'XyJlR6AHpn': No such file or directory

Luckily $IFS is re-set for each shell to its default value, but do keep that in mind when you fiddle with $IFS. My advise is to do that only in functions and to use local IFS there to have the change confined to only inside the function.

quoting

To prevent $IFS-splitting you have to quote arguments. So let’s try with this:

tmp=$(mktemp -d)
trap 'rm -r "$tmp"' EXIT

You may wonder, why I didn’t quote tmp=$(…) as the spitting occurs after command substitution? For that you have to read man:bash very carefully. In section parameters you have this:

All values undergo tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, and quote removal.

Compare that to section expansion:

There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.

The important difference here is, that parameter assignment expects a single argument and this word splitting does not occurs there. So no quoting is needed for parameter assignments, but you can do it for consistency — it does not hurt.

late vs. early evaluation

While shellcheck is happy, there is a lingering problem: The trap is executed only later on when the shell exits. $tmp might get changed (by accident) or be used for something else. In that case the rm will delete whatever file $tmp points too.

That is because the outer quotes are single quotes while the inner quotes are double quotes: single quotes prevent evaluation of the command when the trap statement is executed. Later on when the trap is executed, the command is evaluated a second time. That is when the double quotes prevent $tmp from being split on $IFS.

So lets look at the following variant:

tmp=$(mktemp -d)
trap "rm -r $tmp" EXIT
tmp+="/subdir"

Double quotes are now use when the trap is setup: $tmp gets inserted here as it is currently defined. If $tmp is changed later on, we still delete file temporary file we just created.

But shellcheck is unhappy now:

trap "rm -r $tmp" EXIT
            ^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled.

Personally I think SC2064 is a bad advise here as we want to evaluate “$tmp” now and not later. I want to delete the file $tmp is pointing to right now, not where it might point to in the future. I’m not alone with that opinion and issue 1945 calls SC2064 questionable.

But there is a bigger problem again: But what will happen, when the trap fires?

late quoting

Remember that $tmp might contain $IFS characters! For example I can set TMPDIR=/tmp/I like blanks. The trap command will be rm -f /tmp/I like blanks. It will fail as there is no file /tmp/I, ./like and ./blanks — hopefully.

So how do we fix that? I give you two variants:

1. Nestes double quoting using backslash-escaping:

   tmp=$(mktemp -d)
   trap "rm -r \"$tmp\"" EXIT
   

2. Single quotes inside double quotes:

   tmp=$(mktemp -d)
   trap "rm -r '$tmp'" EXIT
   

Which one is correct?

The answer is very disappointing: None!

Variant 1 will fail for TMPDIR=/tmp/\" and variante will fail for TMPDIR=/tmp/\'. $tmp will then be a path containing a double quote in variant 1 and a single quote in variant 2. Because of the early evaluation $tmp is inserted as-is during the first evaluation when trap is setup. On the 2nd evaluation when the trap is executed, you will have an odd number of quotes!

correct quoting

So we need a mechanism to quote $tmp correctly, so it survives two rounds of evaluation.

Luckily bash has such a feature:

tmp=$(mktemp -d)
# shellcheck disable=SC2064
trap "rm -r ${tmp@Q}" EXIT

@Q is a operator, which is documented like this in man:bash:

The expansion is a string that is the value of parameter quoted in a format that can be reused as input.

That is exactly what we want:

• the outer quotes prevent $tmp from being split when the trap is setup.
• the @Q adds the necessary escaping to also prevent $tmp from being split when the trap executes.

closing words

Be warned that the operator @Q is a bashism: This is not supported by ash, dash, or busybox sh: There you have to quote " and ' manually. I leave that to you.

I will simply accept bash and use @Q as that is much more readable and — most importantly — correct.