Hey team! I think I’ve found a likely root cause for the Swift full rebuild issue, assuming CodeCrafters’ runtime path has similar timestamp behaviour to what I can reproduce locally.
In short: SwiftPM/llbuild stores nanosecond mtimes in build.db. In my local Docker/OrbStack repro, files created during docker build keep their mtime with a precision of whole seconds after docker run, losing the nanosecond part. That leaves build.db storing, for example, 19:24:22.104643862, while the runtime filesystem says 19:24:22.000000000. llbuild treats that as a stale build state, and recompiles dependencies.
I confirmed this by patching the nanosecond fields in build.db to match the filesystem in the container. Without the patch, SwiftPM rebuilt dependencies. With the patch, calling swift build in a container became a no-op:
[0/2] Write swift-version-24593BA9C3E375BF.txt
Build complete! (0.09s)
Doing the reverse—truncating source file mtimes before swift build—isn’t enough, because SwiftPM creates and updates additional build artefacts during the build, and llbuild records those mtimes too.
Suggested fixes
I don’t think the Python build.db patch I wrote for this proof-of-root-cause is a real fix. It depends on llbuild’s private binary encoding and should only be treated as a proof of root cause.
After swift build runs during image creation, both the build directory and its build.db need to show up in the runtime environment (the container) with the same metadata, especially around file modification time. Specifically, the files in the build directory must keep their nanosecond mtimes; otherwise build.db remembers one timestamp while the runtime filesystem reports another.
Realistically, that could mean using a mounted/cache directory/filesystem for the build directory, or ensuring the OCI layer pipeline preserves sub-second mtimes. I also asked ChatGPT, which suggests reusing a VM/rootfs image that was snapshotted after the priming build (rather than being packed into an OCI layer/tar archive and then unpacked again)—but I understand that suggestion less and will leave it to the team!
Ultimately, the key requirement isn’t a specific technology, but that the files in the build directory and build.db see the same mtimes before and after the image build/container runtime boundary.
If that’s not possible, would could think of an upstream SwiftPM/llbuild change. For example, we could suggest a mode similar in spirit to file-system: device-agnostic, but for timestamp granularity. Given the setup for CodeCrafters, we wouldn’t be worried about the fact that one-second mtimes can miss very fast file changes. Another alternative would be content-hash-based mode, but that’s too expensive for this case in my opinion.
With my limited knowledge of containerisation, I’m not sure if there’s a way to use different backends that would prevent the loss of the nanosecond precision.
Assumptions
My findings are based on two assumptions. If any of them are wrong, please do let me know.
Our goal is to achieve incremental builds for each stage. My understanding is that when we start a container from one of the pre-built images, calling .codecrafters/compile.sh should do an incremental build (ie only build what’s changed for the stage), but instead the current situation is that swift recompiles everything, taking significantly longer.
My repro assumes a backend that loses nanosecond precision. My repro demonstrates that my local Docker/OrbStack backend loses nanosecond precision. I don’t know whether CodeCrafters’ production backend does the same. If it does, this explains the Swift rebuild behaviour.
The long read: journey through my findings
Notes
For this whole investigation:
- I used Swift 6.3:noble instead of CodeCrafters’ current 6.0.1
- I modified the Dockerfile to include sqlite3 + python3 to aid with troubleshooting
Investigating Paul’s findings
As mentioned in this thread, Swift has now fixed the issue of inodes, with the setting file-system: device-agnostic. This is correctly set in release.yaml:
❯ head -3 /tmp/codecrafters-build-redis-swift/release.yaml
client:
name: basic
file-system: device-agnostic
So inode mismatch is unlikely to be the remaining issue, at least given the llbuild manifest generated here. I decided to assume this was working, and moved to Paul’s other findings here
release.yaml has changed
I found the same. The only difference was the addition of -color-diagnostics to many commands. Using TERM=dumb before swift build removed those changes, but didn’t address the issue.
TERM=dumb swift build -c release --build-path /tmp/codecrafters-build-redis-swift
build.db has changed
Looking into this, I found that build.db changed even after running swift build a second time, even though the second build was incremental. For the incremental build, I found a few entries are added in rule_results (N/app/Sources, N/app/Package.resolved, N/app/Package.swift, I/app/Sources).
For the first build (the one triggering a full recompile), most if not all entries in rule_results had their value field changed.
At that point, that’s all I had for build.db. I didn’t know what value represented, and couldn’t easily find it. Because I didn’t know whether it’d be relevant, I decided to come back to it later.
Spoiler alert: it was very relevant.
Investigating modification times
I wanted to first rule in or out the modification times of source files. After trial and error, I realised that the containers created from the image lost their nanosecond precision:
❯ cat > /tmp/mtime-test.dockerfile << EOF
FROM alpine
RUN touch /tmp/file-under-test && stat -c 'build: %y %n' /tmp/file-under-test
CMD ["stat", "-c", "run: %y %n", "/tmp/file-under-test"]
EOF
❯ docker build --progress=plain --no-cache -f /tmp/mtime-test.dockerfile -t mtime-test /tmp/
[...]
#5 [2/2] RUN touch /tmp/file-under-test && stat -c 'build: %y %n' /tmp/file-under-test
#5 0.146 build: 2026-04-26 23:18:29.266053359 +0000 /tmp/file-under-test
#5 DONE 0.2s
❯ docker run --rm mtime-test
run: 2026-04-26 23:18:29.000000000 +0000 /tmp/file-under-test
In short:
build: 2026-04-26 23:18:29.266053359 +0000 /tmp/file-under-test
run: 2026-04-26 23:18:29.000000000 +0000 /tmp/file-under-test
I’m not an expert in container runtimes, image layers, or snapshotters, but my current understanding is that this loss could happen during OCI layer creation/export, import, unpacking, or snapshotter materialisation. Basically between committing the RUN filesystem changes into the image, and materialising those layers for runtime.
It’s also my understanding that OCI layers are commonly represented as tar-like archives. Classic tar headers only store mtimes at one-second precision (not nanosecond), so that could be the culprit (although pax extensions can store sub-second timestamps. Ultimately, I can’t really prove exactly which part of the Docker/OrbStack path drops the nanoseconds, and don’t know how to. My repro only proves that the end-to-end build → image → container run path does drop the nanoseconds.
Changing the container system was going to be out of my depth, so I stayed away from that, but I wanted to at least find a way to test my theory.
Truncating mtime before compiling
Given the issue above, I tried using touch to remove the nanoseconds from the filesystem before calling swift build. In .codecrafters/compile.sh, I decided to truncate the mtime of all files and folders under /app and /tmp/codecrafters-build-redis-swift/:
find /app /tmp/codecrafters-build-redis-swift/ | while IFS= read -r f; do
touch -d "@$(stat -c '%Y' "$f")" "$f"
done
The files were correctly truncated, but the full recompile still happened. I also hacked compile-sdk to make sure RUN .codecrafters/compile.sh is called after the code added by the backend, just in case COPY . /app modified the mtimes, but that didn’t help either.
Time to look in build.db
I wanted to see what would be different—if anything—before and after a call to swift build.
❯ cp /tmp/codecrafters-build-redis-swift/build.db /tmp/build-before.db
❯ .codecrafters/compile.sh
# ...full rebuild...
❯ cp /tmp/codecrafters-build-redis-swift/build.db /tmp/build-after.db
Looking one file at a time (e.g. NIOCore/ByteBuffer-core.swift):
❯ sqlite3 /tmp/build-before.db "
SELECT hex(r.value)
FROM rule_results r JOIN key_names k ON k.id = r.key_id
WHERE k.key = 'N/tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift';"
020100000000000000000000000000000000000000248100000000000063DC000000000000065AEE6900000000DACC9014000000000000000000000000000000000000000000000000000000000000000000000000
❯ sqlite3 /tmp/build-after.db "
SELECT hex(r.value)
FROM rule_results r JOIN key_names k ON k.id = r.key_id
WHERE k.key = 'N/tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift';"
020100000000000000000000000000000000000000248100000000000063DC000000000000065AEE690000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Something interesting happened here. DACC9014 disappeared, but everything was the same. Interpreted in little endian, DACC9014 is 0x1490CCDA, which is 345033946. Which could be a reasonable value for nanoseconds. That’s not proof though, just a hint.
A good way to verify is to stat from the image build time, and to look at whether the bytes at offset 45 represent the nanoseconds. I tried this out by adding stat ByteBuffer-core.swift in compile.sh:
# ...
TERM=dumb swift build -c release --build-path /tmp/codecrafters-build-redis-swift
stat /tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift
Which outputs:
#12 29.95 File: /tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift
...
#12 29.95 Modify: 2026-04-27 00:26:36.211178382 +0000
Looking at the value in build.db for the file:
❯ sqlite3 /tmp/codecrafters-build-redis-swift/build.db "
SELECT hex(r.value)
FROM rule_results r JOIN key_names k ON k.id = r.key_id
WHERE k.key = 'N/tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift';"
020100000000000000000000000000000000000000248100000000000063DC0000000000003CADEE69000000008E53960C000000000000000000000000000000000000000000000000000000000000000000000000
The value at offset 45 is 8E53960C. Interpreted in little endian, 0x0C96538E is 211178382. Exactly the same as the number of nanoseconds from the stat call!
The value right before, at offset 37, is 3CADEE69. Interpreted in little endian, 0x69EEAD3C is 1777249596, which looks like a UNIX timestamp! And it is indeed the timestamp for 2026-04-27 00:26:36.
At this point, it looked very likely that llbuild stores file mtimes in build.db, with seconds at offset 37 and nanoseconds at offset 45 for this FileInfo-like value.
Root cause almost identified
I ran this test on a few files, and found the same pattern. This was a strong indicator that llbuild stores file mtimes in build.db. Given the earlier issues with nanoseconds being truncated off file mtimes between the time the image is built, and the time the container runs, it explains why a full rebuild happens.
But it doesn’t explain why truncating the nanoseconds off of *.swift files wasn’t enough.
A quick look at build.db explained why. It’s not just *.swift files that are tracked!
❯ sqlite3 /tmp/codecrafters-build-redis-swift/build.db 'SELECT key FROM key_names WHERE key LIKE "%.swift.o";'
N/tmp/[...]/_NIOBase64.build/Base64.swift.o
N/tmp/[...]/InternalCollectionsUtilities.build/Debugging.swift.o
N/tmp/[...]/InternalCollectionsUtilities.build/Descriptions.swift.o
❯ sqlite3 /tmp/codecrafters-build-redis-swift/build.db 'SELECT key FROM key_names WHERE key LIKE "C/%";'
C/tmp/codecrafters-build-redis-swift/aarch64-unknown-linux-gnu/release/CNIOWASI.build/CNIOWASI.c.o
C/tmp/codecrafters-build-redis-swift/aarch64-unknown-linux-gnu/release/CNIODarwin.build/shim.c.o
C/tmp/codecrafters-build-redis-swift/aarch64-unknown-linux-gnu/release/CNIOWindows.build/shim.c.o
...
❯ sqlite3 /tmp/codecrafters-build-redis-swift/build.db 'SELECT key FROM key_names WHERE key LIKE "CC%";'
CC._NIOBase64-aarch64-unknown-linux-gnu-release.module
CC.InternalCollectionsUtilities-aarch64-unknown-linux-gnu-release.module
CC._NIODataStructures-aarch64-unknown-linux-gnu-release.module
...
# After building a second time, another type appears
❯ sqlite3 /tmp/codecrafters-build-redis-swift/build.db 'SELECT key FROM key_names WHERE key LIKE "I/%" LIMIT 5;'
I/app/Sources
Many of those rows had rule_results.value blobs that changed after compilation. That made sense: llbuild stores not just source-file state, but also build artefacts and command results.
I considered pre-fetching and pre-building the SwiftPM dependencies, then truncating the file mtimes before swift build. But it wouldn’t work, because SwiftPM creates or updates files during the build, and records their nanosecond mtimes in build.db. Truncating the files’ mtime after swift build also doesn’t solve the problem, because then the filesystem no longer matches what llbuild just recorded. It’s basically a chicken-and-egg problem.
Root cause hypothesis identified
At this point, the hypothesis was:
- During the image build, SwiftPM/llbuild observes files with nanosecond mtimes.
- llbuild stores those mtimes in
build.db. - Somewhere between image build and container runtime, the filesystem is materialised with mtimes truncated to whole seconds.
- At container runtime, llbuild compares
build.dbagainst the filesystem. - The seconds match, but the nanoseconds do not.
- llbuild invalidates the cached build state, and recompiles dependencies.
The cache is therefore not “missing”. It’s just more precise than the filesystem metadata that survived the image → runtime flow.
Confirming the hypothesis
Because touching files after swift build makes the filesystem disagree with what llbuild just recorded, I tried a different test, just the opposite of touching files: patch build.db itself, so that its recorded mtimes match the runtime filesystem before invoking swift build.
Here is a small proof-of-concept script that patches those mtime fields in build.db:
The script handles CC.* entries differently, because those values can contain multiple embedded file info records, not just one path’s metadata like N/* does.
I then compared a normal build with a build after patching build.db.
# Normal build
❯ docker run --rm -it redis-swift bash
root@22c9f75b1d15:/app# .codecrafters/compile.sh
Build directory already exists. Proceeding with the build...
[0/1] Planning build
Building for production...
[0/18] Write sources
[10/23] Compiling CNIOWindows WSAStartup.c
...
[32/34] Compiling build_your_own_redis main.swift
[32/34] Write Objects.LinkFileList
[33/34] Linking build-your-own-redis
Build complete! (18.58s)
root@22c9f75b1d15:/app# ^D
exit
# Build after patching `build.db`
❯ docker run --rm -it redis-swift bash
root@3d7107140e84:/app# cat > /tmp/patch_build_db_mtimes.py <<EOF
...
EOF
root@bfea6de4998e:/app# python3 /tmp/patch_build_db_mtimes.py /tmp/codecrafters-build-redis-swift/build.db
patched_rows=451 patched_fields=636
root@bfea6de4998e:/app# .codecrafters/compile.sh
Build directory already exists. Proceeding with the build...
[0/1] Planning build
Building for production...
[0/2] Write swift-version-24593BA9C3E375BF.txt
Build complete! (0.88s)
root@bfea6de4998e:/app#
Hypothesis: confirmed 
!
The discrepancy in mtime between build.db and the runtime filesystem causes the first build in the container to become a full rebuild instead of the expected incremental build.
Late addition: Verifying the content of the database blob
I’m editing this message to add further confirmation of our findings. Note: because I’m a new user, I can’t add as many links as I would like to refer to the actual source code.
Let’s work with ByteBuffer-core.swift’s blob:
020100000000000000000000000000000000000000248100000000000063DC0000000000003CADEE69000000008E53960C000000000000000000000000000000000000000000000000000000000000000000000000
We found seconds at offset 37 and nanoseconds at offset 45.
I stumbled upon swift-llbuild’s FileInfo.h where we can see:
struct FileTimestamp {
uint64_t seconds;
uint64_t nanoseconds;
// ...
}
struct FileChecksum {
uint8_t bytes[32] = {0};
// ...
}
struct FileInfo {
/// The device number.
uint64_t device;
/// The inode number.
uint64_t inode;
/// The mode flags of the file.
uint64_t mode;
/// The size of the file.
uint64_t size;
/// The modification time of the file.
FileTimestamp modTime;
/// The checksum of the file.
FileChecksum checksum = {};
// ...
}
With this structure, the file’s seconds are actually stored as a 64-bit unsigned integer at offset 32. Similarly, the nanoseconds are stored as a 64-bit unsigned integer at offset 40.
We found the seconds at offset 37 and the nanoseconds at offset 45, which means the blob has 5 bytes unaccounted for.
But, assuming for now that this is the structure that gets encoded in the blob, this is what it would represent:
00 | 02010000 00 # TBD
05 | 00000000 00000000 # FileInfo::device
13 | 00000000 00000000 # FileInfo::inode
21 | 24810000 00000000 # FileInfo::mode
29 | 63DC0000 00000000 # FileInfo::size
37 | 3CADEE69 00000000 # FileTimestamp::seconds
45 | 8E53960C 00000000 # FileTimestamp::nanoseconds
53 | 00000000 00000000 # FileChecksum bytes[0..7]
61 | 00000000 00000000 # FileChecksum bytes[8..15]
69 | 00000000 00000000 # FileChecksum bytes[16..23]
77 | 00000000 00000000 # FileChecksum bytes[24..31]
We can verify whether this is indeed the right structure that gets encoded by checking whether the mode and size values match.
# stat -c "%f" /tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift
8124
# stat -c "%a" /tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift
444
# python3 -c 'print(oct(0x00008124))'
0o100444
That’s a match! 8124 is the same as 24810000 00000000 in the data.
Now let’s look at the file size.
# stat -c "%s" /tmp/codecrafters-build-redis-swift/checkouts/swift-nio/Sources/NIOCore/ByteBuffer-core.swift
56419
# python3 -c "print(hex(56419))"
0xdc63
Again that’s a match, so we’re on the right path!
Looking at files referencing FileInfo, we find BuildValue.h
It contains a BuildValue::toData() method which encodes the type with:
- the value’s
Kind - a signature if the value’s kind has one
- One or more "output info"s if the value’s kind has them, encoding:
- the number of output infos as a 32-bit value
- the output info values
- string values if the value has any
“Output info” is quite vague. Looking at getNthOutputInfo, it shows that it gets the value from valueData, which is a union of FileInfo and FileInfo*. So “output infos” are FileInfo structures. Looks like we could be on the right path, and that BuildValue could indeed be what gets encoded in the blob!
In summary: BuildValue encodes its kind, a potential signature, possibly a 32-bit value counting the number of FileInfo entries followed by one or more FileInfo entries, and string values.
Crucially, while Kind is defined as a uint32_t, it appears to be encoded as a uint8_t (see basic::BinaryCodingTraits<buildsystem::BuildValue::Kind>::encode()).
Assuming only 8 bits (1 byte) for Kind, the first byte’s value is 02, referring toExistingInput (“A value produced by an existing input file”). This sounds promising. It also expects no signature (kindHasSignature returns false), and is expected to contain one or more output info (kindHasOutputInfo returns true). Therefore, if our assumptions are correct, we expect to see:
- One byte for the kind
- 4 bytes for the 32-bit number of
FileInfobeing encoded - At least 80 bytes for one
FileInfostructure.
And this expectation aligns perfectly with the blob data for ByteBuffer-core.swift
00 | 02 # BuildValue::Kind = ExistingInput
01 | 01000000 # numOutputInfos = 1
05 | 00000000 00000000 # FileInfo::device = 0
13 | 00000000 00000000 # FileInfo::inode = 0
21 | 24810000 00000000 # FileInfo::mode = 0444
29 | 63DC0000 00000000 # FileInfo::size = 56419 / 0xDC63 bytes
37 | 3CADEE69 00000000 # FileTimestamp::seconds = 1777249596 seconds since epoch
45 | 8E53960C 00000000 # FileTimestamp::nanoseconds = 211178382
53 | 00000000 00000000 # FileChecksum bytes[0..7]
61 | 00000000 00000000 # FileChecksum bytes[8..15]
69 | 00000000 00000000 # FileChecksum bytes[16..23]
77 | 00000000 00000000 # FileChecksum bytes[24..31]
Added bonus
Earlier in my writing, I mentioned that the build system sets file-system: device-agnostic, which tells llbuild not to track inode changes. I made the assumption that there was no bug in llbuild and that this was working correctly, meaning that the swift full rebuild issue was therefore not caused by inode changes.
Now that we know what’s in the blob, we know for sure that file-system: device-agnostic works as expected, and that llbuild doesn’t track the device or the inode. Therefore, addressing the mtime issue is likely the main way forward.
