Upcoming change: Build system overhaul

Announcements
1

Hey everyone!

Just wanted to pre-announce a change to the platform that’ll ship in the coming weeks, similar to how Upcoming change: Stages overhaul is rolling out.

Context

When you push code to CodeCrafters we run three steps:

  • Build
  • Compile
  • Run

The “Compile” and “Run” steps run on every code push, so they’re designed to be as fast as possible. The “Build” step only runs if dependencies change, or if your repository has been inactive for a while. More on this here.

The way we designed these steps initially is that “Build” & “Compile” are completely hidden from a user, and “Run” uses scripts like spawn_redis_server.sh that are present in your starter repository.

As we’ve introduced more challenges and languages, we’ve found some problems with this setup:

  • The “Build” step has some assumptions around how your code is structured, so it can fail when your package structure diverges from starter code.
  • The “Compile” step has such assumptions too, and cannot be configured by the user.
  • The script to run your program locally can’t be changed without affecting how remote builds work.
    • Maybe you’ve got python pointing to Python 2 locally and want to use python3 instead. Changing to python3 in your_redis_server.sh will likely cause a remote build failure.

We’re going to fix all of these issues by making a set of backward-compatible changes.

Changes

Roughly, the changes are:

Change 1: Repository-aware caching

Previously, the Build step would run with only few files available (like Cargo.toml and Cargo.lock) so that we could cache builds across multiple users. The downside to this is that other files couldn’t be referenced. We’ll change this so that the full repository context is available during builds.

This fixes issues like [Rust] Trying to make a multi-crates project can't find Cargo.toml but works locally.

This change is already live on select challenges & languages.

Change 2: Add .codecrafters/{compile,run}.sh

  • .codecrafters/compile.sh will contain the code for the “Compile” step, which currently runs “behind-the-scenes” and can cause unexpected errors.
  • .codecrafters/run.sh will be run instead of your_program.sh when tests are run into the cloud. This allows customizing your_program.sh for running locally without affecting remote tests. One can still control how tests are run remotely by editing .codecrafters/run.sh.

Both these files will be available inside your repository and we’ll include extensive comments explaining how they work. This will allow customizing these steps if you need to.

Change 3: Replace your_[whatever].sh with your_program.sh

  • The script to run your program will now be standardized across all challenges, it’ll be named your_program.sh everywhere.
  • Changes to your_program.sh will now not affect remote builds, so the file can be changed to suit your local development needs

These changes will allow more customization over how remote builds/tests are run, and will allow us to support a wider variety of programming languages over time. I’ll post updates once these start rolling out!

3 Likes
2

Quick update on this:

The change has been live on the Shell challenge for a while and we haven’t seen any complaints so far. We’re now expanding support - Interpreter and HTTP server are done, others to follow!

2 Likes