Discussion: Skipping repeated test runs without any changes to code (codecrafters CLI)

General
1

I’m not sure what everyone’s workflow is while using the codecrafters CLI, but my typical workflow includes running the codecrafters test command first, and then codecrafters submit (once all the tests pass).

However, the final tests after submit is redundant - there is no change to the codebase at all after since the previous successful run of test - yet the remote server still re-executes all the tests.

From what I’ve seen in the CLI codebase, the test command simply pushes the local directory to a remote temporary directory, which is removed after testing is complete. What we can maybe do as an additional step is,

  1. Cache the latest SHA256 hash (checksum) of the entire project directory (or alternatively store a git patch file).
  2. Compare the stored checksum (or patch file) for subsequent test runs.
  3. If the checksum has not changed, skip the tests since there is no difference in the codebase.

I can see cases where someone may want to rerun the same tests multiple times without making code changes (for eg, to accurately determine what went wrong). To account for this, we could add a flag to explicitly re-run tests regardless of checksum (something like codecrafters test --force?). Whichever is the non-default workflow could be assigned a flag accordingly.

1 Like
3

Hey @pranjalpokharel7, thanks for the suggestion!

The tests run after submit aren’t entirely redundant. They help catch flakiness or issues that may not show up consistently at first.

Definitely something worth exploring further. Appreciate you taking the time to write this up!