For the last 4 stages of the Evaluating Expressions extension of the Build your own interpreter challenge, the instructions say that the expected output should be:
$ ./your_program.sh evaluate test.lox
Operand must be a number.
[line 1]
However, the tester says that it expects line 1 to be "", in other words empty.
By removing all output sent to stdout, the tester accepts the submission.
I believe that either the instructions should indicate to not print anything, or the tester should test the output string in order for the instructions and the tester to be consistent.
As far I understand, the tester for all the error stages checks the exit code and not the actual content of the printed error message. I personally feel that this approach is the way to go, because in case of deeply nested expressions, error reporting (specially with the correct character and line information) becomes something too non trivial, and maybe too challenging for beginners!
That makes sense. I’m not saying that the output should be tested, only that the instructions for those last 4 stages saying that the string output will be tested. I’m just pointing out that there is a discrepancy between the instructions and what the tester is looking for. I believe that if it’s too difficulty to implement that portion, the instructions for this extension should me similar to the Parser extension that explicitly states the tester will only look for the exit code not the output from stdout.
Oh wait, I misread the issue here - you’re right, our instructions don’t mention that stderr will NOT be tested.
@andy1li can we open a PR for this please? Every “errors” stage must include in the notes that we don’t test stderr (except for the initial “scanning” error stages, those we can’t change without breaking backward compat)