When displaying code examples, we now highlight 1-2 “concise” examples at the top:
These might not be the best approaches, but they let you get a sense of what a quick implementation could look like.
We’ve also added upvote/downvote buttons so you can share feedback on these (we’ll use this information to push them up/down the list):
For those curious, here’s how this works internally:
We pick the last 1000 code examples, sort them by number of lines changed and then run some filters to weed out “irrelevant” code examples. We then pick two from the top of the list (i.e. least lines changed).
Filtering out irrelevant examples was the bulk of the work here. More on this below.
Irrelevant code examples are those where the changes aren’t related to the stage itself. These code examples do pass tests, but the bulk of the code was likely written in earlier stages.
Some examples:
We’ve got ~1600 stage-language combinations, so filtering these out by hand would be a LOT of work. We use LLMs for this instead.
We send the following information in a prompt:
And we get back an evaluation that looks like this:
There will be some false positives with this approach of course, we’ll rectify these over time once the upvotes/downvotes start pouring in.
False negatives are okay, if a “relevant” solution is marked as “irrelevant” it’ll just not show up at the top of the list.
