"Concise" code examples

When displaying code examples, we now highlight 1-2 “concise” examples at the top:

These might not be the best approaches, but they let you get a sense of what a quick implementation could look like.

We’ve also added upvote/downvote buttons so you can share feedback on these (we’ll use this information to push them up/down the list):

Screenshot 2024-05-20 at 11.27.10

1 Like

For those curious, here’s how this works internally:

Overview

We pick the last 1000 code examples, sort them by number of lines changed and then run some filters to weed out “irrelevant” code examples. We then pick two from the top of the list (i.e. least lines changed).

Filtering out irrelevant examples was the bulk of the work here. More on this below.

Irrelevant code examples

Irrelevant code examples are those where the changes aren’t related to the stage itself. These code examples do pass tests, but the bulk of the code was likely written in earlier stages.

Some examples:

Identifying irrelevant code examples

We’ve got ~1600 stage-language combinations, so filtering these out by hand would be a LOT of work. We use LLMs for this instead.

We send the following information in a prompt:

  • The stage instructions
  • The user’s diff
  • An explanation of what “relevant” means to us

And we get back an evaluation that looks like this:

There will be some false positives with this approach of course, we’ll rectify these over time once the upvotes/downvotes start pouring in.

False negatives are okay, if a “relevant” solution is marked as “irrelevant” it’ll just not show up at the top of the list.

2 Likes