Stuck on stage #WS9 when trying to read the superheroes database

[joelburger]

I’m stuck on Stage #WS9 - Retrieve data using a full-table scan.

When reading from the sample superheroes database, I’m getting unexpected varint values back when trying to parse the recordSize from the cell and the schemaBody size from the sqllite schema table.

Here are my logs:

readCell { pageType: 13, cellPointer: 3726, recordSize: 13057 }
parseTableSchema {
  headerSize: 7,
  schemaTypeSize: 5,
  schemaNameSize: 11,
  tableNameSize: 11,
  rootPageSize: 1,
  schemaBodySize: 69
}
{
  tableName: 'superheroes',
  rootPage: 2,
  schemaBody: 'CREATE TABLE "superheroes" (id integer primary key autoincrement, nam'
}

And here’s a snippet of my code:

const readVarInt = (buffer, offset) => {
  let value = 0;
  let bytesRead = 0;
  for (let i = 0; i < 9; i += 1) {
    value |= (buffer[offset + i] & 0x7f) << (7 * i);
    bytesRead += 1;
    if (!(buffer[offset + i] & 0x80)) {
      break;
    }
  }
  return { value, bytesRead };
};

I’m not sure what I’m doing wrong, but I was able to pass the previous tests and I’m also able to successfully parse the included samples.db database from the repository.

[andy1li]

Your readVarInt function looks correct to me.

However, when running the code, I couldn’t replicate the logs in your post because a different error occurred:

Fatal error: Error: Failed to parse columns from "CREATE TABLE "superheroes" (id integer primary key autoincrement, nam".

Additionally, the test in sqlparser.test.js failed:

@joelburger Could you push an up-to-date version of the code (or share a link to GitHub) so I can take a closer look?

[joelburger]

I’ve pushed an update to fix the failing test.

I’ve also added a script command to allow the debug logs to be displayed.

e.g.
npm run debug superheroes.db .tables

[andy1li]

@joelburger Unfortunately, I can’t see your latest push in our system. The most recent one is from 2024-10-12. Could you try again or provide a link to GitHub instead?

[joelburger]

Sorry can you check again please

[andy1li]

Turns out the readVarInt function isn’t correct after all. The bug is in this line:

    value |= (buffer[offset + i] & 0x7f) << (7 * i);

The problem is that (take smaller values of i for example):

1. buffer[offset + i] needs to be shifted further to the left, but
2. << (7 * i) actually shifts less than needed.

One way to fix it is to shift first, and then or:

    value <<= 7;
    value |= buffer[offset + i] & 0x7f;

---

Sanity check after the fix:

0x8166 is 0b10000001 01100110 as raw binary, which as a varint means 0b11100110 == 230 in decimal.

---

As for schemaBody, I’m not quite sure I understand this line:

const schemaBodySize = headerSize === 7 ? convertVarInt(buffer[5] + buffer[6]) : convertVarInt(buffer[5]);

@joelburger Could you elaborate a bit on what the code is doing here?

BTW, the correct schemaBodySize is 195 for the first table of the local superheroes.db:

[joelburger]

After applying this fix, the record size and schema body size are now being calculated correctly.
Thanks so much for the quick response.