HTTP Server #CR8: Gzip compression

I’m stuck on gzip compression stage 3.

I’ve also tried sending the gzip compressed body as hex, base64. I’ve provided my current version of the code snippet below.

Here are my logs:

inner@afterlife:/mnt/d/projects/codecrafters-http-server-csharp$ codecrafters test
Initiating test run...

⚡ This is a turbo test run. https://codecrafters.io/turbo

Running tests. Logs should appear shortly...

remote: [compile] MSBuild version 17.3.2+561848881 for .NET        
remote: [compile]   Determining projects to restore...        
remote: [compile]   Restored /app/codecrafters-http-server.csproj (in 90 ms).        
remote: [compile]   codecrafters-http-server -> /app/bin/Release/net6.0/codecrafters-http-server.dll        
remote: [compile]
remote: [compile] Build succeeded.        
remote: [compile]     0 Warning(s)        
remote: [compile]     0 Error(s)        
remote: [compile]
remote: [compile] Time Elapsed 00:00:03.17        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [http-compression-3] Running tests for HTTP Compression > Stage #3: Gzip compression
remote: [http-compression-3] Running program
remote: [http-compression-3] $ ./your_server.sh
remote: [your_program] Starting custom HTTP Server.        
remote: [your_program] Custom HTTP Server started.        
remote: [http-compression-3] Connected to localhost port 4221
remote: [http-compression-3] $ curl -v -X GET http://localhost:4221/echo/blueberry -H "Accept-Encoding: gzip"
remote: [http-compression-3] > GET /echo/blueberry HTTP/1.1
remote: [http-compression-3] > Host: localhost:4221
remote: [http-compression-3] > Accept-Encoding: gzip
remote: [http-compression-3] > 
remote: [http-compression-3] Sent bytes: "GET /echo/blueberry HTTP/1.1\r\nHost: localhost:4221\r\nAccept-Encoding: gzip\r\n\r\n"
remote: [your_program] TCP Connection 127.0.0.1:38108 established!        
remote: [your_program] Hex string: 1F8B08000000000000034BCA294D4D4A2D2AAA0400D3241E7909000000        
remote: [http-compression-3] Received bytes: "HTTP/1.1 200 OK\r\nContent-Encoding: gzip\r\nContent-Type: text/plain\r\nContent-Length: 29\r\n\r\n\x1f?\b\x00\x00\x00\x00\x00\x00\x03K?)MMJ-*?\x04\x00?$\x1ey\t\x00\x00\x00"
remote: [http-compression-3] < HTTP/1.1 200 OK
remote: [http-compression-3] < Content-Encoding: gzip
remote: [http-compression-3] < Content-Type: text/plain
remote: [http-compression-3] < Content-Length: 29
remote: [http-compression-3] < 
remote: [http-compression-3] < <Binary Content>
remote: [http-compression-3] < 
remote: [http-compression-3] Received response with 200 status code
remote: [http-compression-3] ✓ Content-Encoding header is present
remote: [http-compression-3] ✓ Content-Length header is present
remote: [http-compression-3] Failed to decode gzip: Failed to decompress data: gzip: invalid header
remote: [http-compression-3] Test failed
remote: [http-compression-3] Terminating program
remote: [http-compression-3] Program terminated successfully
remote: 
remote: View our article on debugging test failures: https://codecrafters.io/debug        

And here’s a snippet of my code:

            var echoMessage = serverRequest.Path.Replace(EchoEndpointPath, string.Empty);

            var acceptEncoding = serverRequest.Headers.Where(x =>
                    string.Equals(x.Key, "Accept-Encoding", StringComparison.InvariantCultureIgnoreCase))
                .Select(x => x.Value)
                .FirstOrDefault()?
                .Split(',')
                .Select(x => x.Trim());

            var containsGzipEncoding = acceptEncoding?
                .Contains("gzip", StringComparer.InvariantCultureIgnoreCase);

            var includeGzip = containsGzipEncoding.HasValue && containsGzipEncoding.Value;

            sb.Append("HTTP/1.1 200 OK\r\n");

            if (includeGzip)
            {
                byte[] compressedData;
                using (var memoryStream = new MemoryStream())
                {
                    using (var gzipStream = new GZipStream(memoryStream, CompressionMode.Compress))
                    {
                        var stringBytes = Encoding.UTF8.GetBytes(echoMessage);
                        gzipStream.Write(stringBytes, 0, stringBytes.Length);
                        gzipStream.Flush();
                    }

                    compressedData = memoryStream.ToArray();

                    Console.WriteLine($"Hex string: {BitConverter.ToString(compressedData).Replace("-", string.Empty)}");
                }
                
                sb.Append("Content-Type: text/plain\r\n");
                sb.Append("Content-Encoding: gzip\r\n");
                sb.Append($"Content-Length: {compressedData.Length}\r\n\r\n");
                sb.Append(Encoding.UTF8.GetString(compressedData));
            }
            else
            {
                sb.Append("Content-Type: text/plain\r\n");
                sb.Append($"Content-Length: {echoMessage.Length}\r\n\r\n");
                sb.Append(echoMessage);
            }

I’m getting the same error in the test, I’ve also tried hex, base64 or binary but they all result in the same error.

[http-compression-3] $ curl -v -X GET http://localhost:4221/echo/banana -H "Accept-Encoding: gzip"
[http-compression-3] > GET /echo/banana HTTP/1.1
[http-compression-3] > Host: localhost:4221
[http-compression-3] > Accept-Encoding: gzip
[http-compression-3] >
[http-compression-3] Sent bytes: "GET /echo/banana HTTP/1.1\r\nHost: localhost:4221\r\nAccept-Encoding: gzip\r\n\r\n"
[your_program] Parsed & Split Data;  [
[your_program]   'GET /echo/banana HTTP/1.1',
[your_program]   'Host: localhost:4221',
[your_program]   'Accept-Encoding: gzip',
[your_program]   '',
[your_program]   ''
[your_program] ]
[your_program] 11/05/24 15:19:06 Extracted path banana
[your_program] Gzipped data: <Buffer 1f 8b 08 00 00 00 00 00 00 03 4b 4a cc 03 42 00 cf 67 8b 03 06 00 00 00>
[your_program] Gzipped data as a Base64 string: H4sIAAAAAAAAA0tKzANCAM9niwMGAAAA
[your_program] 11/05/24 15:19:06 HTTP/1.1 200 OK
[your_program] Content-Encoding: gzip
[your_program] Content-Type: text/plain
[your_program] Content-Length: 50
[your_program]
[your_program] 1f8b08000000000000034b4acc034200cf678b0306000000
[your_program]
[http-compression-3] Received bytes: "HTTP/1.1 200 OK\r\nContent-Encoding: gzip\r\nContent-Type: text/plain\r\nContent-Length: 50\r\n\r\n1f8b08000000000000034b4acc034200cf678b0306000000\r\n"
[http-compression-3] < HTTP/1.1 200 OK
[http-compression-3] < Content-Encoding: gzip
[http-compression-3] < Content-Type: text/plain
[http-compression-3] < Content-Length: 50
[http-compression-3] <
[http-compression-3] < <Binary Content>
[http-compression-3] <
[http-compression-3] Received response with 200 status code
[http-compression-3] ✓ Content-Encoding header is present
[http-compression-3] ✓ Content-Length header is present
[http-compression-3] Failed to decode gzip: Failed to decompress data: gzip: invalid header
[http-compression-3] Test failed
[http-compression-3] Terminating program
[http-compression-3] Program terminated successfully
function gzipString(inputString) {
    return new Promise((resolve, reject) => {
        // Convert the string to a Buffer since zlib works with binary data
        const buffer = Buffer.from(inputString, 'utf-8');
        zlib.gzip(buffer, (err, result) => {
            if (err) {
                reject(err);
            } else {
                resolve(result);
            }
        });
    });
}

    async _send200(data, content_type, encoding) {
        data = data ? data : '';
        if (encoding[0] === 'gzip') {
            try {
                const gzippedData = await gzipString(data)
                encoding = `Content-Encoding: ${encoding[0]}${CRLF}`
                const encoded = gzippedData.toString('hex') + CRLF
                console.log('Gzipped data:', gzippedData);
                console.log('Gzipped data as a Base64 string:', gzippedData.toString('base64'));
                return `${HTTPResponse[200]}${encoding}Content-Type: ${content_type}${CRLF}Content-Length: ${encoded.length}${CRLF}${CRLF}${encoded}`;
            } catch (e) {
                console.error(e);
                return `${HTTPResponse[500]}Content-Type: text/plain${CRLF}Content-Length: 21${CRLF}${CRLF}Internal Server Error${CRLF}`;
            }
        } else {
            return `${HTTPResponse[200]}Content-Type: ${content_type}${CRLF}Content-Length: ${data.length}${CRLF}${CRLF}${data}${CRLF}`;
        }
    }

Okay I managed to pass the test, my issue was that the gzip encoded data that I was passing wasn’t encoded properly in my response. The test is expecting gzip encoded data that it will open with gunzip and check that it returns the same string in the echo

thanks for the hint on gunzip. Tests passed

Hey! could you please post a snippet of your solution? I’ve already checked that my encoded response indeed returns the same input string after decompression - yet I am still getting the same error…

@noe-lc you’ll find code snippets from users who passed the stage in the “Code Examples” tab:

Thanks, I did check the examples - the logic looks is exactly same but just some implementation differences. Still do not know what in my code is wrong ,so I wanted to see how someone with the exact same issue worked around it :+1:

In my case, the Encoding.UTF8.GetString(compressedData) was impacting the result as this function call was converting the compatible gzip output to incompatible

2 Likes

For those stumbling upon this error while using TS and plain socket writes, you need to write the response in two parts:

  1. First write the startLine + headers: socket.write(startLineAndHeaders)
  2. Write the compressed data socket.write(compressedData)
  3. End the socket with socket.end()

This is done so that the client is prepared to receive gzip encoded data.

Thanks! This worked for me.
I was under the impression that I needed to send the hex representation of gzip zlib.gzipSync(string).toString('hex') and I would write the entire response with interpolated values. It seems I didn’t need the toString('hex') after all I just needed a 2nd write.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.

Note: I’ve updated the title of this post to include the stage ID (#CR8). You can learn about the stages rename here: Upcoming change: Stages overhaul.