Inconsistent ResponseHeader required - v0 vs V1 - ApiVersions vs Fetch

Some tests require response header in v0 (ApiVersions) other in v1 (Fetch).
When I updated to v1 - e.g. added empty TAG BUFFER it fails with ApiVersions, but is required for Fetch

Why? Found nothing in Kafka protocol docs, that would explain it

Log:

your_program] 2024-10-12T20:04:14.810041Z DEBUG kafka_starter_rust: accepted new connection peer_addr="127.0.0.1:39744"
[your_program] 2024-10-12T20:04:14.810525Z DEBUG kafka_starter_rust: Got frame frame_length=48
[your_program] 2024-10-12T20:04:14.810766Z DEBUG kafka_starter_rust: RequestHeader request_header=RequestHeader { request_api_key: 1, request_api_version: 16, correlation_id: 933664922, client_id: NullableString("kafka-tester"), tagged_fields: 0 }
[tester::#DH6] Hexdump of sent "Fetch" request: 
[tester::#DH6] Idx  | Hex                                             | ASCII
[tester::#DH6] -----+-------------------------------------------------+-----------------
[tester::#DH6] 0000 | 00 00 00 30 00 01 00 10 37 a6 98 9a 00 0c 6b 61 | ...0....7.....ka
[tester::#DH6] 0010 | 66 6b 61 2d 74 65 73 74 65 72 00 00 00 01 f4 00 | fka-tester......
[tester::#DH6] 0020 | 00 00 01 03 20 00 00 00 00 00 00 00 00 00 00 00 | .... ...........
[tester::#DH6] 0030 | 01 01 01 00                                     | ....
[tester::#DH6] 
[tester::#DH6] Hexdump of received "Fetch" response: 
[tester::#DH6] Idx  | Hex                                             | ASCII
[tester::#DH6] -----+-------------------------------------------------+-----------------
[tester::#DH6] 0000 | 37 a6 98 9a 00 00 00 00 00 00 00 00 00 00 00 01 | 7...............
[tester::#DH6] 0010 | 00                                              | .
[tester::#DH6] 
[tester::#DH6] [Decoder] - .ResponseHeader
[tester::#DH6] [Decoder]   - .correlation_id (933664922)
[tester::#DH6] [Decoder]   - .TAG_BUFFER
[tester::#DH6] [Decoder] - .ResponseBody
[tester::#DH6] [Decoder]   - .throttle_time_ms (0)
[tester::#DH6] [Decoder]   - .error_code (0)
[tester::#DH6] [Decoder]   - .session_id (0)
[tester::#DH6] [Decoder]   - .num_responses (0)
[tester::#DH6] [Decoder]   - .TAG_BUFFER
[tester::#DH6] āœ“ Correlation ID: 933664922
[tester::#DH6] āœ“ Throttle Time: 0
[tester::#DH6] āœ“ Error Code: 0 (NO_ERROR)
[tester::#DH6]   āœ“ TopicResponses: []
[tester::#DH6] Test passed.
[tester::#DH6] Terminating program
[tester::#DH6] Program terminated successfully

[tester::#GS0] Running tests for Stage #GS0 (Consuming Messages - Include Fetch in APIVersions)
[tester::#GS0] $ ./your_program.sh /tmp/server.properties
[tester::#GS0] [Serializer] Writing log files to: /tmp/kraft-combined-logs
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/server.properties
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/meta.properties
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/.kafka_cleanshutdown
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/foo-0/partition.metadata
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/pax-0/partition.metadata
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/quz-0/partition.metadata
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/quz-1/partition.metadata
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/partition.metadata
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/foo-0/00000000000000000000.log
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/pax-0/00000000000000000000.log
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/quz-0/00000000000000000000.log
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/quz-1/00000000000000000000.log
[tester::#GS0] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
[tester::#GS0] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
[tester::#GS0] Connecting to broker at: localhost:9092
[your_program] Debug mode enabled
[your_program] 2024-10-12T20:04:14.821377Z  INFO kafka_starter_rust: Started server listen_addr=127.0.0.1:9092
[tester::#GS0] Connection to broker at localhost:9092 successful
[tester::#GS0] Sending "ApiVersions" (version: 4) request (Correlation id: 1410041126)
[your_program] 2024-10-12T20:04:15.815535Z DEBUG kafka_starter_rust: accepted new connection peer_addr="127.0.0.1:39772"
[tester::#GS0] Hexdump of sent "ApiVersions" request: 
[tester::#GS0] Idx  | Hex                                             | ASCII
[tester::#GS0] -----+-------------------------------------------------+-----------------
[tester::#GS0] 0000 | 00 00 00 23 00 12 00 04 54 0b 85 26 00 09 6b 61 | ...#....T..&..ka
[tester::#GS0] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
[tester::#GS0] 0020 | 6c 69 04 30 2e 31 00                            | li.0.1.
[tester::#GS0] 
[tester::#GS0] Hexdump of received "ApiVersions" response: 
[tester::#GS0] Idx  | Hex                                             | ASCII
[tester::#GS0] -----+-------------------------------------------------+-----------------
[tester::#GS0] 0000 | 54 0b 85 26 00 00 00 03 00 12 00 02 00 04 00 00 | T..&............
[tester::#GS0] 0010 | 01 00 0e 00 10 00 00 00 00 00 00                | ...........
[tester::#GS0] 
[tester::#GS0] [Decoder] - .ResponseHeader
[tester::#GS0] [Decoder]   - .correlation_id (1410041126)
[tester::#GS0] [Decoder] - .ResponseBody
[tester::#GS0] [Decoder]   - .error_code (0)
[tester::#GS0] [Decoder]   - .num_api_keys (0)
[tester::#GS0] [Decoder]   - .throttle_time_ms (50336256)
[tester::#GS0] [Decoder]   - .TAG_BUFFER
[your_program] 2024-10-12T20:04:15.815977Z DEBUG kafka_starter_rust: Got frame frame_length=35
[your_program] 2024-10-12T20:04:15.816024Z DEBUG kafka_starter_rust: RequestHeader request_header=RequestHeader { request_api_key: 18, request_api_version: 4, correlation_id: 1410041126, client_id: NullableString("kafka-cli"), tagged_fields: 0 }
[tester::#GS0] Received:
[tester::#GS0] Hex (bytes 15-26)                               | ASCII
[tester::#GS0] ------------------------------------------------+------------------
[tester::#GS0] 00 01 00 0e 00 10 00 00 00 00 00 00             | ............
[tester::#GS0]                 ^                                      ^
[tester::#GS0] Error: unexpected 7 bytes remaining in decoder after decoding ApiVersionsResponse
[tester::#GS0] Context:
[tester::#GS0] - ApiVersions v3
[tester::#GS0]   - Response Body
[tester::#GS0] 
[tester::#GS0] Test failed
1 Like

As a general rule, use the v1 ResponseHeader in almost all situations, as outlined in KIP-482:

In order to have flexible version support across all requests and responses, we will bump the version of all requests and responses.
ā€¦
The new ResponseHeader version will be version 1, superseding version 0.

The only exception is that ApiVersionsResponse uses a v0 header.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.