Question about #wq2 List for multiple topics

silversword05 · December 6, 2024, 6:14am

Seems there is a bug WQ2. Fails randomly.

[tester::#WQ2] Running tests for Stage #WQ2 (Listing Partitions - List for multiple topics)
[tester::#WQ2] $ ./your_program.sh /tmp/server.properties
[tester::#WQ2] [Serializer] Writing log files to: /tmp/kraft-combined-logs
[tester::#WQ2] [Serializer]   - Wrote file to: /tmp/server.properties
[tester::#WQ2] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/meta.properties
[tester::#WQ2] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/.kafka_cleanshutdown
[tester::#WQ2] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/partition.metadata
[tester::#WQ2] [Serializer]   - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
[tester::#WQ2] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
[tester::#WQ2] Connecting to broker at: localhost:9092
[your_program] Logs from your program will appear here!
[your_program] Waiting for a client to connect...
[your_program] Reading Cluster Metadata
[your_program] Unsupported record type: 12
[your_program] Topic Record: TopicRecord{topic_name=bar, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , tagged_fields_count=0}
[your_program] Partition Record: PartitionRecord{partition_id=0, topic_uuid=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, removing_replica_array=CompactArray{Values=}, adding_replica_array=CompactArray{Values=}, leader_id=1, leader_epoch=0, partition_epoch=0, directories_array=CompactArray{Values=10 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 01 ,}, tagged_fields_count=0}
[your_program] Topic Record: TopicRecord{topic_name=baz, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 25 , tagged_fields_count=0}
[your_program] Partition Record: PartitionRecord{partition_id=0, topic_uuid=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 25 , replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, removing_replica_array=CompactArray{Values=}, adding_replica_array=CompactArray{Values=}, leader_id=1, leader_epoch=0, partition_epoch=0, directories_array=CompactArray{Values=10 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 01 ,}, tagged_fields_count=0}
[your_program] Topic Record: TopicRecord{topic_name=foo, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , tagged_fields_count=0}
[your_program] Partition Record: PartitionRecord{partition_id=0, topic_uuid=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, removing_replica_array=CompactArray{Values=}, adding_replica_array=CompactArray{Values=}, leader_id=1, leader_epoch=0, partition_epoch=0, directories_array=CompactArray{Values=10 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 01 ,}, tagged_fields_count=0}
[your_program] Partition Record: PartitionRecord{partition_id=1, topic_uuid=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, removing_replica_array=CompactArray{Values=}, adding_replica_array=CompactArray{Values=}, leader_id=1, leader_epoch=0, partition_epoch=0, directories_array=CompactArray{Values=10 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 01 ,}, tagged_fields_count=0}
[tester::#WQ2] Connection to broker at localhost:9092 successful
[tester::#WQ2] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 2010429749)
[tester::#WQ2] Hexdump of sent "DescribeTopicPartitions" request:
[tester::#WQ2] Idx  | Hex                                             | ASCII
[tester::#WQ2] -----+-------------------------------------------------+-----------------
[tester::#WQ2] 0000 | 00 00 00 2d 00 4b 00 00 77 d4 b9 35 00 0c 6b 61 | ...-.K..w..5..ka
[tester::#WQ2] 0010 | 66 6b 61 2d 74 65 73 74 65 72 00 04 04 62 61 72 | fka-tester...bar
[tester::#WQ2] 0020 | 00 04 62 61 7a 00 04 66 6f 6f 00 00 00 00 04 ff | ..baz..foo......
[tester::#WQ2] 0030 | 00                                              | .
[tester::#WQ2]
[your_program] Client connected
[your_program] File descriptor already closed -1
[your_program] File descriptor already closed -1
[your_program] Attaching a new client thread. client-fd: 4
[your_program] Received 49 bytes from client
[your_program] Received Request: RequestHeader{message_size=45, request_api_key=75, request_api_version=0, corellation_id=2010429749, client_id=kafka-tester}
[tester::#WQ2] Hexdump of received "DescribeTopicPartitions" response:
[tester::#WQ2] Idx  | Hex                                             | ASCII
[tester::#WQ2] -----+-------------------------------------------------+-----------------
[tester::#WQ2] 0000 | 00 00 01 27 77 d4 b9 35 00 00 00 00 00 04 00 00 | ...'w..5........
[tester::#WQ2] 0010 | 04 62 61 72 00 00 00 00 00 00 40 00 80 00 00 00 | .bar......@.....
[tester::#WQ2] 0020 | 00 00 00 73 00 04 00 00 00 00 00 00 00 00 00 01 | ...s............
[tester::#WQ2] 0030 | 00 00 00 00 02 00 00 00 01 02 00 00 00 01 01 01 | ................
[tester::#WQ2] 0040 | 01 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 | ................
[tester::#WQ2] 0050 | 02 00 00 00 01 02 00 00 00 01 01 01 01 00 00 00 | ................
[tester::#WQ2] 0060 | 00 00 00 01 00 00 00 01 00 00 00 00 02 00 00 00 | ................
[tester::#WQ2] 0070 | 01 02 00 00 00 01 01 01 01 00 00 00 0d f8 00 00 | ................
[tester::#WQ2] 0080 | 00 04 62 61 7a 00 00 00 00 00 00 40 00 80 00 00 | ..baz......@....
[tester::#WQ2] 0090 | 00 00 00 00 25 00 02 00 00 00 00 00 00 00 00 00 | ....%...........
[tester::#WQ2] 00a0 | 01 00 00 00 00 02 00 00 00 01 02 00 00 00 01 01 | ................
[tester::#WQ2] 00b0 | 01 01 00 00 00 0d f8 00 00 00 04 66 6f 6f 00 00 | ...........foo..
[tester::#WQ2] 00c0 | 00 00 00 00 40 00 80 00 00 00 00 00 00 73 00 04 | ....@........s..
[tester::#WQ2] 00d0 | 00 00 00 00 00 00 00 00 00 01 00 00 00 00 02 00 | ................
[tester::#WQ2] 00e0 | 00 00 01 02 00 00 00 01 01 01 01 00 00 00 00 00 | ................
[tester::#WQ2] 00f0 | 00 00 00 00 00 01 00 00 00 00 02 00 00 00 01 02 | ................
[tester::#WQ2] 0100 | 00 00 00 01 01 01 01 00 00 00 00 00 00 01 00 00 | ................
[tester::#WQ2] 0110 | 00 01 00 00 00 00 02 00 00 00 01 02 00 00 00 01 | ................
[tester::#WQ2] 0120 | 01 01 01 00 00 00 0d f8 00 ff 00                | ...........
[tester::#WQ2]
[tester::#WQ2] [Decoder] - .ResponseHeader
[tester::#WQ2] [Decoder]   - .correlation_id (2010429749)
[tester::#WQ2] [Decoder]   - .TAG_BUFFER
[tester::#WQ2] [Decoder] - .ResponseBody
[tester::#WQ2] [Decoder]   - .throttle_time_ms (0)
[tester::#WQ2] [Decoder]   - .topic.length (3)
[tester::#WQ2] [Decoder]   - .Topics[0]
[tester::#WQ2] [Decoder]     - .error_code (0)
[tester::#WQ2] [Decoder]     - .name (bar)
[tester::#WQ2] [Decoder]     - .topic_id (00000000-0000-4000-8000-000000000073)
[tester::#WQ2] [Decoder]     - .is_internal (false)
[tester::#WQ2] [Decoder]     - .num_partitions (3)
[tester::#WQ2] [Decoder]     - .Partitions[0]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (0)
[your_program] Received Describe Topic Partitions Request: DescribeTopicPartitionsRequest{RequestHeader{message_size=45, request_api_key=75, request_api_version=0, corellation_id=2010429749, client_id=kafka-tester}, array_length=4, topics=Topics{Topic{topic_name=bar, tagged_fields=TaggedFields{fieldCount=0}}, Topic{topic_name=baz, tagged_fields=TaggedFields{fieldCount=0}}, Topic{topic_name=foo, tagged_fields=TaggedFields{fieldCount=0}}, }, responsePartitionLimit=4, cursor=255, tagged_fields=TaggedFields{fieldCount=0}}
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .Partitions[1]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (0)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .Partitions[2]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (1)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .topic_authorized_operations (3576)
[tester::#WQ2] [Decoder]     - .TAG_BUFFER
[tester::#WQ2] [Decoder]   - .Topics[1]
[tester::#WQ2] [Decoder]     - .error_code (0)
[tester::#WQ2] [Decoder]     - .name (baz)
[tester::#WQ2] [Decoder]     - .topic_id (00000000-0000-4000-8000-000000000025)
[tester::#WQ2] [Decoder]     - .is_internal (false)
[tester::#WQ2] [Decoder]     - .num_partitions (1)
[tester::#WQ2] [Decoder]     - .Partitions[0]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (0)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .topic_authorized_operations (3576)
[tester::#WQ2] [Decoder]     - .TAG_BUFFER
[tester::#WQ2] [Decoder]   - .Topics[2]
[tester::#WQ2] [Decoder]     - .error_code (0)
[tester::#WQ2] [Decoder]     - .name (foo)
[tester::#WQ2] [Decoder]     - .topic_id (00000000-0000-4000-8000-000000000073)
[tester::#WQ2] [Decoder]     - .is_internal (false)
[tester::#WQ2] [Decoder]     - .num_partitions (3)
[tester::#WQ2] [Decoder]     - .Partitions[0]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (0)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .Partitions[1]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (0)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .Partitions[2]
[tester::#WQ2] [Decoder]       - .error_code (0)
[tester::#WQ2] [Decoder]       - .partition_index (1)
[tester::#WQ2] [Decoder]       - .leader_id (1)
[tester::#WQ2] [Decoder]       - .leader_epoch (0)
[tester::#WQ2] [Decoder]       - .replica_nodes ([1])
[tester::#WQ2] [Decoder]       - .isr_nodes ([1])
[tester::#WQ2] [Decoder]       - .eligible_leader_replicas ([])
[tester::#WQ2] [Decoder]       - .last_known_elr ([])
[tester::#WQ2] [Decoder]       - .offline_replicas ([])
[tester::#WQ2] [Decoder]       - .TAG_BUFFER
[tester::#WQ2] [Decoder]     - .topic_authorized_operations (3576)
[tester::#WQ2] [Decoder]     - .TAG_BUFFER
[tester::#WQ2] [Decoder]   - .next_cursor (null)
[tester::#WQ2] [Decoder]   - .TAG_BUFFER
[tester::#WQ2] ✓ Correlation ID: 2010429749
[tester::#WQ2] ✓ Throttle Time: 0
[tester::#WQ2]   ✓ TopicResponse[0] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[0] Topic Name: bar
[tester::#WQ2]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000073
[tester::#WQ2]   ✓ TopicResponse[1] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[1] Topic Name: baz
[tester::#WQ2]   ✓ TopicResponse[1] Topic UUID: 00000000-0000-4000-8000-000000000025
[tester::#WQ2]     ✓ PartitionResponse[0] Error code: 0
[tester::#WQ2]     ✓ PartitionResponse[0] Partition Index: 0
[tester::#WQ2]   ✓ TopicResponse[2] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[2] Topic Name: foo
[tester::#WQ2]   ✓ TopicResponse[2] Topic UUID: 00000000-0000-4000-8000-000000000073
[tester::#WQ2] Expected partitions.length to be 2, got 3
[tester::#WQ2] Test failed
[tester::#WQ2] Terminating program
[your_program] Sending msg to client: DescribeTopicPartitionsResponse{message_size=0, corellation_id=2010429749, tagged_fields=TaggedFields{fieldCount=0}, throttle_time=0, array_length=4, topics=Topics{Topic{error_code=0, topic_name=bar, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , boolInternal=0, array_length=4, partitions=Partitions{Partition{error_code=0, partition_id=0, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, Partition{error_code=0, partition_id=0, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, Partition{error_code=0, partition_id=1, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, }, authorizedOperations=00 00 0d fffffff8 , tagged_fields=TaggedFields{fieldCount=0}}, Topic{error_code=0, topic_name=baz, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 25 , boolInternal=0, array_length=2, partitions=Partitions{Partition{error_code=0, partition_id=0, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, }, authorizedOperations=00 00 0d fffffff8 , tagged_fields=TaggedFields{fieldCount=0}}, Topic{error_code=0, topic_name=foo, topic_id=00 00 00 00 00 00 40 00 ffffff80 00 00 00 00 00 00 73 , boolInternal=0, array_length=4, partitions=Partitions{Partition{error_code=0, partition_id=0, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, Partition{error_code=0, partition_id=0, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, Partition{error_code=0, partition_id=1, leader_id=1, leader_epoch=0, replica_array=CompactArray{Values=1,}, in_sync_replica_array=CompactArray{Values=1,}, eligible_leader_replica_array=CompactArray{Values=}, last_known_elr_array=CompactArray{Values=}, offline_replica_array=CompactArray{Values=}, tagged_fields=TaggedFields{fieldCount=0}}, }, authorizedOperations=00 00 0d fffffff8 , tagged_fields=TaggedFields{fieldCount=0}}, }, cursor=255, tagged_fields=TaggedFields{fieldCount=0}}
[your_program] Message sent to client: 299 bytes
[tester::#WQ2] Program terminated successfully

Topics foo and bar has the same topic-id which IIUC should not happen. Please correct it.

[tester::#WQ2] ✓ Correlation ID: 2010429749
[tester::#WQ2] ✓ Throttle Time: 0
[tester::#WQ2]   ✓ TopicResponse[0] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[0] Topic Name: bar
[tester::#WQ2]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-4000-8000-000000000073
[tester::#WQ2]   ✓ TopicResponse[1] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[1] Topic Name: baz
[tester::#WQ2]   ✓ TopicResponse[1] Topic UUID: 00000000-0000-4000-8000-000000000025
[tester::#WQ2]     ✓ PartitionResponse[0] Error code: 0
[tester::#WQ2]     ✓ PartitionResponse[0] Partition Index: 0
[tester::#WQ2]   ✓ TopicResponse[2] Error code: 0
[tester::#WQ2]   ✓ TopicResponse[2] Topic Name: foo
[tester::#WQ2]   ✓ TopicResponse[2] Topic UUID: 00000000-0000-4000-8000-000000000073
[tester::#WQ2] Expected partitions.length to be 2, got 3
[tester::#WQ2] Test failed
[tester::#WQ2] Terminating program

silversword05 · December 6, 2024, 6:16am

Also, while creating the cluster file, ideally the test suite should first create and write the cluster files and then run the binaries under test. Often, I saw that the test suite started the binaries before writing the cluster file (the first request was sent after the files are written).

andy1li · December 6, 2024, 6:23am

Hi @silversword05, thanks for highlighting the issue! We’ll take a closer look and get back to you as soon as possible.

andy1li · December 9, 2024, 10:57am

We’ve fixed the Topic ID collision issue in this PR. Thanks again for highlighting it! @silversword05

silversword05 · December 9, 2024, 5:18pm

Happy to help!

system · December 16, 2024, 5:18pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Question about ea7: List for a single partition - log file load Challenges challenge:kafka	3	31	January 6, 2025
New extension: Listing Partitions (Kafka) Announcements	4	108	October 30, 2024
Kafka : Looks like your program has terminated. A Kafka server is expected to be a long-running process Challenges challenge:kafka	3	40	October 10, 2024
Question about HN6 (Fetch with an unknown topic) Challenges challenge:kafka	11	130	October 4, 2024
Stack on question #wa6 Challenges challenge:kafka	6	95	September 30, 2024

Question about #wq2 List for multiple topics

Related topics