Headers:
Request Header => api_key api_version correlation_id client_id
api_key => INT16
api_version => INT16
correlation_id => INT32
client_id => NULLABLE_STRING
Field |
Description |
---|
api_key | The id of the request type. |
api_version | The version of the API. |
correlation_id | A user-supplied integer value that will be passed back with the response |
client_id | A user specified identifier for the client making the request. |
Response Header => correlation_id
correlation_id => INT32
Field |
Description |
---|
correlation_id | The user-supplied value passed in with the request |
Produce API (Key: 0):
Requests:
Produce Request (Version: 0) => acks timeout [topic_data]
acks => INT16
timeout => INT32
topic_data => topic [data]
topic => STRING
data => partition record_set
partition => INT32
record_set => BYTES
Field |
Description |
---|
acks | The number of nodes that should replicate the produce before returning. -1 indicates the full ISR. |
timeout | The time to await a response in ms. |
topic_data | |
topic | |
data | |
partition | |
record_set | |
Produce Request (Version: 1) => acks timeout [topic_data]
acks => INT16
timeout => INT32
topic_data => topic [data]
topic => STRING
data => partition record_set
partition => INT32
record_set => BYTES
Field |
Description |
---|
acks | The number of nodes that should replicate the produce before returning. -1 indicates the full ISR. |
timeout | The time to await a response in ms. |
topic_data | |
topic | |
data | |
partition | |
record_set | |
Responses:
Produce Response (Version: 0) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code base_offset
partition => INT32
error_code => INT16
base_offset => INT64
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | |
error_code | |
base_offset | |
Produce Response (Version: 1) => [responses] throttle_time_ms
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code base_offset
partition => INT32
error_code => INT16
base_offset => INT64
throttle_time_ms => INT32
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | |
error_code | |
base_offset | |
throttle_time_ms | Duration in milliseconds for which the request was throttled due to quota violation. (Zero if the request did not violate any quota.) |
Fetch API (Key: 1):
Requests:
Fetch Request (Version: 0) => replica_id max_wait_time min_bytes [topics]
replica_id => INT32
max_wait_time => INT32
min_bytes => INT32
topics => topic [partitions]
topic => STRING
partitions => partition fetch_offset max_bytes
partition => INT32
fetch_offset => INT64
max_bytes => INT32
Field |
Description |
---|
replica_id | Broker id of the follower. For normal consumers, use -1. |
max_wait_time | Maximum time in ms to wait for the response. |
min_bytes | Minimum bytes to accumulate in the response. |
topics | Topics to fetch. |
topic | Topic to fetch. |
partitions | Partitions to fetch. |
partition | Topic partition id. |
fetch_offset | Message offset. |
max_bytes | Maximum bytes to fetch. |
Fetch Request (Version: 1) => replica_id max_wait_time min_bytes [topics]
replica_id => INT32
max_wait_time => INT32
min_bytes => INT32
topics => topic [partitions]
topic => STRING
partitions => partition fetch_offset max_bytes
partition => INT32
fetch_offset => INT64
max_bytes => INT32
Field |
Description |
---|
replica_id | Broker id of the follower. For normal consumers, use -1. |
max_wait_time | Maximum time in ms to wait for the response. |
min_bytes | Minimum bytes to accumulate in the response. |
topics | Topics to fetch. |
topic | Topic to fetch. |
partitions | Partitions to fetch. |
partition | Topic partition id. |
fetch_offset | Message offset. |
max_bytes | Maximum bytes to fetch. |
Responses:
Fetch Response (Version: 0) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code high_watermark record_set
partition => INT32
error_code => INT16
high_watermark => INT64
record_set => BYTES
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
high_watermark | Last committed offset. |
record_set | |
Fetch Response (Version: 1) => throttle_time_ms [responses]
throttle_time_ms => INT32
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code high_watermark record_set
partition => INT32
error_code => INT16
high_watermark => INT64
record_set => BYTES
Field |
Description |
---|
throttle_time_ms | Duration in milliseconds for which the request was throttled due to quota violation. (Zero if the request did not violate any quota.) |
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
high_watermark | Last committed offset. |
record_set | |
Offsets API (Key: 2):
Requests:
Offsets Request (Version: 0) => replica_id [topics]
replica_id => INT32
topics => topic [partitions]
topic => STRING
partitions => partition timestamp max_num_offsets
partition => INT32
timestamp => INT64
max_num_offsets => INT32
Field |
Description |
---|
replica_id | Broker id of the follower. For normal consumers, use -1. |
topics | Topics to list offsets. |
topic | Topic to list offset. |
partitions | Partitions to list offset. |
partition | Topic partition id. |
timestamp | Timestamp. |
max_num_offsets | Maximum offsets to return. |
Responses:
Offsets Response (Version: 0) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code [offsets]
partition => INT32
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
offsets | A list of offsets. |
Metadata API (Key: 3):
Requests:
Metadata Request (Version: 0) => [topics]
Field |
Description |
---|
topics | An array of topics to fetch metadata for. If no topics are specified fetch metadata for all topics. |
Responses:
Metadata Response (Version: 0) => [brokers] [topic_metadata]
brokers => node_id host port
node_id => INT32
host => STRING
port => INT32
topic_metadata => topic_error_code topic [partition_metadata]
topic_error_code => INT16
topic => STRING
partition_metadata => partition_error_code partition_id leader [replicas] [isr]
partition_error_code => INT16
partition_id => INT32
leader => INT32
Field |
Description |
---|
brokers | Host and port information for all brokers. |
node_id | The broker id. |
host | The hostname of the broker. |
port | The port on which the broker accepts requests. |
topic_metadata | |
topic_error_code | The error code for the given topic. |
topic | The name of the topic |
partition_metadata | Metadata for each partition of the topic. |
partition_error_code | The error code for the partition, if any. |
partition_id | The id of the partition. |
leader | The id of the broker acting as leader for this partition. |
replicas | The set of all nodes that host this partition. |
isr | The set of nodes that are in sync with the leader for this partition. |
LeaderAndIsr API (Key: 4):
Requests:
LeaderAndIsr Request (Version: 0) => controller_id controller_epoch [partition_states] [live_leaders]
controller_id => INT32
controller_epoch => INT32
partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas]
topic => STRING
partition => INT32
controller_epoch => INT32
leader => INT32
leader_epoch => INT32
zk_version => INT32
live_leaders => id host port
id => INT32
host => STRING
port => INT32
Field |
Description |
---|
controller_id | The controller id. |
controller_epoch | The controller epoch. |
partition_states | |
topic | Topic name. |
partition | Topic partition id. |
controller_epoch | The controller epoch. |
leader | The broker id for the leader. |
leader_epoch | The leader epoch. |
isr | The in sync replica ids. |
zk_version | The ZK version. |
replicas | The replica ids. |
live_leaders | |
id | The broker id. |
host | The hostname of the broker. |
port | The port on which the broker accepts requests. |
Responses:
LeaderAndIsr Response (Version: 0) => error_code [partitions]
error_code => INT16
partitions => topic partition error_code
topic => STRING
partition => INT32
error_code => INT16
Field |
Description |
---|
error_code | Error code. |
partitions | |
topic | Topic name. |
partition | Topic partition id. |
error_code | Error code. |
StopReplica API (Key: 5):
Requests:
StopReplica Request (Version: 0) => controller_id controller_epoch delete_partitions [partitions]
controller_id => INT32
controller_epoch => INT32
delete_partitions => INT8
partitions => topic partition
topic => STRING
partition => INT32
Field |
Description |
---|
controller_id | The controller id. |
controller_epoch | The controller epoch. |
delete_partitions | Boolean which indicates if replica's partitions must be deleted. |
partitions | |
topic | Topic name. |
partition | Topic partition id. |
Responses:
StopReplica Response (Version: 0) => error_code [partitions]
error_code => INT16
partitions => topic partition error_code
topic => STRING
partition => INT32
error_code => INT16
Field |
Description |
---|
error_code | Error code. |
partitions | |
topic | Topic name. |
partition | Topic partition id. |
error_code | Error code. |
UpdateMetadata API (Key: 6):
Requests:
UpdateMetadata Request (Version: 0) => controller_id controller_epoch [partition_states] [live_brokers]
controller_id => INT32
controller_epoch => INT32
partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas]
topic => STRING
partition => INT32
controller_epoch => INT32
leader => INT32
leader_epoch => INT32
zk_version => INT32
live_brokers => id host port
id => INT32
host => STRING
port => INT32
Field |
Description |
---|
controller_id | The controller id. |
controller_epoch | The controller epoch. |
partition_states | |
topic | Topic name. |
partition | Topic partition id. |
controller_epoch | The controller epoch. |
leader | The broker id for the leader. |
leader_epoch | The leader epoch. |
isr | The in sync replica ids. |
zk_version | The ZK version. |
replicas | The replica ids. |
live_brokers | |
id | The broker id. |
host | The hostname of the broker. |
port | The port on which the broker accepts requests. |
UpdateMetadata Request (Version: 1) => controller_id controller_epoch [partition_states] [live_brokers]
controller_id => INT32
controller_epoch => INT32
partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas]
topic => STRING
partition => INT32
controller_epoch => INT32
leader => INT32
leader_epoch => INT32
zk_version => INT32
live_brokers => id [end_points]
id => INT32
end_points => port host security_protocol_type
port => INT32
host => STRING
security_protocol_type => INT16
Field |
Description |
---|
controller_id | The controller id. |
controller_epoch | The controller epoch. |
partition_states | |
topic | Topic name. |
partition | Topic partition id. |
controller_epoch | The controller epoch. |
leader | The broker id for the leader. |
leader_epoch | The leader epoch. |
isr | The in sync replica ids. |
zk_version | The ZK version. |
replicas | The replica ids. |
live_brokers | |
id | The broker id. |
end_points | |
port | The port on which the broker accepts requests. |
host | The hostname of the broker. |
security_protocol_type | The security protocol type. |
Responses:
UpdateMetadata Response (Version: 0) => error_code
error_code => INT16
Field |
Description |
---|
error_code | Error code. |
UpdateMetadata Response (Version: 1) => error_code
error_code => INT16
Field |
Description |
---|
error_code | Error code. |
ControlledShutdown API (Key: 7):
Requests:
ControlledShutdown Request (Version: 1) => broker_id
broker_id => INT32
Field |
Description |
---|
broker_id | The id of the broker for which controlled shutdown has been requested. |
Responses:
ControlledShutdown Response (Version: 1) => error_code [partitions_remaining]
error_code => INT16
partitions_remaining => topic partition
topic => STRING
partition => INT32
Field |
Description |
---|
error_code | |
partitions_remaining | The partitions that the broker still leads. |
topic | |
partition | Topic partition id. |
OffsetCommit API (Key: 8):
Requests:
OffsetCommit Request (Version: 0) => group_id [topics]
group_id => STRING
topics => topic [partitions]
topic => STRING
partitions => partition offset metadata
partition => INT32
offset => INT64
metadata => STRING
Field |
Description |
---|
group_id | The group id. |
topics | Topics to commit offsets. |
topic | Topic to commit. |
partitions | Partitions to commit offsets. |
partition | Topic partition id. |
offset | Message offset to be committed. |
metadata | Any associated metadata the client wants to keep. |
OffsetCommit Request (Version: 1) => group_id group_generation_id member_id [topics]
group_id => STRING
group_generation_id => INT32
member_id => STRING
topics => topic [partitions]
topic => STRING
partitions => partition offset timestamp metadata
partition => INT32
offset => INT64
timestamp => INT64
metadata => STRING
Field |
Description |
---|
group_id | The group id. |
group_generation_id | The generation of the group. |
member_id | The member id assigned by the group coordinator. |
topics | Topics to commit offsets. |
topic | Topic to commit. |
partitions | Partitions to commit offsets. |
partition | Topic partition id. |
offset | Message offset to be committed. |
timestamp | Timestamp of the commit |
metadata | Any associated metadata the client wants to keep. |
OffsetCommit Request (Version: 2) => group_id group_generation_id member_id retention_time [topics]
group_id => STRING
group_generation_id => INT32
member_id => STRING
retention_time => INT64
topics => topic [partitions]
topic => STRING
partitions => partition offset metadata
partition => INT32
offset => INT64
metadata => STRING
Field |
Description |
---|
group_id | The group id. |
group_generation_id | The generation of the consumer group. |
member_id | The consumer id assigned by the group coordinator. |
retention_time | Time period in ms to retain the offset. |
topics | Topics to commit offsets. |
topic | Topic to commit. |
partitions | Partitions to commit offsets. |
partition | Topic partition id. |
offset | Message offset to be committed. |
metadata | Any associated metadata the client wants to keep. |
Responses:
OffsetCommit Response (Version: 0) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code
partition => INT32
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
OffsetCommit Response (Version: 1) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code
partition => INT32
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
OffsetCommit Response (Version: 2) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition error_code
partition => INT32
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
error_code | |
OffsetFetch API (Key: 9):
Requests:
OffsetFetch Request (Version: 0) => group_id [topics]
group_id => STRING
topics => topic [partitions]
topic => STRING
partitions => partition
partition => INT32
Field |
Description |
---|
group_id | The consumer group id. |
topics | Topics to fetch offsets. |
topic | Topic to fetch offset. |
partitions | Partitions to fetch offsets. |
partition | Topic partition id. |
OffsetFetch Request (Version: 1) => group_id [topics]
group_id => STRING
topics => topic [partitions]
topic => STRING
partitions => partition
partition => INT32
Field |
Description |
---|
group_id | The consumer group id. |
topics | Topics to fetch offsets. |
topic | Topic to fetch offset. |
partitions | Partitions to fetch offsets. |
partition | Topic partition id. |
Responses:
OffsetFetch Response (Version: 0) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition offset metadata error_code
partition => INT32
offset => INT64
metadata => STRING
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
offset | Last committed message offset. |
metadata | Any associated metadata the client wants to keep. |
error_code | |
OffsetFetch Response (Version: 1) => [responses]
responses => topic [partition_responses]
topic => STRING
partition_responses => partition offset metadata error_code
partition => INT32
offset => INT64
metadata => STRING
error_code => INT16
Field |
Description |
---|
responses | |
topic | |
partition_responses | |
partition | Topic partition id. |
offset | Last committed message offset. |
metadata | Any associated metadata the client wants to keep. |
error_code | |
GroupCoordinator API (Key: 10):
Requests:
GroupCoordinator Request (Version: 0) => group_id
group_id => STRING
Field |
Description |
---|
group_id | The unique group id. |
Responses:
GroupCoordinator Response (Version: 0) => error_code coordinator
error_code => INT16
coordinator => node_id host port
node_id => INT32
host => STRING
port => INT32
Field |
Description |
---|
error_code | |
coordinator | Host and port information for the coordinator for a consumer group. |
node_id | The broker id. |
host | The hostname of the broker. |
port | The port on which the broker accepts requests. |
JoinGroup API (Key: 11):
Requests:
JoinGroup Request (Version: 0) => group_id session_timeout member_id protocol_type [group_protocols]
group_id => STRING
session_timeout => INT32
member_id => STRING
protocol_type => STRING
group_protocols => protocol_name protocol_metadata
protocol_name => STRING
protocol_metadata => BYTES
Field |
Description |
---|
group_id | The group id. |
session_timeout | The coordinator considers the consumer dead if it receives no heartbeat after this timeout in ms. |
member_id | The assigned consumer id or an empty string for a new consumer. |
protocol_type | Unique name for class of protocols implemented by group |
group_protocols | List of protocols that the member supports |
protocol_name | |
protocol_metadata | |
Responses:
JoinGroup Response (Version: 0) => error_code generation_id group_protocol leader_id member_id [members]
error_code => INT16
generation_id => INT32
group_protocol => STRING
leader_id => STRING
member_id => STRING
members => member_id member_metadata
member_id => STRING
member_metadata => BYTES
Field |
Description |
---|
error_code | |
generation_id | The generation of the consumer group. |
group_protocol | The group protocol selected by the coordinator |
leader_id | The leader of the group |
member_id | The consumer id assigned by the group coordinator. |
members | |
member_id | |
member_metadata | |
Heartbeat API (Key: 12):
Requests:
Heartbeat Request (Version: 0) => group_id group_generation_id member_id
group_id => STRING
group_generation_id => INT32
member_id => STRING
Field |
Description |
---|
group_id | The group id. |
group_generation_id | The generation of the group. |
member_id | The member id assigned by the group coordinator. |
Responses:
Heartbeat Response (Version: 0) => error_code
error_code => INT16
Field |
Description |
---|
error_code | |
LeaveGroup API (Key: 13):
Requests:
LeaveGroup Request (Version: 0) => group_id member_id
group_id => STRING
member_id => STRING
Field |
Description |
---|
group_id | The group id. |
member_id | The member id assigned by the group coordinator. |
Responses:
LeaveGroup Response (Version: 0) => error_code
error_code => INT16
Field |
Description |
---|
error_code | |
SyncGroup API (Key: 14):
Requests:
SyncGroup Request (Version: 0) => group_id generation_id member_id [group_assignment]
group_id => STRING
generation_id => INT32
member_id => STRING
group_assignment => member_id member_assignment
member_id => STRING
member_assignment => BYTES
Field |
Description |
---|
group_id | |
generation_id | |
member_id | |
group_assignment | |
member_id | |
member_assignment | |
Responses:
SyncGroup Response (Version: 0) => error_code member_assignment
error_code => INT16
member_assignment => BYTES
Field |
Description |
---|
error_code | |
member_assignment | |
DescribeGroups API (Key: 15):
Requests:
DescribeGroups Request (Version: 0) => [group_ids]
Field |
Description |
---|
group_ids | List of groupIds to request metadata for (an empty groupId array will return empty group metadata). |
Responses:
DescribeGroups Response (Version: 0) => [groups]
groups => error_code group_id state protocol_type protocol [members]
error_code => INT16
group_id => STRING
state => STRING
protocol_type => STRING
protocol => STRING
members => member_id client_id client_host member_metadata member_assignment
member_id => STRING
client_id => STRING
client_host => STRING
member_metadata => BYTES
member_assignment => BYTES
Field |
Description |
---|
groups | |
error_code | |
group_id | |
state | The current state of the group (one of: Dead, Stable, AwaitingSync, or PreparingRebalance, or empty if there is no active group) |
protocol_type | The current group protocol type (will be empty if the there is no active group) |
protocol | The current group protocol (only provided if the group is Stable) |
members | Current group members (only provided if the group is not Dead) |
member_id | The memberId assigned by the coordinator |
client_id | The client id used in the member's latest join group request |
client_host | The client host used in the request session corresponding to the member's join group. |
member_metadata | The metadata corresponding to the current group protocol in use (will only be present if the group is stable). |
member_assignment | The current assignment provided by the group leader (will only be present if the group is stable). |
ListGroups API (Key: 16):
Requests:
ListGroups Request (Version: 0) =>
Responses:
ListGroups Response (Version: 0) => error_code [groups]
error_code => INT16
groups => group_id protocol_type
group_id => STRING
protocol_type => STRING
Field |
Description |
---|
error_code | |
groups | |
group_id | |
protocol_type | |