Tiered Storage Configs
Below is the Tiered Storage configuration.
log.local.retention.bytes
The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. Default value is -2, it represents
log.retention.bytesvalue to be used. The effective value should always be less than or equal tolog.retention.bytesvalue.Type: long Default: -2 Valid Values: [-2,...] Importance: medium log.local.retention.ms
The number of milliseconds to keep the local log segments before it gets eligible for deletion. Default value is -2, it represents
log.retention.msvalue is to be used. The effective value should always be less than or equal tolog.retention.msvalue.Type: long Default: -2 Valid Values: [-2,...] Importance: medium remote.fetch.max.wait.ms
The maximum amount of time the server will wait before answering the fetch request containing remote fetch partitions. It's important to be aware that the request will only be responded after all remote partitions have been successfully fetched, have failed, or this timeout is exceeded.
Type: int Default: 500 Valid Values: [1,...] Importance: medium remote.list.offsets.request.timeout.ms
The maximum amount of time the server will wait for the remote list offsets request to complete.
Type: long Default: 30000 (30 seconds) Valid Values: [1,...] Importance: medium remote.log.manager.copier.thread.pool.size
Size of the thread pool used in scheduling tasks to copy segments.
Type: int Default: 10 Valid Values: [1,...] Importance: medium remote.log.manager.copy.max.bytes.per.second
The maximum number of bytes that can be copied from local storage to remote storage per second. This is a global limit for all the partitions that are being copied from local storage to remote storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be copied per second.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: medium remote.log.manager.copy.quota.window.num
The number of samples to retain in memory for remote copy quota management. The default value is 11, which means there are 10 whole windows + 1 current window.
Type: int Default: 11 Valid Values: [1,...] Importance: medium remote.log.manager.copy.quota.window.size.seconds
The time span of each sample for remote copy quota management. The default value is 1 second.
Type: int Default: 1 Valid Values: [1,...] Importance: medium remote.log.manager.expiration.thread.pool.size
Size of the thread pool used in scheduling tasks to clean up the expired remote log segments.
Type: int Default: 10 Valid Values: [1,...] Importance: medium remote.log.manager.fetch.max.bytes.per.second
The maximum number of bytes that can be fetched from remote storage to local storage per second. This is a global limit for all the partitions that are being fetched from remote storage to local storage. The default value is Long.MAX_VALUE, which means there is no limit on the number of bytes that can be fetched per second.
Type: long Default: 9223372036854775807 Valid Values: [1,...] Importance: medium remote.log.manager.fetch.quota.window.num
The number of samples to retain in memory for remote fetch quota management. The default value is 11, which means there are 10 whole windows + 1 current window.
Type: int Default: 11 Valid Values: [1,...] Importance: medium remote.log.manager.fetch.quota.window.size.seconds
The time span of each sample for remote fetch quota management. The default value is 1 second.
Type: int Default: 1 Valid Values: [1,...] Importance: medium remote.log.manager.follower.thread.pool.size
Size of the thread pool used in scheduling follower tasks to read the highest-uploaded remote-offset for follower partitions.
Type: int Default: 2 Valid Values: [1,...] Importance: medium remote.log.manager.thread.pool.size
Size of the thread pool used in scheduling follower tasks to read the highest-uploaded remote-offset for follower partitions. This config is deprecated since 4.2, please use
remote.log.manager.follower.thread.pool.sizeinstead.Type: int Default: 2 Valid Values: [1,...] Importance: medium remote.log.metadata.manager.class.name
Fully qualified class name of
RemoteLogMetadataManagerimplementation.Type: string Default: org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager Valid Values: non-empty string Importance: medium remote.log.metadata.manager.class.path
Class path of the
RemoteLogMetadataManagerimplementation. If specified, the RemoteLogMetadataManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.Type: string Default: null Valid Values: Importance: medium remote.log.metadata.manager.impl.prefix
Prefix used for properties to be passed to RemoteLogMetadataManager implementation. For example this value can be
rlmm.config..Type: string Default: rlmm.config. Valid Values: non-empty string Importance: medium remote.log.metadata.manager.listener.name
Listener name of the local broker to which it should get connected if needed by RemoteLogMetadataManager implementation.
Type: string Default: null Valid Values: non-empty string Importance: medium remote.log.reader.max.pending.tasks
Maximum remote log reader thread pool task queue size. If the task queue is full, fetch requests are served with an error.
Type: int Default: 100 Valid Values: [1,...] Importance: medium remote.log.reader.threads
Size of the thread pool that is allocated for handling remote log reads.
Type: int Default: 10 Valid Values: [1,...] Importance: medium remote.log.storage.manager.class.name
Fully qualified class name of
RemoteStorageManagerimplementation.Type: string Default: null Valid Values: non-empty string Importance: medium remote.log.storage.manager.class.path
Class path of the
RemoteStorageManagerimplementation. If specified, the RemoteStorageManager implementation and its dependent libraries will be loaded by a dedicated classloader which searches this class path before the Kafka broker class path. The syntax of this parameter is same as the standard Java class path string.Type: string Default: null Valid Values: Importance: medium remote.log.storage.manager.impl.prefix
Prefix used for properties to be passed to RemoteStorageManager implementation. For example this value can be
rsm.config..Type: string Default: rsm.config. Valid Values: non-empty string Importance: medium remote.log.storage.system.enable
Whether to enable tiered storage functionality in a broker or not. When it is true broker starts all the services required for the tiered storage functionality.
Type: boolean Default: false Valid Values: Importance: medium remote.log.index.file.cache.total.size.bytes
The total size of the space allocated to store index files fetched from remote storage in the local storage.
Type: long Default: 1073741824 (1 gibibyte) Valid Values: [1,...] Importance: low remote.log.manager.task.interval.ms
Interval at which remote log manager runs the scheduled tasks like copy segments, and clean up remote log segments.
Type: long Default: 30000 (30 seconds) Valid Values: [1,...] Importance: low remote.log.metadata.custom.metadata.max.bytes
The maximum size of custom metadata in bytes that the broker should accept from a remote storage plugin. If custom metadata exceeds this limit, the updated segment metadata will not be stored, the copied data will be attempted to delete, and the remote copying task for this topic-partition will stop with an error.
Type: int Default: 128 Valid Values: [0,...] Importance: low
RLMM Configs
Below is the configuration for TopicBasedRemoteLogMetadataManager, which is the default implementation of RemoteLogMetadataManager.
All configurations here should start with the prefix defined by remote.log.metadata.manager.impl.prefix, for example, rlmm.config.remote.log.metadata.consume.wait.ms.
remote.log.metadata.consume.wait.ms
The amount of time in milliseconds to wait for the local consumer to receive the published event.
Type: long Default: 120000 (2 minutes) Valid Values: [0,...] Importance: low remote.log.metadata.initialization.retry.interval.ms
The retry interval in milliseconds for retrying RemoteLogMetadataManager resources initialization again.
Type: long Default: 100 Valid Values: [0,...] Importance: low remote.log.metadata.initialization.retry.max.timeout.ms
The maximum amount of time in milliseconds for retrying RemoteLogMetadataManager resources initialization. For TopicBasedRemoteLogMetadataManager's initialization, the timer starts after this local broker is ready to process requests (primarily for ensuring the local cluster is ready when metadata is stored locally as an internal topic). If initialization fails within this timeout, this broker process will terminate.
Type: long Default: 120000 (2 minutes) Valid Values: [0,...] Importance: low remote.log.metadata.topic.num.partitions
The number of partitions for remote log metadata topic.
Type: int Default: 50 Valid Values: [1,...] Importance: low remote.log.metadata.topic.replication.factor
Replication factor of remote log metadata topic.
Type: short Default: 3 Valid Values: [1,...] Importance: low remote.log.metadata.topic.retention.ms
Retention of remote log metadata topic in milliseconds. Default: -1, that means unlimited. Users can configure this value based on their use cases. To avoid any data loss, this value should be more than the maximum retention period of any topic enabled with tiered storage in the cluster.
Type: long Default: -1 Valid Values: Importance: low
The implementation of TopicBasedRemoteLogMetadataManager needs to create admin, producer, and consumer clients for the internal topic __remote_log_metadata.
Additional configurations can be provided for different types of clients using the following configuration properties:
# Configs for admin, producer, and consumer clients
<rlmm.prefix>.remote.log.metadata.common.client.<kafka.property> = <value>
# Configs only for producer client
<rlmm.prefix>.remote.log.metadata.producer.<kafka.property> = <value>
# Configs only for consumer client
<rlmm.prefix>.remote.log.metadata.consumer.<kafka.property> = <value>