Interface RemoteStorageManager
- All Superinterfaces:
AutoCloseable
,Closeable
,Configurable
Each upload or copy of a segment is initiated with RemoteLogSegmentMetadata
containing RemoteLogSegmentId
which is universally unique even for the same topic partition and offsets.
RemoteLogSegmentMetadata
is stored in RemoteLogMetadataManager
before and after copy/delete operations on
RemoteStorageManager
with the respective RemoteLogSegmentState
. RemoteLogMetadataManager
is
responsible for storing and fetching metadata about the remote log segments in a strongly consistent manner.
This allows RemoteStorageManager
to have eventual consistency on metadata (although the data is stored
in strongly consistent semantics).
-
Nested Class Summary
Modifier and TypeInterfaceDescriptionstatic enum
Type of the index file. -
Method Summary
Modifier and TypeMethodDescriptionvoid
copyLogSegmentData
(RemoteLogSegmentMetadata remoteLogSegmentMetadata, LogSegmentData logSegmentData) Copies the givenLogSegmentData
provided for the givenremoteLogSegmentMetadata
.void
deleteLogSegmentData
(RemoteLogSegmentMetadata remoteLogSegmentMetadata) Deletes the resources associated with the givenremoteLogSegmentMetadata
.fetchIndex
(RemoteLogSegmentMetadata remoteLogSegmentMetadata, RemoteStorageManager.IndexType indexType) Returns the index for the respective log segment ofRemoteLogSegmentMetadata
.fetchLogSegment
(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition) Returns the remote log segment data file/object as InputStream for the givenRemoteLogSegmentMetadata
starting from the given startPosition.fetchLogSegment
(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition, int endPosition) Returns the remote log segment data file/object as InputStream for the givenRemoteLogSegmentMetadata
starting from the given startPosition.Methods inherited from interface org.apache.kafka.common.Configurable
configure
-
Method Details
-
copyLogSegmentData
void copyLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata, LogSegmentData logSegmentData) throws RemoteStorageException Copies the givenLogSegmentData
provided for the givenremoteLogSegmentMetadata
. This includes log segment and its auxiliary indexes like offset index, time index, transaction index, leader epoch index, and producer snapshot index.Invoker of this API should always send a unique id as part of
RemoteLogSegmentMetadata.remoteLogSegmentId()
even when it retries to invoke this method for the same log segment data.- Parameters:
remoteLogSegmentMetadata
- metadata about the remote log segment.logSegmentData
- data to be copied to tiered storage.- Throws:
RemoteStorageException
- if there are any errors in storing the data of the segment.
-
fetchLogSegment
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition) throws RemoteStorageException Returns the remote log segment data file/object as InputStream for the givenRemoteLogSegmentMetadata
starting from the given startPosition. The stream will end at the end of the remote log segment data file/object.- Parameters:
remoteLogSegmentMetadata
- metadata about the remote log segment.startPosition
- start position of log segment to be read, inclusive.- Returns:
- input stream of the requested log segment data.
- Throws:
RemoteStorageException
- if there are any errors while fetching the desired segment.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.
-
fetchLogSegment
InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition, int endPosition) throws RemoteStorageException Returns the remote log segment data file/object as InputStream for the givenRemoteLogSegmentMetadata
starting from the given startPosition. The stream will end at the smaller of endPosition and the end of the remote log segment data file/object.- Parameters:
remoteLogSegmentMetadata
- metadata about the remote log segment.startPosition
- start position of log segment to be read, inclusive.endPosition
- end position of log segment to be read, inclusive.- Returns:
- input stream of the requested log segment data.
- Throws:
RemoteStorageException
- if there are any errors while fetching the desired segment.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.
-
fetchIndex
InputStream fetchIndex(RemoteLogSegmentMetadata remoteLogSegmentMetadata, RemoteStorageManager.IndexType indexType) throws RemoteStorageException Returns the index for the respective log segment ofRemoteLogSegmentMetadata
.- Parameters:
remoteLogSegmentMetadata
- metadata about the remote log segment.indexType
- type of the index to be fetched for the segment.- Returns:
- input stream of the requested index.
- Throws:
RemoteStorageException
- if there are any errors while fetching the index.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.
-
deleteLogSegmentData
void deleteLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata) throws RemoteStorageException Deletes the resources associated with the givenremoteLogSegmentMetadata
. Deletion is considered as successful if this call returns successfully without any errors. It will throwRemoteStorageException
if there are any errors in deleting the file.- Parameters:
remoteLogSegmentMetadata
- metadata about the remote log segment to be deleted.- Throws:
RemoteResourceNotFoundException
- if the requested resource is not foundRemoteStorageException
- if there are any storage related errors occurred.RemoteResourceNotFoundException
- when there are no resources associated with the given remoteLogSegmentMetadata.
-