Package org.apache.kafka.server.log.remote.storage

Interface RemoteStorageManager

All Superinterfaces:
AutoCloseable, Closeable, Configurable
@Evolving public interface RemoteStorageManager extends Configurable, Closeable
This interface provides the lifecycle of remote log segments that includes copy, fetch, and delete from remote storage.

Each upload or copy of a segment is initiated with RemoteLogSegmentMetadata containing RemoteLogSegmentId which is universally unique even for the same topic partition and offsets.

RemoteLogSegmentMetadata is stored in RemoteLogMetadataManager before and after copy/delete operations on RemoteStorageManager with the respective RemoteLogSegmentState. RemoteLogMetadataManager is responsible for storing and fetching metadata about the remote log segments in a strongly consistent manner. This allows RemoteStorageManager to have eventual consistency on metadata (although the data is stored in strongly consistent semantics).

  • Method Details

    • copyLogSegmentData

      void copyLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata, LogSegmentData logSegmentData) throws RemoteStorageException
      Copies the given LogSegmentData provided for the given remoteLogSegmentMetadata. This includes log segment and its auxiliary indexes like offset index, time index, transaction index, leader epoch index, and producer snapshot index.

      Invoker of this API should always send a unique id as part of RemoteLogSegmentMetadata.remoteLogSegmentId() even when it retries to invoke this method for the same log segment data.

      Parameters:
      remoteLogSegmentMetadata - metadata about the remote log segment.
      logSegmentData - data to be copied to tiered storage.
      Throws:
      RemoteStorageException - if there are any errors in storing the data of the segment.

    • fetchLogSegment

      InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition) throws RemoteStorageException
      Returns the remote log segment data file/object as InputStream for the given RemoteLogSegmentMetadata starting from the given startPosition. The stream will end at the end of the remote log segment data file/object.
      Parameters:
      remoteLogSegmentMetadata - metadata about the remote log segment.
      startPosition - start position of log segment to be read, inclusive.
      Returns:
      input stream of the requested log segment data.
      Throws:
      RemoteStorageException - if there are any errors while fetching the desired segment.
      RemoteResourceNotFoundException - when there are no resources associated with the given remoteLogSegmentMetadata.

    • fetchLogSegment

      InputStream fetchLogSegment(RemoteLogSegmentMetadata remoteLogSegmentMetadata, int startPosition, int endPosition) throws RemoteStorageException
      Returns the remote log segment data file/object as InputStream for the given RemoteLogSegmentMetadata starting from the given startPosition. The stream will end at the smaller of endPosition and the end of the remote log segment data file/object.
      Parameters:
      remoteLogSegmentMetadata - metadata about the remote log segment.
      startPosition - start position of log segment to be read, inclusive.
      endPosition - end position of log segment to be read, inclusive.
      Returns:
      input stream of the requested log segment data.
      Throws:
      RemoteStorageException - if there are any errors while fetching the desired segment.
      RemoteResourceNotFoundException - when there are no resources associated with the given remoteLogSegmentMetadata.

    • fetchIndex

      InputStream fetchIndex(RemoteLogSegmentMetadata remoteLogSegmentMetadata, RemoteStorageManager.IndexType indexType) throws RemoteStorageException
      Returns the index for the respective log segment of RemoteLogSegmentMetadata.
      Parameters:
      remoteLogSegmentMetadata - metadata about the remote log segment.
      indexType - type of the index to be fetched for the segment.
      Returns:
      input stream of the requested index.
      Throws:
      RemoteStorageException - if there are any errors while fetching the index.
      RemoteResourceNotFoundException - when there are no resources associated with the given remoteLogSegmentMetadata.

    • deleteLogSegmentData

      void deleteLogSegmentData(RemoteLogSegmentMetadata remoteLogSegmentMetadata) throws RemoteStorageException
      Deletes the resources associated with the given remoteLogSegmentMetadata. Deletion is considered as successful if this call returns successfully without any errors. It will throw RemoteStorageException if there are any errors in deleting the file.

      Parameters:
      remoteLogSegmentMetadata - metadata about the remote log segment to be deleted.
      Throws:
      RemoteResourceNotFoundException - if the requested resource is not found
      RemoteStorageException - if there are any storage related errors occurred.
      RemoteResourceNotFoundException - when there are no resources associated with the given remoteLogSegmentMetadata.