Package org.apache.kafka.connect.source

Class SourceConnector

java.lang.Object
org.apache.kafka.connect.connector.Connector
org.apache.kafka.connect.source.SourceConnector
All Implemented Interfaces:
Versioned
Direct Known Subclasses:
MockSourceConnector, SchemaSourceConnector, VerifiableSourceConnector
public abstract class SourceConnector extends Connector
SourceConnectors implement the connector interface to pull data from another system and send it to Kafka.

Kafka Connect may discover implementations of this interface using the Java ServiceLoader mechanism. To support this, implementations of this interface should also contain a service provider configuration file in META-INF/services/org.apache.kafka.connect.source.SourceConnector.

  • Constructor Details

    • SourceConnector

      public SourceConnector()

  • Method Details

    • exactlyOnceSupport

      public ExactlyOnceSupport exactlyOnceSupport(Map<String,String> connectorConfig)
      Signals whether the connector supports exactly-once semantics with a proposed configuration. Connector authors can assume that worker-level exactly-once support is enabled when this method is invoked.

      For backwards compatibility, the default implementation will return null, but connector authors are strongly encouraged to override this method to return a non-null value such as SUPPORTED or UNSUPPORTED.

      Similar to validate, this method may be called by the runtime before the start method is invoked when the connector will be run with exactly-once support.

      Parameters:
      connectorConfig - the configuration that will be used for the connector.
      Returns:
      ExactlyOnceSupport.SUPPORTED if the connector can provide exactly-once support with the given configuration, and ExactlyOnceSupport.UNSUPPORTED if it cannot. If this method is overridden by a connector, should not be null, but if null, it will be assumed that the connector cannot provide exactly-once semantics.
      Since:
      3.3

    • canDefineTransactionBoundaries

      public ConnectorTransactionBoundaries canDefineTransactionBoundaries(Map<String,String> connectorConfig)
      Signals whether the connector implementation is capable of defining the transaction boundaries for a connector with the given configuration. This method is called before Connector.start(Map), only when the runtime supports exactly-once and the connector configuration includes transaction.boundary=connector.

      This method need not be implemented if the connector implementation does not support defining transaction boundaries.

      Parameters:
      connectorConfig - the configuration that will be used for the connector
      Returns:
      ConnectorTransactionBoundaries.SUPPORTED if the connector will define its own transaction boundaries, or ConnectorTransactionBoundaries.UNSUPPORTED otherwise; may never be null. The default implementation returns ConnectorTransactionBoundaries.UNSUPPORTED.
      Since:
      3.3
      See Also:

    • alterOffsets

      public boolean alterOffsets(Map<String,String> connectorConfig, Map<Map<String,?>,Map<String,?>> offsets)
      Invoked when users request to manually alter/reset the offsets for this connector via the Connect worker's REST API. Connectors that manage offsets externally can propagate offset changes to their external system in this method. Connectors may also validate these offsets to ensure that the source partitions and source offsets are in a format that is recognizable to them.

      Connectors that neither manage offsets externally nor require custom offset validation need not implement this method beyond simply returning true.

      User requests to alter/reset offsets will be handled by the Connect runtime and will be reflected in the offsets returned by any OffsetStorageReader instances provided to this connector and its tasks.

      Note that altering / resetting offsets is expected to be an idempotent operation and this method should be able to handle being called more than once with the same arguments (which could occur if a user retries the request due to a failure in writing the new offsets to the offsets store, for example).

      Similar to validate, this method may be called by the runtime before the start method is invoked.

      Parameters:
      connectorConfig - the configuration of the connector
      offsets - a map from source partition to source offset, containing the offsets that the user has requested to alter/reset. For any source partitions whose offsets are being reset instead of altered, their corresponding source offset value in the map will be null. This map may be empty, but never null. An empty offsets map could indicate that the offsets were reset previously or that no offsets have been committed yet.
      Returns:
      whether this method has been overridden by the connector; the default implementation returns false, and all other implementations (that do not unconditionally throw exceptions) should return true
      Throws:
      UnsupportedOperationException - if it is impossible to alter/reset the offsets for this connector
      ConnectException - if the offsets for this connector cannot be reset for any other reason (for example, they have failed custom validation logic specific to this connector)
      Since:
      3.6