org.apache.kafka.connect.source

Class SourceConnector

java.lang.Object

org.apache.kafka.connect.connector.Connector

org.apache.kafka.connect.source.SourceConnector

All Implemented Interfaces:

Versioned

Direct Known Subclasses:

MockSourceConnector, SchemaSourceConnector, VerifiableSourceConnector

public abstract class SourceConnector
extends Connector

SourceConnectors implement the connector interface to pull data from another system and send it to Kafka.

Kafka Connect may discover implementations of this interface using the Java ServiceLoader mechanism. To support this, implementations of this interface should also contain a service provider configuration file in META-INF/services/org.apache.kafka.connect.source.SourceConnector.

Field Summary

Fields inherited from class org.apache.kafka.connect.connector.Connector

context

Constructor Summary

Constructors

Constructor and Description
SourceConnector()

Method Summary

Modifier and Type	Method and Description
boolean	alterOffsets(Map<String,String> connectorConfig, Map<Map<String,?>,Map<String,?>> offsets) Invoked when users request to manually alter/reset the offsets for this connector via the Connect worker's REST API.
ConnectorTransactionBoundaries	canDefineTransactionBoundaries(Map<String,String> connectorConfig) Signals whether the connector implementation is capable of defining the transaction boundaries for a connector with the given configuration.
protected SourceConnectorContext	context() Returns the context object used to interact with the Kafka Connect runtime.
ExactlyOnceSupport	exactlyOnceSupport(Map<String,String> connectorConfig) Signals whether the connector supports exactly-once semantics with a proposed configuration.

Methods inherited from class org.apache.kafka.connect.connector.Connector

config, initialize, initialize, reconfigure, start, stop, taskClass, taskConfigs, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.connect.components.Versioned

version

Constructor Detail

SourceConnector

public SourceConnector()

Method Detail

context

protected SourceConnectorContext context()

Description copied from class: Connector

Returns the context object used to interact with the Kafka Connect runtime.

Overrides:

context in class Connector

Returns:

the context for this Connector.

exactlyOnceSupport

public ExactlyOnceSupport exactlyOnceSupport(Map<String,String> connectorConfig)

Signals whether the connector supports exactly-once semantics with a proposed configuration. Connector authors can assume that worker-level exactly-once support is enabled when this method is invoked.

For backwards compatibility, the default implementation will return null, but connector authors are strongly encouraged to override this method to return a non-null value such as SUPPORTED or UNSUPPORTED.

Similar to validate, this method may be called by the runtime before the start method is invoked when the connector will be run with exactly-once support.

Parameters:

connectorConfig - the configuration that will be used for the connector.

Returns:

ExactlyOnceSupport.SUPPORTED if the connector can provide exactly-once support with the given configuration, and ExactlyOnceSupport.UNSUPPORTED if it cannot. If this method is overridden by a connector, should not be null, but if null, it will be assumed that the connector cannot provide exactly-once semantics.

Since:

3.3

canDefineTransactionBoundaries

public ConnectorTransactionBoundaries canDefineTransactionBoundaries(Map<String,String> connectorConfig)

Signals whether the connector implementation is capable of defining the transaction boundaries for a connector with the given configuration. This method is called before Connector.start(Map), only when the runtime supports exactly-once and the connector configuration includes transaction.boundary=connector.

This method need not be implemented if the connector implementation does not support defining transaction boundaries.

Parameters:

connectorConfig - the configuration that will be used for the connector

Returns:

ConnectorTransactionBoundaries.SUPPORTED if the connector will define its own transaction boundaries, or ConnectorTransactionBoundaries.UNSUPPORTED otherwise; may never be null. The default implementation returns ConnectorTransactionBoundaries.UNSUPPORTED.

Since:

3.3

See Also:

TransactionContext

alterOffsets

public boolean alterOffsets(Map<String,String> connectorConfig,
                            Map<Map<String,?>,Map<String,?>> offsets)

Invoked when users request to manually alter/reset the offsets for this connector via the Connect worker's REST API. Connectors that manage offsets externally can propagate offset changes to their external system in this method. Connectors may also validate these offsets to ensure that the source partitions and source offsets are in a format that is recognizable to them.

Connectors that neither manage offsets externally nor require custom offset validation need not implement this method beyond simply returning true.

User requests to alter/reset offsets will be handled by the Connect runtime and will be reflected in the offsets returned by any OffsetStorageReader instances provided to this connector and its tasks.

Note that altering / resetting offsets is expected to be an idempotent operation and this method should be able to handle being called more than once with the same arguments (which could occur if a user retries the request due to a failure in writing the new offsets to the offsets store, for example).

Similar to validate, this method may be called by the runtime before the start method is invoked.

Parameters:

connectorConfig - the configuration of the connector

offsets - a map from source partition to source offset, containing the offsets that the user has requested to alter/reset. For any source partitions whose offsets are being reset instead of altered, their corresponding source offset value in the map will be null. This map may be empty, but never null. An empty offsets map could indicate that the offsets were reset previously or that no offsets have been committed yet.

Returns:

whether this method has been overridden by the connector; the default implementation returns false, and all other implementations (that do not unconditionally throw exceptions) should return true

Throws:

UnsupportedOperationException - if it is impossible to alter/reset the offsets for this connector

ConnectException - if the offsets for this connector cannot be reset for any other reason (for example, they have failed custom validation logic specific to this connector)

Since:

3.6