org.apache.kafka.connect.tools

Class VerifiableSourceTask

java.lang.Object

org.apache.kafka.connect.source.SourceTask

org.apache.kafka.connect.tools.VerifiableSourceTask

All Implemented Interfaces:

Task

public class VerifiableSourceTask
extends SourceTask

A connector primarily intended for system tests. The connector simply generates as many tasks as requested. The tasks print metadata in the form of JSON to stdout for each message generated, making externally visible which messages have been sent. Each message is also assigned a unique, increasing seqno that is passed to Kafka Connect; when tasks are started on new nodes, this seqno is used to resume where the task previously left off, allowing for testing of distributed Kafka Connect.

If logging is left enabled, log output on stdout can be easily ignored by checking whether a given line is valid JSON.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.kafka.connect.source.SourceTask

SourceTask.TransactionBoundary

Field Summary

Fields

Modifier and Type	Field and Description
static String	COMPLETE_RECORD_DATA_CONFIG
static String	ID_CONFIG
static String	NAME_CONFIG
static String	THROUGHPUT_CONFIG
static String	TOPIC_CONFIG

Fields inherited from class org.apache.kafka.connect.source.SourceTask

context, TRANSACTION_BOUNDARY_CONFIG

Constructor Summary

Constructors

Constructor and Description
VerifiableSourceTask()

Method Summary

Modifier and Type	Method and Description
void	commitRecord(SourceRecord record, RecordMetadata metadata) Commit an individual SourceRecord when the callback from the producer client is received.
List<SourceRecord>	poll() Poll this source task for new records.
void	start(Map<String,String> props) Start the Task.
void	stop() Signal this SourceTask to stop.
String	version() Get the version of this task.

Methods inherited from class org.apache.kafka.connect.source.SourceTask

commit, commitRecord, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME_CONFIG

public static final String NAME_CONFIG

See Also:

Constant Field Values

ID_CONFIG

public static final String ID_CONFIG

See Also:

Constant Field Values

TOPIC_CONFIG

public static final String TOPIC_CONFIG

See Also:

Constant Field Values

THROUGHPUT_CONFIG

public static final String THROUGHPUT_CONFIG

See Also:

Constant Field Values

COMPLETE_RECORD_DATA_CONFIG

public static final String COMPLETE_RECORD_DATA_CONFIG

See Also:

Constant Field Values

Constructor Detail

VerifiableSourceTask

public VerifiableSourceTask()

Method Detail

version

public String version()

Description copied from interface: Task

Get the version of this task. Usually this should be the same as the corresponding Connector class's version.

Returns:

the version, formatted as a String

start

public void start(Map<String,String> props)

Description copied from class: SourceTask

Start the Task. This should handle any configuration parsing and one-time setup of the task.

Specified by:

start in interface Task

Specified by:

start in class SourceTask

Parameters:

props - initial configuration

poll

public List<SourceRecord> poll()

Description copied from class: SourceTask

Poll this source task for new records. If no data is currently available, this method should block but return control to the caller regularly (by returning null) in order for the task to transition to the PAUSED state if requested to do so.

The task will be stopped on a separate thread, and when that happens this method is expected to unblock, quickly finish up any remaining processing, and return.

Specified by:

poll in class SourceTask

Returns:

a list of source records

commitRecord

public void commitRecord(SourceRecord record,
                         RecordMetadata metadata)

Description copied from class: SourceTask

Commit an individual SourceRecord when the callback from the producer client is received. This method is also called when a record is filtered by a transformation or when "errors.tolerance" is set to "all" and thus will never be ACK'd by a broker. In both cases metadata will be null.

SourceTasks are not required to implement this functionality; Kafka Connect will record offsets automatically. This hook is provided for systems that also need to store offsets internally in their own system.

The default implementation just calls SourceTask.commitRecord(SourceRecord), which is a nop by default. It is not necessary to implement both methods.

Overrides:

commitRecord in class SourceTask

Parameters:

record - SourceRecord that was successfully sent via the producer, filtered by a transformation, or dropped on producer exception

metadata - RecordMetadata record metadata returned from the broker, or null if the record was filtered or if producer exceptions are ignored

stop

public void stop()

Description copied from class: SourceTask

Signal this SourceTask to stop. In SourceTasks, this method only needs to signal to the task that it should stop trying to poll for new data and interrupt any outstanding poll() requests. It is not required that the task has fully stopped. Note that this method necessarily may be invoked from a different thread than SourceTask.poll() and SourceTask.commit().

For example, if a task uses a Selector to receive data over the network, this method could set a flag that will force SourceTask.poll() to exit immediately and invoke wakeup() to interrupt any ongoing requests.

Specified by:

stop in interface Task

Specified by:

stop in class SourceTask