Package org.apache.kafka.connect.tools

Class VerifiableSourceTask

java.lang.Object
org.apache.kafka.connect.source.SourceTask
org.apache.kafka.connect.tools.VerifiableSourceTask
All Implemented Interfaces:
Task
public class VerifiableSourceTask extends SourceTask
A connector primarily intended for system tests. The connector simply generates as many tasks as requested. The tasks print metadata in the form of JSON to stdout for each message generated, making externally visible which messages have been sent. Each message is also assigned a unique, increasing seqno that is passed to Kafka Connect; when tasks are started on new nodes, this seqno is used to resume where the task previously left off, allowing for testing of distributed Kafka Connect.

If logging is left enabled, log output on stdout can be easily ignored by checking whether a given line is valid JSON.

  • Field Details

  • Constructor Details

    • VerifiableSourceTask

      public VerifiableSourceTask()

  • Method Details

    • version

      public String version()
      Description copied from interface: Task
      Get the version of this task. Usually this should be the same as the corresponding Connector class's version.
      Returns:
      the version, formatted as a String

    • start

      public void start(Map<String,String> props)
      Description copied from class: SourceTask
      Start the Task. This should handle any configuration parsing and one-time setup of the task.
      Specified by:
      start in interface Task
      Specified by:
      start in class SourceTask
      Parameters:
      props - initial configuration

    • poll

      public List<SourceRecord> poll()
      Description copied from class: SourceTask
      Poll this source task for new records. If no data is currently available, this method should block but return control to the caller regularly (by returning null) in order for the task to transition to the PAUSED state if requested to do so.

      The task will be stopped on a separate thread, and when that happens this method is expected to unblock, quickly finish up any remaining processing, and return.

      Specified by:
      poll in class SourceTask
      Returns:
      a list of source records

    • commitRecord

      public void commitRecord(SourceRecord record, RecordMetadata metadata)
      Description copied from class: SourceTask

      Commit an individual SourceRecord when the callback from the producer client is received. This method is also called when a record is filtered by a transformation or when "errors.tolerance" is set to "all" and thus will never be ACK'd by a broker. In both cases metadata will be null.

      SourceTasks are not required to implement this functionality; Kafka Connect will record offsets automatically. This hook is provided for systems that also need to store offsets internally in their own system.

      The default implementation just calls SourceTask.commitRecord(SourceRecord), which is a nop by default. It is not necessary to implement both methods.

      Overrides:
      commitRecord in class SourceTask
      Parameters:
      record - SourceRecord that was successfully sent via the producer, filtered by a transformation, or dropped on producer exception
      metadata - RecordMetadata record metadata returned from the broker, or null if the record was filtered or if producer exceptions are ignored

    • stop

      public void stop()
      Description copied from class: SourceTask
      Signal this SourceTask to stop. In SourceTasks, this method only needs to signal to the task that it should stop trying to poll for new data and interrupt any outstanding poll() requests. It is not required that the task has fully stopped. Note that this method necessarily may be invoked from a different thread than SourceTask.poll() and SourceTask.commit().

      For example, if a task uses a Selector to receive data over the network, this method could set a flag that will force SourceTask.poll() to exit immediately and invoke wakeup() to interrupt any ongoing requests.

      Specified by:
      stop in interface Task
      Specified by:
      stop in class SourceTask