@InterfaceStability.Unstable public class KafkaStreams extends Object
The computational logic can be specified either by using the TopologyBuilder
to define a DAG topology of
Processor
s or by using the KStreamBuilder
which provides the high-level DSL to define transformations.
One KafkaStreams
instance can contain one or more threads specified in the configs for the processing work.
A KafkaStreams
instance can co-ordinate with any other instances with the same
application ID
(whether in the same process, on other processes on this
machine, or on remote machines) as a single (possibly distributed) stream processing application.
These instances will divide up the work based on the assignment of the input topic partitions so that all partitions
are being consumed.
If instances are added or fail, all (remaining) instances will rebalance the partition assignment among themselves
to balance processing load and ensure that all input topic partitions are processed.
Internally a KafkaStreams
instance contains a normal KafkaProducer
and KafkaConsumer
instance
that is used for reading input and writing output.
A simple example might look like this:
Map<String, Object> props = new HashMap<>();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-stream-processing-application");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
StreamsConfig config = new StreamsConfig(props);
KStreamBuilder builder = new KStreamBuilder();
builder.stream("my-input-topic").mapValues(value -> value.length().toString()).to("my-output-topic");
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
KStreamBuilder
,
TopologyBuilder
Modifier and Type | Class and Description |
---|---|
static class |
KafkaStreams.State
Kafka Streams states are the possible state that a Kafka Streams instance can be in.
|
static interface |
KafkaStreams.StateListener
Listen to
KafkaStreams.State change events. |
Constructor and Description |
---|
KafkaStreams(TopologyBuilder builder,
Properties props)
Create a
KafkaStreams instance. |
KafkaStreams(TopologyBuilder builder,
StreamsConfig config)
Create a
KafkaStreams instance. |
KafkaStreams(TopologyBuilder builder,
StreamsConfig config,
KafkaClientSupplier clientSupplier)
Create a
KafkaStreams instance. |
Modifier and Type | Method and Description |
---|---|
Collection<StreamsMetadata> |
allMetadata()
Find all currently running
KafkaStreams instances (potentially remotely) that use the same
application ID as this instance (i.e., all instances that belong to
the same Kafka Streams application) and return StreamsMetadata for each discovered instance. |
Collection<StreamsMetadata> |
allMetadataForStore(String storeName)
Find all currently running
KafkaStreams instances (potentially remotely) that
use the same application ID as this instance (i.e., all
instances that belong to the same Kafka Streams application)
and that contain a StateStore with the given storeName
and return StreamsMetadata for each discovered instance. |
void |
cleanUp()
Do a clean up of the local
StateStore directory (StreamsConfig.STATE_DIR_CONFIG ) by deleting all
data with regard to the application ID . |
void |
close()
Shutdown this
KafkaStreams instance by signaling all the threads to stop, and then wait for them to join. |
boolean |
close(long timeout,
TimeUnit timeUnit)
Shutdown this
KafkaStreams by signaling all the threads to stop, and then wait up to the timeout for the
threads to join. |
<K> StreamsMetadata |
metadataForKey(String storeName,
K key,
Serializer<K> keySerializer)
Find the currently running
KafkaStreams instance (potentially remotely) that
use the same application ID as this instance (i.e., all
instances that belong to the same Kafka Streams application)
and that contain a StateStore with the given storeName
and the StateStore contains the given key
and return StreamsMetadata for it. |
<K> StreamsMetadata |
metadataForKey(String storeName,
K key,
StreamPartitioner<? super K,?> partitioner)
Find the currently running
KafkaStreams instance (potentially remotely) that
use the same application ID as this instance (i.e., all
instances that belong to the same Kafka Streams application)
and that contain a StateStore with the given storeName
and the StateStore contains the given key
and return StreamsMetadata for it. |
Map<MetricName,? extends Metric> |
metrics()
Get read-only handle on global metrics registry.
|
void |
setStateListener(KafkaStreams.StateListener listener)
An app can set a single
KafkaStreams.StateListener so that the app is notified when state changes. |
void |
setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh)
Set the handler invoked when a
internal thread abruptly
terminates due to an uncaught exception. |
void |
start()
Start the
KafkaStreams instance by starting all its threads. |
KafkaStreams.State |
state()
Return the current
KafkaStreams.State of this KafkaStreams instance. |
<T> T |
store(String storeName,
QueryableStoreType<T> queryableStoreType)
Get a facade wrapping the local
StateStore instances with the provided storeName if the Store's
type is accepted by the provided queryableStoreType . |
String |
toString()
Produce a string representation containing useful information about this
KafkaStream instance such as
thread IDs, task IDs, and a representation of the topology DAG including StateStore s (cf. |
String |
toString(String indent)
Produce a string representation containing useful information about this
KafkaStream instance such as
thread IDs, task IDs, and a representation of the topology DAG including StateStore s (cf. |
public KafkaStreams(TopologyBuilder builder, Properties props)
KafkaStreams
instance.builder
- the processor topology builder specifying the computational logicprops
- properties for StreamsConfig
public KafkaStreams(TopologyBuilder builder, StreamsConfig config)
KafkaStreams
instance.builder
- the processor topology builder specifying the computational logicconfig
- the Kafka Streams configurationpublic KafkaStreams(TopologyBuilder builder, StreamsConfig config, KafkaClientSupplier clientSupplier)
KafkaStreams
instance.builder
- the processor topology builder specifying the computational logicconfig
- the Kafka Streams configurationclientSupplier
- the Kafka clients supplier which provides underlying producer and consumer clients
for the new KafkaStreams
instancepublic void setStateListener(KafkaStreams.StateListener listener)
KafkaStreams.StateListener
so that the app is notified when state changes.listener
- a new state listenerpublic KafkaStreams.State state()
KafkaStreams.State
of this KafkaStreams
instance.public Map<MetricName,? extends Metric> metrics()
public void start() throws IllegalStateException, StreamsException
KafkaStreams
instance by starting all its threads.
Note, for brokers with version 0.9.x
or lower, the broker version cannot be checked.
There will be no error and the client will hang and retry to verify the broker version until it
times out
.
IllegalStateException
- if process was already startedStreamsException
- if the Kafka brokers have version 0.10.0.xpublic void close()
KafkaStreams
instance by signaling all the threads to stop, and then wait for them to join.
This will block until all threads have stopped.public boolean close(long timeout, TimeUnit timeUnit)
KafkaStreams
by signaling all the threads to stop, and then wait up to the timeout for the
threads to join.
A timeout
of 0 means to wait forever.timeout
- how long to wait for the threads to shutdowntimeUnit
- unit of time used for timeouttrue
if all threads were successfully stopped—false
if the timeout was reached
before all threads stoppedpublic String toString()
KafkaStream
instance such as
thread IDs, task IDs, and a representation of the topology DAG including StateStore
s (cf.
TopologyBuilder
and KStreamBuilder
).public String toString(String indent)
KafkaStream
instance such as
thread IDs, task IDs, and a representation of the topology DAG including StateStore
s (cf.
TopologyBuilder
and KStreamBuilder
).indent
- the top-level indent for each linepublic void cleanUp()
StateStore
directory (StreamsConfig.STATE_DIR_CONFIG
) by deleting all
data with regard to the application ID
.
May only be called either before this KafkaStreams
instance is started
or after the
instance is closed
.
Calling this method triggers a restore of local StateStore
s on the next application start
.
IllegalStateException
- if the instance is currently runningpublic void setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh)
internal thread
abruptly
terminates due to an uncaught exception.eh
- the uncaught exception handler for all internal threads; null
deletes the current handlerpublic Collection<StreamsMetadata> allMetadata()
KafkaStreams
instances (potentially remotely) that use the same
application ID
as this instance (i.e., all instances that belong to
the same Kafka Streams application) and return StreamsMetadata
for each discovered instance.
Note: this is a point in time view and it may change due to partition reassignment.
StreamsMetadata
for each KafkaStreams
instances of this applicationpublic Collection<StreamsMetadata> allMetadataForStore(String storeName)
KafkaStreams
instances (potentially remotely) that
application ID
as this instance (i.e., all
instances that belong to the same Kafka Streams application)StateStore
with the given storeName
StreamsMetadata
for each discovered instance.
Note: this is a point in time view and it may change due to partition reassignment.
storeName
- the storeName
to find metadata forStreamsMetadata
for each KafkaStreams
instances with the provide storeName
of
this applicationpublic <K> StreamsMetadata metadataForKey(String storeName, K key, Serializer<K> keySerializer)
KafkaStreams
instance (potentially remotely) that
application ID
as this instance (i.e., all
instances that belong to the same Kafka Streams application)StateStore
with the given storeName
StateStore
contains the given key
StreamsMetadata
for it.
This will use the default Kafka Streams partitioner to locate the partition.
If a custom partitioner
has been
configured
via StreamsConfig
,
KStream.through(StreamPartitioner, String)
, or KTable.through(StreamPartitioner, String, String)
,
or if the original KTable
's input topic
is partitioned
differently, please use metadataForKey(String, Object, StreamPartitioner)
.
Note:
StateStore
; this method provides a way of finding which host it
would exist onK
- key typestoreName
- the storeName
to find metadata forkey
- the key to find metadata forkeySerializer
- serializer for the keyStreamsMetadata
for the KafkaStreams
instance with the provide storeName
and
key
of this application or StreamsMetadata.NOT_AVAILABLE
if Kafka Streams is (re-)initializingpublic <K> StreamsMetadata metadataForKey(String storeName, K key, StreamPartitioner<? super K,?> partitioner)
KafkaStreams
instance (potentially remotely) that
application ID
as this instance (i.e., all
instances that belong to the same Kafka Streams application)StateStore
with the given storeName
StateStore
contains the given key
StreamsMetadata
for it.
Note:
StateStore
; this method provides a way of finding which host it
would exist onK
- key typestoreName
- the storeName
to find metadata forkey
- the key to find metadata forpartitioner
- the partitioner to be use to locate the host for the keyStreamsMetadata
for the KafkaStreams
instance with the provide storeName
and
key
of this application or StreamsMetadata.NOT_AVAILABLE
if Kafka Streams is (re-)initializingpublic <T> T store(String storeName, QueryableStoreType<T> queryableStoreType)
StateStore
instances with the provided storeName
if the Store's
type is accepted by the provided queryableStoreType
.
The returned object can be used to query the StateStore
instances.T
- return typestoreName
- name of the store to findqueryableStoreType
- accept only stores that are accepted by QueryableStoreType.accepts(StateStore)
StateStore
instancesInvalidStateStoreException
- if Kafka Streams is (re-)initializing or a store with storeName
and
queryableStoreType
doesnt' exist