org.apache.kafka.streams.processor

Class LogAndSkipOnInvalidTimestamp

java.lang.Object

org.apache.kafka.streams.processor.LogAndSkipOnInvalidTimestamp

All Implemented Interfaces:

TimestampExtractor

@InterfaceStability.Evolving
public class LogAndSkipOnInvalidTimestamp
extends java.lang.Object

Retrieves embedded metadata timestamps from Kafka messages. If a record has a negative (invalid) timestamp value the timestamp is returned as-is; in addition, a WARN message is logged in your application. Returning the timestamp as-is results in dropping the record, i.e., the record will not be processed.

Embedded metadata timestamp was introduced in "KIP-32: Add timestamps to Kafka message" for the new 0.10+ Kafka message format.

Here, "embedded metadata" refers to the fact that compatible Kafka producer clients automatically and transparently embed such timestamps into message metadata they send to Kafka, which can then be retrieved via this timestamp extractor.

If the embedded metadata timestamp represents CreateTime (cf. Kafka broker setting message.timestamp.type and Kafka topic setting log.message.timestamp.type), this extractor effectively provides event-time semantics. If LogAppendTime is used as broker/topic setting to define the embedded metadata timestamps, using this extractor effectively provides ingestion-time semantics.

If you need processing-time semantics, use WallclockTimestampExtractor.

See Also:

FailOnInvalidTimestamp, UsePreviousTimeOnInvalidTimestamp, WallclockTimestampExtractor

Constructor Summary

Constructors

Constructor and Description
LogAndSkipOnInvalidTimestamp()

Method Summary

Modifier and Type	Method and Description
long	extract(ConsumerRecord<java.lang.Object,java.lang.Object> record, long previousTimestamp) Extracts the embedded metadata timestamp from the given ConsumerRecord.
long	onInvalidTimestamp(ConsumerRecord<java.lang.Object,java.lang.Object> record, long recordTimestamp, long previousTimestamp) Writes a log WARN message when the extracted timestamp is invalid (negative) but returns the invalid timestamp as-is, which ultimately causes the record to be skipped and not to be processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LogAndSkipOnInvalidTimestamp

public LogAndSkipOnInvalidTimestamp()

Method Detail

onInvalidTimestamp

public long onInvalidTimestamp(ConsumerRecord<java.lang.Object,java.lang.Object> record,
                               long recordTimestamp,
                               long previousTimestamp)

Writes a log WARN message when the extracted timestamp is invalid (negative) but returns the invalid timestamp as-is, which ultimately causes the record to be skipped and not to be processed.

Parameters:

record - a data record

recordTimestamp - the timestamp extractor from the record

previousTimestamp - the latest extracted valid timestamp of the current record's partition˙ (could be -1 if unknown)

Returns:

the originally extracted timestamp of the record

extract

public long extract(ConsumerRecord<java.lang.Object,java.lang.Object> record,
                    long previousTimestamp)

Extracts the embedded metadata timestamp from the given ConsumerRecord.

Specified by:

extract in interface TimestampExtractor

Parameters:

record - a data record

previousTimestamp - the latest extracted valid timestamp of the current record's partition˙ (could be -1 if unknown)

Returns:

the embedded metadata timestamp of the given ConsumerRecord