You are viewing documentation for an older version (0.7) of Kafka. For up-to-date documentation, see the latest version.
Important configuration properties for the producer
Important configuration properties for the producer:
More details about producer configuration can be found in the scala class kafka.producer.ProducerConfig.
| property | default | description |
|---|---|---|
serializer.class | kafka.serializer.DefaultEncoder. This is a no-op encoder. The serialization of data to Message should be handled outside the Producer | class that implements the kafka.serializer.Encoder<T> interface, used to encode data of type T into a Kafka message |
partitioner.class | kafka.producer.DefaultPartitioner<T> - uses the partitioning strategy hash(key)%num_partitions. If key is null, then it picks a random partition. | class that implements the kafka.producer.Partitioner<K>, used to supply a custom partitioning strategy on the message key (of type K) that is specified through the ProducerData<K, T> object in the kafka.producer.Producer<T> send API |
producer.type | sync | this parameter specifies whether the messages are sent asynchronously or not. Valid values are - |
asyncfor asynchronous batching send throughkafka.producer.AyncProducersyncfor synchronous send throughkafka.producer.SyncProducer
broker.list | null. Either this parameter or zk.connect needs to be specified by the user. | For bypassing zookeeper based auto partition discovery, use this config to pass in static broker and per-broker partition information. Format-brokerid1:host1:port1, brokerid2:host2:port2. If you use this option, the partitioner.class will be ignored and each producer request will be routed to a random broker partition.zk.connect | null. Either this parameter or broker.partition.info needs to be specified by the user | For using the zookeeper based automatic broker discovery, use this config to pass in the zookeeper connection url to the zookeeper cluster where the Kafka brokers are registered.buffer.size | 102400 | the socket buffer size, in bytesconnect.timeout.ms | 5000 | the maximum time spent by kafka.producer.SyncProducer trying to connect to the kafka broker. Once it elapses, the producer throws an ERROR and stops.socket.timeout.ms | 30000 | The socket timeout in millisecondsreconnect.interval | 30000 | the number of produce requests after which kafka.producer.SyncProducer tears down the socket connection to the broker and establishes it again; this and the following property are mainly used when the producer connects to the brokers through a VIP in a load balancer; they give the producer a chance to pick up the new broker periodicallyreconnect.time.interval.ms | 10 * 1000 * 1000 | the amount of time after which kafka.producer.SyncProducer tears down the socket connection to the broker and establishes it again; negative reconnect time interval means disabling this time-based reconnect featuremax.message.size | 1000000 | the maximum number of bytes that the kafka.producer.SyncProducer can send as a single message payloadcompression.codec | 0 (No compression) | This parameter allows you to specify the compression codec for all data generated by this producer.compressed.topics | null | This parameter allows you to set whether compression should be turned on for particular topics. If the compression codec is anything other than NoCompressionCodec, enable compression only for specified topics if any. If the list of compressed topics is empty, then enable the specified compression codec for all topics. If the compression codec is NoCompressionCodec, compression is disabled for all topics.zk.read.num.retries | 3 | The producer using the zookeeper software load balancer maintains a ZK cache that gets updated by the zookeeper watcher listeners. During some events like a broker bounce, the producer ZK cache can get into an inconsistent state, for a small time period. In this time period, it could end up picking a broker partition that is unavailable. When this happens, the ZK cache needs to be updated. This parameter specifies the number of times the producer attempts to refresh this ZK cache.
Options for Asynchronous Producers (producer.type=async)queue.time | 5000 | maximum time, in milliseconds, for buffering data on the producer queue. After it elapses, the buffered data in the producer queue is dispatched to the event.handler.queue.size | 10000 | the maximum size of the blocking queue for buffering on the kafka.producer.AsyncProducerbatch.size | 200 | the number of messages batched at the producer, before being dispatched to the event.handlerevent.handler | kafka.producer.async.EventHandler<T> | the class that implements kafka.producer.async.IEventHandler<T> used to dispatch a batch of produce requests, using an instance of kafka.producer.SyncProducer.event.handler.props | null | the java.util.Properties() object used to initialize the custom event.handler through its init() APIcallback.handler | null | the class that implements kafka.producer.async.CallbackHandler<T> used to inject callbacks at various stages of the kafka.producer.AsyncProducer pipeline.callback.handler.props | null | the java.util.Properties() object used to initialize the custom callback.handler through its init() API