The Apache Software Foundation has officially released version 2.3 of Apache Kafka. The update to the Message Broker, which is designed for real-time processing of large amounts of data, contains many new features, which can be traced back to Kafka Improvement Proposals (KIP). In addition, bug fixes and improvements from a number of JIRA tickets have been incorporated.

Elastic Stack 7.2 controls security information and event management (SIEM)

Kafka Connect with incremental cooperative rebalancing

With necessary configurations of the connectors, Kafka Connect now masters an uninterrupted realignment of the worker threads (KIP-415). The so-called rebalancing of the worker tasks distributed across the worker nodes ensures that the load of all workers remains evenly distributed across the connect cluster. In Kafka versions up to 2.2, deploying a new connector, reconfiguring it, or adding or removing workers always resulted in downtime during rebalancing. As of Kafka 2.3, this process is “incremental cooperative” and therefore without interruptions.

Programming Language Julia, an alternative to Python, R and Matlab

More meaningful logs for Kafka Connect

KIP-449 will provide better insights into the Worker Logs. The log messages receive additional information that makes it easier to read the status of a single connector. Due to the asynchronous operation, the logs originating from the thread pools distributed in the connect cluster so far allowed only limited conclusions about the individual logical operations.

In Kafka Streams, users can now also save timestamps in RocksDB. In older versions, it was only possible to store keys and values ​​in the status memory. KIP-258 is also intended to provide the basis for handling out-of-order messages and TTLs in KTables in the future. For cases in which higher performance is more important than persistence on the disk, Kafka Streams also has in-memory implementations for the Window and the Session Store. These were previously restricted to the state store.

Here is Raspberry Pi 4, with 4 GB of RAM and 4K output

Improved handling of replicas

Among the innovations in the Kafka Core are mainly improvements in dealing with Replicas to mention. KIP-351 and KIP-427 provide advanced information about monitoring the backup replicas distributed across different brokers. In order to keep track of the critical size of the minimum required number of in-sync replicas (minimum ISR) that Kafka requires to allow write processes to the partitions, KIP-427 provides supplementary metrics that alert users in a timely manner. In addition, KIP-351 extends the kafka-topics command with the -under-min-isr flag, making it easier for users to spot topics with too few in-sync replicas.

For more information about what’s new in the Big Data framework, see the Apache Software Foundation blog post announcing the new release, and the Apache Kafka project page, which includes Release 2.3 for download.

The results of the Galaxy Zoo project force us to revise the theory of the formation of spiral arms of galaxies