Apache Kafka Plug-in
Apache Kafka Connection
The Apache Kafka Software from the Apache Foundation is a free software for streaming data packets. A Kafka system is composed of clusters that connect and interlink to provide a messaging system that allows data to be transported losslessly from sender (producer) to receiver (consumer). The system works according to the logic of Publish/Subscribe. Brokers hold data for consumers so that processing can be controlled by the client. The order of messages remains the same. Messages are marked in Kafka with a topic and optionally with a key. Sent messages contain a payload whose format is not provided, similar to MQTT.
You can find a dedicated video about this topic by visiting our tutorial stream.
These and other customers rely on the OPC Router
That’s how easy it is.
Apache Kafka connection via drag & drop
Connecting your systems via the Apache Kafka software is straightforward and intuitive with the OPC Router using drag & drop. The two screenshots show you an example of the Kafka application. In the first screenshot, data is fetched with the help of KepServerEX, translated into JSON and sent to an Apache Kafka Broker. The second screenshot shows how data is received from an Apache Kafka, evaluated in JSON and transferred to an MS-SQL database.
Bringing Industry 4.0 to life
- Achieving more with coupled systems:
Coupled systems can exchange information in real time and thus immediately adapt to all requirements. Waiting times and inefficiencies are eliminated by including and viewing all available information. - Industry 4.0 with Apache Kafka:
Industry 4.0 means the best possible networking, self-control and self-optimisation of plants based on information. Apache Kafka provides the necessary infrastructure to achieve this goal on a large scale. - About OPC Router:
OPC Router provides a powerful link for industrial systems to Apache Kafka. Its graphical configuration and robust runtime make it ideal for industrial use.
Apache Kafka Plug-in application examples
Master data distribution across all plants
For many processes, material master data is an important basis for the parameterisation of individual production processes. For a fast and autonomous decision, these are kept close to the plant as shadow copies to avoid problems and delays when retrieving central data. With Apache Kafka, master data can be made available securely to all interested “consumers”. A cyclical transmission, for example as a JSON package, ensures that the current status is always available directly in the databases at the plants. There is no dependence on the central system. Current production parameters can also be sent back from the plants in the same way, so that an intelligent evaluation and processing can take place centrally (Big Data/AI).
Process Data Streaming
Kafka offers a high degree of security and tolerance towards connection problems for central monitoring over large geographical distances and across a wide variety of network architectures. If OPC Router is used to transfer current status/fault and runtime information from systems to Kafka, the cluster system ensures that data is transferred correctly and seamlessly to the central monitoring system. Compared to lightweight IoT infrastructures such as MQTT, Kafka ensures reliable transmission, as is necessary for industrial messaging systems.
Connecting Systems with the OPC Router Apache Kafka Plug-in
Transfer-Objects
The transfer object maps the Kafka producer in the OPC Router. It offers the fields Topic, Key and Payload. Messages are marked with topic and key so that consumers are informed about new messages if they have subscribed to the respective topic. Any data block can be transferred to the payload in OPC Router Connection. Common sources are the JSON or the XML object.
Trigger
If OPC Router should act as a Kafka Consumer, the Kafka Trigger is used. A topic configured for the trigger, which is used by OPC Router to register as a subscriber in Kafka. The trigger activates its connection, in which it is configured, as soon as a new message is available for the topic. If several messages are available, the trigger is retriggered for each message. Even after a system restart, the trigger retrieves all messages not yet retrieved from broker.
Browsing
Browsing of the topics in the broker is possible in the transfer object and in the trigger. After successful configuration of the broker connection, available topics are loaded and made accessible to the operator in list form for selection.