As mentioned on the following blog post by Lucas Jellema, Kafka is going to play a part in several Oracle products. For some usecases it might eventually even replace JMS. In order to allow for easy integration with Kafka, you can use Oracle Service Bus to create a virtualization layer around Kafka. Ricardo Ferreira from Oracle's A-Team has done some great work on making a custom Kafka Service Bus transport available to us. Read more about this here: http://www.ateam-oracle.com/
In this blog I summarize what I have done to get the end to end sample working for SOA Suite 12.2.1.2.0 and Kafka 0.10.1.0 based on the blogs I mentioned. This allows you to quickly start developing against Apache Kafka.
Setting up Apache Kafka
Setting up Apache Kafka for development is easy. You follow the quickstart on: https://kafka.apache.org/
- Download Apache Kafka: https://www.apache.org/dyn/
- Unzip it: tar -xzf kafka_2.11-0.10.1.0.tgz
- Go to the Kafka directory: cd kafka_2.11-0.10.1.0
- Start ZooKeeper: bin/zookeeper-server-start.sh config/zookeeper.properties
- Start a new console
- Start the Kafka broker: bin/kafka-server-start.sh config/server.properties
- Create a topic: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Setting up the Kafka transport in OSB
Copy the following files:
– $KAFKA_HOME/libs/slf4j-api-1.7.21.jar
– $KAFKA_HOME/libs/kafka-clients-0.10.1.0.jar
To $OSB_DOMAIN/lib.
In my case this was /home/oracle/.jdeveloper/system12.2.1.2.42.161008.1648/DefaultDomain/lib. I'm using the JDeveloper IntegratedWebLogicServer
In my case this was /home/oracle/.jdeveloper/system12.2.1.2.42.161008.1648/DefaultDomain/lib. I'm using the JDeveloper IntegratedWebLogicServer
Download the Kafka transport from here: http://www.ateam-oracle.com/
Extract the zip file.
Copy kafka-transport.ear and kafka-transport.jar to $MW_HOME/osb/lib/transports.
Start the domain
Execute install.py from the kafka-transport zipfile. Use wlst.sh in my case from: /home/oracle/Oracle/Middleware12212/Oracle_Home/oracle_common/common/bin/wlst.sh
Provide the required information. It will ask for Url, username, password of your WebLogic server and deploy the kafka-transport.jar and kafka-transport.ear to the specified server (AdminServer + cluster targets). If the deployments are already there, they are first undeployed by the script.
Stop the domain
Locate the following file: $MW_HOME/osb/lib/osbconsoleEar/webapp/WEB-INF/lib/adflib_osb_folder.jar.
Add the following entries:
desc.res.gallery.kafka=The Kafka transport allows you to create proxy and business services that communicate with Apache Kafka brokers.
desc.res.gallery.kafka.proxy=The Kafka transport allows you to create proxy services that receive messages from Apache Kafka brokers.
desc.res.gallery.kafka.business=The Kafka transport allows you to create business services that route messages to Apache Kafka brokers.
ZIP up the result as a new adflib_osb_folder.jar
Check the Service Bus console
After the above steps are completed, you can start the domain and use the Kafka transport from the servicebus console.
Setting up JDeveloper
Copy the JDeveloper plugin descriptor (transport-kafka.xml) to the plugins folder:
$MW_HOME/osb/config/plugins. In my case this is: /home/oracle/Oracle/
You will not see possible options for consumer or producer settings but you can use the settings from: https://kafka.apache.org/documentation#consumerapi
Apache Kafka provides shell scripts to test producing and consuming messages:
- Producing: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
- Consuming: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
It helps to add a report, log or alert action to your Service Bus
pipeline so you can see messages which have passed. As a report key I
have used the Kafka offset from $inbound:
./ctx:transport/ctx:request/tp:headers/kafka:offset
As you can see, several steps need to be performed to install this custom transport. It is only supported on a best-effort basis by the A-Team. I could not see options for properties in the Service Bus Console as was shown in the blog posts mentioned at the start of this post, but that is not a real issue since if a fixed set would be provided and more options would become available in a new version of Kafka, this might become limiting. It is a shame custom transports are not visible in the component palette in JDeveloper. Once you know however you can use the Kafka transport by creating Proxy and Business services from File, New this also becomes a non-issue.
There are of course other solutions to take care of the integration with Kafka such as using Kafka connectors or create a custom service to wrap Kafka, but I like the way this custom transport allows you to integrate it with Service Bus. This allows you to make Kafka only available through this channel. This offers options like easily applying policies, monitoring, alerting, etc. I do expect in Oracle's Cloud offering interaction with Kafka products running in the Oracle Cloud such as the Event Hub, will of course be much easier. We're looking forward to it.
There are of course other solutions to take care of the integration with Kafka such as using Kafka connectors or create a custom service to wrap Kafka, but I like the way this custom transport allows you to integrate it with Service Bus. This allows you to make Kafka only available through this channel. This offers options like easily applying policies, monitoring, alerting, etc. I do expect in Oracle's Cloud offering interaction with Kafka products running in the Oracle Cloud such as the Event Hub, will of course be much easier. We're looking forward to it.
No comments:
Post a Comment