Challenges of moving from Kafka Mirror Maker to Brooklin for SSL Kafka Brokers

Problem

Moving from Kafka mirror maker to Brooklin has its advantages written by me here. But doing this migration is not easy-breezy as it should have been. Major challenge I faced was: Making SSL connection between consumer Kafka broker and Brooklin

Solution

SSL Problem

This problem turned out to be more tricker than I had anticipated.

I have been using 1.0.2 version of Brooklin for my work. And the problem i faced was i was unable to create a SSL connection between kafkaMirroringConnector ( connector used to create a Kafka consumer to be listening to Kafka Broker ) and Kafka Broker and problem lies in the limitation of the code. After going through code, I realized the only option was to fix it and create a new build.

Technically the change belongs to just 2 file.

KafkaMirrorMakerConnectorTask.java

//Add following line to createKafkaConsumer method
properties.putIfAbsent(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, _mirrorMakerSource.isSecure() ? "SSL" : "PLAINTEXT");

server.properties

//Add below line under 'Kakfa Mirror connector Configs' . //This would be used by createConsumer under //PartitionDisciveryThread
brooklin.server.connector.kafkaMirroringConnector.consumer.security.protocol=SSL

Build datastream-kafka-connector module and replace the jar int he brooklin lib directory.

Now when creating a new brooklin task you could use kafkassl in place of kafka in source string.

Example :
bin/brooklin-rest-client.sh -o CREATE -u http://localhost:32311/ -n first-mirroring-stream -s "kafkassl://localhost:9093/^(first|second)-topic$" -c kafkaMirroringConnector -t kafkaTransportProvider -m '{"owner":"test-user","system.reuseExistingDestination":"false"}' 2>/dev/null

Note: Another method which i tried and didn’t work for was the use of consumerFactoryClassName param. This can help you create custom consumer as per the code but it caused me more issues.