The Kafka Router allows you to consume data from multiple Kafka instances / topics and route them to other Kafka instances / topics.
- clone Kafka data e.g. consume data from a demo environment and replicate it to your local Kafka.
- proxy external Kafka instances e.g. have one whitelisted host consume from external Kafka sources and replicate data to internal Kafka instances, can also go the other way by routing data from internal Kafka sources to external Kafka instances.
- merge messages from multiple topics into one aggregate topic.
Two environment variables are used:
CONFIG_FILE: the path to the config file. Default:
/config/config.yaml. The config file can be YAML (
*.yml) or JSON (
RESSOURCE_PATH: the path to the directory for additional resources (truststores, keystores, etc.). Default:
The config file format can be YAML (
*.yml) or JSON (
*.json), auto-detected over the file suffix.
consumer-groupName of the consumer group for the Kafka Router (allowing multiple instances running concurrently and resuming data processing from committed offsets). Default:
backoff-strategy.backoff-time-secondsWhen messages were read but could not be delivered (e.g. when the target Kafka is not reachable), the messages will not be committed in the source, but message delivery is retried instead. The backoff times (list of doubles for seconds) state after how many seconds a next attempt at delivering the messages should be made for the first, second, …, attempt. Default:
0.5, 1, 2, 3, 5, 10(after the 5th retry, wait 10 seconds before trying again).
consumer-group: kafka-router backoff-strategy: backoff-time-seconds: 1,2,3,4,5,10,30,60
Configure the Kafka instances you want to work with (as a consumer or provider, or both).
kafka: map of Kafka instance key (used for routing config) with Kafka properties.
bootstrap-servers: Kafka bootstrap servers, required.
truststore-path: for TLS connections: path to the X.509 truststore.
truststore-password: password of the truststore.
keystore-path: for MTLS connections: path to the X.509 keystore.
keystore-password: password of the keystore.
properties: additional Kafka properties, as key/value pairs
kafka: city-winterthur: bootstrap-servers: localhost:9092 country-switzerland: bootstrap-servers: external.kafka-host.org:443 truststore-path: example-truststore.p12 truststore-password: SeCrEt-007! keystore-path: example-keystore.p12 keystore-password: SeCrEt-007!
Configure the message routing (from which sources you want to consume data, and where it should be published):
routes: list of routes, with the following properties:
source: Kafka instance key (see above) from where data should be consumed.
source-topic: Source topic name, regular expression (allowing to specify several source topics)
target: Kafka instance key (see above) to which data should be routed.
target-topic: Optional target topic. If absent, the same topic as the source of the message is used.
routes: # copy the sales topic from winterthur to switzerland - source: city-winterthur source-topic: sales target: country-switzerland target-topic: sales-winterthur # copy the sales topic from chur to switzerland - source: city-chur source-topic: sales target: country-switzerland target-topic: sales-chur # collect the sales from all locations to a sales-all topic - source: country-switzerland source-topic: sales-.+ target: country-switzerland target-topic: sales-all # collect all sales globally - source: country-switzerland source-topic: sales-all target: global target-topic: all-sales
Maven Build / run
Build with Maven (default goals:
This builds an Uber JAR (portable Java application with libraries included), which you can run with:
java -jar target/kafka-router.jar
Docker build / run
Build the docker image:
mvn && docker build -t pwalser75/kafka-router .
Run the docker container:
docker run -it --volume ./config:/config pwalser75/kafka-router
Complete example with Kafka-Router, Kafka and Redpanda Console (Kafka viewer):
or in detached mode:
docker-compose up -d
You can then view the logs with:
docker logs -f kafka-router
__ __ _____ ___ __ / //_/__ _/ _/ /_____ _ / _ \___ __ __/ /____ ____ / ,< / _ `/ _/ '_/ _ `/ / , _/ _ \/ // / __/ -_) __/ /_/|_|\_,_/_//_/\_\\_,_/ /_/|_|\___/\_,_/\__/\__/_/ https://github.com/pwalser75/kafka-router 13:15:49.439 INFO [main] | c.f.tools.kafkarouter.ResourceLoader - reading configuration from /config/config.yaml 13:15:49.593 INFO [main] | c.f.tools.kafkarouter.KafkaRouterMain - Configuring routes: 13:15:49.593 INFO [main] | c.f.tools.kafkarouter.KafkaRouterMain - - kafka (sales-winterthur) -> kafka (sales-switzerland) 13:15:49.712 INFO [main] | c.f.tools.kafkarouter.KafkaRouterMain - - kafka (sales-chur) -> kafka (sales-switzerland) 13:15:49.719 INFO [main] | c.f.tools.kafkarouter.KafkaRouterMain - Starting routes: 13:15:49.720 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - Joining consumer group... 13:15:49.720 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - Subscribing to initial topics: 13:15:52.758 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - - sales-winterthur 13:15:52.759 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - Joining consumer group... 13:15:52.759 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - Subscribing to initial topics: 13:15:52.763 INFO [main] | c.f.tools.kafkarouter.KafkaRouter - - sales-chur 13:15:52.764 INFO [main] | c.f.tools.kafkarouter.KafkaRouterMain - Startup complete, ready to route messages...
The Redpanda Kafka Viewer is accessible over http://localhost:9000. Here you see the topics, and can also upload test messages (select topic, then Actions>Publish Message) and check if they’re routed.
The Kafka Router also adds an additional header
stating from which source / topic / partition / offset the messages was routed: