Skip to content

ops-baidu/jruby-kafka

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Jruby::Kafka

Prerequisites

About

This gem is primarily used to wrap most of the Kafka 0.8.0 high level consumer and Kafka 0.8.0 producer API into jruby. The Kafka Consumer Group Example is pretty much ported to this library.

Installation

This package is now distruted via rubygems.org but you can build it using the following instructions.

From the root of the project run:

$ rake package

You can run the following to install the resulting package:

$ gem install jruby-kafka*.gem

Add this line to your application's Gemfile:

gem 'jruby-kafka'

Usage

If you want to run the tests, make sure you already have downloaded Kafka 0.8.0, followed the kafka quickstart instructions and have KAFKA_PATH set in the environment.

Using in irb

make a producer

jar_dir = "path/to/dir/with/kafka/jars"

include Java
Dir.glob(File.join(jar_dir, "*.jar")) { |jar|
  $CLASSPATH << jar
}
require 'jruby-kafka'

producer_options = {:zk_connect => "localhost:2181", :topic_id => "test", :broker_list => "localhost:9092"}
producer = Kafka::Producer.new(producer_options)
producer.connect()
producer.sendMsg(nil, "heres a test")

then a consumer

include Java
Dir.glob(File.join(jar_dir, "*.jar")) { |jar|
  $CLASSPATH << jar
}
require 'jruby-kafka'
queue = SizedQueue.new(20)
group = Kafka::Group.new(options)
group.run(1,queue)
Java::JavaLang::Thread.sleep 3000

#just gets first 20 things & prints out
until queue.empty?
  puts(queue.pop)
end

group.shutdown()

Using in logstash:

Check out this repo: https://github.com/joekiller/logstash-kafka

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published