Skip to content

Latest commit

 

History

History
42 lines (28 loc) · 1.32 KB

README.md

File metadata and controls

42 lines (28 loc) · 1.32 KB
Raw

cenote-write

Apache Storm Topology used by cenote for data writing

Travis license

Pipeline

  • The kafka-spout Spout is connected to a kafka topic called cenoteIncoming.
  • Everytime it consumes a message, it passes it down to forwardToCassandra Bolt.
  • This bolt (which is written in Python) connects to a cassandra keyspace called cenote, and handles the data writing accoring to cenote's specs.

Run to a cluster

  • Clone the repository:
$ git clone --recurse-submodules -j8 https://github.com/AuthEceSoftEng/cenote-write.git
  • Install the requirements of cenote-cockroach:
$ cd python-src/resources/CockroachHandler
$ pip3 install -r requirements.txt

  • Set up the required .env files according to the provided samples in:

    • root directory of the repo
    • python-src/resources/CockroachHandler directory

  • Compile the source code:

$ mvn clean install package
  • Submit the topology to the cluster:
$ storm jar path/to/write-0.1.0-jar-with-dependencies.jar com.issel.cenote.WriteTopology WriteTopology