Skip to content

Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)

License

Notifications You must be signed in to change notification settings

sparklingpandas/sparklingml

Repository files navigation

sparkling ml logo buildstatus codecov.io

sparklingml

Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)

Why?

SparklingML's goal is to expose additional machine learning stages for Spark with the pipeline interface.

Status

Super early! Come join!

Dev mailing list: https://groups.google.com/forum/#!forum/sparklingml-dev

Building

Sparkling ML consists of two components, a Python component and a Java/Scala component. The Python component depends on having the Java/Scala component pre-build which can be done by running ./build/sbt package.

The Python component depends on the package listed in requirements.txt (as well as part of setup.py). Development and testing also requires spacy, nose, codecov, pylint, and flake8.

The script build_and_package.sh builds & tests both the Scala and Python code.

For now this only works with Spark 2.3.2, it needs some changes to support other versions.

Tests

Are your DocTests failing with

Expected nothing Got: Warning: no model found for 'en' Only loading the 'en' tokenizer.

Make sure you've installed spacy & the en language pack (python -m spacy download en)

Including in your build

SparklingML is not yet ready for production use.

License

SparklingML is licensed under the Apache 2 license. Some additional components may be under a different license.

About

Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published