Installation / Deployment

Build egg file

$ cd spooq2
$ python setup.py bdist_egg

The output is stored as dist/Spooq2-<VERSION_NUMBER>-py2.7.egg

Build zip file

$ cd spooq2
$ rm temp.zip
$ zip -r temp.zip src/spooq2
$ mv temp.zip Spooq2_$(grep "__version__" src/spooq2/_version.py | \
    cut -d " " -f 3 | tr -d \").zip

The output is stored as Spooq2-<VERSION_NUMBER>.zip.

Include pre-build package (egg or zip) with Spark

For Submitting or Launching Spark:

$ pyspark --py-files Spooq2-<VERSION_NUMBER>.egg

The library still has to be imported in the pyspark application!

Within Running Spark Session:

>>> sc.addFile("Spooq2-<VERSION_NUMBER>.egg")
>>> import spooq2

Install local repository as package

$ cd spooq2
$ python setup.py install

Install Spooq2 directly from git

$ pip install git+https://github.com/breaka84/spooq@master

Development, Testing, and Documenting

Please refer to Setup for Development, Testing, Documenting.