Setup for Development, Testing, Documenting¶
Attention: The current version of Spooq is designed (and tested) for Python 2.7/3.7/3.8 on ubuntu, manjaro linux and WSL2 (Windows Subsystem Linux).
- python 2.7 or python 3.7/3.8
- Java 8 (jdk8-openjdk)
- Latex (for PDF documentation)
Setting up the Environment¶
The requirements are stored in the file Pipfile separated for production and development packages.
To install the packages needed for development and testing run the following command:
$ pipenv install --dev
This will create a virtual environment in ~/.local/share/virtualenvs.
If you want to have your virtual environment installed as a sub-folder (.venv) you have to set the environment variable PIPENV_VENV_IN_PROJECT to 1.
To remove a virtual environment created with pipenv just change in the folder where you created it and execute pipenv –rm.
Activate the Virtual Environment¶
$ pipenv shell
$ exit # or close the shell
For more commands of pipenv call pipenv -h.
Creating Your Own Components¶
Implementing new extractors, transformers, or loaders is fairly straightforward. Please refer to following descriptions and examples to get an idea:
The tests are implemented with the pytest framework.
$ pipenv shell $ cd tests $ pytest
Those are the most useful plugins automatically used:
Shuffles the order of execution for the tests to avoid / discover dependencies of the tests.
Randomization is set by a seed number. To re-test the same order of execution where you found an error, just set the seed value to the same as for the failing test. To temporarily disable this feature run with pytest -p no:random-order -v
Generates an HTML for the test coverage
$ pytest --cov-report term --cov=spooq2
$ pytest --cov-report html:cov_html --cov=spooq2
This project uses Sphinx for creating its documentation. Graphs and diagrams are produced with PlantUML.
The main documentation content is defined as docstrings within the source code. To view the current documentation open docs/build/html/index.html or docs/build/latex/spooq2.pdf in your application of choice. There are symlinks in the root folder for symplicity:
Although, if you are reading this, you have probably already found the documentation…
For generating the graphs and diagrams, you need a working plantuml installation on your computer! Please refer to sphinxcontrib-plantuml.
$ cd docs $ make html $ chromium build/html/index.html
For generating documentation in the PDF format you need to have a working (pdf)latex installation on your computer! Please refer to TexLive on how to install TeX Live - a compatible latex distribution. But beware, the download size is huge!
$ cd docs $ make latexpdf $ evince build/latex/Spooq2.pdf
Themes, plugins, settings, … are defined in docs/source/conf.py.
Allows linking to other projects’ documentation. E.g., PySpark, Python2 To add an external project, at the documentation link to intersphinx_mapping in conf.py
This allows you to write CommonMark (Markdown) inside of Docutils & Sphinx projects instead of rst.