Welcome to Spooq’s documentation!

Spooq is your PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

Extractors, Transformers, and Loaders are independent components which can be plugged-in into a pipeline instance or used separately.

Table of Content

• Changelog
  • 3.3.8 (2022-08-11)
  • 3.3.7 (2022-03-15)
  • 3.3.6 (2021-11-19)
  • 3.3.5 (2021-11-16)
  • 3.3.4 (2021-07-21)
  • 3.3.3 (2021-06-30)
  • 3.3.2
  • 3.3.1 (2021-06-22)
  • 3.3.0 (2021-04-22)
  • 3.2.0 (2021-04-13)
  • 3.1.0 (2021-01-27)
  • 3.0.1 (2021-01-22)
  • 3.0.0b (2020-12-09)
  • 2.3.0 (2020-11-23)
  • 2.2.0 (2020-10-02)
  • 2.1.1 (2020-09-04)
  • 2.1.0 (2020-08-17)
  • 2.0.0 (2020-05-22)
  • 0.6.2 (2019-05-13)
  • 0.6.1 (2019-03-26)
• Installation / Deployment
  • Via Pip
  • Build wheel file
  • Build egg file
  • Build zip file
  • Include pre-build package (egg or zip) with Spark
  • Install local repository as package
  • Install Spooq directly from git
  • Development, Testing, and Documenting
• Setup for Development, Testing, Documenting
  • Prerequisites
  • Setting up the Environment
  • Activate the Virtual Environment
  • Creating Your Own Components
  • Running Tests
    • Test Plugins
  • Generate Documentation
    • Diagrams
    • HTML
    • PDF
    • Configuration
• Releasing a new Version on PyPi
  • Things to consider
    • Version Bump
    • Documentation
    • Changelog
  • Automatic Publishing via Github Action
  • Manual Publishing from Command Line
    • Create the Distribution Files
    • Upload to Test-PyPi
    • Upload to Real PyPi
• Examples
  • JSON Files to Partitioned Hive Table
    • Sample Input Data:
    • Sample Output Tables
    • Application Code for Updating the Users Table
    • Application Code for Updating the Friends_Mapping Table
    • Application Code for Updating Both, the Users and Friends_Mapping Table, at once
• Extractors
  • JSON Files
  • JDBC Source
  • Class Diagram of Extractor Subpackage
  • Create your own Extractor
• Transformers
  • Exploder
  • Sieve (Filter)
  • Mapper
    • Class
    • Activity Diagram
    • Available Custom Mapping Methods
    • Custom Mapping Methods Details
  • Threshold-based Cleaner
    • Base Class
  • Enumeration-based Cleaner
    • Base Class
  • Null Cleaner
    • Base Class
  • Newest by Group (Most current record per ID)
  • Class Diagram of Transformer Subpackage
  • Create your own Transformer
• Loaders
  • Hive Database
    • Activity Diagram
  • Class Diagram of Loader Subpackage
  • Create your own Loader
• Pipeline
  • Pipeline
  • Pipeline Factory
  • Class Diagram of Pipeline Subpackage
• Spooq Base
  • Global Logger
  • Extractor Base Class
    • Create your own Extractor
  • Transformer Base Class
    • Create your own Transformer
  • Loader Base Class
    • Create your own Loader
• Architecture Overview
  • Typical Data Flow of a Spooq Data Pipeline
  • Simplified Class Diagram

Indices and tables

• Module Index
• Search Page