Welcome to Spooq’s documentation!

Spooq is your PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

Extractors, Transformers, and Loaders are independent components which can be plugged-in into a pipeline instance or used separately.

Table of Content

• Installation / Deployment
  • Build egg file
  • Build zip file
  • Include pre-build package (egg or zip) with Spark
  • Install local repository as package
  • Install Spooq2 directly from git
  • Development, Testing, and Documenting
• Examples
  • JSON Files to Partitioned Hive Table
    • Sample Input Data:
    • Sample Output Tables
    • Application Code for Updating the Users Table
    • Application Code for Updating the Friends_Mapping Table
    • Application Code for Updating Both, the Users and Friends_Mapping Table, at once
• Extractors
  • JSON Files
  • JDBC Source
  • Class Diagram of Extractor Subpackage
  • Create your own Extractor
• Transformers
  • Exploder
  • Sieve (Filter)
  • Mapper
    • Class
    • Activity Diagram
    • Custom Mapping Methods
  • Threshold-based Cleaner
  • Newest by Group (Most current record per ID)
  • Class Diagram of Transformer Subpackage
  • Create your own Transformer
• Loaders
  • Hive Database
    • Activity Diagram
  • Class Diagram of Loader Subpackage
  • Create your own Loader
• Pipeline
  • Pipeline
  • Pipeline Factory
  • Class Diagram of Pipeline Subpackage
• Spooq Base
  • Global Logger
  • Extractor Base Class
    • Create your own Extractor
  • Transformer Base Class
    • Create your own Transformer
  • Loader Base Class
    • Create your own Loader
• Setup for Development, Testing, Documenting
  • Prerequisites
  • Setting up the Environment
  • Activate the Virtual Environment
  • Creating Your Own Components
  • Running Tests
    • Test Plugins
  • Generate Documentation
    • Diagrams
    • HTML
    • PDF
    • Configuration
• Architecture Overview
  • Typical Data Flow of a Spooq Data Pipeline
  • Simplified Class Diagram

Indices and tables

• Module Index
• Search Page