ETL Components

Simplified UML Class Diagram

• Extractors
  • Extractor
    • Extractor.name
    • Extractor.logger
    • Extractor.extract()
  • JSON Files
    • JSONExtractor
      • JSONExtractor.extract()
  • JDBC Source
    • JDBCExtractor
    • JDBCExtractorFullLoad
      • JDBCExtractorFullLoad.extract()
    • JDBCExtractorIncremental
      • JDBCExtractorIncremental.extract()
  • Create your own Extractor
• Transformers
  • Annotator (Load and Update Column Comments)
    • AnnotatorMode
    • MissingColumnHandling
    • ColumnNotFound
    • load_comments_from_metastore_table()
    • update_comment()
    • Annotator
      • Annotator.transform()
  • Exploder
    • Exploder
      • Exploder.transform()
  • Sieve (Filter)
    • Sieve
      • Sieve.transform()
  • Mapper
    • Class
      • MapperMode
      • MissingColumnHandling
      • MapperParameterNotSupported
      • DataTypeValidationFailed
      • ColumnMappingNotSupported
      • Mapper
        • Mapper.transform()
    • Custom Transformations
      • spooq.transformer.mapper_transformations.as_is
        • as_is()
      • spooq.transformer.mapper_transformations.to_num
        • to_num()
      • spooq.transformer.mapper_transformations.to_bool
        • to_bool()
      • spooq.transformer.mapper_transformations.to_timestamp
        • to_timestamp()
      • spooq.transformer.mapper_transformations.str_to_array
        • str_to_array()
      • spooq.transformer.mapper_transformations.map_values
        • map_values()
      • spooq.transformer.mapper_transformations.meters_to_cm
        • meters_to_cm()
      • spooq.transformer.mapper_transformations.has_value
        • has_value()
      • spooq.transformer.mapper_transformations.apply
        • apply()
      • spooq.transformer.mapper_transformations.to_json_string
        • to_json_string()
      • spooq.transformer.mapper_transformations.to_str
        • to_str()
      • spooq.transformer.mapper_transformations.to_int
        • to_int()
      • spooq.transformer.mapper_transformations.to_long
        • to_long()
      • spooq.transformer.mapper_transformations.to_float
        • to_float()
      • spooq.transformer.mapper_transformations.to_double
        • to_double()
  • Threshold-based Cleaner
    • ThresholdCleaner
      • ThresholdCleaner.transform()
    • Base Class
  • Enumeration-based Cleaner
    • EnumCleaner
      • EnumCleaner.transform()
    • Base Class
  • Null Cleaner
    • NullCleaner
      • NullCleaner.transform()
    • Base Class
  • Newest by Group (Most current record per ID)
    • NewestByGroup
      • NewestByGroup.transform()
  • Create your own Transformer
• Loaders
  • Hive Database
    • HiveLoader
      • HiveLoader.load()
    • Activity Diagram
  • Create your own Loader
• Pipeline
  • Pipeline
  • Pipeline Factory
    • PipelineFactory
      • PipelineFactory.url
      • PipelineFactory.execute()
      • PipelineFactory.get_metadata()
      • PipelineFactory.get_pipeline()
  • Class Diagram of Pipeline Subpackage
• Spooq Base
  • Global Logger
    • initialize()
    • get_logging_level()
  • Extractor Base Class
    • Create your own Extractor
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation
  • Transformer Base Class
    • Create your own Transformer
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation
  • Loader Base Class
    • Create your own Loader
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation