ETL Components

Simplified UML Class Diagram

• Extractors
  • Extractor
    • Extractor.name
    • Extractor.logger
    • Extractor.extract()
  • JSON Files
    • JSONExtractor
      • JSONExtractor.extract()
  • JDBC Source
    • JDBCExtractor
    • JDBCExtractorFullLoad
      • JDBCExtractorFullLoad.extract()
    • JDBCExtractorIncremental
      • JDBCExtractorIncremental.extract()
  • Create your own Extractor
• Transformers
  • Exploder
    • Exploder
      • Exploder.transform()
  • Sieve (Filter)
    • Sieve
      • Sieve.transform()
  • Mapper
    • Class
      • Mapper
    • Custom Transformations
      • spooq.transformer.mapper_transformations.as_is
        • as_is()
      • spooq.transformer.mapper_transformations.to_num
        • to_num()
      • spooq.transformer.mapper_transformations.to_bool
        • to_bool()
      • spooq.transformer.mapper_transformations.to_timestamp
        • to_timestamp()
      • spooq.transformer.mapper_transformations.str_to_array
        • str_to_array()
      • spooq.transformer.mapper_transformations.map_values
        • map_values()
      • spooq.transformer.mapper_transformations.meters_to_cm
        • meters_to_cm()
      • spooq.transformer.mapper_transformations.has_value
        • has_value()
      • spooq.transformer.mapper_transformations.apply
        • apply()
      • spooq.transformer.mapper_transformations.to_json_string
        • to_json_string()
      • spooq.transformer.mapper_transformations.to_str
        • to_str()
      • spooq.transformer.mapper_transformations.to_int
        • to_int()
      • spooq.transformer.mapper_transformations.to_long
        • to_long()
      • spooq.transformer.mapper_transformations.to_float
        • to_float()
      • spooq.transformer.mapper_transformations.to_double
        • to_double()
    • Custom Mapping Functions as Strings [DEPRECATED]
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_IntBoolean
        • _generate_select_expression_for_IntBoolean()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_IntNull
        • _generate_select_expression_for_IntNull()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_StringBoolean
        • _generate_select_expression_for_StringBoolean()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_StringNull
        • _generate_select_expression_for_StringNull()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_TimestampMonth
        • _generate_select_expression_for_TimestampMonth()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_as_is
        • _generate_select_expression_for_as_is()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_boolean
        • _generate_select_expression_for_extended_string_to_boolean()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_date
        • _generate_select_expression_for_extended_string_to_date()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_double
        • _generate_select_expression_for_extended_string_to_double()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_float
        • _generate_select_expression_for_extended_string_to_float()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_int
        • _generate_select_expression_for_extended_string_to_int()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_long
        • _generate_select_expression_for_extended_string_to_long()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_timestamp
        • _generate_select_expression_for_extended_string_to_timestamp()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_unix_timestamp_ms_to_date
        • _generate_select_expression_for_extended_string_unix_timestamp_ms_to_date()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_unix_timestamp_ms_to_timestamp
        • _generate_select_expression_for_extended_string_unix_timestamp_ms_to_timestamp()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_has_value
        • _generate_select_expression_for_has_value()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_json_string
        • _generate_select_expression_for_json_string()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_keep
        • _generate_select_expression_for_keep()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_meters_to_cm
        • _generate_select_expression_for_meters_to_cm()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_no_change
        • _generate_select_expression_for_no_change()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_ms_to_ms
        • _generate_select_expression_for_timestamp_ms_to_ms()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_ms_to_s
        • _generate_select_expression_for_timestamp_ms_to_s()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_s_to_ms
        • _generate_select_expression_for_timestamp_s_to_ms()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_s_to_s
        • _generate_select_expression_for_timestamp_s_to_s()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_for_unix_timestamp_ms_to_spark_timestamp
        • _generate_select_expression_for_unix_timestamp_ms_to_spark_timestamp()
      • spooq.transformer.mapper_custom_data_types._generate_select_expression_without_casting
        • _generate_select_expression_without_casting()
  • Threshold-based Cleaner
    • ThresholdCleaner
      • ThresholdCleaner.transform()
    • Base Class
  • Enumeration-based Cleaner
    • EnumCleaner
      • EnumCleaner.transform()
    • Base Class
  • Null Cleaner
    • NullCleaner
      • NullCleaner.transform()
    • Base Class
  • Newest by Group (Most current record per ID)
    • NewestByGroup
      • NewestByGroup.transform()
  • Create your own Transformer
• Loaders
  • Hive Database
    • HiveLoader
      • HiveLoader.load()
    • Activity Diagram
  • Create your own Loader
• Pipeline
  • Pipeline
  • Pipeline Factory
    • PipelineFactory
      • PipelineFactory.url
      • PipelineFactory.execute()
      • PipelineFactory.get_metadata()
      • PipelineFactory.get_pipeline()
  • Class Diagram of Pipeline Subpackage
• Spooq Base
  • Global Logger
    • initialize()
    • get_logging_level()
  • Extractor Base Class
    • Create your own Extractor
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation
  • Transformer Base Class
    • Create your own Transformer
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation
  • Loader Base Class
    • Create your own Loader
      • Exemplary Sample Code
      • References to include
      • Tests
      • Documentation