Transformers

Transformers take a DataFrame as an input, transform it accordingly and return a DataFrame.

Each Transformer class has to have a transform method which takes no arguments and returns a DataFrame.

Possible transformation methods can be Selecting the most up-to-date record by id, Exploding an array, Filter (on an exploded array), Apply basic threshold cleansing or Map the incoming DataFrame to at provided structure.

• Annotator (Load and Update Column Comments)
  • AnnotatorMode
  • MissingColumnHandling
  • ColumnNotFound
  • load_comments_from_metastore_table()
  • update_comment()
  • Annotator
    • Annotator.transform()
• Exploder
  • Exploder
    • Exploder.transform()
• Sieve (Filter)
  • Sieve
    • Sieve.transform()
• Mapper
  • Class
    • MapperMode
    • MissingColumnHandling
    • MapperParameterNotSupported
    • DataTypeValidationFailed
    • ColumnMappingNotSupported
    • Mapper
      • Mapper.transform()
  • Custom Transformations
    • spooq.transformer.mapper_transformations.as_is
      • as_is()
    • spooq.transformer.mapper_transformations.to_num
      • to_num()
    • spooq.transformer.mapper_transformations.to_bool
      • to_bool()
    • spooq.transformer.mapper_transformations.to_timestamp
      • to_timestamp()
    • spooq.transformer.mapper_transformations.str_to_array
      • str_to_array()
    • spooq.transformer.mapper_transformations.map_values
      • map_values()
    • spooq.transformer.mapper_transformations.meters_to_cm
      • meters_to_cm()
    • spooq.transformer.mapper_transformations.has_value
      • has_value()
    • spooq.transformer.mapper_transformations.apply
      • apply()
    • spooq.transformer.mapper_transformations.to_json_string
      • to_json_string()
    • spooq.transformer.mapper_transformations.to_str
      • to_str()
    • spooq.transformer.mapper_transformations.to_int
      • to_int()
    • spooq.transformer.mapper_transformations.to_long
      • to_long()
    • spooq.transformer.mapper_transformations.to_float
      • to_float()
    • spooq.transformer.mapper_transformations.to_double
      • to_double()
• Threshold-based Cleaner
  • ThresholdCleaner
    • ThresholdCleaner.transform()
  • Base Class
• Enumeration-based Cleaner
  • EnumCleaner
    • EnumCleaner.transform()
  • Base Class
• Null Cleaner
  • NullCleaner
    • NullCleaner.transform()
  • Base Class
• Newest by Group (Most current record per ID)
  • NewestByGroup
    • NewestByGroup.transform()

Create your own Transformer

Please see the Create your own Transformer for further details.