Transformers
Transformers take a DataFrame as an input, transform it accordingly
and return a DataFrame.
Each Transformer class has to have a transform method which takes no arguments
and returns a DataFrame.
Possible transformation methods can be Selecting the most up-to-date record by id, Exploding an array, Filter (on an exploded array), Apply basic threshold cleansing or Map the incoming DataFrame to at provided structure.
- Exploder
- Sieve (Filter)
- Mapper
- Class
- Custom Transformations
- spooq.transformer.mapper_transformations.as_is
- spooq.transformer.mapper_transformations.to_num
- spooq.transformer.mapper_transformations.to_bool
- spooq.transformer.mapper_transformations.to_timestamp
- spooq.transformer.mapper_transformations.str_to_array
- spooq.transformer.mapper_transformations.map_values
- spooq.transformer.mapper_transformations.meters_to_cm
- spooq.transformer.mapper_transformations.has_value
- spooq.transformer.mapper_transformations.apply
- spooq.transformer.mapper_transformations.to_json_string
- spooq.transformer.mapper_transformations.to_str
- spooq.transformer.mapper_transformations.to_int
- spooq.transformer.mapper_transformations.to_long
- spooq.transformer.mapper_transformations.to_float
- spooq.transformer.mapper_transformations.to_double
- Custom Mapping Functions as Strings [DEPRECATED]
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_IntBoolean
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_IntNull
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_StringBoolean
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_StringNull
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_TimestampMonth
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_as_is
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_boolean
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_date
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_double
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_float
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_int
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_long
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_to_timestamp
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_unix_timestamp_ms_to_date
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_extended_string_unix_timestamp_ms_to_timestamp
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_has_value
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_json_string
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_keep
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_meters_to_cm
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_no_change
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_ms_to_ms
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_ms_to_s
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_s_to_ms
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_timestamp_s_to_s
- spooq.transformer.mapper_custom_data_types._generate_select_expression_for_unix_timestamp_ms_to_spark_timestamp
- spooq.transformer.mapper_custom_data_types._generate_select_expression_without_casting
- Threshold-based Cleaner
- Enumeration-based Cleaner
- Null Cleaner
- Newest by Group (Most current record per ID)
Create your own Transformer
Please see the Create your own Transformer for further details.