datasketches-hive

This is a maintenance release to make this Apache Hive component work with the latest versions of datasketches-java-3.1.0 and datasketches-memory-2.0.0

• This release fixes critical bug
• updates datasketches-java dependency to 1.3.0-incubating
• minor licensing fixes
• minor code cleanup.

This is the initial Apache release for this component.

• The Java package structure has been changed to org.apache.datasketches
• The file license headers have been updated with the Apache license header
• The LICENSE, NOTICE, and DISCLAIMER-WIP files have been added and/or updated.

No other significant code changes from the prior version.

• Based on sketches-core-0.13.0
• CPC sketch UDFs
• KLL sketch UDFs
• additional quantiles sketch UDFs: toString, getN, getCDF
• additional HLL sketch UDFs: SketchToString, getEstimateAndErrorBounds

Compatibility with sketches-core-0.11.0

• based on sketches-core-0.10.3
• support HLL sketch late init from Hive
• support char and varchar types as HLL and Theta sketch input

This is a maintenance release to use the latest sketches-core-0.10.2

• This is based on sketches-core-0.10.1 and memory-0.10.3
• New Tuple sketch UDFs: ArrayOfDoublesSketchesTTestUDF, ArrayOfDoublesSketchToMeansUDF, ArrayOfDoublesSketchToVariancesUDF, ArrayOfDoublesSketchToEstimateAndErrorBoundsUDF, ArrayOfDoublesSketchToNumberOfRetainedEntriesUDF, ArrayOfDoublesSketchToQuantilesSketchUDF
• Performance improvement: wrap() is used instead of heapify() in HLL UDFs

• HLL DataToSketchUDAF: Input strings are converted to char[] before passing to HllSketch. This is substantially faster than passing strings due to avoiding UTF-8 conversion process. Warning: effectively a different hash function is used for strings. So unions of sketches produced by this version and the previous version will have no overlap, and therefore produce incorrect results. We recommend upgrading to this version, and, if any sketches have been created with string inputs and stored, we recommend recomputing them from the raw data.

• added DataToSketchUDAF, UnionSketchUDAF, SketchToEstimateUDF and UnionSketchUDF for HLL sketch

• This is based on sketches-core-0.10.0 and memory-0.10.1
• Tuple Sketch: added ArrayOfDoublesSketchToValuesUDTF to dump values

• Fixed Memory shading problem. The shaded hive jar now includes both the shaded core and shaded memory code.
• Added TupleSketch UDFs

• Added frequent items sketch UDFs
• Added parameter K to quantiles sketch union UDAFs
• Added UDFs to get K from quantiles sketches
• Heavy refactoring of theta sketch UDFs, added seed support, changed the default number of nominal entries to be consistent with core library, fixed treatment of empty sketches in union
• Added UDFs to get PMF from quantiles sketches
• Code style and documentation improvements

Major Additions

Doubles Quantiles Sketch

Generic Quantiles Sketch with additional String implementation

Fixes

• In DataToSketchUDAF, it was restricted to only Writable types, now allows any input types.

Other

• Minor performance improvements.
• Javadoc improvements

• Update Set Operations

• Updated Set Operations for Hive

Initial Open Source Release aligned with sketches-core 0.2.0