low-level helpers for Apache Spark libraries and tests
APACHE-2.0 License
Spark, Hadoop, and Kryo utilities
Classes that implement the Registrar interface can use various shorthands for registering classes with Kryo.
Adapted from RegistrationTest:
register(
cls[A], // comes with an AlsoRegister that loops in other classes
arr[Foo], // register a class and an Array of that class
cls[B] → BSerializer(), // use a custom Serializer
CDRegistrar // register all of another Registrar's registrations
)
Serializer
s and AlsoRegister
s are picked up implicitly if not provided explicitly.AlsoRegister
s are recursive, allowing for much easier and more robust accountability about what is registered and why, and ensurance that needed registrations aren't overlooked.Configuration
: serializable Hadoop-Configuration
wrapperContext
: SparkContext
wrapper that is also a Hadoop Configuration
, for unification of "global configuration access" patternsConf
: load a SparkConf
with settings from file(s) specified in the SPARK_PROPERTIES_FILES
environment variableSparkConfBase
: trait that brokers setting config key-values and creating a SparkConf
KeyPartitioner
/ Partitioner
: shorthands for common Spark-Partitioner
-creation patterns
Histogram
accumulator