Bot releases are hidden (Show)
This release includes initial support for running Spark against HBase with a richer feature set than was previously possible with MapReduce bindings:
* Support for Spark and Spark Streaming against Spark 2.1.1
* RDD/DStream formation from scan operations
* convenience methods for interacting with HBase from an HBase backed RDD / DStream instance
* examples in both the Spark Java API and Spark Scala API
* support for running against a secure HBase cluster
For user configurable parameters for HBase datasources. Please refer to org.apache.hadoop.hbase.spark.datasources.HBaseSparkConf for details.
User can either set them in SparkConf, which will take effect globally, or configure it per table, which will overwrite the value set in SparkConf. If not set, the default value will take effect.
Currently three parameters are supported.
Before this patch, users of the spark HBaseContext would fail due to lack of privilege exceptions.
Note:
* It is preferred to have spark in spark-on-yarn mode if Kerberos is used.
* This is orthogonal to issues with a kerberized spark cluster via InputFormats.
Right now the timestamp is always latest. With this patch, users can select timestamps they want.
In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they can also select a time range with minimum timestamp and maximum timestamp.
Run test under root dir or hbase-spark dir
Run specified test case, since we have two plugins, we need specify both java and scala.
When only test scala or jave test case, disable the other one use -Dxx=None as follow:
The integration module for Apache Spark now includes Java-friendly equivalents for the bulkLoad
and bulkLoadThinRows
methods in JavaHBaseContext
.
HBase now ships with an integration test for our integration with Apache Spark.
You can run this test on a cluster by using an equivalent to the below, e.g. if the version of HBase is 2.0.0-alpha-2
spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-spark-it-2.0.0-alpha-2-tests.jar -Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic
As of this JIRA, Spark version is upgraded from 1.6 to 2.1.1
Adds a kafka proxy that appears to hbase as a replication peer. Use to tee table edits to kafka. Has mechanism for dropping/routing updates. See https://github.com/apache/hbase-connectors/tree/master/kafka for documentation.
Cleaned up kafka submodule dependencies. Added used dependencies to pom and removed the unused. Depends explicitly on hadoop2. No messing w/ hadoop3 versions.
Updates our hbase-spark integration so defaults spark 2.4.0 (October 2018) from 2.1.1 and Scala 2.11.12 (from 2.11.8).
This commit adds a kafka connector. The connectors acts as a replication peer and sends modifications in HBase to kafka.
For further information, please refer to kafka/README.md.
New features in hbase-spark:
* native type support (short, int, long, float, double),
* support for Dataframe writes,
* avro support,
* catalog can be defined in json.