sparklyr

CRAN 0.6.2 - Patch to address the following issues:

• Fix connection_spark_shinyapp() under RStudio 1.1 to avoid error while listing Spark installation options for the first time when: https://github.com/rstudio/sparklyr/pull/914

CRAN 0.6.1 - Patch to address the following issues:

• Fix spark_connect() failing for concurrent connections: https://github.com/rstudio/sparklyr/issues/880.
• Fix spark_apply() under empty partitions with https://github.com/rstudio/sparklyr/issues/878, race conditions using more than on CPU per executor, support for explicit column types and avoid persisting any data: https://github.com/rstudio/sparklyr/pull/899.
• Support for Spark 1.6.3: https://github.com/rstudio/sparklyr/pull/889.

CRAN 0.6 - Major release with support to:

• Distribute R computations using spark_apply() to execute arbitrary R code across your cluster over rows and groups.
• Connect to External Data using spark_read_source(), spark_write_source, spark_read_jdbc() and spark_write_jdbc().
• Use the Latest Frameworks including dplyr 0.7, DBI 0.7, RStudio 1.1 and Spark 2.2.
• Spark Connections add a new Databricks connection that enables using sparklyr in Databricks through mode="databricks", add support for Yarn Cluster through master="yarn-cluster" and connection speed was also improved.
• Dataframes add support for sdf_pivot(), sdf_broadcast(), cbind(), rbind(), sdf_separate_column(), sdf_bind_cols(), sdf_bind_rows(), sdf_repartition() and sdf_num_partitions().
• Machine Learning adds support for multinomial regression in ml_logistic_regression(), weights.column for GLM, ml_model_data() and a new ft_count_vectorizer() function for ml_lda().
• Many other features, initial support for broom over ml_linear_regression() and ml_generalized_linear_regression(), dplyr support for %like%, %rlike% and%regexp%, extensions now support download_scalac() to help you install the required Scala compilers while developing extensions, database management got simplified with tbl_change_db() and src_databases() to query and switch between HIVE databases and started a joint effort with Microsoft to improve Spark installations under github.com/rstudio/spark-install.

Additional changes and improvements can be found in the sparklyr NEWS file and the sparklyr 0.6 release blog post.

CRAN 0.5.6 - Patch to address the following issues:

• dplyr joins with same variable names, see https://github.com/tidyverse/dplyr/issues/2825
• Import dplyr 0.7 to avoid manual install of dbplyr
• sample_n performance https://github.com/rstudio/sparklyr/issues/271
• Fix for https://github.com/rstudio/sparklyr/issues/744 by avoiding override of now supported LIMIT statement, see: https://github.com/rstudio/sparklyr/commit/72b2b97000c1c5676467b8f42e5ea5e0de4c8a12
• Workaround for https://github.com/tidyverse/dplyr/issues/2848 and https://github.com/rstudio/sparklyr/issues/744 see https://github.com/rstudio/sparklyr/commit/939dc0a51eb96b4aa6a4c999eb04c496fda97798

CRAN 0.5.5 - Patch release over 0.5.4 with compatibility fixes for RStudio 1.1 and mode="databricks" connections.

CRAN 0.5.4 - Patch release over 0.5.3 with compatibility fixes for dplyr 0.6 and Spark 2.1.

CRAN 0.5.3 - Patch release over 0.5.2 with fix for new DBI:: sqlInterpolate #351 and mutate under dplyr 0.6 #523.

CRAN 0.5.2 - Patch release over 0.5.1 with fixes for connections under Windows that require path normalization https://github.com/rstudio/sparklyr/pull/492 and Livy connections to handle states properly while connecting https://github.com/rstudio/sparklyr/pull/421.

Additional changes and improvements can be found in the sparklyr NEWS file and the sparklyr 0.5 release blog post.

Additional changes and improvements can be found in the sparklyr NEWS file and the sparklyr 0.4 release blog post.