Project to benchmark and profile Error-Prone performance to uncover bottlenecks.
APACHE-2.0 License
Current test subject is RxJava 2.x project (Apache 2.0)
Upstream issue about performance: google/error-prone/issues/994
I integrate Error Prone with RxJava and provide ways to benchmark and profile builds with Error Prone, with slightly optimized Error Prone configuration and without Error Prone.
Table of contents:
Error Prone (2.2.0) seems to significantly slow down compilation comparing to JDK compiler.
Initially this popped out while I was integrating Error Prone into our projects at Lyft and then was reproduced with very similar results on RxJava.
Quote from "Building Useful Program Analysis Tools Using an Extensible Java Compiler" White Paper published by Research at Google in 2012:
We found that error-prone incurs a 9.0% time overhead and a 0.95% memory overhead, which are tolerable for our build environment. We have not yet invested effort in optimizing error-prone, and it should be possible to improve its performance.
I'm specifically interested in:
9.0% time overhead and a 0.95% memory overhead
and
We have not yet invested effort in optimizing error-prone, and it should be possible to improve its performance.
Unfortunately, I wasn't able to find any other discussions of Error Prone performance (PRs welcome).
Benchmarks, profilings and real-world integration experience show that Error Prone (2.2.0) has much more significant overhead on build times (see details below).
According to method call profiling (see below), Error Prone spends significant amount of time on this subset of checks:
PrivateSecurityContractProtoAccess.matchMethodInvocation
2.8% - 663 ms (Protobuf)ImmutableModification.matchMethodInvocation
2.4% - 551 ms (Guava)ShouldHaveEvenArgs.matchMethodInvocation
0.8% - 195 ms (Truth)CollectionIncompatibleType.matchMethodInvocation
0.6% - 142 ms (JDK)NamedParameterChecker.matchMethodInvocation
0.3% - 62,342 s (JDK)While these checks might be relevant to lots of Google projects, only 2 of them are relevant to RxJava project.
Profiling Lyft projects where I tried to intgrate Error Prone gives similar results: lots of time is spent on checks that are not relevant to the codebase in the first place.
I hope it's possible to optimize these checks by checking compilation classpath and excluding irrelevant checks from analysis. For example, whole category of Guava checks can be excluded if Guava is not found in compilation classpath.
I hope it's possible to parallelize AST analysis since Error Prone doesn't modify it by default.
Profiling suggests that compilation step where Error Prone does its work only uses single thread.
You can look at JProfiler output that I've put in assets
folders to get more insights about expensive method calls.
Warmups: 5, iterations: 10.
Environment:
javac
)4.173
sec14.922
sec, 3.57
x slower14.680
sec, 3.51
x slower10.358
sec, 2.48
x slowerjavac
)0.89
GB2.3
GB, 2.58
x memory used2.08
GB, 2.3
x memory usedN/A
(haven't measured yet)Looks like Parallel GC doesn't play well with Error Prone and max memory usage spikes significantly, please see below results with G1 GC on JDK 9.
javac
plugin)4.963
sec10.699
sec, 2.15
x slowerN/A
(haven't measured yet)N/A
(haven't measured yet)javac
plugin)1.1
GB1.08
GB, 0.98
x memory usedN/A
(haven't measured yet)N/A
(haven't measured yet)Looks like G1 GC is more agressive than Parallel GC and able to keep Error Prone's memory use even lower than baseline JDK compiler, however you can see on graphs that Error Prone caused more often garbage collection (I think it's fine if it doesn't affect performance, Error Prone does lots of useful work).
javac
)4.918
sec16.565
sec, 3.36
x slower3.33
x slowerN/A
(haven't measured yet)javac
)1.14
GB1.01
GB, 0.88
x memory used1.07
GB, 0.93
x memory usedN/A
(haven't measured yet)Same situation as above w/ JDK 9.
javac
)javac
plugin)javac
)javac
)javac
)javac
plugin)scripts/
: scripts to ease benchmarking/profiling for you, they're intended to be used by you to achieve reproducible and comparable resultsscenarios/
: benchmark/profiling scenarios in format supported by gradle-profiler
docker/
: Dockerfiles and misc required for isolated benchmark/profile environmentrxjava/
: project under testassets/
: benchmark/profiling files for README/etcprofile-out*/
: directories with benchmark/profiling results generated by gradle-profiler
To ease environment setup (specifically switching between JDK versions and building gradle-profiler) I've Dockerized project. However you still welcome to run profiling directly on local environment.
I recommend using *-docker-*
scripts, as they don't require you to manage JDK and gradle-profiler
installation. Moreover, Docker seems to reproducibly improve disk reads for these benchmarks (see Docker WTH) which allows pretty much exclude IO from benchmarks and align final build times with compiler performance which is what we're interested in.
*-local-*
scripts expect you to have gradle-profiler
built and available in PATH
, as well as JDK installed depending on requirements.
Notes on Docker:
macOS can't run Docker on macOS kernel natively, thus it runs it through macOS virtualization. It has ridiculous IO overhead which can affect runs.
Linux machine with minimal amount of processes running is recommended.
scripts/benchmark-rxjava-docker-oracle-jdk8.sh
javac
) in RxJava project in local environmentscripts/benchmark-rxjava-local.sh
javac
plugin) in RxJava project with Oracle JDK 9 in a Docker containerscripts/benchmark-rxjava-docker-oracle-jdk9-javac-plugin.sh
javac
plugin) in RxJava project in local environmentscripts/benchmark-rxjava-local-javac-plugin.sh
Note 1:
I personally use JProfiler, but it's a paid product and you need to have a license for it. It gives deep insigts into method calls, memory allocations and so on and I consider it the most useful tool for profiling in this project. I expect people interested in deep profiling data to use JProfiler or any other reasonable profiler of their choise (see Note 3).
Note 2:
chrome-trace
gives very limited profiling information, however it clearly shows java compilation step duration and it's free. For some reason it doesn't work in provided Docker env, see this issue (PRs welcome).
Note 3:
PRs with more profiler integrations are welcomed! Especially if they're open-source, free to use and provide method call analysis.
You can check -*profile-*
scripts, copy/modify them and add profiler of your choice.
# YOU MUST HAVE A PROPER LICENSE TO USE JPROFILER
# YOU MUST ACCEPT JPROFILER'S TERMS OF USE
# REPOSITORY MAINTANERS ARE NOT RESPONSIBLE FOR YOUR VIOLATION OF LICENSE OR TERMS OF USE
scripts/profile-rxjava-docker-oracle-jdk8-jprofiler.sh
This project is made to provide reproducible, open-source setup to uncover real-world performance issues of Error Prone (2.2.0).
I was trying to integrate Error Prone in our Android project at Lyft and was surprised by 3.85x
slower build times which is obviously unnaceptable.
These findings were initially discussed in Buck Build system Slack, specifically with @kageiit (a fellow build-engineer at Uber).
Obviously, guys from Error Prone will want setup that'll allow them reproduce these results, so here it is :)
During Dockerization of this benchmarking/profiling setup I've noticed interesting thing:
Running same benchmarks/profiling in Docker container results in much faster execution.
Reproduced on 2 different Linux machines.
It does however produce stable results and I still recommend Docker for benchmarks and profiling as it removes hassle of environment setup.
Environment:
Warmups: 5, iterations: 10.
8.454
sec4.173
sec, 2.02
x faster (!)18.998
sec14.922
sec, 1.27
x faster (!)18.722
sec14.680
sec, 1.27
x faster (!)You can see it on following graphs:
Warmups: 3, iterations: 1.
java.io.FileInputStream.readBytes
9.824
sec1.495
sec, 6.57
x faster (!)(JProfiler)
(JProfiler)
I don't have an accurate answer yet, only guesses.
Profiling clearly shows that disk reads are much faster in Docker container, they're fast like data is either in RAM or very well cached in RAM.
Note that the way benchmark script is made is that it mounts project dir to the Docker container and then copies it inside container fs, so benchmark runs against files that are inside container fs.
Docker/Linux experts are very welcomed to reproduce it and update this section with explanaition and perf profiling + references to Docker docs/sources.
Note that you need Oracle JDK 8 and
gradle-profiler
installed, ideally same version as Docker image resolves.
scripts/benchmark-rxjava-local-jdk8.sh
scripts/benchmark-rxjava-docker-oracle-jdk8.sh
Please carefully read NOTICE.md before using this project.