DPE University Training

Maintaining Gradle Build Cache Performance Exercise

This is a hands-on exercise to go along with the Maintaining Optimal Gradle Build Cache Performance training module. In this exercise you will go over the following:

Using the Develocity Build Validation Scripts to identify build caching issues

Prerequisites

Finished going through the relevant sections in the training course

Clone this Repository

This repository contains the project with build caching issues. We will use the Develocity Build Validation Scripts to identify these issues and address them.

Clone this repository in a new directory location on your machine

In the rest of this README the location will be referred to as /path/to/dpeuni-gradle-maintain-cache-perf. Whenever you see this, replace it with the location you are using.

Develocity Authentication

We will use the DPE University Develocity instance as the remote cache. If you haven't already done so, you can authenticate with the Develocity service by running:

./gradlew provisionGradleEnterpriseAccessKey

The output of the task will indicate a browser window will come up from which you can complete the authentication:

Once the browser window comes up you can enter a title for the access key that will be created or go with the suggested title:

Once confirmed you will see the following message and you can close the browser window and return to the editor:

Downloading the Develocity Build Validation Scripts

Open the Develocity Build Validation Scripts installation instructions
Copy the installation command
Open a terminal, create a new directory for the installation location and run the command there. The installation location should be outside the project directory.
Go to the directory containing the Develocity Build Validation Scripts, you will see contents similar to:

$ ls -ltr
total 192
-rw-r--r--   1 adayal  wheel    862 Oct 25  2023 network.settings
-rw-r--r--   1 adayal  wheel    438 Oct 25  2023 mapping.example
drwxr-xr-x  27 adayal  wheel    864 Oct 25  2023 lib
-rw-r--r--   1 adayal  wheel      6 Oct 25  2023 VERSION
-rw-r--r--   1 adayal  wheel    404 Oct 25  2023 README.md
-rw-r--r--   1 adayal  wheel  11356 Oct 25  2023 LICENSE
-rwxr-xr-x   1 adayal  wheel  16628 Oct 25  2023 05-validate-remote-build-caching-ci-local.sh
-rwxr-xr-x   1 adayal  wheel  10954 Oct 25  2023 04-validate-remote-build-caching-ci-ci.sh
-rwxr-xr-x   1 adayal  wheel  11958 Oct 25  2023 03-validate-local-build-caching-different-locations.sh
-rwxr-xr-x   1 adayal  wheel  10874 Oct 25  2023 02-validate-local-build-caching-same-location.sh
-rwxr-xr-x   1 adayal  wheel  11336 Oct 25  2023 01-validate-incremental-building.sh

Identify Local Build Caching Issues (Same Location)

Let's run experiment 2 to identify local caching issues when the build is run from the same location.

Go to the directory where the Develocity Build Validation Scripts are installed
Run the second script in interactive mode: ./02-validate-local-build-caching-same-location.sh -i
Press the Enter key to start the experiment as the following prompt indicates:

Press <Enter> to get started with the experiment.

4. The project does not use the Common Custom User Data Gradle plugin, and we will proceed without it. Press the Enter key when you see the following prompt:

Press <Enter> once you have (optionally) configured your build with the Common Custom User Data Gradle plugin and pushed the changes.

You should have already authenticated with Develocity, so press
the Enter key when you see the following prompt:

Press <Enter> once you have (optionally) adjusted your access permissions and configured the API credentials on your machine.

Enter the file path to the project using the file:// URL. Replace /path/to/dpeuni-gradle-maintain-cache-perf with the actual location:

What is the URL for the Git repository that contains the project to validate? file:///path/to/dpeuni-gradle-maintain-cache-perf

For the next 4 questions use the default values (as denoted by the values in parentheses) by pressing the Enter key without typing anything else:

What is the branch for the Git repository that contains the project to validate? <the repository's default branch>
What is the commit id for the Git repository that contains the project to validate? <the branch's head>
What are additional options to use when cloning the Git repository? --depth=1
What is the directory to invoke the build from? <the repository's root directory>

When the script asks what task to invoke, specify the build task:

What are the Gradle tasks to invoke? build

For the last question, press Enter without typing anything else:

What are additional cmd line arguments to pass to the build invocation? <none>

When you see the following prompt, press Enter:

Press <Enter> to check out the project from Git.

It will create a separate clone of the repository for the experiment.

NOTE: If you get an error here, check the file URL of the repository.

When it prompts to run the first build press Enter:

Press <Enter> to run the first build of the experiment.

It will run the build task on the project.

When it prompts to run the second build press Enter:

Press <Enter> to run the second build of the experiment.

It will run the clean task followed by the build task on the project.

After it has finished, it will prompt you to continue and look at the results. Press the Enter key:

Press <Enter> to measure the build results.

In the Performance Characteristics section, it will inform you there is one task with a potential build caching issue.

Open the Executed cacheable tasks link.
It will inform you the :app:test task was executed even though it should have
gotten it's output from the cache.

Open the Task inputs comparison link.
It will inform you that the model-1.0.jar file from the model subproject changed.
This is an input to the :app:test task.

Clear the filters on the top right of the page

Expand the jar task on the model subproject. You will see the model/build/tmp/jar/MANIFEST.MF file
which is an input to this task changed.

In the terminal, notice the Experiment artifact dir in the Summary section:

All the files for the two builds are in this directory.

Run a diff to see how the model/build/tmp/jar/MANIFEST.MF file changed between the builds:

NOTE: For each experiment there exists a latest symlink that always points to the most recent experiment's artifact directory. Otherwise, adjust the path for the diff as per your Experiment artifact dir.

$ diff .data/02-validate-local-build-caching-same-location/latest/first*/model/build/tmp/jar/MANIFEST.MF .data/02-validate-local-build-caching-same-location/latest/second*/model/build/tmp/jar/MANIFEST.MF
2c2
< Implementation-Timestamp: 2024-05-17T13:49:25.219892Z
---
> Implementation-Timestamp: 2024-05-17T13:50:52.688382Z

We can see a timestamp is changing in the file. You may recognize this issue from a previous training.

We can use runtime classpath normalization to ignore the timestamp attribute. Open app/build.gradle.kts and add the following:

normalization {
    runtimeClasspath {
        metaInf {
            ignoreAttribute("Implementation-Timestamp")
        }
    }
}

Commit the change to the local repository:

$ git commit -am "Use runtime classpath normalization to ignore timestamp attribute in manifest file"

Run the experiment again. At the end of the output you will see a command you can run to do this:

You should see no issues found this time. If you do, be sure your changes are committed. This is required because the experiments always use a fresh clone of the repository.

Identify Local Build Caching Issues (Different Locations)

Each experiment identifies different kinds of build caching issues. Using experiment 2 we identified and fixed one issue. Now let's run the next experiment to see if any other issues are identified.

Go to the directory where the Develocity Build Validation Scripts are installed
You can run the third script in interactive mode, however since it is similar to the second script, we can run it using the command as follows:

NOTE: Replace /path/to/dpeuni-gradle-maintain-cache-perf with the location of the project.

./03-validate-local-build-caching-different-locations.sh -r file:///path/to/dpeuni-gradle-maintain-cache-perf -t build

The experiment will run and find 1 task with a potential build caching issue:

Open the Executed cacheable tasks link.
It will inform you the :app:countSrc task was executed even though it should have
gotten it's output from the cache.

Open the Task inputs comparison link.
It will inform you that the app/src input for the task has different absolute paths.

That tells us that instead of using relative paths, this input has it's path sensitivity set to absolute. This experiment runs the two builds in different directories, so the absolute paths would have been different for this input.

Let us confirm this is the issue.

5. Open app/build.gradle.kts and go to the countSrc task configuration. You will see the srcDir input has been set to use absolute path sensitivity

When you encounter this, you should consider if the input really requires the full absolute path, or if using the relative path is good enough.

Change the path sensitivity to relative.
Commit the change to the local repository:

$ git commit -am "countSrc task's srcDir input should have relative path sensitivity"

Run the experiment again:

NOTE: Replace /path/to/dpeuni-gradle-maintain-cache-perf with the location of the project.

./03-validate-local-build-caching-different-locations.sh -r file:///path/to/dpeuni-gradle-maintain-cache-perf -t build

You should no longer see any issues. If you do, be sure your changes are committed. This is required because the experiments always use a fresh clone of the repository.