GEApiData
Intermediate layer for DV API
MIT License
ge-api-data
Intermediate layer for Develocity API:
- Paginates
/api/buildsallowing to request more than 1000 builds - Consolidate attributes and cache performance endpoints
- Supports advanced query language requests for GE 2023.3
This is not an official Develocity library. This is a free interpretation of a layer that allows massive requests and consolidates builds information to be used in tooling/reports to help you understand better the state of the build in your projects
Dependency
dependencies {
implementation("io.github.cdsap:geapi-data:0.3.0")
}
Simple Usage
Define Repository
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
We need to specify the Gradle Enterprise url and the API token
Retrieve Build Scans with Attributes
val getBuildScans = GetBuildsWithAttributesRequest(repository).get()
getBuildScans.forEach {
...
}
If you are using Gradle Enterprise 2023.3 you can use the variant:
val getBuildScans = GetBuildsFromQueryWithAttributesRequest(repository).get()
getBuildScans.forEach {
...
}
Returns a list of ScanWithAttributes:
| Property | Description |
|---|---|
| id | Build Scan id |
| projectName | Project name |
| requestedTasksGoals | Requested tasks or goals |
| tags | List of tags |
| hasFailed | Build failed |
| environment | Environment user |
| buildDuration | Duration in ms |
| buildTool | Build Tool |
| buildStartTime | Build start time |
| values | Custom values |
Retrieve Build Scans with Cache Performance
val buildScans = GetBuildsWithAttributesRequest(repository).get()
val filter = Filter(requestedTask = "assembleDebug")
val builds = GetBuildsWithCachePerformanceRequest(repository).get(buildScans, filter)
build.forEach {
...
}
Returns a list of Build:
| Property | Description |
|---|---|
| builtTool | Build Tool |
| taskExecution | List of tasks |
| tags | List of tags |
| requestedTask | Requested Tasks |
| id | Build Scan id |
| buildDuration | Build Duration |
| avoidanceSavingsSummary | Avoidance Savings Summary |
| buildStartTime | Build start time |
| projectName | Project name |
| goalExecution | List of goals |
| values | Custom values |
Both Request functions are suspend.
Retrieve Artifact Transforms
// Single Build Scan
GetSingleBuildArtifactTransformRequest(repository).get(buildScanId)
// Multiple Build Scans
val getBuildScans = GetBuildsFromQueryWithAttributesRequest(repository).get(filter)
GetBuildsWithArtifactTransformRequest(repository).get(getBuildScans, filter)
Returns a list of ArtifactTransform:
| Property | Description |
|---|---|
| artifactTransformExecutionName | Name of the artifact transform |
| transformActionType | Transform Action type |
| inputArtifactName | Input Artifact Name |
| outcome | Outcome of the unit of work |
| avoidanceOutcome | Avoidance outcome of the uint of work |
| duration | Duration of the transform execution |
| avoidanceSavings | Avoidance savings in ms |
| fingerprintingDuration | Fingerprinting duration in ms |
| changedAttributes | Requested attributes for the transform |
| buildScanId | Build Scan id |
| cacheArtifactSize | Cache output size artifact |
Note: Artifact transform endpoint is available in Develocity 2023.4+
Retrieve Resource usage build
val getBuildScans = GetBuildsFromQueryWithAttributesRequest(repository).get(filter)
GetBuildsResourceUsageRequest(repository).get(getBuildScans, filter)
Returns a list of BuildWithResourceUsage:
| Property | Description |
|---|---|
| totalMemory | Total memory in the machine running the build |
| total | Total performance metrics |
| nonExecution | Non execution performance metrics |
| execution | Execution performance metrics |
| builtTool | Build Tool |
| tags | List of tags |
| requestedTask | Requested Tasks |
| id | Build Scan id |
| buildDuration | Build duration |
| buildStartTime | Build start time |
| projectName | Project name |
| values | Custom values |
| enviroment | Environment |
where PerformanceMetrics is a map with the following properties returning a Metric:
| Property |
|---|
| allProcessesCpu |
| buildProcessCpu |
| buildChildProcessesCpu |
| allProcessesMemory |
| buildProcessMemory |
| buildChildProcessesMemory |
| diskReadThroughput |
| diskWriteThroughput |
| networkUploadThroughput |
| networkDownloadThroughput |
Metric contains
| Property |
|---|
| max |
| average |
| median |
| p25 |
| p75 |
| p95 |
Notes:
- Resource usage endpoint is available in Develocity 2024.2+.
- Version 0.2.8 supports only gradle build resource usage
Available Requests
| Request | Return type |
|---|---|
| GetBuildsFromQueryWithAttributesRequest | List<ScanWithAttributes> |
| GetBuildsWithArtifactTransformRequest | List<ArtifactTransform> |
| GetBuildsWithAttributesRequest | List<ScanWithAttributes> |
| GetBuildsWithCachePerformanceRequest | List<Build> |
| GetScanAttribute | List<ScanWithAttributes> |
| GetSingleBuildArtifactTransformRequest | List<ArtifactTransform> |
| GetSingleBuildCachePerformanceRequest | Build |
| GetSingleBuildScanAttributesRequest | ScanWithAttributes |
| GetBuildsResourceUsageRequest | List<BuildWithResourceUsage> |
| GetSingleBuildResourceUsageRequest | BuildWithResourceUsage |
Filter
Once we have seen the simple usage, we need to introduce the Filter entity. It's optional for GetBuildsWithAttributesRequest
and required for GetBuildsWithCachePerformanceRequest.
Filter is critical to reduce the overhead of the process. Cache requests are expensive, and we want to limit the requests based
in parameters like project,tags or requestedTask.
| Property | Description | Default |
|---|---|---|
| maxBuilds | Max builds requested | 50 |
| sinceBuildId | Build Id where we'll start the request. If null, we start from the last build | null |
| project | Project name | null |
| includeFailedBuilds | Include failed builds in the request | false |
| requestedTask | Requested Task | null |
| tags | List of tags, , negation tags should be marked with !. Example: !main
|
emptyList() |
| user | User Build Scan | null |
| exclusiveTags | List of tags must match with the available tags in the Build Scan | false |
| concurrentCalls | Concurrent calls for the Attributes request | 10 |
| concurrentCallsConservative | Concurrent calls for Cache request | 5 |
| clientType | Type of client, API or CLI | ClientType.API |
Real Examples
Last 5000 ScanAttributes filtering by project
Requires GE 2023.3
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsFromQueryWithAttributesRequest(repository).get(Filter(maxBuilds = 5000)
Last 5000 ScanAttributes
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsWithAttributesRequest(repository).get(Filter(maxBuilds = 5000)
Last 5000 ScanAttributes filtering by project
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsWithAttributesRequest(repository).get(Filter(maxBuilds = 5000, project = "nowinandroid")
Last 5000 ScanAttributes filtering by project including CI tag
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsWithAttributesRequest(repository).get(Filter(maxBuilds = 5000, project = "nowinandroid", tags = listOf("CI","main), exclusiveTags = true)
Last 5000 ScanAttributes filtering by project including CI and main tag being exclusive
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsWithAttributesRequest(repository).get(Filter(maxBuilds = 5000, project = "nowinandroid", tags = listOf("CI"))
5000 ScanAttributes since BuildId x
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val getBuildScans = GetBuildsWithAttributesRequest(repository).get(Filter(maxBuilds = 5000, sinceBuildId = "x"))
Get Builds cache Performance from the last 5000 builds in project
val repository = GradleRepositoryRequest(GEClient(apiKey, url))
val filter = Filter(maxBuilds = 5000, project = "nowinandroid")
val scans = GetBuildsWithAttributesRequest(repository).get(filter)
val builds = GetBuildsWithCachePerformanceRequest(repository).get(scans, filter)
Network Client Configuration
We can extend the GEClient configuration using ClientConf. This entity allows to change the default configuration
for retries and exponential backoff configuration depending on the scenario/server load.
| Property | Default |
|---|---|
| maxRetries | 200 |
| exponentialBase | 2.0 |
| exponentialMaxDelay | 60000 |
Client Types
We define two different client types:
- API
- CLI
CLI provides a console feedback ideal for building CLI's(see next section):
Calculating Build Scans to retrieve
100% ################################################## -
Date first Build scan processed: 23/08/2023 15:35:36
Date last Build scan processed: 21/08/2023 23:53:42
Getting 50000 Build Scans Attributes
Existing tooling using ge-api-data
ProjectReport
https://github.com/cdsap/ProjectReport
CLI providing reports by user/task/project in a Develocity instance
Using GetBuildsWithAttributesRequest and GetBuildsFromQueryWithAttributesRequest
TaskReport
https://github.com/cdsap/TaskReport
This CLI provides general reports for duration/fingerprinting by type and path. Additionally, it offers single reports for
specific tasks.
Uses GetBuildsWithAttributesRequest, GetBuildsFromQueryWithAttributesRequest and GetBuildsWithCachePerformanceRequest
CompareGEBuilds
https://github.com/cdsap/CompareGEBuilds
This CLI compares two different build sequences and generates reports with durations based on metrics by variant. Ideal for Performance Regressions.