arcore-android-sdk

New APIs and capabilities

Flash Mode

ARCore now supports enabling the device's torch mode, allowing for better perception capabilities in dark environments.

Kotlin/Java:

• Config.FlashMode: configuration option for enabling or disabling the flash unit on the device.
• See the Developer guide for more details on how to detect flash capabilities, and how to enable the torch mode.

C:

• ArFlashMode: configuration option for enabling or disabling the flash unit on the device.
• See the Developer guide for more details on how to detect flash capabilities, and how to enable the torch mode.

Known issues

• None.

Breaking & behavioral changes

• None.

New APIs and capabilities

• None.

Deprecations

• None.

Bug fixes

• Fixed situations where the normal vector of vertical planes would sometimes face away from the camera. google/model-viewer#4771.
• Various bug fixes and performance improvements.

Known issues

• None.

Breaking & behavioral changes

• None.

New APIs and capabilities

• None.

Deprecations

• None.

Bug fixes

• Various bug fixes and performance improvements.

Bug fixes

• Fixed an issue when installing ARCore on certain devices using Android 14.

Other changes

• Updated okhttp dependency from 3.12.0 to 4.10.0 in the ml_kotlin sample.

Bug fixes

• #1271: Fixed an issue that could cause methods of the ArCoreApk class (Java) and ArCoreApk_* functions (NDK) to return errors or incorrect non-availability when running in dev builds or other otherwise as a side-loaded app.

Breaking & behavioral changes

• ARCore's targetSdkVersion has been updated to Android API level 34. If your app does not specify a targetSdkVersion, your app's targetSdkVersion will become 34 due to manifest merging.

Bug fixes

• #1602: Fixed a NullPointerException when closing an ARCore session with a SharedCamera that was already closed.

Known issues

• None.

Breaking & behavioral changes

• None.

New APIs and capabilities

• None.

Deprecations

• None.

Other changes

• Upgraded sample apps using de.javagl:obj to version 0.4.0 from version 0.2.1.

Bug fixes

• Various bug fixes and performance improvements.

Known issues

• #1573: On certain Android devices, enabling 60FPS mode causes the tracking to become unsynchronized with the camera feed. The 60FPS mode has been temporarily disabled on these devices and will not appear in supported camera configurations.

Other changes

• Removed Streetscape Geometry shaders from samples that do not use them.

Bug fixes

• Various bug fixes and performance improvements.

Known issues

• #1573: On certain devices, enabling 60FPS mode causes the tracking to become unsynchronized with the camera feed. The 60FPS mode has been temporarily disabled on these devices and will not appear in supported camera configurations.

New APIs and capabilities

Scene Semantics

Scene Semantics provides richer details about the surrounding scene, making it easier to understand the world around the user. Scene Semantics runs an ML model on the camera image feed and provides a semantic image with each pixel corresponding to one of 11 labels of outdoor concepts.

See Introduction to Scene Semantics and the semantics_java sample app for more information.

Kotlin/Java:

• Frame.acquireSemanticImage()
• Frame.getSemanticLabelFraction()
• Frame.acquireSemanticConfidenceImage()
• SemanticLabel

C:

• ArFrame_acquireSemanticImage()
• ArFrame_getSemanticLabelFraction()
• ArFrame_acquireSemanticConfidenceImage()
• ArSemanticLabel

Streetscape Geometry

Streetscape Geometry is a new ARCore Geospatial API that provides the geometry of buildings and terrain in an area around the user when the Geospatial API is enabled.

See Streetscape Geometry Developer Guide (Kotlin/Java) and Streetscape Geometry Developer Guide (C) for more information. Additionally, the geospatial_java sample app has been updated to include Streetscape Geometry.

Kotlin/Java:

• Config.StreetscapeGeometryMode
• StreetscapeGeometry
• StreetscapeGeometry.Quality
• StreetscapeGeometry.Type
• Mesh

C:

• ArStreetscapeGeometryMode
• ArStreetscapeGeometry
• ArStreetscapeGeometryQuality
• ArStreetscapeGeometryType
• ArMesh

Rooftop anchors

Rooftop anchors are a new Geospatial anchor type that helps you anchor content to a rooftop.

See Geospatial anchors (Java) and Geospatial anchors (C) for more information. Additionally, the geospatial_java sample app has been updated to include Rooftop anchors.

Kotlin/Java:

• ResolveAnchorOnRooftopFuture
• Earth.resolveAnchorOnRooftopAsync()
• Anchor.RooftopAnchorState

C:

• ArEarth_resolveAnchorOnRooftopAsync()
• ArResolveAnchorOnRooftopFuture
• ArRooftopAnchorState

Geospatial Depth

When both the Depth API and Streetscape Geometry modes are enabled in an ARCore session in areas with VPS coverage, the generated depth images are improved using Streetscape Geometry. Streetscape Geometry is integrated into each generated depth image out to a range of 65.535 meters. No changes are needed when calling the Depth API to see this benefit.

See Geospatial Depth (Java) and Geospatial Depth (C) for more information.

Vulkan rendering support

ARCore now provides Vulkan rendering support by exposing the camera image through an AHardwareBuffer. You can use this Hardware Buffer by binding it to a VkImage.

See Rendering your ARCore application using Vulkan and the hello_ar_vulkan_c sample app for more information.

Java:

• Config.TextureUpdateMode
• Frame.getHardwareBuffer()

C:

• ArTextureUpdateMode
• ArFrame_getHardwareBuffer()

Electronic Image Stabilization (EIS)

You can configure ARCore to use Electronic Image Stabilization, which smooths camera frames for a more fluid user experience.

See Enabling Electronic Image Stabilization and the hello_eis_kotlin sample app for more information.

Kotlin/Java:

• Config.ImageStabilizationMode
• Frame.transformCoordinates3d()
• Session.isImageStabilizationModeSupported()
• There is a known issue where calling Frame.transformCoordinates3d() with 2D coordinates other than OPENGL_NORMALIZED_DEVICE_COORDINATES will cause a crash instead of throwing IllegalArgumentException. This will be resolved in the following release.

C:

• ArImageStabilizationMode
• ArFrame_transformCoordinates3d()
• ArSession_isImageStabilizationModeSupported()
• There is a known issue where calling ArFrame_transformCoordinates3d() with 2D coordinates other than AR_COORDINATES_2D_OPENGL_NORMALIZED_DEVICE_COORDINATES will cause your app to abort instead of returning AR_ERROR_INVALID_ARGUMENT. This will be resolved in the following release.

ARCore async APIs

This release introduces new ARCore async APIs to improve ergonomics when working with asynchronous operations, following the Future paradigm.

Java:

• Resolving Terrain anchors: Use Earth.resolveAnchorOnTerrainAsync() to obtain a ResolveAnchorOnTerrainFuture. The old symbol Earth.resolveAnchorOnTerrain() has been deprecated and may be removed in a later version of ARCore.
• Resolving Cloud anchors: Use Session.resolveCloudAnchorAsync() to obtain a ResolveCloudAnchorFuture. The old symbol Session.resolveCloudAnchor() has been deprecated and may be removed in a later version of ARCore.
• Hosting Cloud anchors: Use Session.hostCloudAnchorAsync() to obtain a HostCloudAnchorFuture. The old symbols Session.hostCloudAnchor() and Session.hostCloudAnchorWithTtl() have been deprecated and may be removed in a later version of ARCore.
• Check APK availability: Use ArCoreApk.checkAvailabilityAsync() to register a callback. The old symbol ArCoreApk.checkAvailability() has not been deprecated.

C:

• Resolving Terrain anchors: Use ArEarth_resolveAnchorOnTerrainAsync() to obtain a ArResolveAnchorOnTerrainFuture or use ArResolveAnchorOnTerrainCallback. The old symbol ArEarth_resolveAndAcquireNewAnchorOnTerrain() has been deprecated and may be removed in a later version of ARCore.
• Resolving Cloud anchors: Use ArSession_resolveCloudAnchorAsync() to obtain a ArResolveCloudAnchorFuture or use ArResolveCloudAnchorCallback. The old symbol ArSession_resolveAndAcquireNewCloudAnchor() has been deprecated and may be removed in a later version of ARCore.
• Hosting Cloud anchors: Use ArSession_hostCloudAnchorAsync() to obtain a ArHostCloudAnchorFuture or use ArHostCloudAnchorCallback. The old symbols ArSession_hostAndAcquireNewCloudAnchor() and ArSession_hostAndAcquireNewCloudAnchorWithTtl() have been deprecated and may be removed in a later version of ARCore.
• Check APK availability: Use ArCoreApk_checkAvailabilityAsync() to register a callback. The old symbol ArCoreApk_checkAvailability() has not been deprecated.

Deprecations

C:

• ArVpsAvailabilityFuture_getState() has been deprecated. Use ArFuture_getState() instead.
• ArVpsAvailabilityFuture_cancel() has been deprecated. Use ArFuture_cancel() instead.
• ArVpsAvailabilityFuture_release() has been deprecated. Use ArFuture_release() instead.
• ArCheckVpsAvailabilityCallback has been deprecated. Use ArVpsAvailabilityCallback instead.

Other changes

• There are now base types for all async operations:
  • In Kotlin/Java, an interface Future, with the methods Future.getState() and Future.cancel(). The class VpsAvailabilityFuture now implements Future.
  • In C, a base struct type ArFuture, with functions ArFuture_getState(), ArFuture_cancel() and ArFuture_release(). All future structs in C can be cast to ArFuture. In C++, you can use ArAsFuture() to do the cast.

Bug fixes

• arcore-unity-extensions#90: Fixed a crash when destroying an ARSession while a Cloud Anchor is being resolved.

Other changes

• Fixes issue #1561: binary artifacts in the repository by adding exemptions.
• Adds copyright statements added to files that were inadvertently missing them previously.

New APIs and capabilities

Geospatial pose accuracy

All Geospatial poses now expose their orientation accuracy of the Yaw rotation, replacing heading accuracy.

• Kotlin/Java reference
  • GeospatialPose.getOrientationYawAccuracy()
• C reference
  • ArGeospatialPose_getOrientationYawAccuracy()

Deprecations

• Kotlin/Java

  • GeospatialPose.getHeading() has been deprecated. Please use GeospatialPose.getEastUpSouthQuaternion() instead.
  • GeospatialPose.getHeadingAccuracy() has been deprecated. Please use GeospatialPose.getOrientationYawAccuracy() instead.

• C

  • ArGeospatialPose_getHeading() has been deprecated. Please use ArGeospatialPose_getEastUpSouthQuaternion() instead.
    ArGeospatialPose_getHeadingAccuracy() has been deprecated. Please use ArGeospatialPose_getOrientationYawAccuracy() instead.

Other changes

• ARCore Geospatial API Session.checkVpsAvailabilityAsync() and ArSession_checkVpsAvailabilityAsync() will continue to retry silently in the background if it is unable to establish a connection to the ARCore service for 5 seconds.

New APIs and capabilities

Check Geospatial VPS availability

A new ARCore Geospatial API has been added to check for Visual Positioning System (VPS) availability at a specified horizontal position at runtime. This API can be used without an active AR session, for example, to present an "Enter AR" button only when VPS is available.

• Kotlin/Java
  • Developer guide: Check VPS Availability
  • Session.checkVpsAvailabilityAsync()
• C
  • Developer guide: Check VPS Availability
  • ArSession_checkVpsAvailabilityAsync() determines if VPS is available in a given area. This will initiate a query with a remote ARCore API service.

Geospatial pose orientation

Geospatial poses now expose their orientation in 3D space.

• Kotlin/Java reference
  • GeospatialPose.getEastUpSouthQuaternion()
• C reference
  • ArEarth_getGeospatialPose()

Geospatial pose conversions

Geospatial poses can now be converted to and from world-space poses.

• Kotlin/Java reference

  • Earth.getGeospatialPose() converts a Pose to a GeospatialPose in the same position.
  • Earth.getPose() converts a geospatial location and orientation to a Pose in the same position.

• C reference

  • ArEarth_getGeospatialPose() converts the input pose to a ArGeospatialPose in the same position.
  • ArEarth_getPose() converts a geospatial location and orientation to an ArPose in the same position.

Note that Geospatial poses obtained from these functions have a heading value set to zero. Instead, use its EUS quaternion instead.

Other changes

• The geospatial_java sample has been adjusted to check for VPS availability.
• The geospatial_java sample now supports creating a Geospatial anchor by tapping on the AR camera view.
• Updated Kotlin samples to Kotlin version 1.5.30 from 1.4.20.

Bug fixes

• Fixed "Device Not Supported" being reported for emulator images on ARM host machines.
• ArEarth_getEarthState() and ArAnchor_getTerrainAnchorState() now correctly return ERROR_NOT_AUTHORIZED when there is an authentication issue instead of returning ERROR_INTERNAL.

Breaking & behavioral changes

• To use Cloud Anchors, you must now enable the ARCore API instead of the older ARCore Cloud Anchor API, which is now deprecated. To support older app versions, you may enable both while making the transition. Apps built with ARCore SDK 1.32.0 and lower will target the older API, while apps built with ARCore SDK 1.33.0 and higher will target the new API. Note the following:
  • The new API uses the domain name arcore.googleapis.com instead of arcorecloudanchor.googleapis.com.
  • If you are using an API Key which is restricted by API, you must allow the ARCore API.
  • If you send requests to the ARCore Cloud Anchor Management API from your backend, you must also change the domain name to arcore.googleapis.com, after enabling the ARCore API.
  • The old API/endpoint will continue to be supported through August 2023.
  • See ARCore 1.33 Cloud Anchor endpoint changes for more details.
• ArImage has been removed. Use android.media.Image instead.
• targetSdkVersion inside the ARCore aar has been updated to API level 33. Specifying a targetSdkVersion in your project's build.gradle or AndroidManifest.xml will override the ARCore value.

New APIs and capabilities

ARCore Geospatial Terrain anchor API - new

The new ARCore Geospatial Terrain anchor API can create an anchor at a specified horizontal position and altitude relative to the horizontal position's terrain.

• Kotlin/Java reference

  • Earth.resolveAnchorOnTerrain() creates a new Anchor at the specified latitude, longitude, altitude above the terrain and orientation.
  • Anchor.TerrainAnchorState describes the current Terrain anchor state of Anchor.

• C reference

  • ArEarth_resolveAndAcquireNewAnchorOnTerrain() creates a new Anchor at the specified latitude, longitude, altitude above the terrain and orientation.
  • ArTerrainAnchorState describes the current Terrain anchor state of ArAnchor.

Other changes

• Updated Java and native sample apps to targetSdkVersion 33.

Breaking & behavioral changes

• ARCore's targetSdkVersion has been updated to Android API level 32. If your app does not specify a targetSdkVersion, your app's targetSdkVersion will become 32 due to manifest merging. Specifying a targetSdkVersion in your project's build.gradle or AndroidManifest.xml will override ARCore's targetSdkVersion value.

Other changes

• Updated Java and C sample apps to targetSdkVersion 32.

Bug fixes

• The known issue in Google Play Services for AR (ARCore) version 1.31 where configuring a session with GeospatialMode.ENABLED while the session is running may cause the session to hang has been fixed.

Breaking & behavioral changes

• Updated ARCore's minimum supported Android API level to 19 (from 18).
  • The upcoming Android NDK version (r24) will remove support for Android API levels v16-18.
  • For more information, see the Android NDK r23 changelog.

New APIs and capabilities

ARCore Geospatial API - new

The new ARCore Geospatial API utilizes data from Google Earth 3D models and Street View image data from Google Maps to enable your app for immersive, global-scale, location-based augmented reality experiences.

• Kotlin/Java reference

  • Earth provides localization ability in geospatial coordinates.
    • Earth.createAnchor() creates a new Anchor at the specified latitude, longitude, altitude, and orientation.
  • Earth.Earthstate describes the current state of Earth, including its TrackingState.
  • GeospatialPose describes a specific latitude, longitude, altitude, and compass heading.

• C reference

  • ArEarth provides localization ability in geospatial coordinates.
    • ArEarth_acquireNewAnchor() creates a new Anchor at the specified latitude, longitude, altitude, and orientation.
    • ArEarthState describes the current state of ArEarth, including its ArTrackingState.
  • ArGeospatialPose describes a specific latitude, longitude, altitude, and compass heading.

ARCore Depth API improvements

The maximum range of Depth measurements has increased from 8191mm to 65535mm when building with ARCore SDK 1.31 and later. The Depth API now uses all 16-bits to represent depth, where previously only 13 bits were used for depth values.

To reflect this change in behavior, some Depth API functions have new names and behavior changes. The old versions of these functions will have no change in behavior, though it is recommended to make use of the longer depth range. For more information, see ARCore Depth changes in ARCore 1.31.

• Kotlin/Java
  • The Depth API function calls have changed:
    • Frame.acquireDepthImage() has been deprecated in favor of Frame.acquireDepthImage16Bits().
    • Frame.acquireRawDepthImage() has been deprecated in favor of Frame.acquireRawDepthImage16Bits().
    • The output image formats for both calls have changed from
      android.graphics.ImageFormat.DEPTH16 to
      android.hardware.HardwareBuffer.D_16.
    • Depth is still represented as a 16-bit integer in units of
      millimeters, but now all 16-bits are used to represent depth, allowing
      for a maximum expressible range of 65535mm (previously 8191mm).
• C
  • The Depth API function calls have changed:
    • ArFrame_acquireDepthImage() has been deprecated in favor of ArFrame_acquireDepthImage16Bits().
    • ArFrame_acquireRawDepthImage() has been deprecated in favor of ArFrame_acquireRawDepthImage16Bits().
    • The output image formats for both calls have changed from AR_IMAGE_FORMAT_DEPTH16 to AR_IMAGE_FORMAT_D_16.
    • Depth is still represented as a 16-bit integer in units of
      millimeters, but now all 16-bits are used to represent depth, allowing
      for a maximum expressible range to go 65535mm (previously 8191mm).

Deprecations

• Kotlin/Java

  • The Depth API function calls Frame.acquireDepthImage() and
    Frame.acquireRawDepthImage() are deprecated. Please use
    Frame.acquireDepthImage16Bits() and Frame.acquireRawDepthImage16Bits()
    instead.

• C

  • The Depth API function calls ArFrame_acquireDepthImage() and
    ArFrame_acquireRawDepthImage() are deprecated. Please use
    ArFrame_acquireDepthImage16Bits() and
    ArFrame_acquireRawDepthImage16Bits() instead.

Known issues

• Configuring a session with GeospatialMode.ENABLED while the session is running may cause the session to hang. This issue will be resolved in Google Play Services for AR (ARCore) version 1.32. In the meantime, ensure your app only sets a configuration with GeospatialMode.ENABLED while the session is paused.

Other changes

• HelloAR sample apps now visualize the Depth API representing longer-range
  depth values with new colors.

Known issues

• None

Other changes

• Annotated API return values with @NonNull and @Nullable.
• Updated the build tool versions used for building samples: Gradle to 7.0.2 and Android Gradle plugin to 7.0.4. Upgrading these tools is not required for existing ARCore applications. For new features, known issues, incompatibility information, see Android documentation on Android Gradle plugin.

Known issues

• None

Breaking & behavioral changes

• Beginning in November 2022, AR-enabled apps will no longer be able to acquire NDK images or image metadata. The SDK versions impacted depend on the specific function being called, see the deprecation announcement to learn more.
  • C: Calls to ArImage_getNdkImage and ArImage_getNdkCameraMetadata will always return nullptr for the AImage and ACameraMetadata objects.
  • Java: The Image object returned by Frame#acquireCameraImage() will be of size 0x0 pixels. Calls to Frame#getImageMetadata() will always throw IllegalArgumentException.
  • Unity (AR Foundation):
    • When using AR Foundation 2.1 (Unity 2019 LTS), calls to XRCameraSubsystem.TryGetLatestImage(out XRCameraImage) will always return false.
    • AR Foundation 4.x and later versions are unaffected, calls to XRCameraSubsystem.TryGetLatestImage(out XRCameraImage) will behave normally.
  • ARCore SDK for Unity (deprecated): Calls to Frame.CameraImage.AcquireCameraImageBytes() will be of size 0x0 pixels. Calls to Frame.CameraMetadata.GetAllCameraMetadataTags() will return an empty List<CameraMetadataTag>.
• ArImage#getCropRect() now returns a full-sized crop rectangle, i.e. all pixels in the Image are valid. See android.media.Image#getCropRect() for additional information.

Other changes

• hello_ar_java and hello_ar_kotlin: Changed an object's color when it is placed using SCREENSPACE_WITH_APPROXIMATE_DISTANCE to better distinguish objects placed using Instant Placement.

Bug fixes

• persistent_cloud_anchor_java: Added a missing build dependency to the sample. Fixes a bug where hosting any anchor would have state CloudAnchorState ERROR_NOT_AUTHORIZED even if the project was set up correctly.

Known issues

• None

Breaking & behavioral changes

• Beginning in November 2022, AR-enabled apps will no longer be able to acquire NDK images or image metadata. The SDK versions impacted depend on the specific function being called, see the deprecation announcement to learn more.
  • C: Calls to ArImage_getNdkImage and ArImage_getNdkCameraMetadata will always return nullptr for the AImage and ACameraMetadata objects.
  • Java: The Image object returned by Frame#acquireCameraImage() will be of size 0x0 pixels. Calls to Frame#getImageMetadata() will always throw IllegalArgumentException.
  • Unity (AR Foundation):
    • When using AR Foundation 2.1 (Unity 2019 LTS), calls to XRCameraSubsystem.TryGetLatestImage(out XRCameraImage) will always return false.
    • AR Foundation 4.x and later versions are unaffected, calls to XRCameraSubsystem.TryGetLatestImage(out XRCameraImage) will behave normally.
  • ARCore SDK for Unity (deprecated): Calls to Frame.CameraImage.AcquireCameraImageBytes() will be of size 0x0 pixels. Calls to Frame.CameraMetadata.GetAllCameraMetadataTags() will return an empty List<CameraMetadataTag>.
• ArImage#getCropRect() now returns a full-sized crop rectangle, i.e. all pixels in the Image are valid. See android.media.Image#getCropRect() for additional information.

New APIs and capabilities

• None

Deprecations

• None

Other changes

• None

Bug fixes

• None

Known issues

• None

Breaking & behavioral changes

• None

New APIs and capabilities

• None

Deprecations

• None

Other changes

• Added hello_ar_kotlin to demonstrate best practices when using Kotlin. For more information see the QuickStart for Android.
• Added ml_kotlin to demonstrate how an image classification model can be used with ARCore. For more information see Use ARCore as input for Machine Learning models.

Bug fixes

• None

Known issues

• None

Breaking & behavioral changes

• targetSdkVersion inside the ARCore aar has been updated to API level 31. Specifying a targetSdkVersion in your project's build.gradle or AndroidManifest.xml will override the ARCore value.

New APIs and capabilities

• Added support for describing ARCore dataset locations with URIs for Recording and Playback.
  • C: Developer guide
  • Java: Developer guide

Deprecations

• The file path string based Recording and Playback methods RecordingConfig#getMp4DatasetFilePath() and RecordingConfig#setMp4DatasetFilePath() have been deprecated in favor of URI based methods RecordingConfig#getMp4DatasetUri() and RecordingConfig#setMp4DatasetUri().

Other changes

• Updated Java and native sample apps to targetSdkVersion 31 (Android 12).

Bug fixes

• None