imagebitmap-getimagedata-demo

Demos for performance of capturing and processing webcam frames using several methods.

This repo serves to prove out the usefulness of adding a ImageBitmap.getImageData method to the spec, with the option of reusing an existing ArrayBuffer to avoid allocating large amounts of large size garbage.

It's worth noting that Firefox fails all versions of the current implementation due to running the test machine out of memory in under 5 seconds.

Proposal

dictionary GetImageDataOptions {
  ArrayBuffer? buffer;
  boolean neuter = false;
}

partial interface ImageBitmap {
  [NewObject] Promise<ImageData> getImageData(int sx, int sy, int sw, int sh, optional GetImageDataOptions = {});
}

ImageBitmap Patches

There's a patch for both Chromium and Firefox. They can be found under the patches directory. The Firefox patch does not implement ImageBitmap.getImageData with a promise return value as I didn't spend the time to find out how to do promises in Firefox's code. The Chromium patch does return a promise, but all work is done on the main thread.

Testing

All testing is done on a Raspberry Pi Model 3B+, which has 1 GB of memory. Camera is a Logitech C920 Pro. Swapfile is disabled during testing to avoid the uncertainty that might cause with the performance results.

Average CPU and memory usage data were gathered at 5 second intervals over a 2 minute period using sar. Baseline memory usage for the system is 140-180 MB.

The test demo uses the ImageData to compute the average color of the frame. It uses striding so that it takes less than 30 MS for a 1080p image so that it is not the bottleneck in the test.

All test runs involved a fresh start of the browser.

Results

As of the time of this testing (July 2019), Firefox is unable to run this test scenario on the Pi without crashing the system due to OOM. On Chromium the new ImageBitmap.getImageData code path offers superior performance to OffscreenCanvas, with the worker FPS clocking in at 60 FPS vs 5 FPS. The improvement is significant enough that the bottleneck is moved to elsewhere in the Chromium code, with the getImageData code path performing well enough to handle 1080p@30.

Chromium

The neuter option appears to have little effect on either CPU or memory usage.

1080p@30

The results for ImageBitmap.getImageData usage with ImageCapture.grabFrame seem to be bottlenecked by ImageCapture.grabFrame converting the frame using libyuv. chrome://tracing shows this takes ~25-35 MS per frame, which eats through most of the time needed to process a frame at 30 FPS. If the bottleneck was alleviated, it looks like the FPS could likely hit the 30 FPS mark, given that the worker processes frames fast enough for 45+ FPS.

720p@30

NOTE: The ImageCapture.grabFrame implementation seems to limit frame requests to the camera frame rate, so an FPS of 30 using that capture method is effectively 'maxxed out'.

Firefox

Firefox implementation of ImageCapture is out of spec and only implements takePhoto, which returns a Blob, so there's an extra step using createImageBitmap. It also only supports WebGL for OffscreenCanvas, so we can't use that either.

Interestingly average memory usage is lower when the neuter option is not used.

WARNING: With Firefox the CanvasRenderingContext2D method tends to run off the rails quite quickly and will use a lot of memory and likely lock up the process. It will crash a Raspberry Pi, or a low-memory Android phone.

1080p@30

720p@30