3D face landmark detection using MediaPipe's Facemesh and Iris tracking models
APACHE-2.0 License
Face landmark detection using two MediaPipe models: Facemesh and Iris, implemented inside Unity using over 120 Custom Render Textures.
Designed for mobile platforms, the two networks are perfect for use in VR. Both networks are capped at 60 FPS in this implementation but can be modified to run faster.
Figure 1. Facemesh Outputs
The original Facemesh model has three main outputs: face flag, 468 3D face points, and 133 2D contour points (edge points of face, eyes, mouth). My implementation of Facemesh throws out the 133 2D contour outputs to save computation time since I won't be using those.
To retrieve the head rotations, I use SVD (singular value decomposition) on three points outputted from Facemesh to get the rotation matrix. The eye positions are also used to feed into the Iris tracking model.
Figure 2. Iris Outputs
The Iris model has two outputs, same as the original, 71 3D brows and contour points and 5 3D iris points.
Right eye inputs must be flipped horizontally during input and flipped back at output.
At the final step, a shader takes Facemesh and Iris model outputs, calculates blendshape values based on key points and smooths the results.
Location: .../FaceLandmarkDetection/CRTs/FaceMesh
Name | Location (X, Y) |
---|---|
Rotation Matrix Top | 0, 0 |
Rotation Matrix Middle | 1, 0 |
Rotation Matrix Bottom | 2, 0 |
Scale and Frobenius norm | 3, 0 |
Target Centroid | 4, 0 |
Source Centroid | 5, 0 |
Location: .../FaceLandmarkDetection/CRTs/Iris
One iris network is used for both eyes. To keep track of which output is left or right, each layer of the network adds a 10000000.0 to output location (0, 0) if it's the right eye.
Right eyes inputs are flipped horizontally.
Location: .../FaceLandmarkDetection/CRTs/
Name | Location (X, Y) |
---|---|
Rotation Matrix Top | 0, 0 |
Rotation Matrix Middle | 1, 0 |
Rotation Matrix Bottom | 2, 0 |
Mouth open, shrink, smile | 3, 0 |
Eyes blink left, right | 4, 0 |
Brow left up/down, right up/down | 5, 0 |
Iris left XY, right XY position | 6, 0 |
The rotation matrix is copied over from the Procrustes Analysis CRT.
This is more of a tech demo, not made for actual use. Setting this up correctly is a tedious process for an impractical effect. If you wish to continue, you'll need to know how to edit shaders.
At least two meshes, the face must be a completely separate mesh from the rest of the body. No more than 16384 vertices on the face because the blendshapes are baked onto 128 x 128 textures.
Start creating the blendshapes in Blender:
Save as FBX, import into Unity.
Open up the Bake Blendshapes editor window located under Tools -> SCRN -> Bake Blendshapes in your menu.
Hit "Bake" when you're done filling out the blendshapes. The texture should save into Assets/FaceLandmarkDetection/
as xxxx_blendBaked.asset
If you want to use the example shaders in the Demo folder, put the FaceTrack material on the face mesh and HeadRotation on everything else.
xxxx_blendBaked.asset
as the Baked Blendshape Texture in the FaceTrack material.Create a mask for HeadRotation, it uses UV0, 0.0 is no rotation, 1.0 is full rotation. Ex. if we want the neck to gradually follow the rotation, put a gradient on the neck UV.
Most likely the example shaders aren't good enough, you can move it into your own shader by copying everything I have between these tags in both the shader into an equivalent Vertex shader
// ------------------- FACE TRACKING START ------------------- //
...
// ------------------- FACE TRACKING END ------------------- //
FaceLandmarkDetection/Demoes/Face Tracking/Prefabs
folder somewhere on your avatar so the CRTs get referenced.The purpose of the Python and C++ code is to help me debug intermediate layers of the networks. It runs very slow because it has to output every single network layer.
I suggest using a virtual environment and a package manager.
Thanks to Merlin, Raliv and orels1 for the help.
If you have questions or comments, you can reach me on Discord: SCRN#8008 or Twitter: https://twitter.com/SCRNinVR