Mapping a New Face with CONNECTED COLORS Tokyo-based WOW studios uses Cinema 4D to combine 3D and live performance with stunning results.

In early 2016, visual design studio WOW in Tokyo produced 'Connected Colors', a project combining animated 3D graphics, projection mapping and live performance. The video was made for a global Intel promotion titled 'Experience Amazing'. Intel requested innovators from around the world to create artworks for the campaign, and selected artist Nobumichi Asai to create a face mapping project.

While Asai took the role of creative director and technical director, WOW worked as the production company for the live action and CG animations. “Face mapping is a new technique of projecting moving images onto a living face, incorporating and using the natural expressions and expressiveness of the face as part of the final result, similar to the art of makeup, which has millenniums of history,” he said.

“Each project in Intel’s promotion was to have a different theme. The theme of ‘Connected Colors’ was inspired by the connected world and Intel's concept of the ‘Internet of Things’, or IoT. The connected world has two sides - a positive side that brings efficiency, optimization, harmony and symbiosis to people’s lives, and another side recognizes the need for operational rules and administration. I wanted the message for this project to question the use of technology, and decided to create an artwork expressing the positive side of connections.

“When we focus technology on life, we see that animals, plants and insects are all based on the same structures – on DNA – and all have different colors. I used the motif of the colors that all forms of life have in nature, including humans, and connected those colors.”

Previous to the Intel project, Asai had worked with projection mapping and virtual reality on his own. His first venture into face mapping was a private project titled ‘OMOTE’, and later he did some face mapping for Japanese TV programming and music videos.

What was different about this project was the level of precision it demanded from their tracking and projection. Lack of precision would affect latency alignment and color reproduction, causing the audience to feel disconnected. Therefore Asai, WOW’s CG artist Shingo Abe and his team made substantial improvements to the latency, alignment and color systems within their production and projection set-ups, giving them much better results compared to Asai’s previous projects.

Using computers with 6th generation Intel Core i7 processors and a light projector, Asai has successfully turned the human face into a new kind of canvas. The processors are tuned with Intel’s overclocking toolkit to push their performance and achieve the level of precision Asai wanted.

Asai and Abe selected their hardware carefully. Asai said, “For tracking we used five OptiTrack motion capture cameras, which capture at 240fps, and Intel PCs. For projection we had a Panasonic projector, looking for the lowest latency, highest resolution and widest dynamic range we could achieve.

Abe said, “For the animations and elements of the animated textures I used Cinema 4D, compositing in After Effects, which is a very fast and natural workflow for motion graphics. I could produce many iterations very quickly, giving me time to get closer to the result and quality I was looking for. MoGraph also has great tools for this kind of project. I use After Effects and Cinema 4D for most of my work, combined with Photoshop and Illustrator.”

First, the artists took a 3D scan of the performer’s face, unwrapped the UVW and captured the texture of the face, and then imported the geometry and the texture into Cinema 4D. After building a 3D model of the face, they created the animations to fit to the UV of this model, and then applied the animation as a texture. This was done for each animation, and these 3D models then became the projection source, or generator.”

At performance time, the position and rotation of the performer’s face were calculated from the tracking data in real time, and we synchronized the position and rotation of the 3D model with it. Thus, an accurate, perspective view of the 3D model’s animated textures was projected onto the real face.

Abe said, "The whole project had to be completed in six months, but because it took us some time to get our ideas and approaches into shape we only had one month to spend on production work. The speed of Cinema 4D really helped us."

“One of the first and most important aspects of face mapping we discovered was how sensitive viewers are to even slight changes to the face that are incongruous or out of harmony with the performance. Compared to small alterations made to landscapes or product shots, for instance, audience reactions to faces are much stronger. So we had to do a lot of testing and work rapidly through many trials.”

Cinema 4D’s MoGraph tools were especially useful in three of the animations. The first is the scanning sequence at the start of the video. Later on we see a transition from the model’s skin to a black-and-yellow lizard’s skin, and a few seconds later a black-and-red frog’s skin forms and slides off the face. 12sec, 50sec, 1.03sec

MoGraph’s Inheritance Effector was used in the scanning sequence to transition a composited wireframe as a 3D effect as it moves over the face, and then spreads out from the nose. When this was rendered with the Cel Renderer as a post effect, it created a complex pattern.

For the lizard’s skin, the Shader Effector forced a gradual change across the texture of the face’s surface. The Random Effector was used to produce the randomized, fractured black-and-red pieces of the frog skin, while the slide-off effect was created using Dynamics, which have parameters that can be set through the Effectors.

Once all of these animations, looks and effects were completed as WOW wanted them to be, precise projection data was generated using original software the team developed for this purpose. Interestingly, although this production gives you an impression of interactivity and spontaneity, the actual projected movie is fixed. Only the projection, driven by the motion capture, was interactive. This made it possible to storyboard the piece in a deliberate way.

At times, the model appears to blink and open up her eyes to reveal an animal’s eyes, or to suddenly smile. In those cases, Abe explained, she did not actually blink in real-time. Viewers are watching a projected animation - only. “We have done a lot of research and created numerous simulations for eye expressions, adjusting them over and over,” he said. “People can immediately sense even the smallest incongruity around the eyes. In daily life, they watch other people’s eyes and, from there, read slight changes in feelings. To test those projections, we used a mockup of the scanned face model, Cinema 4D and other software until we could finally create very natural blinks and smiles.”

Asai commented, “Modifying details of the face is very sensitive work. It is fairly easy to express humor or fear, or achieve grotesque results. Conversely, beautiful expressions on faces are very difficult.”

WOW website:
http://www.w0w.co.jp/en/index.html

Nobumichiasai website:
http://www.nobumichiasai.com/