Modelling the Songbird's Vocalization Apparatus
Researchers at the University of Southern Denmark are tackling the virtual voice problem from the other end. They are attempting to understand how the zebra finch makes the noises it does, and accomplishes its range of sounds, by capturing the existing organs in its throat, modelling them in 3D, converting to a virtual environment and attempting to animate the model to produce the same sounds via fluid dynamics.
In short, they are creating a virtual voicebox by reverse engineering a physical one.
Lead researcher Dr Coen Elemans and team used high-field magnetic resonance imaging and micro-computed tomography of several zebra finch bodies to create a high resolution data set, which was used to create a single 3D model of the trachea.
They did this because, like with humans, we have made significant strides in decoding the neural codes used by the brain to order the voice box to produce sounds efforts in humans have met with some success, to decode the neural codes to produce synthetic speech when the human voice box has been destroyed.
However, unlike with humans, the anatomy of the complex physical structures of the avian throat, are not well understood. Back in common with humans, we don't entirely understand how the bird's anatomy leads it to create the complex waveform patterns it is capable of. So, by simulating the anatomy and changing various parts, whilst sending the same control signals to the virtual larynx, the scientists are hoping to understand exactly how the throat works, and which part modifies the sound in what ways.
Although their 3D models are not currently available to the general public and would be of limited use without the proper rigging and fluid dynamics setup in the simulation their work is leading us towards the point where we will be able to create a larynx inside the avatar itself, and when the user wishes to speak, the larynx inside the avatar treats the user's input as neural codes and moves in response. In short, the avatar would create the voice, each one basically designed for that avatar, produced as part of it.
So far, the multinational team has generated interactive 3D PDF models of the syringeal skeleton, soft tissues, cartilaginous pads, and muscles affecting sound production. These models show in detail the delicate balance between strength, and lightness of bones and cartilage required to support and alter the vibrating membranes of the syrinx at super fast speeds.
Dr Elemans stated in a release that This study provides the basis to analyze the micromechanics, and exact neural and muscular control of the syrinx. For example, we describe a cartilaginous structure which may allow the zebra finch to precisely control its songs by uncoupling sound frequency and volume .
In addition, the researchers found a previously unrecognised Y-shaped structure on the sternum which corresponds to the shape of the syrinx and could help stabilize sound production. By modelling the human larynx similarly, we are certain to discover if there are any unexpected contributions from soft tissue which we did not previously categorise as part of the process.
Fluid dynamics simulations which include airflow are very demanding simulations which cannot be run in real-time on even high-end home hardware. For the moment, such an approach to generating a virtual voice must remain strictly the purview of the supercomputer. However, of course the more we learn, the more we can pair down the simulation to its simplest elements, and predict how they would behave under different airflows. That will make it possible to construct such simulations without resorting to a fluid dynamics setup or we will reach the point where fluid dynamics is omnipresent in virtual environments anyhow. At either point, this approach to producing a virtual voice will become viable, for all our simulations.