Lego AR Kiosks: Colour Magic Symbol
At the beginning of 2009, Lego announced in a press release that they were teaming with German AR developer Metaio to develop an augmented reality solution for Lego kits. This went on sale in Germany and California, as test markets. So far, the test is not over, but from all indications they are doing very well.
The technology is simple, and is magic symbol based. Atop all LEGO kit boxes distributed in the test areas, is a colour magic symbol - a high contrast printed square, framed in a single colour, which is a set size, and possesses a simple design that a camera can read, and compare to its database. It also serves as a way of fixing the location and orientation of the square, in space.
Lego's system is unusual, in that they have gone for a colour magic symbol rather than pure black and white. It is not unusual enough to be unique, but it is still not the mainstream solution.
The idea is if you hold a 'digital box' lego kit up to a viewing kiosk, the kiosk will read the symbol printed on the top of the box, if you hold the box flat. Two cameras at the top of the kiosk handle this task - it requires two cameras for stereoscopy, to triangulate depth. One of these cameras is a high megapixel model, the other is not. The second is necessary only to provide a second perspective for triangulation, not to produce a display, so it does not have to have a high bit rate. You can see both cameras in that shiny black square on the photo of a kiosk below.
Below the cameras is an augmented reality mirror, much the same as seen in a few other places in recent years. It consists of a display screen which outputs the view of the physical world from one of the cameras, the higher megapixel one, shown on the left in this photo. Then the internal computer renders the 3D data to match the desired position and orientation, and superimposes it on top of the physical.
In this case, it is not superimposed on top of the symbol itself, which remains visible. Instead, it uses the placement of the symbol, together with how straight or rotated it is, to work out where the flat front of the box should be. This is where the virtual data is superimposed.
On the photo above, you can see the magic symbol, as the picture inside a white frame. As long as this stays in view, however you tilt the box, the 3D model stays locked in place. Remove the symbol from view by tilting the box too far, and the virtual data vanishes. To locate the symbol if you hold a box, it is on the end opposite the bar code. Contrary to some initial reports, the bar code is not used as a magic symbol at all, in this particular application of the technology.
When in place, each Lego box displays the completed model inside, sitting atop the box. If the model has moving parts, such as the police station car in the video below, those parts will move about. The car drives about the scene. In the case of a helicopter, its blades will whirl, and a crane may raise its arm and swing about. Its designed to add a sense of excitement to the product, showing the completed model inside, in a better way than a picture possibly could.
The technology does not detect anything other than the symbol, so you can put your arm straight through the model without bothering it. Once the cameras have a lock on the magic symbol, you can partially cover it with your hand, and it will still work - as long as enough of the symbol is visible, once it is locked in memory, the computer can extrapolate the rest.
As an interesting side note, the models built up and shown outside the box, were made with Lego's own Digital Designer CAD program, which they released free to the public in 2007.