The frontiers of Augmented Reality

We met Mauro Rubin in one of his workshops during the 2012 edition of Frontiers of Interaction that took place this year in a wonderful location: Cinecittà, Rome.

by Andrea Paraggio

The frontiers of Augmented Reality
The frontiers of Augmented Reality

We met Mauro Rubin in one of his workshops during the 2012 edition of Frontiers of Interaction that took place this year in a wonderful location: Cinecittà, Rome.

Mauro is one of the most greatest Italian experts in the augmented reality field (that from now on we will call "AR"), founder/CEO of the startup JoinPad where he makes AR applications for advertising/promotional purposes as well as for complex industrial purposes.

In his workshop we had the opportunity to clear out our ideas about AR and above all we released the stereotypical idea that often people have of AR such as the fact that it is used only for the interaction of 3D objects displayed on the basis of 2D markers present in the real world. Although it is considered a fairly trivial use of a technology far more powerful, the world of advertising and promotion seems to have taken advantage of good opportunities offered by the AR. The world is full of excellent examples of use in this direction.

Below a list of possible applications of AR which we discussed during the workshop:

  • industry, automation (image/object recognition, complex visual patterns recognition);
  • advertising, promotion, sales support (image recognition, marker recognition, RFID);
  • art, museums, galleries, installations (image recognition, marker recognition, RFID);
  • tourism (image recognition, marker recognition, geo-localization);
  • gaming (interaction between real environment & virtual world);
  • traditional publishing augmented (image recognition, marker recognition es. Layar);
  • medical imaging (image/object recognition, complex visual patterns recognition).
What is AR?

The often-cited definition of AR was given by Ronald Azuma in "A Survey of Augmented Reality", 1997.

AR is about augmenting the real world environment with virtual information by improving people's senses and skills. AR mixes virtual characters with the actual world. [...] AR supplements reality, rather than completely replacing it.

It lists three indispensable requirements for an AR application:

  • the AR system combines real and virtual information;
  • the AR system is interactive in real-time;
  • virtual content is registered in 3D.

According to the criteria above, Google Goggles could not be called an application AR, as it is not in real time (you take a picture and send it to Google for processing), and additional information are not 3 dimensions.

Another thing that seems important is the fact that the AR is a complement to reality, it's an integration, not a complete replacement. And that's what makes it fundamentally different from virtual reality.

Reality-Virtuality continuum on wikipedia

Let us now turn to a more practical question, what are the reasons that make the AR attractive for companies that look to digital for the business support?

  • it is simple to use;
  • it is applicable to every kind of object;
  • it is in real time.

In fact it is not necessary that the end-user AR application possesses special skills. Furthermore, it is possible to increase the information capacity of any type of object, from the apple to the engine of a car. All this in real time: the AR overlays the view of the real world, using the flow of a camera or, even better, a dedicated pair of glasses, with camera and translucent display.

As mentioned previously, there seems to be mainly two major applications of AR: one based on the recognition of 2D markers, another more specialized, used in technical / industrial and military purposes, based on visual pattern recognition algorithms, images and objects, much more complex.

Marker-based AR

A marker is an image in two dimensions belonging to the real/analogical world. Many mistakenly believe that the QR-code is a marker: it is more like a simple bar code (such as EAN-13) that is recognized by the machine through proper software, returning a meaningful output.

If we wanted to put the QR-code within a theory of communication, it would be a signifier that is "something that stands for something else" and that constitutes a link between the real world and digital world (the QR-code is printed and subsequently recognized by a digital device).

Augmented reality marker-based is the most simple to implement as there are already existing algorithms that need only to be instructed, from time to time, to recognize the different markers. In fact this mode, although at a first glance, seems attractive, it is extremely self-referential and devoid of any practical use. Probably it is much used in the field of advertising/promotion for the fact that it is very useful in triggering in those who use the popular effect of "wow", whose purpose is to fan positive word of mouth within the communication target.

An example of application of AR in the field of advertising is provided by General Electric with the campaign launch of its smart grid.

Screenshots taken by General Electric Smart Grid promotional video

Besides being the classic example of representation of complex 3D objects from markers, smart grid allows interaction with the physical user: blowing near the microphone, wind turbines start spinning.

Another example of the use of AR in the sales promotion of products is provided by LEGO.

Image by IntelFreePress

In the picture you can see a potential client that interacts with a box of construction: placing it under a special camera, a display returns the 3D image of the product being developed, displaying it in all its aspects.

Advanced use of AR

The advanced use of AR is certainly the most charming one and able to stimulate imaginations from films set in a not too distant future after all. It mainly concerns the return of detailed information in real time, of objects more or less complex, including procedures for manipulation, modification and interaction with these objects.

The most beautiful and significant works are those in which the user interacts with the special glasses, manufactured specifically to become a real prosthesis with which the human being expands its capacity to understand the world, certainly to an extent greater than that augmented reality could be enjoyed by a device more distant from the eyes, such as a smartphone or tablet.

Screenshots taken by BMW research projects on augmented reality concept video

In the above concept, the BMW technician is wearing a pair of special lenses: through the lens with a translucent screen he can see both the real world and the augmented reality. The latter is further enhanced by an audio stream containing information about the procedures to be carried out in real time.

The glasses have a camera at the center of the eyes, and a display for each eye, the latter obviously translucent, gives the opportunity to look at the information and the real world, all at the same time. There are those who have innovated to include this technology in contact lenses: Innovega has developed the AR by including it in contact lenses, up to when their technology was appreciated by DARPA and the Pentagon, organizations that use technology for military purposes. (source: Wired US).

According to Mauro Rubin, usually in the concept video we see a misuse of 3D objects that are to "augment" the real world. In reality this is not always applicable nor useful and effective if it gives the AR a pure enrichment informational purposes, here is that this technology begins to enter into a symbiotic relationship with other disciplines such as cognitive ergonomics, information design and human factors.

It is much more effective from this point of view the display of information labels clearly visible, with different levels of opacity. This is more effective not only from the point of view of the disciplines mentioned above, but also from the point of view of the optimization and sustainable use of the hardware available to applications.

Augmented reality for medical visualization

Given the complexity of medicine, it isn't difficult to imagine how a technology such as AR may help those professionals who deal with medical specialists providing increasingly better quality and a higher definition of images to support the diagnosis. Up to today radiological technologies such as computed tomography, or magnetic resonance imaging, are able to return images in three dimensions from the combination of a high number of scans 2D. Let's imagine you view the images obtained with these technologies directly on the patient's body. The result obtained is something that today is called the in-vivo medical visualization, a young field of research that deals with the AR applied to the medical images, displaying them in real time on the body of the living patient, using special glasses.

Video courtesy of CAMP Computer Assisted Medical Procedures Institute at the Technische Universitat Munchen aims to bring technological innovation to medicine.

A research group Medical Augmented Reality (MAR) very young and motivated at the Technische Universität München, for years have been conducting research on the possible interaction between AR and medicine. Dozens of projects are still open, and many of them not only are very interesting, they also answer to relevant clinical issues, both from a medical/surgical and from an educational/training point of view.

There are, in fact, more educational uses of AR, a little less complex, based on software and hardware easily found even by non-professionals. Just think of Augmented Reality Magic Mirror using the Kinect, a project designed to offer support for training in the medical field and based on a well-known hardware (especially gamers), the Microsoft Kinect: combining a db of CT images to a skeleton you can use the context and focus of kinect to hook a person and see inside organs and bone structure through a virtual mask.

Screenshots taken by Technische Universität München CAMP project: Augmented Reality Magic Mirror video

We like to imagine the combination of this project with a project for which we have already discussed before in Geeks of Health, ZygoteBody, former Google Body: real-time visualization of the inside of a person, with different levels of augmented reality, combined with a filtration capacity of organs and structures for function and for apparatuses.

For a list of interesting projects in Medical AR field, take a look at TUM CAMP webpage. "Technische Universitat Munchen" is a leading university regarding the application of AR to medical visualization.

Interview with Mauro Rubin

After this overview of what has been already done, we ask Mauro in what direction, according to him, the AR technology applied to medical visualization will develop in the near future:

Andrea: In which of the two perspectives above, in your opinion, AR will have more possibilities of application in the medical field?

Mauro: Well, in both cases it depends on what level of quality the AR software will be developed on... The UX in the field of AR is very critical and it will considerably affect the outcome. Personally I trust in the clinical perspective because it could substantially resolve a real need: reducing human errors. I heard that some prototypes mounted on helmets already are used to gain scan/x-rays of wounded soldiers on battlefield (because, at this moment, evacuating wounded soldiers in hot spots is one of the most critical missions: saving a wounded soldier often involves the loss of more men and equipment). Such tools like this could help practitioners and physicians in decision-making and get the real state of the situation.

Andrea: In your workshop you have asserted that one of the reasons to dive into the business of AR is due to its incredible intuitiveness and ease of use by all. Particular expertise and skills are not required to the application's user. In your opinion, do you believe that the same issue is valid for a possible use by physicians, within the clinical perspective?

Mauro: I think AR could help to extend the perception of reality, but one should never consider in relying completely on the technology... AR is just a new and intuitive vector to gain information, such as google 10 years ago. But I wouldn't get someone that has studied on google to do me a surgery operation!

Andrea: I like to think of a zygotebody (ex Google Body) in-vivo on real people's bodies, open-source and with publicly available data. A sort of wiki-APP. In your opinion, is it possible to have an AR app with a live filtering for apparatuses, levels and human body's functions?

Mauro: As JoinPad, I've been in contact with European enterprises that are working on very advanced clinical solutions: there are systems that can provide in real time all patient's medical reports, directly during the surgery operation, in overlay with the physician's visual field... There aren't still produced, but I don't think this is an hypothesis far away from the present.

Frameworks and apps for the development of AR apps

We have already said that the AR is a combined technology, the result of the interaction of multiple technologies:

  • pattern recognition;
  • 3D environment;
  • geolocalization.

Below is a list of frameworks and apps to start developing small AR applications:

Framework/App Description License
ARToolkit libs A simple framework for creating real-time augmented reality applications, multiplatform library (Win, Linux, OsX, SGI) FREE (non-commercial use)
FLARToolkit Porting of ARToolkit from C++ to Actionscript (flash) FREE (non-commercial use)
Aurasma Particular as it works directly from your smartphone or tablet iOs & Android app with single user license (non-commercial use)
Metaio Perfect for publishers, with an easy drag & drop interface Commercial license (PAY)
D'Fusion studio Very professional and easy to use, allows object-oriented programming Commercial license (PAY)
Vuforia by Qualcomm Great combination of ease of use and professional results for free FREE (commercial use too)

This article has been written also thanks to the valuable contribution of Alessio Cataldo, Account and Content Assistant, who attended to the AR Workshop at FoI2012 together with Andrea Paraggio.

contact us

© 2016 Healthware. All rights reserved.
Healthware Group s.r.l. VAT 07854130635
Headquarters: Salerno, Amalfi Coast (Via A. Amato, 15) Italy
Offices: London - Milan - New York - Rome