Abstract

With the development of mixed-reality experiences and the corresponding interface devices multiple frameworks for classification of these experiences have been proposed. However these past attempts have mostly been developed alongside and with the intent of capturing specific projects (Marsh 2015, Billinghurst 2001) or are nevertheless very focused on existing methods and technologies (Milgram 1994). The existing taxonomies also all assume physical reality as a fixpoint and constant and are thereby not suited to describe many fictional mixed-reality environments and altered states of consciousness. In this paper we describe a new model for describing such experiences and examplify it's use with currently existing as well as idealized technologies from popular culture.

Terminology

We propose the following terms and definitions that will be used extensively for the remainder of the paper:

Introduction

We identify two different types of relationships between layers in multi-reality environments. The first is layer nesting. Layer nesting describes how some layers are contained in other layers; i.e. they exist within and can be represented fully by the parent layer's world model and the child layer's rules emerge natively from the parent layer's dynamics. Layer nesting is visualized on the vertical axis in the following diagrams. For each layer of reality on the bottom of the diagram the nested parent layers can be found by tracing a line upwards to the top of the diagram. Following a materialistic point of view, physical reality therefore must completely encompass the top of each diagram.

The second type of relationship describes the information flow between a subject and the layers of reality the subject is immersed in. In a multi-reality experience the subject has access to multiple layers of reality and their corresponding world models simultaneously.
Depending on the specific experience, different types of and directions for information exchange can exist between these layers and the subject's internal representation of the experience. For the sake of this paper we distinguish only between input and output data flow (from the perspective of the subject); categorized loosely as information the subject receives from the environment (input, e.g. visual stimuli) and actions the subject can take to influence the state of the world model (output, e.g. motor actions) respectively.

In the following diagrams, information flow is visualized horizontally, in the region below the dashed line at the bottom of the diagram. The subject's internal mental model and layer of reality are placed on the bottom left side of the diagram. The layers of reality that the subject experiences directly and that mirror it's internal representations are placed on the far right. There may be multiple layers of reality sharing this space, visualized as a vertical stack of layers. Since the subject must necessarily have a complete internal model of the multi-reality experience around him to feel immersed, the subject's mental layer of reality must span the full height of all the layers visible on the right side of the diagram.
Information flow itself is now visualized concretely using arrows that cross layer boundaries in the lower part of the diagram as described above. Arrows pointing leftwards denote input flow, whilst arrows pointing rightwards denote output-directed information flow. In some cases information doesn't flow directly between the layers the subject is directly aware of and the subject's internal representation and instead traverses intermediate layers first.

Before we take a look at some reality stacks corresponding to current VR and AR technology, we can take a look at waking life as a baseline stack. To illustrate the format of the diagram we will compare it to the stack corresponding to a dreaming state:

In both cases, the top of the diagram is fully occupied by the physical layer of reality, colored in green. This is due to the fact that, according to the materialistic theory of mind, human consciousness owes its existance to the physical and chemical dynamics of neurons in our brains. Therefore our mental reality must be considered fully embedded in the physical reality, and consequently it may only appear underneath it in the diagram.

During waking life, we concern ourselves mostly with the physical reality surrounding us. For this reason the physical reality is placed in the lower right corner of the diagram as the layer holding the external world model relevant to the subject. Information flows in both directions between the physical world model and the subject's mental model, as denoted by the two white arrows: Information about the state of the world model enter the subjects mind via the senses (top arrow, pointing leftwards), and choices the subject makes inside of and based on his mental model can feed back into the physical layer through movements (lower arrow, pointing rightwards).

In the dreaming state on the other hand, the subject is unaware of the physical layer of reality, though the mind remains embedded inside it. When dreaming, subjects' mental models don't depend on external models, hence the mental layer of reality must be the only layer along the bottom of the diagram.

Current Technologies

Since recent technological advancements have enabled the development of VR and AR consumer devices, AR and VR have been established as the potential next frontier of digital entertainment.
As the names imply, the notion of reality is at the core of both technologies. In the following section we will take a look at the respective stacks of both experience types:

In both cases we find the physical layer of reality as an intermediate layer between the mental and digital layers. Actions taken by the subject have to be acted out physically (corresponding to the information traversing the barrier between mental and physical reality) before they can be again digitized using the various tracking and input technologies (which in turn carry the information across the boundary of the physical and digital spaces).

The difference between AR and VR lies in the fact that in AR the subject experiences a mixture of the digital and physical world models. This can be seen in the diagram, where we find that right of the diagram origin and the mental model, the diagram splits and terminates in both layers: while information reaches the subject both from the digital reality through the physical one, as well as directly from the physical reality, the subject only directly manipulates state in the physical reality.

The data conversions necessary at layer boundaries incur at the least losses in quality and accuracy of information for purely technical reasons. However intermediate layers come at a cost larger than just an additional step of conversion: For information to flow through a layer, it must be encodable within that layer’s world model. This means that the 'weakest link' in a given reality stack determines the upper bound of information possible to encode within said stack and thereby limits the overall expressivity of the stack.
As a practical example we can consider creating an hypothetical VR application that allows users to traverse a large virtual space by flying. While the human mind is perfectly capable of imagining to fly and control the motion appropriately, it is extremely hard to devise and implement a satisfying setup and control scheme because the physical body of the user needs to be taken into account and it, unlike the corresponding representations in the mental and digital world models, cannot float around freely.

Future Developments

In the previous section we found that the presence of the physical layer in the information path of VR and AR stacks limits the experience as a whole. It follows that the removal of that indirection should be an obvious goal for future developments:

In the action movie 'The Matrix' (1999), users of the titular VR environment interface with it by plugging cables into implanted sockets that connect the simulation directly to their central nervous system.
While these cables and implanted devices are physical devices, they don't constitute the presence of the physical layer of reality in the information path because while they do transmit information, the information remains in either the encoding of the mental model (neural firing patterns) or the encoding of the digital model (e.g. a numeric encoding of a player character's movement in digital space) and the conversion is made directly between those two - the data never assumes the native encoding of the physical layer (e.g. as a physical motion).

While we are currently far from being able to read arbitrary high-level information from the brain or to synthesize sensual input in human perception by bypassing the sensory organs, brain-computer interfaces (BCI) are a very active area of research with high hopes for comparable achievements in the near future.

Applying this same step of removing the physical layer of reality from AR, we end up with something similar to the nano-particle drug in Nexus (Naam 2012). However this does not grant the user a similar amount of control over his experience as the holy grail of VR does, since the user and the physical part of the environment remain bound by the physical layer of reality's laws.
Instead the holy grail of AR is reached with the creation of a god machine that can manipulate the state of the physical world according to the user's wishes. In this way the digital and physical realities become unified and fully 'augmented'.

Despite the similarities of VR and AR, the two can be considered polar opposites, as becomes evident when we compare their respective utopian implementations: they share the goal of allowing us to experience realities different from the one we naturally inhabit, but while VR seeks to accomplish this by creating a new, nested reality inside ours, thus giving us full control over it. AR, on the other hand, is instead an attempt to retrofit our specific needs directly into the very reality we exist in.
This is in direct contrast with the popular notion of the 'reality-virtuality continuum' (Milgram 1994): the reality-virtuality continuum places common reality and VR (virtuality) as the two extreme poles, while AR is represented as an intermediate state between the two. Here however we propose to view instead AR and VR as the respective poles and find instead reality at the centerpoint, where the two opposing influences 'cancel out'.

Conclusion and Further Work

In this paper we have proposed a taxonomy and visualization style for multi-reality experiences, as well as demonstrated it's flexibility by applying them as examples. Through the application of the proposed theory, we have also gained a new and contrasting view on preceding work such as the reality-virtuality-continuum. We have also found that the taxonomy can be used outside the research field of media studies and its use may extend as far as philosophy of consciousness (see Appendix below).

Further research could enhance the proposed theory with better and more concrete definitions. In the future, the proposed taxonomy might be used to create a more extensive and complete classification of reality stacks and to analyse the relationships between them.

References

1. P. Milgram, H. Takemura, A. Utsumi, F. Kishino (1994): Augmented Reality: A class of displays on the reality-virtuality continuum, in SPIE Vol. 2351
2. W. Marsh, F. Mérienne (2015): Nested Immersion: Describing and Classifying Augmented Virtual Reality, IEEE Virtual Reality Conference 2015
3. M. Billinghurst, H. Kato, I. Poupyrev (2001): The MagicBook: a transitional AR interface, in Computer & Graphics 25
4. L. Wachowski, A. Wachowski (1999): The Matrix (movie)
5. R. Naam (2012): Nexus, Angry Robot (novel)

Appendix: Relation to Theories of Mind

This paper starts from a deeply materialistic point of view that borders on microphysicalism. However it should be noted that the diagram style introduced above lends itself also to display other philosophical theories of mind. As an example, the following graphics show a typical VR stack as interpreted by Materialism, Cartesian Dualism and Solipsism respectively:

However these philosophical theories of minds also constitute reality stacks by themselves and as such can be compared directly: