You have just discovered an essay outlining some of my thoughts on Immersive Virtual Environments (IVE's).   I am currently undertaking a Masters Research Project at the Royal Melbourne Institute of Technology, in Melbourne Australia. The focus of this research is the creation of IVE's that do not require the inhabitant to be encumbered with any technology.
I welcome feedback on this essay and associated issues: mail here

 

Immersive Virtual Environments:

In Search of dynamic realtime interaction

 

Garth Paine

This paper outlines some of the ways in which my art practice, creating Immersive Virtual Environment (IVE) installations, aspect of the human condition. Most notably how we develop relationships, specifically relationships with our physical environment. This paper also outlines some of my work in exploring digital media as an interactive form.

I begin with a definition of the type of IVE being discussed, then outline issues being explored within the work, detailing how the elements of an IVE relate to specific changes in social attitudes and scientific thinking. I will outline three installations, MAP1, Ghost in the Machine(GITM), Moments of a Quiet Mind(MQM). A RealVideo clip of MAP1 is available at the above link.

The Immersive Virtual Environment

I define an IVE as a space, room or simulator where normal sensory stimulus is heightened by the addition of sounds, images and other sensations induced by interaction with the user through movement or other triggers.

IVEs often require the user to don a hardware interface in the form of data gloves, a head mounted display or a body suit. The experience of the user is communicated through this additional level of sensory receptors. There are, consequently, concurrent encounters described by the users relationship with the interface technologies they are required to wear, which admit the computer generated stimulus to the human sensory receptors, and the qualities and aesthetic of the visual and audio content that is descriptive of the mediated experience of an IVE. It can be argued that this multi-faceted approach is too dependent on technology and not sufficiently focused on human interaction and experience. The creation of IVEs that encourage unencumbered interaction serves to make human relationship and experience the central goal. It is for these reasons that I have chosen to explore the creation of IVEs that do not require the user to be encumbered by technology.

As an artist I am interested in the relationships between:

• Technology,
• The Human condition - emotions, relationships, behavior, and
• The definition and experience of physical space.

IVEs provide an opportunity to explore the interdependence of these three elements. Figure 1 illustrates a closed feedback loop comprised of the three interactive elements of an IVE.

Fig. 1

Mediation within an IVE is an aggregation of the human response to space and its ongoing transformation. The intelligence designed into the response patterns of the installation reflects a vital consideration in developing an IVE where there are multiple feedback loops between the user, the space, and the technology. A closed feedback loop is one where any change in one element of the loop effects all elements within the system. The only influences within a closed feedback loop are the elements it contains.

An IVE exemplifies a closed causal loop. Closed causal loops represent a relationship of interdependence as illustrated in Figure 1. The elements of the loop each have know properties:

• The technology element is concerned with the application of computer technologies to the creation and mediation of the IVE. The sound and video content of an IVE are created using a modern computer based aesthetic and are varied as a response to human presence and movement.
• The space is a vessel for containment of the IVE and acts to provide a physical body for the visual and audio content.
• Human movement is an expression of individual and communal responses to the environment and the other inhabitants of the space.

These three elements display emergent properties when united in this way.

It is in this light that I give a more detailed appraisal of their role within the IVE.

Technology: Technology is applied to mediate and deliver the characteristics of the physical space. Software being developed for each installation that analyses the movement and behavior patters of the participant(s). This data is used to vary the visual and audio qualities of the environment by selecting from a large number of possibilities:

• predefine playback files,
• Algorithms for realtime generation of audio and video
• a combination of both approaches

These outcomes are categorised according to a scale defined as appropriate for the installation. This scale is defined by the desired change in the quality of the installation, for instance, a change in:

• movement, from still to chaotic,
• colour, from black and white to saturated iridescent brilliance,
• a variation in texture or intensity of another kind.

The characteristics of the material that forms the definition of the IVE needs to be devised early in the development process so that they communicate something integral to the overall artistic objective.

Space: The word space is used here in it's generic form. It is descriptive of both the physical site, that is the room or outdoor area the installation occupies, and the cumulative effect of the application to that site of the interactive visual and audio content of the installation. It is not possible to separate these facets of the IVE. The physical site brings to each work a power and ambience of it's own. The characteristics of the walls, the ceiling height, the ambient light levels, or the methods used to reduce ambient light penetration into the installation, the size and shape of the room, the textures of the wall and ceiling surfaces. The interactive visual and audio content of the installation is a substantial part of the artistic expression of the work. The symbology, colour, texture and movement characteristics of the video or animation, the textural qualities, intensity, representational characteristics and spatialisation of the sound are the determination of the artistic aesthetic and a large part of the experience of engaging with such an installation.

Human Movement: An IVE depends on the analysis of human behavior and response patterns to it as the communication channel between the technology and the environment. The IVE doe not exist without human interaction.

Figure 1 characterised a self-balancing, or negative feedback loop. A variation in the movement or behavior patterns of the user causes a variation in the response of the technology and therefore a variation in the quality of the space.

The installations MQM and GITM, were designed to bring about a balance within the space. An increase in human activity caused the technology to deliver more intense visual and audio material. The user could expand this response relationship by escalating the intensity of their behavior in concord with the space. The upper range of the visual and audio responses were however, very intense and loud. They were designed to encourage the user to alleviate the environment by easing the intensity of their movement. There was in this way a level of conditioning built into the installation that advocated an intermediate level of behavioral intensity, a middle way. This is, I believe, a reflection of the structure and expectations of society.

The intelligence of the installation is thereby a response to societal patterns and values as observed by the artist.

The social issues explored in these works are:

• The nature of individual and communal relationships in western society.
• The organisational structures within our society that perpetuate significant perceptual foundations for cultural values.
• The process of cognition and the ways in which this influences: Preferential behavior patterns, Preferential environmental conditions.
• The propensity in western cultures to incisively divide the individual into separate elements, mind and body. Furthermore, the way in which we use computer based technologies in pursuit of activities that attest to the possibility of escape from the human bodily form to a nirvana like province of the mind.

An outline of the way in which these issues are addressed within my work can be drawn from the programme notes for Moments of a Quiet Mind(MQM), an IVE installation I mounted at Linen Gallery, Melbourne in July, 1996:

I envisage MQM as a piece that explores the personal experience of presence, the language of conscious existence. Who, What, Where, and When am I ? The experience(s) of this piece will encourage a heightened awareness of the basses of cognition - Reason, Perception, Intuition. Each individual will develop, through interacting with the constructed space, a unique mental map of their environment. ³

This piece focuses on systemic beliefs; the paradoxical relationship between perceived inherent reality and interdependent systems. It is intended that the participant will alter within the installation their contextual basis for momentary existence.

The IVE provides an instrument for the participant to explore personal associations with their environment. The individual develops a cognitive map of the response patterns of the installation.

The creation of an environment that does not necessarily reference the natural characteristics of our everyday surroundings, visually or aurally, leaves the user without many of the social morays that determine the extent of acceptable behavior within society.

This forces those in the installation to consider the basis of a preferential environment:

• What qualities are to be chosen from the many available within the installation to form the foundation for their chosen habitat?;

Additionally, because the characteristics of the environment are so tightly associated with behavior patterns:

• What behavioral qualities generate a preferential environment? Are these preferential behavior patterns for the individual? If not how are the two reconciled?

These issues are further complicated when the installation is habituated by a number of people. MQM and GITM treated movement within the space as cumulative, multiple inhabitants had no option but to develop a communal language for the exploration of the installation. In so doing they were required to develop communal agreements as to what distinguishes a preferential environment, and what characterises preferential behavior.

The nature and context for the definition of preferential environment and behavior patterns constantly changes as the space responds to the input of the participants, and the participants themselves change their perceptions of their relationship with the environment, or leave or enter the installation.

Each person brings with them a large number of expectations, preferences, social and cultural conditioning. This pre-existent perceptual framework is an important basis for individual choices. It would be a complex but fascinating task to establish an empirical measurement representative of the extent to which social conditioning effects choices make within an IVE.

An IVE can be an individual or shared experience. It provides an opportunity for the exploration of communal computing. Computer networks currently being used for communal computing do not provide physical interaction. IVEs present a vehicle for the development of alternative approaches to multi-user interactions, offering the possibility that engagement with technology be as rich and rewarding as other inter-relationships within our lives.

Feedback from a number of people who experienced MQM, and GITM indicated that they were unsure what the outcome of their individual activity was within the space. These people wanted to see a direct and immediate relationship between their movements and the visual or audio evolution of the environment. They wanted to be able to control and manipulate the space quite separately from the input stimuli of others. Neither MQM, nor GITM track individual movement. The activity within the installations was cumulative. This meant that individual activity was added to the activity of other people who were active within the space at the same time. This design meant that the opportunity for individuals to be aware of their own input was inversely proportional to the number of people interacting. The greatest chance of observing individual input was occupying the space alone. Many people attended these pieces on more than one occasion in an attempt to be the sole occupant in order to explore the interactive potential of the installation to it's fullest.

The expectation of individualised activity within what is essentially a communal space is, I postulate a peculiarity of western cultural psychology. The focus on the individual does not parallel the scientific communities changing focus to a network or system based paradigm.

This kind of art practice is in keeping with developments in scientific thinking. Scientific practice shifted, with the introduction of Systems Thinking, Chaos Theory, Quantum Theory and Fractal mathematics, from the study of smaller and smaller particles as a way of acquiring an understanding of the whole, to the study of parts as integral to a network which forms the whole.

In system science, every structure is seen as the manifestation of underlying processes. System thinking is always process thinking. 4

This characterises a major shift from the introspective practices of science, based on the Cartesian model, to an new paradigm based in the interconnectedness of all phenomena. This new paradigm, founded on the precepts of systems thinking, accepts as one of its fundamentals that there are no intrinsic properties associated with parts of the whole. The properties of the parts can only be understood within the context of the whole. All attributes of each part are not necessarily discernible from the surface perspective.

Norbert Wiener, a key proponent of cybernetics, noted during work modeling social systems:

It is certainly true that the social system is an organisation like the individual, that is bound together by a system of communication, and that it has a dynamic in which circular processes of a feedback nature play an important role. ¹

Fritjof Capra comments:

the discovery of feedback as the pattern of life, applicable to organisms and social systems . .[helped] . . social scientists observed many examples of circular causality implicit in social phenomena, . . . the dynamics of these phenomena were made explicit in a coherent unifying pattern.²

Details of complexity emerge as the observer explores deeper levels of connectivity. These properties are called emergent properties. They only have relevance within the context of an inseparable web of relationships. The relationships illustrate causal feedback loops.

MAP1 extends the work done in MQM and GITM by exploring the application of realtime sound generation. This is done by using a realtime granular synthesis process that takes as it's source material, the sounds made by those within the space. Sophisticated sensing technologies in map 1 (David Rockeby's fabulous VNS) observe the physical movements of people within the installation providing Information about the mass, dynamic and direction of movement within predefine independent regions. The movement data is fed to artificial intelligence software that has been trained with a huge range of possible outcomes. This control data is used to drive a realtime sound (granular synthesis) process that takes as it's source material, the sounds made by those within the installation. James McCartney's wonderful Supercollider software is used to do the granulation. Other audio, including text elements and prepared piano, are introduced into the space when visitors are in close proximity to one of four corner speakers. These pre-constructed sound elements hint at the viscous, intimate nature of sound as a medium and are intended to lull the visitor into a more receptive frame of mind.

"MAP1 focuses on sound using the immersive, fluid and emotive qualities of the medium to generate a rich, enveloping and ever evolving environment. The sounds are made more fluid by the use of a system capable of moving the apparent source of the sound through the physical environment. This creates a dynamic relationship between the presence and position of a body and the position and movement trajectory of the sounds. The ability to move sound through space affords the sound a physicality and in so doing the sense that the sound becomes another physical character or presence within the installation.

Those within the installation will sense a physical interaction with the sound. A wide range of different aural qualities are mapped in qualitative groupings to different regions within the installation space, generating a plethora of aural textures and densities, chosen on the basis of the quality of movement of the body within that region of the exhibition"

I hope that the reader appreciates, as a result of the points raised in this paper, the issues that inspire my work with IVEs. My current research explores ways or making IVEs more spontaneous and variable, using realtime sound and vision generation, the source of which is derived from the sounds and movements patterns produced by the visitors within the installaton.

=======

1. Wiener, Norbert, Cybernetics, MIT Press, 1948; reprinted 1961. (p.24) return to text

2. Capra, Fritjof, The Web of Life, HarperCollins, London, 1996. (p. 62) return to text

3. Paine, Garth - Excerpt from programme notes, exhibition catalog, MQM , solo exhibition, Linden Gallery, St. Kilda, Melbourne, Australia, July 1996. return to text

4. Capra, Fritjof, The Web of Life, HarperCollins, London, 1996. (p. 42) return to text

I welcome feedback on this essay and associated issues: mail here

return to the top of this document

Back to the Papers Index