Choreographing in VR
INTRODUCING ‘SUBSTITUTE PERFORMERS’ AS INFORMANTS IN THE CHOREOGRAPHIC PROCESS
Author Hanna Pajala-Assefa
Abstract
This paper is situated at the intersection of digital choreography and human-centered interaction within the scope of virtual reality. Both disciplines are discussed under the frame of contemporary artistic practice, an endeavor to make an artistic work while appropriating skills in crafting and composition. The paper focuses on the creation practice in and for virtual reality and the making of an artwork rooted in movement and with a choreographic idea of embedding an improvisatory score into the interaction design. Therefore, the thinking of choreography expands into the design of the interactions in the virtual realm and falls into the subfield of digital choreography.
To reflect on the agent-entangled practice of digital choreography within VR and to trace the roles of various performative, epistemic agents active within the creative process, the author introduces a notion of substitute performers. The methods of phenomenology of lived mediated experience and digital choreography are synthesized to optimize the artistic endeavor to design for diverse bodies. These are then combined with various ethnographic methods to provide insights into moving and dancing bodies in a mediated performative realm. These intertwined methods were an elemental part of the creative process of the VR artwork Skeleton Conductor XR Art which was designed to induce pleasure through improvisational engagement and cultivate kinaesthetic awareness.
CCS CONCEPTS: Applied computing ~ Arts and humanities ~ Performing arts •Human-centered computing ~ Interaction design ~ Interaction design process and methods ~ User centered design
Additional Keywords and Phrases: Digital Choreography, Improvisatory systems, Embodied design practice, Expanded Ethnography
1 Introduction
This paper addresses the modes of choreographic practice when entering the interactive mediation realm of virtual reality. It does it by tracing the process of making an artwork Skeleton Conductor XR Art (later referred to as SC), where VR was not only a creative tool but also an artistic medium. The artwork addressed here treats the virtual space as a site for dance improvisation and therein the embodied knowledge of the performer as an epistemic agent became the conceptual and compositional foundation for digital choreography. The aim is to focus on theories and concepts relevant when bringing the two practices of VR design and choreography together, not as merging into one another but viewing them as a Mobius strip [12] where the two disciplines are entangled into one another.
1.1 Bringing together VR and choreography
The choreography discussed here is understood as an interdisciplinary art form and compositional practice that brings together and organizes spatiotemporal activities and materials, which then create a world of performance. The frame is underpinned and inspired by the author’s variety of practical knowledge on dance and choreography while cultivating the moving, thinking body. Hence, the view on choreography is a form of situated and relational activity of placing bodies relationally to other bodies, materials, and space, and the structuring of the attentive awareness and the sentient body. The choreography here is a composition of the dialog between the “flesh of the body and the flesh of the world” [13] and attending to how they make each other move.
I, the author, situate myself on Butterworth’s Didactic-Democratic framework [3] as a facilitator-collaborator, negotiating the composition through improvisation with prescriptive scores [18] and focusing on the actuality of performance rather than authored movement sequences and fixed compositions. Over the years I have included new media technologies as a constitutive element in performance, particularly applying movement computation technologies to augment dance into audio and visual simulation within live performances. Interactivity in live performance distributes creative agency to performers and technologies and shifts the focus of choreography to relationalities. Bringing interactive elements into performance has shifted my thinking of dramaturgic elements through a similarly attentive and attuned attitude as the lived, performing body. Through interactivity, the constitutive material, and immaterial elements, such as light, sound, and visual objects, are viewed as co-performers in interconnected unfolding flux with performing bodies. This is in line with the choreography referred to as extended choreography [1] or what Johannes Birringer calls a choreographic system [3] where the composition is expanded to include the clusters of tools, designed elements, and choreographic practices that are encompassed in the process and that influence the emergence of the performance.
The virtual reality considered in this paper is a simulated 3D environment that allows the perceiver to explore and interact with the virtual surroundings (1). Its array of applications is used in a myriad of fields and purposes in entertainment, product design, training, education, well-being, and research to name a few. My interest in VR arose from the technology’s apparent ability to immerse the viewer into a digitally produced world conductible in real-time through motion tracking. The interest was not in the theatrical storytelling potential of VR but rather in the horizon of subjective imaginary potentials and the immediate perceptual feedback loop intrinsic to VR. It was the potential to design the virtual simulation based on embodied interactivity that inspired me to pursue a choreographic approach to VR.
With VR and other live movement computation systems, the human body is in close connection with the designed apparatus. Therein the interaction design is in practice done through and for the body. The world the body encounters is situational and relational as it influences the body but is also influenced by the body, and the meaning-making arises from this entanglement. Paul Dourish describes embodied interaction “Action both produces and draws upon meaning, meaning both gives rise to and arises from action” [6]. This coevolvement is a key notion in movement-based VR and connects to Merleau-Ponty’s principle of reversibility, the body’s ability to touch and be touched, to see and be seen [12]. Therefore, in movement-based VR the coupling of movement and augmented simulations becomes the very fabric of the piece itself. The choreographic task at hand is not about organising bodies and objects in space but thinking of the potential coupling of potential movements with virtual objects which become properties of the body. Therein, the choreography extends into the interaction design and the VR system can be thought of as a choreographic score. Within which exists a dramaturgy of interactivity that relies upon perceptual reversibility and operates by structuring and prompting the improvisatory performance through directing and attuning the performer's attention between the moving body and the sensed world it is immersed in and with which it is performing. Hence, the process of creating in and for VR shifts the modes of choreography radically. The artwork extrapolated here, does not have a ‘known’ performer, but an intersectional imaginary one. In the medium of VR, the future audience member is the performer to-be, the person who ignites the audio and visual stimulation, which reacts to their movements through 5-point tracking of head, hands, and feet, according to the virtual score. The virtual apparatus creates a temporary audio-visual virtual world, a digitally induced space where the perceiver is the performer and audience for oneself. Therein, the digital choreography, that I examine here, is a choreographic idea that includes imaginary objects and subjects and the organisation of potential attentiveness in conjunction with the body-mind, virtual space, and interaction technology. This performer-to-be, the unknown, imagined performer is the key element that shifts the creative practice that relies on various agential bodies as substitute performers.
1. The interactivity in VR is based on tracking the headset and controllers, simulating the user’s actions in the virtual world in real-time, which is perceived from the 1st person point of view. The full-body tracking enables also spatial movement in the physical space to be reflected 3-dimensionally in the virtual space The real-time interactivity creates a perceptual feedback loop, an action-computation-perception cycle.
1.2 The VR artwork and artistic objectives
The SC artwork is a single-user movement-based virtual reality artwork. It is a 12-minute, two-episode VR piece, experienced through PC based VR device with a head-mounted display, hand-held controllers, and additional feet trackers. The artwork was created using the Unity game engine during the years 2019-20 by a team of a choreographer, visual designer, and sound designer. Movement-based design here means that the experiencer is conducting the audio and visual simulation through physically moving within the virtual space. Thus, while sculpting out the visual augmentation and playing the spatialized sonic environment they are influenced by it. The artwork is not narrative nor representing the known ‘real’ world, rather creates a mutually responsive illusory, singular world with the participant.
This means that the simulated objects exist relationally to the subject, the performer’s moving body, not as distinct objects existing in the virtual space. According to theorist Karen Barad this ontological entanglement and the mutual constitution of entangled agents, that here forms the material-discursive phenomena of performance is intra-active [2]. Hence, the choreographic intra-active frame is ontologically connected to the feedback loop, the action-computation-perception -cycle that takes place in immediate real-time interactivity. A leading principle for the choreographic concept was being movement-based or primacy of kinaesthetic (2) experience. Therefore, the composition of the aesthetics of interaction design was always bound up with embodied activity and the mapping of sound and visual simulations onto the schematic metaphorical relationalities of the body.
The chosen open-ended dramaturgic approach supports free play and engagement in exploration without a predetermined end goal, narrative, or gamified structure. Similar open-ended structures are very common in improvisation-based choreography, where a prescriptive score or a task gives a structure for the unfolding dramaturgy of the dance without a distinct predefined form for the performance. The aim here was to foster exploratory creative activity for a variant audience, hence, to be inclusive of different bodies, sensitivities, and approaches. This polysolutional aim was to not impose a desired style of movement or activity, but rather to facilitate ambivalent multi- or polysolutional approaches. This type of feminist design practice for multiple bodies [9] manifested itself in composition through interpretative ambivalence, comprising an abstract approach in visual design, multiple sonification schemas, and plurality in movement mapping filtering such as velocity, spatiality, and dynamics. These principles of kinaesthetic primacy, open-endedness, and polysolutionality guided the design to foster self-expressive, explorative engagement. To motivate exploration and lingering, sought after aesthetic and interaction design paths were to evoke pleasure. The source of pleasure was found not only in the discovery of audio-visual properties and creative agency but also in the discursive and porous bodily reflexivity of the feedback loop.
2. My understanding of kinaesthesia encompasses the neurophysiologically defined proprioception as an ability to sense the position and location, orientation, and movement of the body and a more psychosomatic ability to incorporate the affective and interoceptive processes of the self while moving.
2 THE PROCESS
In viewing the VR system as a choreographic score, the choreographer faces a paradoxical problem regarding the intertwined interactivity of the virtual performer and the virtual reality apparatus. The ‘virtual’, can be considered here along the lines of Deleuze as something with the potentiality to emerge or to be actualized, with ‘the virtual encircling the actual’ [5]. An improvised dance performance deals with similar virtual potentialities when a performer moves in space-time according to internal and external impulses while the outcome, the performance is in a process of actualization. The hereby designed virtual artifact will perform its encoded procedures, the to-be perceived external impulses in a circuit with a body that it is entangled with, “continually retracing from one to the other” [5]. The digital apparatus therefore conceptually becomes a co-performer for the human performer-to-be, whom for the creator, has a yet unknown range of sensitivities and performer tendencies. Hence, the constitutive paradoxical problem is the missing of both performers the digital and the human.
In the practice, the interactive, moving body is also the actuator and verifier of the designed properties. Therefore, I propose a notion of ‘substitute performer’ as an agential body for investigating the performing bodies in the process of making and to illustrate how the perceiving, thinking body becomes the epistemic agent for techne and praxis. This attending to the discursive, imaginary potential singularities of embodied agent, takes place in several phases. For the sake of methodological distinction and clarity, they are addressed in separate sections although in practice they happen as mutually informing nested epistemic frames, as all potential bodies, the embodiedly experienced, reflected, and imagined bodies are all informing the process. I start with the investigation of the dancer-choreographer as a substitute performer.
2.1 The dancer as an epistemic agent
The core of the process and the essential conceptual ground is the first-hand experiential, sensorial knowledge of the virtual world one is making. Hence, the verification of any interactivity depends on an engaged body activating the encoded procedures, it is only natural that the artist’s own body is one to obtain first-hand knowledge of “what is it like to be a performer for oneself”. While being the virtual world-inducing performer, or ‘the dancer’, the choreographer simultaneously reflects upon the performer-object with a compositional eye. Therein, the making is grounded in the knowledge of two distinct agents within one person, the choreographer and the dancer. This double role requires attending to the subject-object dichotomy. In creating VR, the maker adopts the role of the Dancer, the subject performing with the system. The choreographer reflects on her experience and the system-perceiver intra-activity with the intentionality of the maker. This is a conscious act of situating oneself in both positionalities and articulating them throughout the process of making. A systematic attentiveness to the modes of performing-in and compositional articulation is necessary, to scrutinise the design's emotional and affective qualities and to open the horizon for more inclusive and ethical approaches.
To make the artwork the choreographer needs to understand what being the dancer in and with the virtual environment is like. Accessing this knowledge requires a first-person method of reflecting on the embodied experience. This means thoroughly attending to the perceptions, affects, flows of movement, and sense-making experience of the dancer. Following an ontological notion of being-in-the-world [17], the performer is the subject consciously and intentionally attuning to the being-in-the-virtual-world and listening to the somatic inner sensation of the body. The subjective body is the one enjoying the act of interacting with the world and immersing oneself in the affective state of experiencing it while adopting a phenomenological attitude of intentionally bracketing out the preconceptions of what it might entail [10, 12, 19]. As if one would experience it for the first time, relying on the embodied knowledge to lead and guide the way in the discovery and intentionally suspending the intentions of the choreographer. Yet, engaging in an interactive environment requires a practical learning of being with and enacting within the interface. This corporeal figuring out of the system is a type of knowledge gathering of the ‘praxis’, the understanding of the practice of the interface, through which the expressive behaviour can take place. The subjective body is navigating in the world by moving and through adopting the virtual affordances the objective body is incorporating them into the ‘I can’ [16] And, while most of this habituation of the ‘I can’ takes place pre-consciously, through operational intentionality, “the structure through which a graspable and sensible world first emerges out of the ambiguity of experience” [19] the perception loop of the virtual apparatus, the responsive environment mirrors it back to the subject highlighting the reflective states. Merleau-Ponty’s reversibility [12] is extended into moving the world and being moved by the virtual world. A reversible relationality with the virtual world of sensing it and being sensed by it, “a sentient and sensed body” [13]. The enacting performing subject is attentive to both, the affordances of the virtual environment, a molding and sense-making of the digital animate world, while it in return feeds the body that is organising itself with the system. A dynamic ontological coinciding relationship between the reversibility of the body and the reversibility of the virtual world is what makes up being in the VR world. While the bodily reversibility, the innate ability to explore the subjective and objective self is hard to capture in the habitual world and to bring to focal consciousness, the sensible interactive simulations of VR enhance the sensing and sense-making during its unfolding, and hence also allow a reflexiveness of it. And therefore, the following step of reflecting, the detailed verbalising of the embodied experience by the dancer, can take place. The system supports attending to intentions, perceptions, affects, corporeality, and sense-making arising from within the body from the feedback, as they together create a chiasmatic whole of mutual entanglement. The dancer explores it by being-in-the-virtual-world and reflects on it both corporeally and consciously.
I was trying out a new version of the visual particle system. The visual aesthetics were familiar to me as I had been there before, still, they were somehow more dense and powerful. I felt as if standing in a plane or an undulating sea of organic matter, consisting of thousands of moving particles. I was able to gather them into my hand, clump them up in more dense clouds, and sculpt the particle clouds into elongated forms. My body just followed along and facilitated the project of sculpting the visual world around me. It occupied my body fully. My body felt soft and attentive even when moving relatively fast and dynamically. The visual world was pervasive, reaching out to far distance, yet it felt intimate, almost as if touching my body or going through it. It was electrifying, I felt a tingling sensation in my skin when moving with the swarming visual particle world. It moved me on emotional and physical levels. My movement affected the matter, made it change its direction, that sparked a new turn in my movement. As this mutually reflective exchange continued, I became more familiar with it and created a feeling of intimacy. As I moved the matter, I moved with it, and that moved me, and that synchrony made us partners. I was moving with the digital matter that had its agency, its own peculiar distinct movements. We were not separate from each other, neither the same, but entangled and emergent together. (an excerpt from process diary)
The dancer's reflections on her experience were crucial knowledge to transpose the compositional reflection into the systemic performative score. In other words, the choreographer reflects on herself as the epistemic agent, the kinaesthetic thinking being, and therefore is able to consider compositional choices embedded in the interactivity design. The above excerpt concretizes distinct compositional and design choices based on reversibility in action. The chosen visual design induced altered movement paths while heightening the perceptual process. This reversible ability to sculpt the matter and it the mover reassured that a direct visual representation of the body itself was redundant, as the visual response alone emphasized the preferred qualities, the proprioceptive bodily sensing reinforcing kinaesthetic awareness.
After opening the depths of the dancer’s epistemic role behind compositional and design choices we can attend to the other layer of substitute performers, the design-oriented-performers. Which in this case consists of a team of 2 designers with the choreographer. Here the focus is shifted to language, to the description and articulation of the experienced.
Attentiveness and translation from the experiential body to the language of words and further into the language of computation. This process requires abstraction and distilling of ‘how the actualized was felt’ into compositional ideas and further iterations of the design. Therein, the forms of practical knowledge with the two factors of ‘techne’, a bringing-forth something yet not discovered through computation, and ‘praxis’, the chiasmatic discovery of the bodily experience, are both present.
2.2 The collaborative design practice
In real-time movement-based interactivity, two computation design factors constitute the outcome. One is the computational architecture of the user-extracted movement data flow, and the other is the digital simulations the system generates, which in this case, were restricted to sound and visuals. To distinguish these two compositional modes, I call them design of motion data flow and design of aesthetics. Framing these as modes of choreography is to conceptually link the movement data to the simulations in VR, such as connecting the properties of movement, dynamics, flow, orientation, and posture with audio and visuals through embodied metaphors [11]. All compositional implementations are intricately intertwined since the system and the performing body are interdependent, and layering multiple couplings creates a complex system. Therein, the pattern of composing, coding, engaging, and observing became the method of design resulting in an experience-based-design practice, constituting a workflow relying on the practical knowledge accumulated through the kinaesthesia – ‘the chain of perception-movement-cognition underpinning embodiment’ [16]. During the iteration process, many aesthetic choices came quite intuitively through embodied experimenting, design improvisation, and what some designers have called ‘body storming [9]. To allow co-creative critical reflection and compositional thinking, and a shared language of aesthetics, it was important to include the embodied reflection of all team members – the design-oriented-performers. In this case, the team was small with a sound designer focusing mostly on details of sonification and a visual designer and coder operating in Unity game engine. Designers were interacting with the system similarly to the dancer-choreographer but with distinct design knowledge and a focus on verifying the technical viability. This iterative process of coding and testing with the team constituted the reflective grounds for the choreography in VR.
Here the phenomenological method of epoché [10, 12] the bracketing of preconceived knowledge of the design, suspending the expectations, and assumptions of it comes to the fore, to allow noticing and attending to ‘how the coupling was actualized in the perception-loop’. The reflective observations come from two stances. Firstly, a body-to-body approach, reading the other body through kinaesthetic empathy [8] as inter-corporeal reflections while witnessing another body within the system, then describing and discussing them with the others. Secondly, a more analytical one, critically reflecting on aspired compositional features and aesthetic affordances. This process requires a constant openness to perception, clarity in the articulation of observations, and translations into new iterations of mapping constellations. As the two team members' professional frames were in music and visual design, they had radically different ‘performer techniques’ therefore attuning to their behavior intercorporeally [19], through kinaesthetic empathy was extremely relevant in interpreting their reflections. Noticing variant behavior and articulations of experiential tones was a path to accessing potential imaginary performers and allowed an iteration of the concept for better inclusivity.
Observing D & J in prototype: D is enthusiastically engaging with the visuals, with almost a child-like joy. He’s moving constantly at a swift pace, rotating both hands, mainly in unison. There is a clear tendency to gather and catch the particles as bundles rather than sculpting them as material. Lesser focus on sounds. He later reflects the sound was so organically following him, that he had no need to focus on it. J is much more gentle, slow, and isolated with his movements. He uses his hands independently, with articulation in dynamics and listening to the space with subtlety. His head and gaze do not so much follow the environment, seemingly head movements are driven by listening. He seems to navigate mainly by sound, only occasionally verifying the happening through the visuals. (from process diary)
However, with the iterative embodied practice the process of reflection was exhaustive over time because it was innately so bound to prior and emerging knowledge of the team. The further in the process of making the less reliable performer specimens and objective subjects the team were, as the affordances began to be habituated into the body and the bracketing became more and more difficult. Therefore, a need to rely on audience inquiry, the third frame of substitute performers, came forth.
In human-centered computing and VR design, the user-experience (3) testing is a standard developmental phase. It is to gather insight into the functionality of the user interface, storyline, and aesthetics with the future user in mind. During the design process, the team invited occasional user testers to clarify pending design issues. These field-specific-performers had expertise in VR design or new media arts and gave feedback on design choices upon request, but therefore they would also not qualify as ‘regular audience’.
3. User is a term in HCI with connotations of appropriating a device and software built for a defined purpose with efficiency and desired functionality. Here, the aim was for open bodily exploration and the reversibility of performer-attendee, instead of user. Therefore, it’s more applicable to use notions of experiencer, perceiver, participant, viewer, attendee, or performer to highlight the entangled, agential nature of the artwork.
2.3 Conceiving the future performers-to-be
The embodied design practice gave us a rough idea of potential variations in performer approaches. Hence, the endeavor was inclusivity for varied performers, the process called for a more thorough critical inquiry into potential audience responses to inform the creative process and to ensure the design met the underlying aims of inclusivity. The interest was more towards whether the design fostered a meaningful and attentive agency and presence and less in the interface functional efficiency, which most of the field-specific-performer feedback was focused on. In other words, choreographically the interest was whether the work dynamically facilitated variant kinaesthetic approaches and if the interface was fostering engagement and pleasure.
We entered the test audience inquiry with a 5-minute prototype of the work which was mature enough to fulfil the main design goals. It had one scene with an autonomously generative particle system appearing as a vast swirling field and an interactive spatial soundscape created from pre-recorded string instrument sound files and a few percussive and non-tonal elements. There was also a time-based score built into the interactivity code that launched new sonic elements and shifts in visual dynamics customized to induce spatial, temporal, and dynamic changes in perceivers’ movements.
To gather the field experience of perceivers, the SC prototype was first presented at various XR industry events using mixed survey methods. This type of iterative-inductive inquiry was an ethnographic approach to VR audience [15]. Inquiry methods were selected to gather information about audience engagement in an authentic context and to minimize disturbing their engagement. The volunteering experiencers were observed, filled out a short questionnaire on experience quality, and briefly interviewed if they initiated spontaneous post-experience comments, which was often the case. The observations and spontaneous verbal feedback were collected on the spot and the qualitative questionnaire results were analysed with grounded methods. Testing the work in XR events for marketing and networking purposes proved challenging. The audience was predominantly technology and design-oriented and therefore likely to have biased, preconceived expectations of VR resulting in skewed results. Also, the events were often very crowded with a general tone of ‘social networking’ resulting in superficial engagement and a constant flow of new viewers challenged focused data collection. Hence, to expand the scope, the prototype was also shown in more media culture and research-oriented contexts to allow more apt sampling (4).
The results from these iterative inquiries were encouraging showing enthusiasm towards the concept and consistency in interface comfortability. However, due to these inquiries being conducted in public, social events there was never sufficient time to engage in a proper in-depth interview, and while the preliminary results showed an indication of developmental needs the suggestions were predominantly focused on a gamified approach. In short, while the audience was generally enjoying the work, they seemed to be confused about its purpose, offering a variation of solutions to address their discomfort. Therefore, a small target-group inquiry with a complimentary method of an in-depth, semi-structured interview was organised. The purposeful sampling of the 5 participants was to represent a potential audience with very little or no prior experience with VR. Experiencers were individually observed by the inquirer and interviewed immediately after.
This inquiry shed light on many details in embodied, contextual, and discursive factors that were vaguely indicated but remained unspecified in other data. Compelling results were how porously the body was perceived in contact with the digital material and the multiple interpretations of the encountered world, referring to it as outer space, swarms of fireflies, being submerged under water, or transitions in transcendental dream-like dimension. Another significant finding was that despite their bodily abilities, physical conduct, or dominance of either auditory or visual sense, the imaginative elements were found meaningful and appealing.
It is worth highlighting how the ethnographic mixed methods [15] were expanded upon the phenomenological practice described earlier. An expansion of ethnographic practice (5) has been argued to better grasp and adapt to the contemporary mediated sociocultural sphere [4, 7, 14]. In conducting audience inquiry of VR, without the embodied and system-compositional knowledge, the perceiver’s behaviour nor reflections do not fully reveal themselves in depth. The singularity of VR is ultimately very private and to interpret the unfolding subjective felt qualities required careful attention to intercorporeality [19], the observer's ability to bodily echo the behaviours of others with knowledge of the perceived phenomena. This in-person attunement and second-person corporeal interpretation may compromise the observer's objectivity, but what is gained by attention to the kinaesthetic subtleties and the subject’s corporeality at hand, is an increased scope, validity, and depth in interpretations and analysis. Bodily attunement was an essential asset in the post-experience interviews, whereas the lived experience still echoed freshly in both bodies and thus allowed for the prompting of recollection and articulation of subtle, almost indescribable micro-level details.
The audience inquiry influenced the proceeding creative process by providing a much wider notion of imaginary performers. The concept was consciously developed towards a performative, open exploration rather than a gamified audio-visual instrument, as the audience study indicated an artwork would provide more hedonistic and aesthetic pleasure, albeit educatory and gamic creative tools seemed to be a prevailing business trend in VR at the time. To achieve this, the results indicated clear needs for improvement in temporality and dramaturgy of interactivity, which were addressed through several compositional and design choices, but also by attending more carefully to our blind spots in supporting inclusion.
An essential need for improvement was to allow more time to establish and enjoy the improvisational nature of the work so that the perceiver could discover the embodied interface and understand the basic concept of kinaesthetic primacy and open-endedness. This was met with a new scene with spatially fixed, discoverable affordances, where the perceiver was able to explore the space with causal and obscured properties embedded in it. Also, adding tracking points to the feet enforced a feel of full-body engagement, and allowed more diversity, especially in sonification aesthetics. With the additional tracking, the potential movement points in space doubled, allowing for an exponential number of relative and dynamic combinations in the motion data stream, providing multiplicity in affordances and increasing ambiguity. It also facilitated the addition of new sonic layers, either to be launched by the agents themself as nested ‘if-then’ causalities (6) or by encoding them into the temporal dramaturgy. This was to strengthen the polysolutionality and ensure there is always complex auditory feedback despite the viewer's bodily approach but also to diminish the dominance of the visual sense.
4. Most of these events took place in Finland with some also overseas. The total number of experiencers within a period of 3 months was close to 100.
5. Contemporary expansions of ethnography such as affect ethnography, performance ethnography, sensory ethnography, and the use of various media in practice.
6. An if-then causality is a coding framework of where the augmentation is launched when certain conditions are met, such as an extreme reduction in movement velocity triggering a percussive sound.
3 REFLECTIONS ON THE METHOD
Through the tracing of a process of the SC artwork, I have drawn out some key features of locating the body in the practice. I have presented an interconnected lens applicable and salient when creating embodied interactivity in VR. I have also argued for the benefits and value of ethnographic methods to be integrated into the process of VR content-making. Finally, I want to return to how the key methods of ethnography and phenomenology intertwine to produce actively synthesized knowledge.
3.1 Ethnography of VR as Medium
As a rapidly developing device- and system-based industry, the practices of VR with use cases of its products are constantly changing. The culture of the VR industry encompasses the culture of creating content for VR with the engineering of the devices themselves, and the sales and marketing of all its products. The features of the hardware and software go hand in hand with content creation. In general, one might say the industry is very business and market-oriented with art playing a marginal role. All these factors come into play when one examines the VR industry as a culture.
To enter this domain and apply choreographic thinking it is important to study and familiarise oneself with VR medium. Not only the conventions of practices but also the perceptual qualities and cultural preconditions the medium might entail. Assessing the domain requires the knowledge of the culture including the utilised hard- and software, to familiarize oneself with the customs, practices, and even ideologies the industry carries, and an awareness of current and envisioned future trends within the industry. This means engaging with and operating within the culture and actively gathering knowledge of it. A type of ethnographic study of the virtual mediation culture and field.
3.2 Ethnography of performers-to-be
Within the endeavor of creating improvisatory, inclusive VR artwork for the audience, in the framework of an agency so intrinsically entangled with the outcome, the real quest is to understand the imaginable experiential qualities of the system. That is to learn about the potential human behavior and affects enabled and induced by the virtual realm. To access and interpret the behaviors, feedback, and reports of the audience for the benefit of the choreographic endeavor is to engage in an extended ethnographic inquiry relying on embodied compositional knowledge. The study preparation, methods deliberation, question formulation, observation making, and result analyses all require first-hand knowledge of the kinds of potentialities and conditions the informants are exposed to.
It is also good to acknowledge that in an audience inquiry, the context matters. Alongside the given pre- and onboarding information, the context the work is presented influences the audience’s expectations and their interpretation. In the art context, where VR is a relatively new medium, the audience tends to have a more open-minded, performative approach, while also gaining more personal pleasure, whereas, in technology contexts, the preconditioning expectations may lead to an overly analytical approach preventing an explorative stance.
Processing the collected data and translating it into artistic conclusions means accumulating complex ethnographic and phenomenological knowledge and ultimately, through interconnected analytic cycles into choreographic, computational, and aesthetic precision. All this requires systematic data collection, iterative induction, and deductive theorizing for design benefits.
3.3 Embodied Design Practice
Ethnography on VR culture is the groundwork for grasping the device-specific, multisensorial, experiential, and singular qualities of the medium and assessing its potentiality for art. The process of acquiring knowledge of the virtual realm and its perceptual and interpretive properties entails having a profound understanding of the medium with its predominant values and customs, the ‘praxis’ of reflecting on the subjective experience, a systemic understanding of ‘techne’, and an encompassing critical analysis of them all. To conceptually think of movement-based mediation circuits choreographically requires lived knowledge of what it is to be a perceiver-performer in it, and therefore, a phenomenological inquiry is embedded in the process. The SC process illuminates, how the practice of digital choreography for VR was intrinsically and multifacetedly rooted in the experience of the choreographer-performer, in her ability to reflect on the surfacing embodied knowledge and to be in an intercorporeal relation with others, human and digital to deliberate the choreographic concepts.
In the making, the bodies of substitute performers are at play. It is not only to consider the performer-choreographer’s perceptual experience at hand but also the bodily knowledge of the design-oriented-performers and the performers-to-be. What is worth pointing out is a deviation from the practices of ethnography as a purely sociological research method. Here the important method is the thick bodily observation and attentive analyses with design knowledge, rather than deductive analyses of means to an end goal, with theoretical underpinning. Rather an inductive extended ethnography with imagining what the artwork could potentially be and the ways it may facilitate the imaginary performer. This requires an open-minded beingness with the very fabric of the virtual potentialities, the unexpected emergent ontology. This is the essence and heart of the artistic endeavor. The beingness with the interplay of the actual physical experience of the designed system, the careful attending to subjective states, the articulation of these experiences, transcorporeal observing and kinaesthetic empathising with others and incorporating all this into the practical making of the choreographic system. It is from this shared, yet individually distinctly different corporeal and pragmatic knowledge a team is able to develop a shared design language and articulate common aesthetic aims. The contribution here is in the detailed and insightful articulation of the subjective experience of all substitute performers at play and extending that insight to constructing a virtual realm for the imaginary performer.
Therefore, the embodied-design approach I’m suggesting here is enclosing the ethnographic research of VR culture at large and the potential audience, as elemental parts of choreography in VR, and that it is simultaneously necessarily phenomenological. A phenomenological attitude is needed because the felt and lived process of entanglement with the system needs to be reflected for it to be critically evaluated and abstracted into computational design schemas. Within a circuit of body and technology, the compositional practice relies upon reflecting on the experienced together with the encoded system, and only with a combination of phenomenological, pragmatic, and ethnographic methods the scrutiny of iterations of the digital techno-choreography can take place.
4 CONCLUSION
In movement-based mediation, the embodied design approach and choreography come together. It pushes choreography into thinking of space-time with imaginary bodies in imaginary spaces, thus expanding its practices. The movement mapping tools and computation systems for designing from and for the body are ever-expanding. These efforts are challenging, yet as I have argued here it is valuable to adopt a type of anthropological attitude toward the mediated human experience in order to imagine and realise choreographic concepts in VR. The process of creating for VR, and argumentatively potentially for any content creation in movement-based mediation environments, benefits from having a synthetic lens on methods and a critically reflective approach. This entails a phenomenological approach with bracketing in attending to the subjective lived experience, an ethnographic approach to the larger cultural frame of VR as a medium for its myriad purposes, and an expanded ethnographic approach to perceiver-performer experience to cultivate intersectional sensibility in design. The analytical and critical stance relies on the creator’s ability to synthesise knowledge and adopt a variety of skills from subjective micro-level practices into wider cultural context analysis.
ACKNOWLEDGMENTS
This research is part of doctoral research on mediated dance and digital choreography. The artwork creation was supported by Finnish Art Promotion Centre and Business Finland. I thank the collaborating artists Daniel Leggat and Janne Storm for their artistic contribution and study participants for their involvement as informants.
REFERENCES
[1] Jane M Bacon and Vida L Midgelow. 2011. Articulating choreographic practices, locating the field: An introduction. Choreographic Practices 1, 1 (January 2011), 3–19. https://doi.org/10.1386/chor.1.3_2
[2] Karen Michelle Barad. 2007. Meeting the universe halfway: quantum physics and the entanglement of matter and meaning. Duke University Press, Durham.
[3] Jo Butterworth and Liesbeth Wildschut. 2012. Contemporary Choreography: A Critical Reader. Routledge.
[4] Maud Ceuterick and Chris Ingraham. 2021. Immersive storytelling and affective ethnography in virtual reality. Review of Communication 21, 1 (January 2021), 9–22. https://doi.org/10.1080/15358593.2021.1881610
[5] G. Deleuze and C. Parnet. 2007. Dialogues II. Columbia University Press.
[6] Paul Dourish. 2001. Where the Action Is: The Foundations of Embodied Interaction. MIT press Cambridge.
[7] Paolo S. H. Favero. 2023. Doing audio/visual/sensory ethnography with and on smartphones – a possible roadmap for an expanded ethnography. International Review of Sociology (July 2023), 1–22. https://doi.org/10.1080/03906701.2023.2233295
[8] Susan Leigh Foster. 2011. Choreographing empathy: kinesthesia in performance. Routledge, London ; New York.
[9] Homewood, Marika Hedemyr, Maja Fagerberg Ranten, and Susan Kozel. 2021. Tracing Conceptions of the Body in HCI: From User to More-Than-Human. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, May 06, 2021. ACM, Yokohama Japan, 1–12. https://doi.org/10.1145/3411764.3445656
[10] Don Ihde. 2007. Listening and voice: phenomenologies of sound (2nd ed ed.). State University of New York Press, Albany.
[11] Mark Johnson. 2007. The meaning of the body: aesthetics of human understanding. University of Chicago Press, Chicago.
[12] Susan Kozel. 2007. Closer: performance, technologies, phenomenology. Massachusetts Institute of Technology, S.l.
[13] Maurice Merleau-Ponty. 2004. The visible and the invisible: The intertwining-the chiasm. The visible and the invisible.
[14] Jean J. Schensul and Margaret D. LeCompte. 2012. Specialized Ethnographic Methods : A Mixed Methods Approach. AltaMira Press, Blue Ridge Summit, USA
[15] Stephen L. Schensul, Jean J. Schensul, and Margaret D. LeCompte. 2012. Initiating Ethnographic Research : A Mixed Methods Approach. AltaMira Press, Blue Ridge Summit, USA
[16] Maxine Sheets-Johnstone. 2011. The primacy of movement. John Benjamins Pub. Co., Amsterdam; Philadelphia.
[17] Timothy Stapleton. 2009. Dasein as being-in-the-world. In Martin Heidegger (1st ed.), Bret W. Davis (ed.). Acumen Publishing Limited, 44–56. https://doi.org/10.1017/UPO9781844654475.004
[18] Emily Elizabeth Sweeney. 2013. Semblances of Form: A Phenomenology of Improvising with Performance Scores. University of Roehampton School of Drama, Theatre, & Performance (2013).
[19] Gail Weiss, Ann V. Murphy, and Gayle Salamon (Eds.). 2020. 50 concepts for a critical phenomenology. Northwestern University Press, Evanston, Illinois.