Handbook of Multimedia for Digital Entertainment and Arts- P24

Chia sẻ: Cong Thanh | Ngày: | Loại File: PDF | Số trang:30

Thêm vào BST

Báo xấu

58
lượt xem 5
download

Download Vui lòng tải xuống để xem tài liệu đầy đủ

Handbook of Multimedia for Digital Entertainment and Arts- P24: The advances in computer entertainment, multi-player and online games, technology-enabled art, culture and performance have created a new form of entertainment and art, which attracts and absorbs their participants. The fantastic success of this new field has influenced the development of the new digital entertainment industry and related products and services, which has impacted every aspect of our lives.

Chủ đề:

Bình luận(0) Đăng nhập để gửi bình luận!

Lưu

Nội dung Text: Handbook of Multimedia for Digital Entertainment and Arts- P24

31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 695 resonance. This aesthetic is an inner resonance recognized from achieved user- experience. Having experienced a vocation as a stage performer I relate to emotions achieved through a satisfying entertainment of an audience. Thus, in the therapy the user entertains him or her-self through the mediating interactive digital media system. This can be from controlling a game or creating art, i.e. music making or painting via digital tools, and is evident as FUN resulting from the created situ- ations. Facilitator knowledge of the potentials of the digital media is essential to user-experience, successful guidance (targeted as via mediating technology), and user self-determined entertainment. The same system that was created speciﬁc for the rehabilitation training was used by me to perform abstract expressionistic stage art and interactive installations at various large exhibitions. These include ongoing tours in Museums of Modern Art as well as at International-National Cultural events (e. g., Danish NeWave, New York 1999; Cultural Olympiad 1996, 2000; and European Cultural Capital events 1996, 2000). Underground events were as informative to the concept as the larger venues (see next section). Insight was gained from these situations. By combining self-reﬂection with collaborative inter-/multi-disciplinary reﬂective analysis of the human performance situations a signiﬁcant input to designing the system, facilitat- ing sessions, and evaluating outcomes evolved. Underground Non-formal Learning An example of how learning was apparent from the entertainment and arts is where I directed, produced and performed in a one-man-show realized as the inaugural Aarhus Festival Fringe (‘Festuge’). This 1999 event took place in an emptied storage rooms that were adjacent to my sizable interactive room installation titled Circle of Interactive Light (COIL), hosted by the Radisson SAS Scandinavia Hotel, Aarhus, Denmark. The Festival committee had declined my proposal to bring together East and West German artists to feature alongside North and South Korean artists as too politically sensitive. The festival theme was ‘THE WALL, Ten Years After’ – and my proposal involved artists of distinction from both sides of the Berlin Wall and the DMZ to discus and showcase how political issues affect art and artists lives. Understandable that the committee declined! The eventual “FESTUGE FRINGE” protest performance featured my digital in- teractive media in the form of the motion sensitive system utilized in various ways with projections inside and outside the human body. The ﬁnal section of the perfor- mance was where I used fourteen infrared sensors to control one sound and a library of image manipulations. The purpose was to explore non-control and subliminal performance by enabling the feedback to control me and this was done by mapping the data to maximum parameter control of a speciﬁc aspect of the sound envelope. In four of the twelve performances I achieved what I refer to as non-control with subliminal interaction. This was where I, as performer, experienced being as one with the feedback stimuli. If conscious intent or query was involved, the experience
696 A.L. Brooks and higher-state of interaction (or inbetweenness – see Kidd) was lost. I relate this targeted non-control and interaction to the therapy work with profoundly disabled where often the user’s abilities are of a non-controlled state so my learning experi- ences as performer/user assist by informing my role as facilitator. Following the performances the wall would open to exhibit the technology used, and then audience walked through the wall into the COIL exhibition space where they could experience the virtual interactive spaces and debate over a drink in the caf´ I had built for the occasion. The piece titled “Behind the Wall” targeted beyond e Berlin and DMZ issues as it reﬂected on human barriers across ages and cultures – and especially committees. The next section introduces the ArtAbilitation2 workshops from 2007 and 2008. The ArtAbilitation movement has evolved from the SoundScapes work. It is where digital interactive media is used to create an entertaining user-experience. ArtAbilitation Workshops, Casa da Musica, Porto, Portugal The “Ao Alcance de Todos - Within Everyone’s Reach” festival was hosted at Casa da M´ sica (Figure 13), Porto, Portugal in April 2008. Eight one hour workshops u were held with 144 disabled children and adults attending with caregivers. The workshops were created in a room 238 square meters ﬂoor area and approximately 20 meters high. Additionally, a symposium for professionals was hosted immedi- ately following the workshops for 35 professionals (international, national, regional, and local // social workers, psychologists, researchers, teachers, and students : : :) - many of whom had attended the workshops. This event was a continuation of the previous year’s inaugural event where the theme was ‘Music, Technology and Disability’, and where six workshops were attended by 91 attendees including 61 from special care institutes, of which 39 had profound disability. 30 student music teachers also attended. A local crew assists in the realization of the workshops (see Petersson and Brooks 2007; Brooks 2008). This section does not detail the workshops but rather it exempliﬁes the design issues where digital interactive media, entertainment and the arts are combined to result in various user-experiences that inform the ongoing research and reﬁnement of the acknowledged international ArtAbilitation movement (Wiederhold 2007). I begin by giving an overview of commonalities in designing both workshops. The 2007 workshop is then detailed in more depth followed by the 2008 workshop. Both workshop designs targeted participant learning from interactive experiences where active participation formed a context for meaning. Offered were opportuni- ties for augmenting learning and awareness. Observations were of self-determined active participation that established goals that in turn engaged interest, curiosity and play. In this way, motivation to achieve goals is considered intrinsic rather than extrinsic, which can result from an activity, task or goal introduced by someone else, 2 http://www.ArtAbilitation.net
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 697 Fig. 13 Casa da Musica, Porto, Portugal (photo with permission Casa da Musica) and or with possibly over zealous guidance/facilitation. Contextual design thus be- came a way to conﬁgure learning resources and interaction (Kress & van Leeuwen, 2001). Exploration, play and transformation were also targeted in both events such that each participant had an ‘action-stillness’ proﬁle that evolved through inter- actions. Each cycle of activity was considered a new creation that contributed to form patterning of actions resulting from the activity. Each activity was designed to increment challenges of confrontation. The resultant ‘play’ and created ‘creativ- ity’ scenarios involved manipulation of provided tools. The manipulation required a degree of competence that was learnt through exploration of the tool’s traits (Bruner 1972). Absence of negative consequences seemingly encouraged participant free ex- pressive explorations, which in turn, can over time result in development of un- employed skill (Beach, 1942). Such development is signiﬁcant even if it is at a micro scale, i.e. micro-development (Yan and Fischer 2002). Responsive environ- ment composition and learning process/outcome evolve as the assessment focus instrument of observers. Realized was new learning spaces and approaches to learn- ing and rehabilitation through emphasizing user’s creation of meaning via serious play. The approach is that no aspect of the learning process and outcome is taken for granted. Rather it is formulated into play and creative activities that are inherent to e.g., games and art making. The activities of play and creativity conceal the embed- ded learning and training involved for the user. An emphasis is on a supplementing tool for traditional practices rather than a replacement. In this sense, learning is at a ‘subliminal’ level for the user as he or she engages in the responsive environment. Thus, motivation is optimized through action and stillness cycles where the user
698 A.L. Brooks Fig. 14 Sala de Ensaio 1, Casa da M´ sica, Oporto, Portugal (C D camera) – (Brooks 2008) u iteratively explores and transforms the feedback media. This process contains choices and decisions that indicate learning, e.g. in the form of increased repertoire of expressions, changes of skills, and new patterns of social interaction (Brooks 2008). In the 2007 workshop a private (2) and a public (1) area were designed – see Sala de Ensaio diagram (Figure 14). This was inspired from the evaluation consul- tations mentioned earlier in this chapter where a caregiver experienced that a private space, without any intervention, was desired before a user felt comfortable enough to express through the SoundScapes system. My previous research (Brooks et al. 2002) presented how certain individuals prefer to explore, play and create without any oth- ers being present (Figures 15–16). Exemplifying the SoundScapes open system and concept a digital video camera was used to create a feedback loop play-space where participant gesture distorted image silhouette and color change to a RGB lighting system that reacted to voice input. Interactions in the private space were video taped and analyzed as achieving a positive ﬂow state (Csikszentmih´ lyi 1996). It a was observed in the private space that the chain of exploration-play-creation began with a curiosity that evolved out of the isolation and initial stillness that was ﬁrst encountered within the created environment. Thereby, stillness became part of the action and vice versa. Indications pointed to how interactive play and creativity that offered choices between interaction and rest in a silent space enhanced the sense of control. A created public space (Figures 17–18) questioned participant perception and awareness where peer-support and scaffolding of exploration was evident. Over- head infrared camera tracking was mapped to auditory music making (Figure 17)
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 699 Figs. 15-16 Private space images of total engagement from ArtAbilitation 2007, (photo credit Augusto Brazio, Casa da Musica) and image effects (Figure 18). Many results occurred from this space. One memo- rable instance was when a woman who was profoundly physically disabled insisted in exploring the interactive space out of her wheelchair – evident was that crawling around required immense effort (Figure 18). Her motions were tracked to open up a digital mask that concealed an image of a famous Portuguese footballer. Her disabled colleagues were supporting by shouting instructions. Motivation was stag- gering to achieve the whole erosion of the mask. Another instance was where an autistic group was in the workshop and one of them began to move and open up the same mask. The ﬁrst exposed facial element of the footballer was the chin and im- mediately another autistic male shouted out the name of the recognized footballer. In both instances (and others) the public virtual interactive space empowered imme- diate learning to be exhibited (Petersson and Brooks 2007). Five physical ‘VIS rooms’ were created in the 2008 workshop. Each was de- signed to offer differing experiences of empowerment through motion creating images and auditory feedback. Large sub-woofer speakers positioned on the wooden ﬂoor gave tactile stimulation which was especially important for the attending class of young deaf children. Findings were that the interactive spaces offered a place where comprehension of tasks could be shared from the strong to the weak in each group. Again wheelchairs were discarded to explore by crawling around the spaces.
700 A.L. Brooks Figs. 17-18 Public interactive space: two designs utilizing overhead camera tracking Targeted tangible outcome was that each participant received a painting that they created through gesture. Reports of how many showed these paintings to family, friends and non-attending staff stated to the powerful effect. Both workshops exhibited action-stillness cycles, which consisted of iterative loops of exploring and transforming, constitute one part of a theoretical map for the purposes of analyzing critical incidents in a non-formal learning process. These cycles are related to the user’s learning experience. The other part of the theoretical map concerns design issues in the form of use qualities relative to the user’s interac- tive experience; transparency, social-action space, user control/autonomy, pliability, playability and seductivity (L¨ wgren and Stolterman 2005). During a whole session o
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 701 cycle of action and stillness, facilitators have the possibility to reﬂect upon the in- dications of learning that occur during the process. The user proﬁle inﬂuences the facilitator’s decisions on how to set up the attributes of the responsive environments relative to the desired learning process and the expected outcome of that process. Once these prerequisites are set, the user is expected to ‘experience from the inter- action and learn from the experience’. Hence, the aim of the workshop events was to investigate the user’s performance in using responsive environments designed to engage participants in experience- learning through action and stillness cycles. A sub-question concerns the ability of perception and the associated learning curve of the attendees with cognitive disabil- ities to be able to easily correlate across dimensions of scale and plane – a matter which inﬂuences the participant’s interactive as well as learning experience. The ﬁnal reported event is one that was built upon previous work where third party performance gesture created visuals to complement classical music situations. Visualizing classical music The previous sections present the motion-sensitive environment and its use as a re- habilitation training supplement within the community of people with a disability, both profound from birth, and acquired. The use of the created interactive system is presented in intimate institute settings for individuals and a later section shares two examples of larger workshop situations where groups attended. This section exem- pliﬁes how a similar concept and technique of using digital interactive media was implemented to elicit dynamic performance data from a situation that featured the ‘Orquestra Nacional do Porto’ - regarded by many as The Portuguese National Or- chestra. This was again in Casa da Musica, Porto, Portugal but this time in the main 1200 seat state-of-the-art auditorium. See publication Interpretation3 (Brooks 2008). A goal of the exploratory study was to dynamically complement the music by offering an experience of inter/multi-sensory stimulation for both audience and per- formers. This built upon my earlier work in Auckland, New Zealand, and Aarhus, Denmark where the different situations, one with orchestra and one with choir, both had a similar stimulation goal. An aim of this work is to offer inclusive access so that even people who are deaf may have an opportunity to appreciate classical mu- sic. Important to mention is how in the choir study three sensors mapped to RGB enabled the conductor to paint the choir backdrop through his gestured interactions with the singers. The session was in the Danish Radio TV studio and six ‘takes’ were recorded. It was evident that his gestures were expanding as he became used to the interaction and – in his own words - “how it felt that the air around my body was activated”. This expansion of gesture relates to the work in the disability sector where targeted in subliminal motivated augmented motion. The following Figs. 19–26 illustrate the result from the visualization experiment. Overhead cameras and stage sensors sourced the performance data and this was 3 http://www.icdvrat.reading.ac.uk/2008/interpretations.htm
702 A.L. Brooks Fig. 19 Data from stage performance – conductor gesture, section/musician expressiveness, lights and music - sourced to dynamically affect the synchronized visuals for environment change in real- time (photo credit Jo˜ o Messias, Casa da Musica) a Fig. 20 Sourced stage data mixed and matched to digitally mirror performance in abstract – here camera and music stand lights - creating and effecting projected visuals (photo credit Jo˜ o Messias, a Casa da Musica)
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 703 Fig. 21 Projected moving image dynamically matched to digitally mirror performance (photo credit Jo˜ o Messias, Casa da Musica) a mapped by created algorithms to effect the environment variables in real-time to complement the music. A technical deﬁcit proved the biggest hurdle and results were unsatisfactory as image mixing was via switching instead of faders on a mixer. Thus, nuance of performance was unable to be matched as I desired. However, as an experiment to explore the concept it did build my knowledge of the concept. The work is planned to evolve further locally for the next phase of experiments due to the problems of accessing the specialized equipment required for technical setup. The work has been accepted for the Cultural Olympiad 2012 London.
704 A.L. Brooks Fig. 22 Projected image (center of screen) shows data capture from camera overhead (photo credit Jo˜ o Messias, Casa da Musica) a Conclusions and Future Directions This article informs of digital interactive media sculpted into a responsive ges- ture driven environment that is used within rehabilitation training. Entertainment achieved through a user creating and playing acts as a stimulant towards augmenting life quality and motivating participation in training that otherwise is boring, tedious and mundane – often for both patient and therapist/facilitator. Artistic composition in the form of music-making and digital painting is empowered through an interface and multi-sensory virtual environment content that is adaptive and ﬂexible enough to address idiosyncratic needs. However, whilst I create for another’s creativity, my
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 705 Fig. 23 Musicians can observe effect of their own performance (photo credit Jo˜ o Messias, Casa a da Musica) art is not the ‘product’ of the users’ interaction, i.e. the music or painting, as many judge, but rather it is the aesthetic resonance that is achieved in and of the user. El- lis (1996, 2004) describes aesthetic resonance achieved within his ‘Sound Therapy’ body of research. This was extemporized in a 2002 article (Brooks et al. 2002) fol- lowing ﬁndings that explored interactive visuals in the form of animations, virtual reality, and robotic lighting devices with colored and patterned gobos. This aesthetic is a representation of a system where affordances were perceived and the targeted action achieved. Resultant sense of self-agency and inner empowerment is realized cumulating to signify a state of ﬂow that is identiﬁed by the facilitator. Investiga- tion is of how to optimize user experience toward self-determined development via the virtual interactive space (VIS) that is created between the user and the system.
706 A.L. Brooks Fig. 24 Conductor engaged in role as he manipulates responsive image artifacts (photo credit Jo˜ o a Messias, Casa da Musica) Fig. 25 Selective use of dynamic shadow effects are merged into environment (photo credit Jo˜ o a Messias, Casa da Musica) Potential micro-development (Yan and Fischer 2002) is a suggested potential even for those who are profoundly impaired. Problematic on the other hand is where a user is ﬁxated on formal therapeutic conditioning and gets tied up in the ‘cor-
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 707 Fig. 26 View from behind the control station where author interacts (photo credit Paul Sharkey) rect way of doing’ instead of ‘just doing’ with a focus on the mediating feedback content rather than on the impaired feed-forward function. User achievements are commonplace. The activities of exploring interactive systems in the ﬁeld of disability are seem- ingly growing. In Denmark commercial suppliers that sell targeted games for this market attempt a monopoly with substandard goods and services that tend to dimin- ish opportunities for end users rather than increase. After two decades of research it is clear that open ﬂexible and affordable systems that have selectable libraries of input device as well as selectable libraries of content stimuli are optimal. Systems that can be adapted, and personalized within customized environments tailored to
708 A.L. Brooks an individual proﬁle offer maximum opportunities and beneﬁt. That beneﬁt is not restricted to those with impairment as fun social interactions can motivate in playing games or making art. Interface technologies have improved and motion detection devices such as the Nintendo Wii offer a data device that permits access to data and thus can advance the ﬁeld in an affordable ﬂexible package – either as a controller for games or music/art making. An approach I made to Sony Computer Entertainment Europe (SCEE), London, after the launch of the EyeToy asking such access to interface data was rejected. Their representatives stated discriminatingly that they did not wish to be seen exploiting disabled people. With such technology available it is a shame that such a commercial proﬁle prevents opportunity inquiry. However, computer vision and graphical card advances makes affordable non-commercial sensor, single or multiple camera systems, a reality and our students in MEDIALOGY4 in Esbjerg are active exploring both user-centered games and art as controlled mediums in the SensoramaLab VR complex. Such diversity in the SoundScapes concept is recognized by Eaglestone & Bamidis (2008) as elements of a hybrid system consisting of networks of complex inter-connected subsystems comprising ‘created technical systems’ and ‘natural hu- man systems’. The MAX software has also undergone huge improvements since I ﬁrst used it as version 2.5 in the early 1990s. Integrated capability to manipulate digital audio signals in real-time from a computer became a reality via the MSP (Max Signal Processing) add-on, thus there was no longer a requirement for exter- nal audio hardware. Subsequently another package developed and released for the MAX software environment in 2003 was Jitter, which enabled control of real-time video, 3D and matrix processing. Latest MAX activities is a partnering with a loop- based sequencer software named Ableton Live that offers intuitive live performance capabilities where MIDI manipulations can slot into a playing song such that it re- mains ‘musical’. Whilst developed for DJs and VJs the potentials in my research for such a tool is acknowledged and this is a future direction when the packages are released. Other explored software includes Touch Designer by Derivative, and Syntetik’s awesome Studio Artist interfaced with a Wacom Cintiq tablet. In this sense a tool to supplement traditional training becomes available if the community of therapists and medical staff are prepared to accept an alternative ap- proach in the form of a hybrid non-formal learning apparatus and method. If they do and are prepared to collaborate in evolving the concept with other interested researchers from the related disciplines then the next-generation of therapists may well be teamed with digital artists, computer scientists, and fMRI neurologists in creating, facilitating, and evaluating the human in new ways, with new methods, and with new apparatus, where FUN is the targeted user-experience and human beneﬁt the ultimate goal. New virtual interactive spaces for learning are exhibited in this work that combines digital interactive media with entertainment and the arts. 4 http://www.medialogy.eu
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 709 References 1. K. E. Misulis. “Essentials of Clinical Neurophysiology.” Butterworth-Heinemann, 1997. 2. A. Lucier. “Statement On:luciermisulis Music for Solo Performer.” In: D. Rosenboom (Ed.), Biofeedback and the Arts, Results of Early Experiments. Aesthetic Research Center of Canada Publications, 1976. 3. D. Warner. “Rock ’n roll Science:warner Playing the Body Electric” [online] Available from http://www.virtualgalen.com/virtualhealing/dr-dave.htm. 4. A. L. Brooks, A. Camurri, N. Canagarajah and S. Hasselblad. “Interaction with shapes and sounds as a therapy for special needs and rehabilitation.” In:brooks International Conference on Disability, Virtual Reality and Associated Technologies (4th). Veszpr´ m, Hungary, 18–20 e September. P. Sharkey, C. S. L´ nyi, and P. Standen, (Eds.). Reading: University of Reading, a pp. 205–212, 2002. 5. A. L. Brooks. Robotic Synchronized to Human Gesture as a Virtual Coach in (Re)habilitation Therapy. In: 3rd International Workshop on Virtual Rehabilitation (IWVR2004), VRlab, EPFL, Lausanne, Switzerland, pp. 17–16, 2004. 6. E. Petersson and A. Brooks. Non-formal therapy and learning potentials through human ges- ture synchronized to robotic gesture. International Journal Universal Access in the Information Society. Springer 6(2), pp. 167–177, 2007. 7. W. M. Marston. Emotions of Normal People, Routledge, 1928. 8. A. L. Brooks. SoundScapes: A concept of Virtual Interactive Space (V.I.S.) [unpub- lished]. World Summit for Social Development/NGO Forum 6–12 March, Holmen, Copenhagen, 1995. 9. A. L. Brooks. Virtual interactive space (V.I.S.) as a movement capture interface tool giving multimedia feedback for treatment and analysis. In: “Bridging Cultures” – pro- gram of The 13th International Congress of the World Confederation for Physical Ther- apy, Yokohama Japan, May 23–28, Science Links Japan: http://sciencelinks.jp/jeast/article/ 200110/000020011001A0418015.php, 1999. 10. A. L. Brooks. Virtual interactive space (V.I.S.). In: ‘Pushing the limits: optimising poten- tial through science and technology’, Congress Program and Abstract Book, 5th Scientiﬁc Congress, Sydney 2000 Paralympic Games, Oct. 11–13, Convention Centre, Darling Harbor, Sydney, Australia, 2000. 11. A. L. Brooks. HUMANICS 1 – a feasibility study to create a home internet based telehealth product to supplement acquired brain injury therapy. In: International Conference on Dis- ability, Virtual Reality and Associated Technologies (5th). Oxford University, UK, 20–22 September 2004. Sharkey, P., McCrindle, R. and Brown, D. (eds.) Reading: University of Reading, pp. 43–50, 2004. 12. A. L. Brooks and E. Petersson. Recursive reﬂection and learning in raw data video analysis of interactive ‘play’ environments for special needs health care. In: Proceedings of 7th Inter- national Workshop on Enterprise networking and Computing in Healthcare Industry, Korea HEALTHCOM 2005, IEEE Signal Processing Society, USA, pp. 83–87, 2005. 13. A. L. Brooks and S. Hasselblad. Creating aesthetically resonant environments for the hand- icapped, elderly and rehabilitation: Sweden. Proceedings of 6th International Conference on Disability, Virtual Reality and Associated Technologies (ICDVRAT), Esbjerg, Denmark, 18th– 20th September, pp. 191–198, 2004. 14. A. L. Brooks. Enhanced Gesture Capture in Virtual Interactive Space. Computers in Art, De- sign, and Education (CADE), 29 June–01 July, Copenhagen Business School, Denmark and Malm¨ University, Sweden, 2004. o 15. A. L. Brooks and E. Petersson. Stillness design attributes in non-formal rehabilitation: CADE2007 - Computers in Art Design and Education. Perth, Curtin University of Technology, pp. 36–44, 2007.
710 A.L. Brooks ˚ 16. G. Goldkuhl and P. J. Agerfalk. “Actability: A way to understand information systems prag- matics.” CMTO Research papers No. 2000:13, Link¨ ping University. Presented at the 3rd o International Workshop on Organisational Semiotics, 4 July 2000, Stafford, UK, 2000. ˚ 17. P. J. Agerfalk and S. Cronholm. “Usability versus Actability: A Conceptual Comparative Anal- ysis.” Presented at the HCI International conference New Orleans, USA, Lawrence Erlbaum and Associates, pp. 235–237, 2001. 18. A. N. Leont’ev. “Activity, consciousness and personality.” NJ: Prentice-Hall, Englewood Cliffs, 1981. 19. R. Davies. “Commentary on P. J. Standen and D. J. Brown – Virtual Reality in Rehabilitation of People with Intellectual Disabilities: Review.” CyberPsychology and Behavior, 8(3), 2005. 20. A. L. Brooks and E. Petersson. “Play Therapy Utilizing the Sony EyeToyr .” In: Annual International Workshop on Presence (8th), London, 21–23 September 2005. Slater, M. (ed.) London: University College London, pp. 303–314, 2005. 21. C. Hummels, T. Djajadiningrapt, and K. Overbeeke. “Knowing, doing and feeling: Communicating with your digital products. Interdisziplin¨ res Kolleg Kognitions und Neu- a rowissenschaften, G¨ nne am M¨ hnesee, March 2–9, pp. 289–308, 2001. u o 22. B. Buxton. “Sketching User Experiences: getting the design right and the right design,” Elsevier, 2007. 23. M. Polanyi. “Personal Knowledge: Towards a Post-Critical Philosophy,” Routledge, 2002 [1958]. 24. R. Wollheim. “Art and Its Objects,” 2nd ed., Cambridge University press, 1980. 25. P. Ellis. “Layered analysis: A video-based qualitative research tool to support the development of a new approach for children with special needs.” The Bulletin for the Council for Research in Music Education. University of Illinois at Urbana-Champaign, USA, 130, pp.65–74, 1996. 26. P. Ellis. “Caress – ‘an endearing touch’.” In: J. Siraj-Blatchford (ed.) Developing New Tech- nologies for Young Children. London: Trentham Books, pp. 113–137, 2004. 27. A. Pollini, Experimenting with an Ubiquitous Computing Open Architecture, [PhD Thesis] http://www.ist-palcom.org/publications/ﬁles/PhD-Thesis-Pollini.pdf, 2008 28. E. Petersson and A. L. Brooks. ArtAbilitationr : An Interactive Installation for the Study of Action and Stillness Cycles in Responsive Environments. Computers in Art, Design, and Education (CADE 2007) http://cedar.humanities.curtin.edu.au/conferences/ cade/pdf/CADE2007Conferenc eprogram&abstracts.pdf, 2007. 38. A. L. Brooks. Towards a platform of alternative and adaptive interactive systems for idiosyn- cratic special needs, Proc. 7th Intl Conf. on Disability, Virtual Reality and Assoc. Technologies with ArtAbilitation, pp 319–326, Maia, Portugal, 8–11 Sept. 2008. 30. B. K. Wiederhold. Virtual Healers, International Association for CyberTherapy & Rehabilita- tion, 2007. 31. G. Kress and T. van Leeuwen. “Reading Images. The Grammar of Visual Design.” London, Routledge, 1996. 32. J. S. Bruner. “Nature and Uses of Immaturity.” In: J. S. Bruner, A. Jolly and K. Sylva (eds.), Play – Its Role in Development and Evolution. Basic Books, 1976 33. F. A. Beach. “Comparison of Copulatory Behavior of Male Rats Raised in Isolation, Cohabi- tation, and Segregation.” Journal of Genetic Psychology, 60, pp. 121–136, 1942. 34. Z. Yan and K. Fischer. “Always Under Construction: Dynamic variations in adult cognitive development,” Human Development, 45, pp. 141–160, 2002. 35. M. Csikszentmih´ lyi. Creativity: Flow and the Psychology of Discovery and Invention. New a York: Harper Perennial, 1996. 36. J. L¨ wgren and E. Stolterman. Thoughtful Interaction Design, Boston, MIT, 2005. o 37. B. Eaglestone and P. D. Bamidis. “Music composition for the multi-disabled: A systems per- spective.” Disability Human Development, 7(1), pp. 19–24, 2008.
31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 711 38. A. L. Brooks. Interpretations: inter-sensory stimulation concept targeting inclusive access of- fering appreciation of classical music for all ages, standing, & disability, Proc. 7th Intl Conf. on Disability, Virtual Reality and Assoc. Technologies with ArtAbilitation, pp 15–22, Maia & Porto, Portugal, 8–11 Sept. 2008. Images are marked with credit to photographer or permitting body. Otherwise copyright is with the author. Additional images are viewed at the author’s online gallery - http://gallery.mac. com/anthony.lewis.brooks
Chapter 32 Natural Interaction in Intelligent Spaces: Designing for Architecture and Entertainment Flavia Sparacino Introduction Designing responsive environments for various venues has become trendy today. Museums wish to create attractive “hands-on” exhibits that can engage and interest their visitors. Several research groups are building an “aware home” that can assist elderly people or chronic patients to have an autonomous life, while still calling for or providing immediate assistance when needed. The design of these smart spaces needs to respond to several criteria. Their main feature is to allow people to freely move in them. Whether they are navigating a 3D world or demanding assistance, we can’t strap users with encumbering sen- sors and limiting tethers to make them interact with the space. Natural interaction, based on people’s spontaneous gestures, movements and behaviors is an essential requirement of intelligent spaces. Capturing the user’s natural input and triggering a corresponding action is however, in many cases, not sufﬁcient to ensure the appro- priate response by the system. We need to be able to interpret the users’ actions in context and communicate to people information that is relevant to them, appropriate to the situation, and adequately articulated (simple or complex) at the right time. On the basis of my work and research I will argue that intelligent spaces need to be supported by three forms of intelligence: perceptual intelligence, which captures people’s presence and movement in the space in a natural and non-encumbering way; interpretive intelligence, which “understands” people’s actions and is capable of making informed guesses about their behavior; and narrative intelligence, which presents us with information, articulated stories, images, and animations, in the right place, at the right time, all tailored to our needs and preferences. All three forms of intelligence need to co-exist and co-operate for an Intelligent Space to be effective. Narrative intelligence is important so that the space provides us with relevant information that matches our interests and needs. We need systems able to select how much information to give, with what detail and composition, and F. Sparacino ( ) Sensing Places and MIT, Santa Monica, CA, USA e-mail: ﬂavia@sensingplaces.com B. Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts, 713 DOI 10.1007/978-0-387-89024-1 32, c Springer Science+Business Media, LLC 2009
714 F. Sparacino how to sequence and articulate various fragments. A lack of interpretive intelligence will produce applications that are unable to determine the appropriate time to get our attention on a speciﬁc matter or story, and which nag the user about whether he or she wants this or that. Context modeling and behavior modeling are ways of approaching interpretive intelligence, the ﬁrst one from the situation’s perspective and the second one from the user’s perspective. Perceptual Intelligence is about endowing spaces with eyes, ears, and sensors that allow them to capture how they are being used by people. This is, in its full complexity, still an open ﬁeld of research: scene interpretation and object recognition are today active and open territories for scientiﬁc investigation. Augmenting a space with all three forms of intelligence can be seen as endowing the space with a mind of its own, which transforms it from a simple container of people and things to an active participant and cooperating agent of our lives. Per- ceptual intelligence represents the bottom layer of this virtual brain, and processes sensorial inputs. Interpretive Intelligence is the middle layer whose role is to “make sense” of the input data: it identiﬁes situations and people’s behaviors. Narrative Intelligence is the upper layer, a bit like the brain cortex, and it regulates the output and communication between the intelligent space and us (Fig. 1). The above description of space intelligence has provided a high level framework for the author’s research. In the following sections of this paper I will ﬁrst illustrate Fig. 1 Layered intelligence model for interactive spaces
32 Designing for Architecture and Entertainment 715 in more detail the three forms of space intelligence. I will then describe incremental contributions to intelligence modeling for spaces I developed in the past years with a focus on applications for architecture and entertainment. The main contribution of this paper is to show that Bayesian networks are an ideal modeling tool for all three forms of space intelligence and to provide a unifying mathematical description for robust sensing, user and context modeling, and articulated information delivery (storytelling). Related Work The work here presented is highly interdisciplinary and it draws from various disciplines. Smart Spaces The author began research in Smart Spaces and Natural Interfaces in collaboration with Alex Pentland at MIT [29, 41, 44, 45]. A variety of research groups today have taken this ﬁeld of investigation further, with a main focus on assisted living and on creating spaces that are useful for ordinary everyday activities. Georgia Tech has developed the Aware Home, a place embedded with technology that enables older adults to age in place, and which helps people communicate with distant relatives and friends [22]. Microsoft’s Easy. Living is an intelligent environment that con- tains myriad devices that work together to provide users access to information and services [7]. These devices may be stationary, as with acoustic speakers or ceiling lights, or they may be mobile, as with laptop computers or mobile telephones. Mi- crosoft’s goal is to allow typical PC-focused activities to move off of a ﬁxed desktop and into the environment [24]. The Stanford Interactive Workspaces project is ex- ploring new possibilities for people to work together in technology-rich spaces with computing and interaction devices on many different scales [19]. Their work aims at augmenting a dedicated meeting space with large displays, wireless/multimodal I/O devices, and seamless integration of mobile and wireless appliances including handheld PCs. Rather than seeing the space as a pro-active facilitator Stanford’s room is a reconﬁgurable platform for running a variety of applications. MIT’s In- telligent Room is a highly interactive environment that uses embedded computation to observe and participate in normal, everyday events. Microphone and camera ar- rays enable it to listen to people and observe what they do. People can speak with, gesture to, and interact with it in several ways [6, 9]. A robust agent-based soft- ware infrastructure supports the operation of these tools. The room’s intelligence is modeled by a multi-agent society that builds a higher level and context based repre- sentation of the user’s activity. Each activity has an associated software agent, called behavior agent, which responds to the user’s actions and performs the corresponding