» 
 » 

Handbook of Multimedia for Digital Entertainment and Arts- P17

Chia sẻ: Cong Thanh | Ngày: | Loại File: PDF | Số trang:30

0
Thêm vào BST
Báo xấu
65
lượt xem 4
download

Handbook of Multimedia for Digital Entertainment and Arts- P17

Mô tả tài liệu
  Download Vui lòng tải xuống để xem tài liệu đầy đủ

Handbook of Multimedia for Digital Entertainment and Arts- P17: The advances in computer entertainment, multi-player and online games, technology-enabled art, culture and performance have created a new form of entertainment and art, which attracts and absorbs their participants. The fantastic success of this new field has influenced the development of the new digital entertainment industry and related products and services, which has impacted every aspect of our lives.

Chủ đề:

Bình luận(0) Đăng nhập để gửi bình luận!

Lưu

Nội dung Text: Handbook of Multimedia for Digital Entertainment and Arts- P17

  1. 21 Projector-Camera Systems in Entertainment and Art 479 Physically Viewing Interaction By projecting images directly onto everyday surfaces, a projector-camera system may be used for creating augmentation effects, such as virtually painting the ob- ject surface with a new color, new texture, or even an animation. Users can interact directly with such projector-based augmentations. For example, they may observe the object from different sides, while simultaneously experiencing consistent occlu- sion effects and depth, or they can move nearer or further from the object, to see local details and global views. Thus, the intuitiveness of physical interaction and advantages of digital presentation are combined. This kind of physically interactive visualization ability is suitable for use in situations when virtual content is mapped as a texture on real object surfaces. View-dependent visual effects such as highlighting to simulate virtually shiny sur- faces require tracking of the users’ view. Multi-user views can also be supported by time-multiplexing the projection for multiple users, with each user wearing a synchronized shutter glass allowing the selection of individual views. But this is only necessary for view-dependent augmentations. Furthermore, view tracking and stereoscopic presentation ability enables virtual objects to be displayed not only on the real surface, but also in front of or behind the surface. A general geometric framework to handle all these variants is described in [26]. The techniques described above, only simulate the desired appearance of an aug- mented object which is supposed to remain fixed in space. To make the projected content truly user-interactive, more information apart from viewpoint changes is required. After turning an ordinary surface into a display, it is further desirable to ex- tend it to become a user interface with an additional input channel. Thereby, cameras can be used for sensing. In contrast to other input technologies, such as embedded electronics for touch screens, tracked wand, or stylus and data gloves often used in virtual environments; vision-based sensing technology has the flexibility to sup- port different types of inputting techniques without modifying the display surface or equipping the users with different devices for different tasks. Differing from in- teraction with special projection screens such as electronically enabled multi-touch or rear-projected screens, some of the primary issues associated with vision-based interaction with front-projected interfaces are the illuminations on the detected hand and object, as well as cast of shadows. In following subsections, two types of typical interaction approaches with spatial projector-camera systems will be introduced, namely near distance interaction and far distance interaction. Vision based interaction techniques will be the main focus and basic interaction operations such as pointing, selecting and manipulation will be considered. Near Distance Interaction In near-distance situations where the projection surface is within arm’s length of the user, finger touching or hand gestures are intuitive ways to select and manipulate the
  2. 480 O. Bimber and X. Yang interface. Apart from this, the manipulation of physical objects can also be detected and used for triggering interaction events. Vision-based techniques may apply a visible light or infrared light camera to capture the projected surface area. To detect finger touching on a projected surface a calibration process, similar to the geometric techniques presented in section “Geo- metric Image Correction”, is needed to map corresponding pixels between projector and camera. Next, fingers, hands and objects need to be categorized as part of the foreground in order to separate them from the projected surface background. When interactions take place on a front-projected surface, the hand is illuminated by the displayed images and thus the appearance of a moving hand changes quickly. This renders segmentation methods, based on skin color or region-growing methods as useless. Frequently, conventional background subtraction methods are also unreliable, since the skin color of a hand may become buried in the projected light. One possible solution to this problem is to expand the capacity of the background subtraction. Despite, its application to an ideal projection screen which assumes enough color differences from skin color as in [27], the background subtraction can also be used to take into account different background and foreground re- flectance factors. When the background changes significantly, a segmentation may fail. An image update can be applied to keep the segmentation robust, where an artificial background may be generated from the known input image for a pro- jector with geometric and color distortions corrected between the projector and camera. Another feasible solution is to detect the changing pixel area between the frames of the captured video to obtain a basic shape of the moving hand or object. Noise can then be removed using image morphology. Following this, a fingertip can be detected by convolution with a fingertip-shaped template over the extracted image, as in [28]. To avoid the complex varying illumination problem for visible light, an infrared camera can be used instead, together with an infrared light source to produce in- visible shadow of a finger on the projected flat surface, as shown in [29]. The shadow of the finger can then be detected by the infrared camera and can thus be singularly used to detect the finger region and fingertip. To enable screen intera- tion by finger touching, the positioning of the finger, either touching the surface or hovering above it, can be further determined by detecting the occlusion ratio of the finger shadow. When the finger is touching the surface, its shadow is fully oc- cluded by the finger itself; while the finger is hovering over the surface, its shadow is larger. It is also possible to exclude the projected content from the captured video by interlacing the projecting images and captured camera frames using synchronized high-speed projectors and cameras, so that more general gesture recognition algo- rithms can be adopted as those reviewed in [30]. To obtain more robust detection results, specific vision hardware can also be utilized, such as real-time depth cam- eras that are based on the time-of-flight principle [31].
  3. 21 Projector-Camera Systems in Entertainment and Art 481 Far Distance Interaction In a situation where the projection surface is beyond the user’s arm length, laser pointer interaction is an intuitive way to select and manipulate projected interface components. Recently, laser pointer interaction has used for interacting with large scale projection display or tiled display at a far distance [32]. To detect and track a laser dot on a projection surface in projector-camera sys- tems, a calibrated camera covering the projecting area is often used. The location and movement of a laser dot can be detected simply by applying an intensity thresh- old to the captured image – assuming that the laser dot is much brighter than the projection. Since the camera and the projector are both geometrically calibrated, the location of the laser dot on the camera image can be mapped to corresponding pixels on projection image. The “on” and “off” status of the laser pointer can be mapped to mouse click events for selecting particular operations. One or more virtual objects that are supposed to be intersected with the laser dot or a corresponding laser ray can be further calculated from the virtual scene geometry. More events for laser pointer interaction can be triggered by temporal or spa- tial gestures, such as encircling, or simply by adding some hardware on laser pointers, such as buttons and embedded electronics for wireless communication. Multiple user laser pointer interaction can also be supported for large projection areas where each user’s laser pointer is distinguishable. This can be supported by time-multiplexing the laser or by using different laser colors or patterns. User stud- ies have been carried out to provide optimized design parameters for laser pointer interaction [33]. Although laser pointing is an intuitive technique, it also suffers from issues such as hand-jittering, inaccuracy and slow interaction speeds. To overcome the hand-jittering problem, which is compounded at greater distances, filtering-based smoothing techniques can be applied, though may lead to discrepancy between the pointing laser dot and the estimated location. Infrared laser pointers may solve this problem, but according to user study results, visible laser lights are still found to be better for interaction. Apart from laser pointing, other tools such as a tracked stylus or specially de- signed passive vision wands [34] tracked by a camera have proven to be flexible and efficient when interacting with large scale projection displays over distances. Gesture recognition provides a natural way for interaction in greater distances without using specific tools. It is mainly based on gesture pattern recognition with or without hand model reconstruction. Evaluating body motions is also an intuitive way for large scale interaction, where the body pose and motion are estimated and behavior patterns may be further detected. When gesture and body motion are the dominant modes of interaction with projector-camera systems, shadows and varying illumination conditions are the main challenges, though shadows can also be utilized for detecting gesture or body motion. In gesture or body interaction, background subtraction is often used for detect- ing the moving body from the difference between the current frame and a reference background image. The background reference image must be regularly updated so
  4. 482 O. Bimber and X. Yang as to adapt to the varying luminance conditions and geometry settings. More com- plex models have extended the concept of background subtraction beyond its literal meaning. A thorough review of the background extraction methods is presented in [35]. Vision-based human action recognition approaches can be generally divided into four phases. The model initialization phase ensures that a system commences its operation with a correct interpretation of the current scene. The tracking phase seg- ments and tracks the human bodies in each camera frame. The pose estimation phase estimates the pose of the users in one or more frames. The recognition phase can recognize the identity of individuals as well as the actions, activities and behaviors performed by one or more user. Details about video based human action detection techniques are reviewed in [36]. Interaction with Handheld Projectors Hand-held projectors may display images on surfaces anywhere at anytime while they are being moved by the user. This is especially useful for mobile projector- based augmentation, which superimposes digital information in physical environ- ments. Unlike other mobile displays such as provided by PDAs or mobile phones, hand-held projectors offer a consistent visual combination of real information gather from physical surfaces with virtual information. This is possible without context switching between information space and real space, thus seamlessly blurring the virtual and real world. They can be used, for instance, as interactive information flashlights [37] – displaying registered image content on surface portions that are illuminated by the projector. Although hand-held projectors provide great flexibility for ubiquitous computing and spontaneous interaction, there are fundamental issues to be addressed before a fluid interaction between the user and the projector is possible. When using a hand- held projector to display on various surfaces in a real environment, the projected image will be dynamically modulated and distorted by the surfaces as the user moves. When the user stops moving the projector, the presented image still suf- fers from shaking by the user’s unavoidable hand-jitter. Thus, a basic requirement for hand-held projector interaction is to produce stable projection. Image Stabilizing One often desired form of image stabilization is to produce a rectangular 2D image on a planar surface – independently of the projector’s actual pose and movement. In this case, the projected image must be continuously warped to keep the correct aspect ratio and to remain undistorted. The warping process is similar to the geo- metric correction techniques described earlier. The difference, however, is that the
  5. 21 Projector-Camera Systems in Entertainment and Art 483 target viewing perspective is usually pointing towards the projection surface along its normal direction, while the position of the hand-held projector may keep on changing. To find the geometric mapping between the projector and the target perspective, the projector’s six degrees of freedom may be obtained from an attached tracking device. The homography is an adequate method to represent this geometric mapping when the projection surface is planar. Instead of using the detected four vertices of the visible projection area to calculate the homography matrix, another practical technique is to identify laser spots displayed from laser-pointers that are attached to the projector-camera system. The laser spots are brighter and therefore easier to detect. In [38], hand-jittering was compensated together with the geometry correc- tion, by continuously tracking the projector’s pose and warping the image at each time-step. A camera attached to the projector detects visual markers on the projec- tion surface, that are used for warping the projected image accordingly. In [42] a similar stabilization approach is described. Here, the projector pose relative to the display surface is recovered up to an unknown translation in the display plane. Pointing Techniques After the stabilization of the projector images, several techniques can be adopted to interact with the displayed content. Controlling a cursor by laser pointing (e.g., with a projector-attached laser pointer) represents one possibility. In this case, common desktop mouse interaction techniques can be mapped directly to hand-held projec- tors. The projector’s center pixel ray can also be used instead of a laser pointer to control the mouse cursor. One of the biggest problems associated with these meth- ods are size reductions and cropping of the display area, caused by the movement of the projector when controlling the cursor. Using a secondary device such as a tracked stylus or a separate laser pointer can overcome these limitations, however the user needs both hands for interaction. Mounting a touch pad or other input devices on the projector is also possible, but might not be as intuitive as a direct pointing with the projector itself. Selection and Manipulation Based on the display and direct pointing ability described above, mouse like interac- tion can be emulated such as selecting a menu or performing a cut-and-paste oper- ation by pointing the cursors on the projected area and pressing buttons mounted on the projector. However, in this scenario, the hand jitter problem, similar to laser pointer interaction, also exists – making it difficult to locate the cursor in specific and small areas. The jitter problem is intensified when cursor pointing is combined with mouse button-pressing operations. Adopting specially designed interaction techniques rather than emulating common desktop GUI methods, can alleviate this problem.
  6. 484 O. Bimber and X. Yang One proven and efficient interaction technique for hand-held projectors is the crossing based widget technique [37]. Crossing based widget is operated by moving the cursor to cross the widget in a specific direction (e.g. from outside to inside, or from top to bottom), while holding the mouse button. This technique avoids point- ing the cursor and pressing a button at the same time. Crossing widget can be used for hand-held projectors to support commonly used desktop GUI elements, such as menus and sliders. Crossing based menu items can be activated by crossing from one direction; and deactivated by crossing from the opposite direction. All actions are executed by releasing the mouse button. Different colors can be used to indicate the crossing directions. Hierarchical menus can also be supported. Similarly, the crossing based slider is activated by crossing the interface in one direction, deacti- vated by crossing it in the opposite direction, and adjusted according to the cursor movement parallel to the slider. Another specially designed interaction technique is called zoom-and-pick wid- get, proposed by [39]. It was designed to implement the simultaneous use of stable high-resolution visualization and pixel-accurate pointing for hand-held projectors. The widget is basically a square magnification area, located around the current pointing cursor position. A circular dead zone is defined within this area. The center of the dead zone is treated as an interaction hot-spot. The widget remains static when the pointing cursor is moving within the dead zone. To gain pixel-accurate pointing ability, a rim is defined around the dead zone. Each crossing of the cursor from the dead zone into the rim triggers a single pixel movement of the widget in the direc- tion of the pointer movement. If the pointer is moving beyond the dead zone and the rim, the widget will be relocated to include the pointer in its dead zone again. Multi-user Interaction Hand-held projectors also pose new chances and challenges for multi-user interac- tion. In contrast to other multi-user devices such as tabletop displays, primarily used for sharing information with others, or other mobile devices such as personal mo- bile phones; hand-held projectors, due to their portability, and personal usage, are suitable both for shared and individual use. Multiple hand-held projectors combine the advantages of public and personal display systems. The main issues associated with multi-user interaction and hand-held projec- tors are primarily concerned with design for ownership, privacy control, sharing, and so on. The name of the owner of a displayed object can be represented by spe- cially designed label widgets placed on the object and operated using crossing based operations. The overlap of two or more cursors can signify consent from multiple users to accomplish collaborative interactive task, such as coping a file or blending two images between the users. Snapping and docking actions can be performed by multiple users in order to quickly view or modify connected information between multiple objects. Multiple displayed images from more than one user can be blended directly or semantically. By displaying high resolution images when the user moves
  7. 21 Projector-Camera Systems in Entertainment and Art 485 closer to the display surface, a focus-and-context experience can be achieved by providing refined local details. More details can be found in [40]. Environment Awareness Due to their portability, hand-held projectors are mainly used spontaneously. There- fore, it is desirable to enhance the hand-held projectors with environment awareness abilities. Geometric and photometric measurement and object recognition and track- ing capacities, would enable the projector to sense and respond to the environment accordingly. Geometric and photometric awareness can be implemented using, for example, structured light techniques, as described in section “Structured Light Scanning”. For object recognition and tracking, the use of a passive fiducial marker (e.g., supported with open source computer vision toolkits such as ARToolkit[41]) is a cheap solu- tion. However, it is not visually attractive which may disturb the appearance of the object and may fail as a result of occlusion or low illumination. Unpowered pas- sive RFID tags can be detected via a radio frequency reader without being visible. They represent another inexpensive solution for object identification. However, they do not support pose tracking. The combination of RFID tags with photo-sensors, called RFIG, has been developed in order to obtain both – object identification and object position. The detection of the object position is implemented by projecting Gray codes onto the photo-sensors. In this way the Gray code is sensed by each photo-sensor and allows computing the projection of the sensors to the projector image plane, and consequently enables projector registration. More details about RFIG are referred to [42]. Interaction Design and Paradigm In the sections above, techniques for human interaction with different configura- tions of projector-camera systems were presented. This subsection, however, will introduce higher level concepts and methods for interaction design and interaction paradigms for such devices. Alternative configurations such as steerable projector and moveable surfaces will also be discussed briefly. Projector-based systems for displaying virtual environments assume high qual- ity, large field of view, and continuous display areas which often evoke feelings of immersion and presence, and provide continuous interaction spaces. In contrast, spatial projector-camera systems that display on everyday surfaces may produce blended and warped images with average quality and a cropped field of view. The cropped view occurs as a result of the constricted display area, discontinuous im- ages on different depth levels, and surfaces with different modulation properties. Due to these discrepancies, it is not always possible to directly adopt interaction techniques from immersive virtual environments or from conventional augmented reality applications.
  8. 486 O. Bimber and X. Yang For example, moving a virtual object using the pointing-and-drag technique, which is often adopted in virtual environments, may not be the preferred method in a projector-based augmented environment, since the appearance of the virtual ob- ject may vary drastically as it is moved and displayed on discontinuous surfaces with different depths and material properties. Instead, grasp-and-drop techniques may be better suited to this situation, as discussed in [43]. Furthermore, the distance between the user and display surface is important for designing and selecting interaction techniques. It was expected that pointing interac- tion is more suitable for manipulating far distance objects, while touching is suitable for near distance objects. However, contradictory findings, derived from user studies for interaction with projector-camera systems aimed for implementing augmented workspace [43], have proven otherwise. Users were found unwilling to touch the physical surfaces even at close range distances after they learned distance gestures such as pointing. Instead, users frequently continued using the pointing method, even for surfaces located in close proximity to them. The reason for this behavior may be two-fold. Firstly, users may prefer to use a consistent technique for manipu- lation such as pointing. Secondly, it seems that the appearance and materials of the surfaces affect the user’s willingness to interact with them [44]. Several interaction paradigms have been introduced with or for projector-camera systems. Tangible user interfaces were developed to manipulate projected content using physical tangible objects [45]. Vision based implicit interaction techniques have also been applied to support subtle and persuasive display concepts derived from ubiquitous computing [46]. The peephole paradigm is discussed as a concept to describe the projected display as a peephole for the physical environment [47]. Varying bubble-like free-form shapes of the projected area based on the environment enables a new interface that moves beyond regular fixed display boundaries [48]. Besides hand-held projectors which enable ubiquitous display, steerable projec- tors also bring new interaction concepts, such as everywhere displays. Such systems enable projections on different surfaces in a room, and to turn them into an interac- tion interfaces. The best way to control a steerable projector during the interaction, however still needs to be determined. Body tracking can be combined with steer- able projections to produce a paradigm called user-following display [49], where the user’s position and pose are tracked. Projection surfaces are then dynamically selected and modulated accordingly, based on a measured and maintained three- dimensional model of the surfaces in the room. Alternatively, laser pointers can be used and tracked by a pan/tilt/zoom camera to control and interact with a steer- able projector unit [50]. Another issue for interaction with steerable projectors is the question of how to support a dynamic interfaces which can change form and location on the fly. A vision-based approach can solve this problem by decoupling interface specifications from its location in space and in the camera image [51]. Besides the projectors themselves, projection surfaces might also be moveable rather than remain static in the environment. They may be rigidly moveable flat screens, semi-rigidly foldable objects such as a fan or an umbrella, or deformable objects such as paper and cloth. Moveable projection surfaces can provide novel interfaces and enable unique interaction paradigms such as foldable displays or
  9. 21 Projector-Camera Systems in Entertainment and Art 487 organic user interfaces [52]. Tracking the pose or deformation of such surfaces, how- ever, is an issue that still needs to be addressed. Cheap hardware trackers have been used recently to support semi-rigid surfaces [53]. Vision-based deformation detec- tion algorithms may be useful in future for supporting deformable display surfaces. Application Examples The basic visualization and interaction techniques that have been presented in the sections above enable a variety of new applications in different domains. In general, projector-camera systems can be applied to interactive or non-interactive visual presentations in situations where the application of projection screens is not possible, or not desired. Several examples are outlined below. Embedded Multimedia Presentations Many historic sites, such as castles, caves, or churches, are open to public. Flat panel displays or projection screens are frequently being used for presenting vi- sual information. These screens, however, are permanently installed features and unnecessarily cover a certain amount of space. They cannot be temporally disas- sembled to give the visitors an authentic impression of the environment’s ambience when required. Being able to project undistorted images onto arbitrary existing surfaces offers a potential solution to this problem. Projectors can display images that are much larger than the device itself. The images can be seamlessly embedded, and turned off any time to provide an unconstrained experience. For these reasons, projector- camera systems and image correction techniques are applied in several professional domains, such as historic sites, theater, festivals, museums, public screen presenta- tions, advertisement displays, theme parks, and many others. Figure 2 illustrates two examples for a theater stage projection at the Karl-May Festival in Elspe (Germany), and an immersive panoramic projection onto the walls of the main tower of castle Osterburg in Weida (Germany). Both are used for displaying multimedia content which is alternately turned on and off during the main stage performance and the museum presentation respectively. Other examples of professional applications can be found at www.vioso.com. Superimposing Museum Artifacts Projector-camera systems can also be used for superimposing museum artifacts with pictorial content. This helps to communicate information about the displayed ob- jects more efficiently than secondary screens.
  10. 488 O. Bimber and X. Yang Fig. 2 Projection onto physical stage setting (top), and 360 degree surround projection onto natu- ral stone walls in castle tower (bottom). Image courtesy: VIOSO GmbH, www.vioso.com In this case, a precise registration of the projector-camera system is not only nec- essary to ensure an adequate image correction (e.g., geometrically, photometrically, and focus), but also for displaying visual content that is geometrically registered to the corresponding parts of the object. Figure 3 illustrates two examples for superimposing visual content, such as color, text and image labels, interactive visualizations of magnifications and un- derdrawings, and visual highlights on replicas of a fossil (primal horse displayed by Senckenberg Museum Frankfurt, Germany) and paintings (Michaelangelo’s Creation of Adam, sanguine and Pontormo’s Joseph and Jacob in Egypt, oil on wood) [22]. In addition to augmenting an arbitrary image content, it is also possible to boost the contrast of low contrast objects, such as paintings whose colors have faded after a long exposure to sun light. The principle techniques describing how this can be achieved are explained in [19]. Spatial Augmented Reality Projector-camera systems cannot only acquire parameters that are necessary for im- age correction, but also higher level information, such as the surrounding scene geometry. This, for instance, enables corrected projections of stereoscopic images
  11. 21 Projector-Camera Systems in Entertainment and Art 489 Fig. 3 Fossil replica superimposed with projected color (top), and painting replicas augmented with interactive pictorial content (bottom) [22] onto real-world surfaces which allows the augmentation of three-dimensional in- teractive content. Active stereoscopic shutter glasses and head-tracking technology supports correct depth viewing of virtual content in precise alignment with the phys- ical environment. This is a projector-based variation of what is referred to as spatial augmented reality [23]. In contrast to mobile augmented realities, the display tech- nology for spatial augmented reality applications is not hand-held or head-worn, but fixed in the environment. This has several technological advantages, but also limits the applications to non-mobile ones. Figure 4 illustrates two projector-based spatial augmented reality examples: An architectural global lighting simulation is projected directly within the real environ- ment enabling a more realistic and immersive visualization than possible with only a monitor. Stereoscopically projected game content can interact with real objects. A physical simulation of the virtual car allows realistic collisions with real items. This is possible through the scanned scene geometry, which also enables correct occlu- sion effects. Object recognition techniques that are applied to the acquired scene geometry and to the captured camera image enable the derivation of contextual information that is used in the game logic. Motorized pan-tilt projector-camera units allow using large parts of an entire room as playground for such spatial augmented reality games. More information on spatial augmented reality can be found in [23]. A free e-book is available at www.SpatialAR.com.
  12. 490 O. Bimber and X. Yang Fig. 4 Examples for spatial augmented reality applications with projector camera systems. An immersive in-place visualization of an architectural lighting simulation (left), and a stereoscopi- cally projected spatial augmented reality game (right). Door, window, illumination and the car are projected Flexible Digital Video Composition Blue screens and chroma keying technology are essential for digital video compo- sition. Professional studios apply tracking technology to record the camera path for perspective augmentations of original video footage. Although this technology is well established, it does not offer a great deal of flexibility. For shootings at non-studio sets, physical blue screens can be installed and takes might have to be recorded twice (with and without blue screens), or parts have to be re-recorded in a studio. In addition, virtual studio technology itself still faces limitations. Chroma-keying and studio illumination, for instance, are difficult to harmonize. Moderators or actors have to spend a fair amount of practice time before interacting with invisible virtual components naturally. Spill on the foreground and disadvantageous foreground col- ors lead to low-quality or even unusable keying results. Temporally synchronized projector-camera systems can be used to project cor- rected keying patterns and other spatial codes onto arbitrary diffuse (real-world) surfaces. Therefore the reflectance of the underlying real surface is widely neutral- ized by applying the image correction techniques that have been explained above. The main difference to the application examples that have been described so far, is that projector-camera systems are used for recording visual effects, and not for presenting corrected visual content directly to human observers. A temporal multiplexing between projection (p-frames) and flash illumination (i-frames) allows capturing the fully lit scene, while still being able to key the fore- ground objects. This is illustrated in figure 5. Since the entire scene is recorded when physical blue screens do not block the view, the footage of the full background scene can be used for video composition. Thus, recordings need not be taken twice, and keying is invariant to foreground colors. In addition, other spatial codes can be embedded into the projected im- ages to enable tracking of the camera, environment matting, and displaying in-place
  13. 21 Projector-Camera Systems in Entertainment and Art 491 Fig. 5 VirtualStudio2Go: Odd (i-) frames record the fully illuminated scene. Even (p-) frames record the non-illuminated scene with projected images that neutralize the appearance of a real background surface and display code patterns. Repeating this at HD scanning speed (59.94Hz) and registering both sub-frames during post-processing supports high quality digital video composition effects for real (non-studio) environments moderator information. Furthermore, the reconstruction of the scene geometry is implicitly supported, and allows special composition effects, such as shadow casts, occlusions and reflections. A concept that combines all of these techniques into one single compact and portable system that is fully compatible with common digital video composition pipelines, and offers an immediate plug-and-play applicability is presented in [24]. It enables professional digital video composition effects in real indoor environments. Interactive Attraction Installations Today, the most popular applications of projector-camera systems are perhaps in- teractive attractions as public installations. By projecting interactive graphics onto everyday surfaces in public places, such as walls in museums, floors in shop- ping mall, subway tunnels, and even dining tables in restaurants, projector-camera systems emerge as an effective attraction tool by creating vivid interactive art, enter- tainment, and advertisement experience for people. Vision based sensing technology can be mainly adopted in such interactive art systems to detect people’s presence and activity in an unobtrusive way, and implicitly engaging people with the artificially augmented environment through large scale human body motions, hand gesture, or finger touching interactions with such installations. Figure 6 illustrates two projector-based interactive attraction installations. In the left example, a realistically rendered water pool is projected directly onto the ground of the Lou Dong Chinese Painting Museum at Tai Cang (China) with a physically built pool boundary, where the rendered water, lotus, and fishes in the pool are all responsive to the visitors who step into it. Ripple effects in the water, blooming lutoses, and escaping fishes, have been rendered in this way [54]. A Chinese painting is mapped as texture on the ground of the pool to compliment the artwork. Since this system has been installed in the museum, its realistic appearance and vivid
  14. 492 O. Bimber and X. Yang Fig. 6 Examples for interactive attraction installations. An interactive water pool installed in a traditional art museum (left) and an interactive augmented physical map installation for tourist attraction (right) interactivity have attracted many visitors, especially young children who have little contact with traditional culture. In another installation which was exhibited for an art-science festival in Shanghai’s Oriental Pearl Tower (China), a shining icon is projected onto a tra- ditional physical tourist map. The tourists can select different sites by hands or props on the map to see related video information. A tour guide can then create a touring path on the map with a laser pointer, or a visitor could produce a path by walking on a projected map on the ground. A three dimensional walk-through of the tour scene can then be triggered along the created path [55]. By integrating the traditional tangible map with the augmented digital information, and enabling vision-based and tangible interaction techniques, the projector-camera system can provide tourists and tour guides with a fresh sightseeing experience. The Future of Projector-Camera Systems Projector-camera systems have already found practical applications in theater, mu- seums, historic sites, open-air festivals, trade shows, advertisement, visual effects, theme parks, and art installations. With advances in technology and techniques, they will be applied in many more areas in future. Future projectors will become more compact in size and will require little power and cooling. Reflective technology (such as DLP or LCOS) will increasingly replace transmissive technology (e.g., LCD). This will leads to an increased brightness and extremely high update rates. GPUs for real-time graphics and vision processing will also be integrated. While resolution, contrast and speed will keep increas- ing, production costs and market prizes will continue to fall. Conventional UHP lamps will be replaced by powerful LEDs or multi-channel lasers. This will make them suitable for mobile applications. Projector-camera technology is currently be- ing integrated into mobile devices, such as cellphones, and supports truly flexible
  15. 21 Projector-Camera Systems in Entertainment and Art 493 presentations methods. Image correction techniques, such as the ones explained above are essential for these devices, since projection screens will most likely not become mobile. But projector-camera systems will not only be used as display devices. In future, they will also enable intelligent, spatially and temporally controllable light sources. Projector-based illumination techniques will not only solve problems in professional domains, such as microscopy or endoscopy, but -one day- might also be applied in more general contexts. Imagine that networked projector-camera systems become as cheap and as com- pact as light bulbs. They could not only be turned on and off, but would allow to offer synthetic room illumination and interactive display capabilities everywhere. For instance, they could produce individual mood profile and ambient light situa- tions, as well as to enable internet access wherever you stand. References 1. O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhoefer, “The Visual Computing of Projector- Camera Systems,” Computer Graphics Forum, Vol. 27, No. 8, pp. 2219–2245, 2008. 2. G. Wetzstein and O. Bimber, “Radiometric Compensation through Inverse Light Transport,” Proceedings of Pacific Graphics, pp. 391–399, 2007. 3. J. Salvi, J. Pag´ s, and J. Batlle, “Pattern Codification Strategies in Structured Light Systems,” e Pattern Recognition, Vol. 37, No.4, pp.827–849, 2004. 4. R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs, “The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays,” Proceedings of ACM SIGGRAPH, pp. 179–188, 1998. 5. O. Bimber, A. Emmering, and T. Klemmer, “Embedded Entertainment with Smart Projectors,” IEEE Computer, Vol.38, No.1, pp.56–63, 2005. 6. S. K. Nayar, H. Peri, M. D. Grossberg, and P. N. Belhumeur, “A Projection System with Radio- metric Compensation for Screen Imperfections,” Proceedings of IEEE International Workshop on Projector-Camera Systems (ProCams), 2003. 7. T. Yoshida, C. Horii, and K. Sato, “A Virtual Color Reconstruction System for Real Heritage with Light Projection,” Proceedings of International Conference on Virtual Systems and Mul- timedia (VSMM), pp. 161–168, 2003. 8. M. Ashdown, T. Okabet, I. Sato, and Y. Sato, “Robust Content-Dependent Photometric Pro- jector Compensation,” Proceedings of IEEE International Workshop on Projector-Camera Systems (ProCams), 2006. 9. A. Grundhoefer and O. Bimber, “Real-Time Adaptive Radiometric Compensation,” IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 14, No. 1, pp. 97–108, 2008. 10. H. Park, M.-H. Lee, S.-J. Kim, and J.-I. Park, “Specularity-Free Projection on Nonplanar Surface,” Proceedings of Pacific-Rim Conference on Multimedia (PCM) (2005), pp. 606–616. 11. R. Sukthankar, C. Tat-Jen, and G. Sukthankar, “Dynamic Shadow Elimination for Multi- Projector Displays,” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. II, pp. 151–157, 2001. 12. C. Jaynes, S. Webb, and R. M. Steele, “Camera-Based Detection and Removal of Shadows from Interactive Multiprojector Displays,” IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 10, No. 3, pp. 290–301, 2004.
  16. 494 O. Bimber and X. Yang 13. M. S. Brown, P. Song, and T. - J. Cham, “Image Pre-Conditioning for Out-of-Focus Projector Blur,” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. II, pp. 1956–1963, 2006. 14. Y. Oyamada and H. Saito, “Focal Pre-Correction of Projected Image for Deblurring Screen Im- age,” Proceedings of IEEE International Workshop on Projector-Camera Systems (ProCams), 2007. 15. L. Zhang and S. K. Nayar, “Projection Defocus Analysis for Scene Capture and Image Dis- play,” ACM Trans. Graph. (Siggraph), Vol. 25, No. 3, pp.907–915, 2006. 16. M. Grosse and O. Bimber, “Coded Aperture Projection,” Proceedings of Emerging Display Technologies and Immersive Projection Technologies 2008 (EDT IPT08), 2008. 17. O. Bimber and A. Emmerling, “Multifocal Projection: A Multiprojector Technique for In- creasing Focal Depth,” IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 12, No. 4, pp. 658–667, 2006. 18. K. Fuji, M. Grossberg, and S. Nayar, “A Projector-Camera System with Real-Time Photomet- ric Adaptation for Dynamic Environments,” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. I, pp. 814–821, 2005. 19. O. Bimber and D. Iwai, “Superimposing Dynamic Range,” ACM Transactions on Graphics (ACM Siggraph Asia), Vol. 27, No. 5, article 150, 2008. 20. D. Cotting, M. Naef, M. H. Gross, and H. Fuchs, “Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display,” Proceedings of IEEE/ACM In- ternational Symposium on Mixed and Augmented Reality (ISMAR), pp. 100–109, 2004. 21. S. Zollmann and O. Bimber, “Imperceptible Calibration for Radiometric Compensation,” Pro- ceedings of Eurographics, pp. 61–64, 2007. 22. O. Bimber, F. Coriand, A. Kleppe, E. Bruns, S. Zollmann, and T. Langlotz, “Superimposing Pictorial Artwork with Projected Imagery,” IEEE MultiMedia, Vol. 12, No.1, pp. 16–26, 2005. 23. O. Bimber and R. Raskar, “Spatial Augmented Reality: Merging Real and Virtual Worlds,” A K Peters LTD (publisher), ISBN: 1-56881-230-2, free e-book: www.SpatialAR.com, July 2005. 24. A. Grundhoefer and O. Bimber, “VirtualStudio2Go: Digital Videocomposition for Real En- vironments,” ACM Transactions on Graphics (ACM Siggraph Asia), Vol. 27, No. 5, article 151, 2008. 25. D. A. Bowman, E. Kruijff, J. J. Laviola, and I. Poupyrev, “3D User Interfaces: Theory and Practice,” Addison-Wesley, 2005. 26. R. Raskar and K. L. Low, “Interacting with spatially augmented reality,” Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation(AFRIGRAPH ’01), pp. 101–108, 2001. 27. A. Licsar and T. Sziranyi, “Hand gesture recognition in camera-projector system,” Proceedings of the ECCV 2004 Workshop on HCI - Computer Vision in Human-Computer Interaction, pp. 83–93, 2004. 28. R. Kjeldsen, C. Pinhanez, G. Pingali, J. Hartman, T. Levas, and M. Podlaseck, “Interacting with Steerable Projected Displays,” Proceedings of the 5th International Conference on Auto- matic Face and Gesture Recognition, pp. 12–17, 2002. 29. A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system,” Pro- ceedings of the 18th annual ACM symposium on User interface software and technology (UIST ’05), pp. 83–92, 2005. 30. Y. Wu, T. S. Huang, and N. Mathews, “Vision-based gesture recognition: A review,” Proceed- ings of the International Workshop on Gesture-Based Communication in Human-Computer Interaction, pp. 103–115, 1999. 31. G. Yahav, G. Iddan, and D. Mandelboum, “3d imaging camera for gaming application,” In- ternational Conference on Consumer Electronics (ICCE 2007) - Digest of Technical Papers, pp. 1–2, 2007. 32. D. R. Olsen and T. Nielsen, “Laser pointer interaction,” Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’01), pp. 17–22, 2001.
  17. 21 Projector-Camera Systems in Entertainment and Art 495 33. B. Myers, R. Bhatnagar, J. Nichols, C. Peck, D. Kong, R. Miller, and A. Long, “Interacting at a distance: measuring the performance of laser pointers and other devices,” Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’02), pp. 33–40, 2002. 34. X. Cao and R. Balakrishnan, “Visionwand: interaction techniques for large displays using a passive wand tracked in 3d,” Proceedings of the 16th annual ACM symposium on User interface software and technology (UIST ’03), pp. 173–182, 2003. 35. M. Piccardi, “Background subtraction techniques: a review,” Proceedings of the IEEE Inter- national Conference on Systems, Man and Cybernetics, Vol. 4, pp. 3099–3104, 2004. 36. T. Moeslund, A. Hilton, and V. Kr¨ ger, “A survey of advances in vision-based human mo- u tion capture and analysis,” Computer Vision and Image Understanding, Vol. 104, No. 2–3, pp. 90–126, 2006. 37. X. Cao and R. Balakrishnan, “Interacting with dynamically defined information spaces using a handheld projector and a pen,” Proceedings of the 19th annual ACM symposium on User interface software and technology (UIST ’06), pp. 225–234, 2006. 38. P. Beardsley, J. Van Baar, R. Raskar, and C. Forlines, “Interaction using a handheld projector,” IEEE Computer Graphics and Applications, Vol. 25, No. 1, pp. 39–43, 2005. 39. C. Forlines, R. Balakrishnan, P. Beardsley, J. van Baar, and R. Raskar, “Zoom-and-pick: facilitating visual zooming and precision pointing with interactive handheld projectors,” Pro- ceedings of the 18th annual ACM symposium on User interface software and technology (UIST ’05), pp. 73–82, 2005. 40. X. Cao, C. Forlines, and R. Balakrishnan, “Multi-user interaction using handheld projectors,” Proceedings of the 20th annual ACM symposium on User interface software and technology (UIST ’07), pp. 43–52, 2007. 41. H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration for a Video-based Aug- mented Reality Conferencing System,” Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, Vol. 99, pp. 85–94, 1999. 42. R. Raskar, P. Beardsley, J. van Baar, Y. Wang, P. Dietz, J. Lee, D. Leigh, and T. Willwacher, “RFIG Lamps: interacting with a self-describing world via photosensing wireless tags and projectors,” Proceedings of SIGGRAPH 2004, pp. 406–415, 2004. 43. S. Voida, M. Podlaseck, R. Kjeldsen, and C. Pinhanez, “A study on the manipulation of 2D ob- jects in a projector/camera-based augmented reality environment,” Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 611–620, 2005. 44. M. Podlaseck, C. Pinhanez, N. Alvarado, M. Chan, and E. Dejesus, “On interfaces projected onto real-world objects,” Proceedings of the SIGCHI Conference on Human Factors in Com- puting Systems, pp. 802–803, 2003. 45. J. Underkoffler, B. Ullmer, and H. Ishii, “Emancipated pixels: real-world graphics in the lumi- nous room,” Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’99), pp. 385–392, 1999. 46. P. Dietz, R. Raskar, S. Booth, J. van Baar, K. Wittenburg, and B. Knep, “Multi-projectors and implicit interaction in persuasive public displays,” Proceedings of the working conference on Advanced visual interfaces, pp. 209–217, 2004. 47. A. Butz and A. Kr¨ ger, “Applying the Peephole Metaphor in a Mixed-Reality Room,” IEEE u Computer Graphics and Applications, Vol. 26, No.1, pp. 56–63, 2006. 48. D. Cotting and M. Gross, “Interactive environment-aware display bubbles,” Proceedings of the 19th annual ACM symposium on User interface software and technology (UIST ’06), pp. 245–254, 2006. 49. G. Pingali, C. Pinhanez, T. Levas, R. Kjeldsen, and M. Podlaseck, “User-Following Displays,” Proceedings of IEEE International Conference on Multimedia and Expo, 2002. 50. D. Kurz, F. Hantsch, M. Grobe, A. Schiewe, and O. Bimber, “Laser pointer tracking in projector-augmented architectural environments,” Proceeding of the 6th IEEE and ACM In- ternational Symposium on Mixed and Augmented Reality (ISMAR), pp.1–8, 2007. 51. R. Kjeldsen, A. Levas, and C. Pinhanez, “Dynamically reconfigurable vision-based user inter- faces,” Machine Vision and Applications, Vol. 16, No. 1, pp. 6–12, 2004.
  18. 496 O. Bimber and X. Yang 52. R. Vertegaal and I. P. Poupyrev, “Introduction to Special Issue of Organic user interfaces,” Communications of ACM, Vol. 51, No. 6, 2008. 53. J. C. Lee, S. E. Hudson, and E. Tse, “Foldable interactive displays,” Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST ’08), pp. 287–290, 2008. 54. M. Jin, H. Zhang, X. Yang, and S. Xiao, “A Real-Time ProCam System for Interaction with Chinese Ink-and-Wash Cartoons,” Proceedings of IEEE International Workshop on Projector- Camera Systems (ProCams), 2007. 55. Y. Yuan, X. Yang, and S. Xiao, “A Framework for Tangible User Interfaces within Projector- based Mixed Reality,” Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), 2007.
  19. Chapter 22 Believable Characters Magy Seif El-Nasr, Leslie Bishko, Veronica Zammitto, Michael Nixon, Athanasios V. Vasiliakos, and Huaxin Wei Introduction The interactive entertainment industry is one of the fastest growing industries in the world. In 1996, the U.S. entertainment software industry reported $2.6 billion in sales revenue, this figure has more than tripled in 2007 yielding $9.5 billion in revenues [1]. In addition, gamers, the target market for interactive entertainment products, are now reaching beyond the traditional 8–34 year old male to include women, Hispanics, and African Americans [2]. This trend has been observed in several markets, including Japan, China, Korea, and India, who has just published their first international AAA title (defined as high quality games with high budget), a 3D third person action game: Ghajini – The Game [3]. The topic of believable characters is becoming a central issue when designing and developing games for today’s game industry. While narrative and character were considered secondary to game mechanics, games are currently evolving to integrate characters, narrative, and drama as part of their design. One can see this pattern through the emergence of games like Assassin’s Creed (published by Ubisoft 2008), Hotel Dusk (published by Nintendo 2007), and Prince of Persia series (published by Ubisoft), which emphasized character and narrative as part of their design. M.S. El-Nasr ( ) School of Interactive Arts and Technology, Simon Fraser University, Vancouver, BC, Canada e-mail: magy@sfu.ca L. Bishko Department of Animation, Emily Carr University of Art and Design, Vancouver, BC, Canada e-mail: lbishko@ecuad.ca V. Zammitto, M. Nixon, and H. Wei School of Interactive Arts and Technology, Simon Fraser University, Vancouver, BC, Canada e-mail: vzammitt@sfu.ca; mna32@sfu.ca; huaxinw@sfu.ca A.V. Vasiliakos University of Peloponnese, Nauplion, Greece e-mail: vasilako@ath.forthnet.gr B. Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts, 497 DOI 10.1007/978-0-387-89024-1 22, c Springer Science+Business Media, LLC 2009
  20. 498 M.S. El-Nasr et al. Beyond the entertainment industry, the use of virtual environments for learning, health therapy, cultural awareness, and training is increasingly becoming a reality. In the recent years, there has been an increase in the number of research initiatives that use simulations and interactive 3D environments for a wide variety of appli- cations [4–11]. Several great examples are displayed in the projects developed by Institute of Creative Technologies at University of Southern California, where they utilize 3D environments with rich characters to teach cultural norms and foreign language, among other subjects. These applications provide a safe and comfortable environment for participants to interact within and learn at their own pace. In or- der to achieve their goals, however, such applications require realistic simulation of culture, people, and space. Thus, again the topic of believable characters is gaining more attention as a central topic that deserves further attention. Since the above mentioned applications are typically interactive, animated be- lievable characters are often required to adapt based on the interaction. Current industry methods, however, rely on heavy scripting, where voice acting, dialogue scripts, hand-coded animation routines, and hard-coded behaviors are used to por- tray the desired character; To mention a few examples of games that employ very detailed motion-captured characters, readers are referred to Assassins’ Creed and Prince of Persia (developed by Ubisoft) and Facade (developed by Mateas and ¸ Stern [12]), see Figure 1. In these games, artists work very diligently to detail characters’ mannerisms and body motion to exhibit the right character character- istics [3]. Such attention to detail of the non-verbal behaviors is a crucial element for character believability [4]. As one can guess, this kind of scripting is labor intensive and rigid, as it does not adapt to all variations induced by interaction. An alternative is to use artificial intel- ligent algorithms and graphics techniques to adapt character behaviors to variations in context induced by interaction. This alternative, however, is not as simple as it sounds, as it has been under research for many years and is still an open problem. Researchers have been working on several fronts to create believable expressive characters that can dynamically adapt within interactive narratives. Graphics re- searchers, for example, explored the integration of emotions and personality as parameters to modify virtual character animations [5–7]. Researchers working on developing conversational agents focus on building articulate virtual characters that can automatically synchronize gesture and speech [8]. Artificial intelligence Fig. 1 Screenshots from games and interactive media featuring characters
THÔNG TIN
TRỢ GIÚP
HỖ TRỢ KHÁCH HÀNG
Theo dõi chúng tôi

Giấy phép ICP số: 670/GP-BTTTT cấp ngày 30/11/2015 Copyright © 2009-2015 TaiLieu.VN. All rights reserved. TaiLieu.VN hiển thị tốt nhất với trình duyệt Chrome, Firefox, Internet Explorer 8.

Đồng bộ tài khoản