This dissertation introduces the concept of malleable media and its implications in the design of new types of shape-changing materials to connect the physical reality and virtual realities. This framework was defined through the design and evaluations of two systems, Vital+Morph and Volflex++. Such projects have been developed using an interdisciplinary approach that crosses engineering, HCI, haptic perception, design, and art. In the thesis, we described the design process, the technical implementations, and their evaluations. We provided a psychophysical experiment and a user study in the context of a media art installation. Also, we proposed a discussion on the characteristics that need to be considered in the design of malleable media and future applications in media arts.
We place our main contribution in emerging tendency in media arts of researching and designing new hybrid materials that can represent digital contents physically. However, by physical qualities, we mean as physically perceived through touch. This marks a difference since most of the previous works concentrate on visual perception. Moreover, we provided several specific applications, such as data physicalization and synchronization between physical and virtual objects.
We also introduce the conceptual framework of malleable media through an explorative study of the design, technological, and perceptual characteristics of this emerging media enabled by the rise of shape-changing materials.
We frame our work in a phenomenon we define as the wrap of the virtual and the physical continuum. Most of the research on Virtual Reality focused on making virtual experiences more physical and real for examples with the use of haptic interfaces. However, with the emergence of shape-changing materials, Internet of Things (IoT) technologies, and robots we witness a situation where physical and real experiences feel and appear to be like the virtual ones.
Malleability is a distinctive characteristic of the digital world. Users can freely navigate, and rearrange symbols on a desktop, or create and manipulate virtual objects. However, such dynamic qualities are not reflected in the objects and interfaces we use to interact with digital processes, which are usually static and rigid.
The Cambridge Dictionary defines malleable as an adjective that describes a substance that can be easily changed into a new shape1. Malleable also refer to objects or materials that can be altered or controlled by outside forces or influences. Malleable media can also be seen as an extension of the idea of ’tangible media’. However, in the definition of Ishii et al. ‘tangible’ just implies the presence of digital data in the physical world to help people relate to their existence. With ’malleable’ we refer to specific qualities of materials and their possibilities of being shaped and manipulated. Malleable media present contents through their materiality. They can be shaped and manipulated either by humans through their hands, and by digital information, which can determine their physical appearance. In malleable media, the haptic qualities are more prominent and important than the visual characteristics or just a physical presence. This thesis aims to provide answers to three main questions.
• Background: We define the conceptual and technological background to identify the landscape where this study is placed. Through this chapter we outline the concept of embodied cognition, the idea of media as developed by Marshall McLuhan, define the characteristics of Virtual Reality and the world as represented by data. We then present how media artists have dealt with virtual reality to create new experiences. Finally, we unfold the main technologies that enable the emergence of malleable media, such as shape-changing interfaces and Internet of Things devices.
• Related Work: We review the main literature in areas that pertain our study. This includes application areas like data physicalization, haptic interfaces, and the different technologies used for achieving different types of shape-change.
• Methods: We present the methodologies used to develop our study. We first describe what bio-inspired design is and which are its implications. Second, we present a specific technology used in one of our studies. Third, we outline the main findings and techniques used for study the haptic perception of materials and objects. Lastly, we define a mix-method of research based on the observations and interview of users in the context of a media art installation.
• Case Study I: We present Vital+Morph, a prototype for a shape-change material. Through this system, we propose an application scenario in the context of remote monitoring and propose a user study to test and understand such scenario in action through an extended study in media arts festivals.
• Case Study II: We present Volflex++, a novel system that can support the creation of physical-virtual materials. Through a psychophysical experiment with twenty subjects, we test the haptic resolution of such a system. We also introduce a proof-of-concept application to show how virtual objects can be presented through this system.
• Discussion: In this last chapter, we outline the common characteristics of Vital+Morph and Volflex++ by the definition of a series of design guidelines. We then discuss how malleable media can be used to enable new types of media artworks.
• How to realize Malleable Media? The creation of shape-changing materials represents an open question in research. Very few technologies have been explored, and all of them depends on the specific type of shape-change a designer wants to enable. In this thesis, we proposed two main ways to realize shape-change. Through Vital+Morph we explored two mechanisms for realizing the change of geometry. As for the Vitals, we proposed a mechanism that can deform a paper cylinder. In contrast with previous works, we proposed a system that is both light, self-powered, and enabled for IoT. In addition, the material is soft and can be held in users hands, while previous works focused on rigid materials. As for the Morph, we proposed a mechanism that can enable not only a change of geometry of a circular shape but this change can be transmitted to a paired device through the internet. The transmission of shape-change is still an underexplored area in research. Through Volflex++ we proposed the Volflex technology as a possible candidate for implementing complex and versatile shape-changing materials. In this case, we concentrate on the change of size and rigidity. While previous works focused on the change of size only on a vertical dimension, we propose the change of size for volumetric surfaces. Regarding the change of rigidity, we propose a mechanism that can present surfaces that can range from soft to stiff. Previous research looked at mechanisms that can produce a surface to became or soft or stiff but cannot offer a transition between these two extremes, and therefore they miss all the rigidities that are in the middle of soft and stiff. Previous systems allow change of size and rigidity as separate. Through Volflex++ it is possible to achieve such changes on a single surface and simultaneously. Most importantly, we propose mechanisms for shape-change oriented to haptic perception. As we showed in the previous chapter, the change of shape is often used to address the visual sense, and many of the systems presented cannot be touched by users. Both Vital+Morph and Volflex++ can be grabbed and directly manipulated by users’ hands. They also produce noticeable haptic feedback and can support both active and passive touch. We believe that the tactile sense should be the first channel of communication of shape-changing materials.
• How Malleable Media are perceived? As noted by many researchers, the perception of shape-change is still not very well understood. In this thesis, we focused on two aspects of perception that have not been covered by previous work. The first is the haptic perception of shape-change. Through Vital+Morph we showed how the proposed mechanism can not only represent digital data through the change of geometry which can also be perceived visually- but that the movements caused by shape-change can produce force-feedback. This adds a new dimension to data physicalization, which was not explored before. Through Volflex++ we performed a user study using psychophysical methods. We wanted to elucidate the haptic perception of the change of size and rigidity. Such kind of studies has not been performed yet for the evaluation of shape-changing materials. Since Volflex++ can represent a large variety of sizes and rigidities is fundamental to understand which is the perceived resolution of the user. Moreover, we found that the cues offered by this system are consistent with the perception of real materials with similar sizes and rigidities. Through this study, we aimed to bridge the methodologies and work done in haptics research for the study of shape-change. The second aspect we wanted to elucidate is the broader perception of malleable media, and try to anticipate which this type of impact these media will have on society. Being a new type of media, it is not really clear how shape-change will be perceived in everyday life, and how possible users interpret this type of communication. In contrast to previous work, we propose to evaluate new technologies through a media art installation. This type of evaluation gave us the possibility to collect a wide range of opinions, that confirmed our design but also warned us for possible misunderstandings and issues that such type of media can present in society. Differently, from previous work, we looked at the perception of shape-change through different perspectives. This helped us to provide additional knowledge that was absent in previous studies, which focused on the evaluation of specific tasks or visual perception only.
• Which content Malleable Media can present and how? Content design and utilization is another open issue in research. It is not clear which content can be presented with shape-change, and for which application applications such media are useful. Through Vital+Morph we proposed data physicalization, which seems to be a suitable application area for shape-change. However, we propose a very specific application domain, such as the communication between a patient and his/her family over distance. Most of the previous works focused only on data, we considered where and how these data are generated, and in which context shape-change is used. We also investigated biomedical data, such as Vital Signs, while previous works looked at very general datasets and never looked at very specific data and their implications. We showed that data sculptures (which are very specific, while researchers on dynamic physicalization tended to create very general representation media, which are not dependent on the dataset displayed. Through our research, we propose to bridge these two aspects, and highlighting that the design choice and aesthetics chosen by the designer can enrich the communication and representation of data. Moreover, we showed that in order to be effective, data has to be interpreted artistically by a designer through the choice of metaphors that has to be consistent with the data displayed. Through Volflex++ we outlined several scenarios where shape- change can be used. We proposed to look not only at data physicalization, but also at virtual musical instruments and digital architecture. Especially, we developed a method that can present virtual objects in their full components, by synchronizing different types of visual, haptic, and auditory feedback. In the context of the research on shape-change, the sound is often omitted. We proposed a method that can interactively represent the sound generated by virtual objects through direct contact with the shape-changing surface. Through our system, the sound model can change dynamically and follow the characteristics of the material presented by the surface. Moreover, both Vital+Morph and Volflex++ suggest how Malleable Media are suitable for presenting both data and virtual objects which are ’organic’. This can open a new way of creating media artworks, such as shape-changing sculptures and installation that can be directly touched and experienced physically by users.
Such systems have been presented both in the form of publications and public presentations, in international conferences, scientific journals, and media arts festivals. They have been validated by the general public and subjects for experiments. While the design and implementations of Vital+Morph and Volflex++ can be seen as engineering contributions, these systems are viewed under the lens of malleable media. The concept of malleable media, and the examples provided in this thesis, point towards a near-future where humans will be exposed to a reality where virtual and physical are mixed, thanks to shape-changing materials. Beyond the idea of ’user interface’, such materials will not be used to mediate our actions with computational processes but will allow us to experience digital information through our body, as we do now with objects and material that composed our environment.
You can download my Ph.D. Thesis here [pdf]