In: International Journal of Human Computer Interaction, 9(2), 151-168.
Towards a Theory of the Effectiveness of Multimedia Systems
Dr. Martijn Hoogeveen (1997)
Short form of title: Towards a Theory of the Effectiveness of Multimedia
This article presents steps towards a theory of the basic effects that multimedia systems have on people. These effects are grouped into knowledge transfer, entertainment and data processing. Although based on empirical investigations, several components of the theory are controversial and in need of further critical empirical investigation. This article criticizes the optimistic multimedia paradigm, the dominant conviction among multimedia system developers and users that adding multimedia functionality to information systems (always) leads to improved information and knowledge transfer. Adding multimedia functionality is not sufficient for these and other learning effects. The role of independent variables that facilitate multimedia effects is reviewed and a more moderate, realistic multimedia paradigm is proposed.
1. Introduction: Why a Theory
A strong paradigmatic belief can be noted in the benevolent effects of multimedia for a wide variety of application domains, particularly Multimedia Assisted Instruction (MAI) (Heller, 1990). Multimedia is used here in the sense of a property of a system or object, indicating that multiple perceptual representation media, such as speech, music, text, graphic, still, animation and video, are used in an integrated manner. Multimedia implies "multisensory", "multimodal", and "multichannel" (Marmolin, 1991). In fig. 1 an overview is given of media types that are used to package and convey an idea or message. For example, the concept "beauty of a flower" can be expressed in a painting (perceptual representation medium), which can be technically represented in a JPEG file and can be stored on the storage medium "CD-ROM" and shown to someone on the presentation medium "PC screen". A more detailed overview of perceptual, technical and physical representation media can be found in fig. 2, elaborating on the ISO work performed within the Multimedia and Hypermedia information coding Expert Group (ISO/IEC, 1992; Hoogeveen, 1996).
Fig. 1. Stylized overview of a message and media used to express and represent the message.
Fig. 2. Overview of examples of perceptual, technical and physical representation media
In many studies the improved learning effectiveness of multimedia in comparison to non-multimedia courses is presumed (Conklin, 1987; Morariu, 1988; Hooper Woolsey, 1991; Marmolin, 1991). However, the true experimental foundation of such assumptions and beliefs is incomplete and often weak (Janda, 1992), and a coherent theoretic basis explaining why multimedia is supposed to work, taking into account the experimental findings, is not (yet) given.
This article presents steps towards a theory of the effectiveness of multimedia systems in the form of a set of hypotheses or propositions. The value of such a theory (see also Huber, 1990) concerning the effectiveness of multimedia systems would be to:
assemble scattered fragments of empirical insight into the added value of Multimedia Systems (MSs);
make hypotheses visible and thus testable;
identify controversial hypotheses which are in need of critical empirical investigation, and thus encourage and focus investigation and debate on the elementary issues;
provide guiding-principles for developing MSs.
In the following sections the multimedia paradigm and alternative interpretations of the effectiveness of MSs are discussed and the theories and evidence in support of this paradigm are reviewed. Next, important dependent variables are analyzed that interact with the multimedia level and may explain the added value of multimedia. Finally, conclusions are drawn about the alternative interpretations of MS effectiveness and a more realistic multimedia paradigm.
2. Multimedia Paradigm
As mentioned in the introduction, one can recognize a set of optimistic convictions in literature and multimedia projects with regard to the effectiveness of multimedia, called here the multimedia paradigm. The word paradigm stresses the aspect of widespread, underlying convictions which often take the form of a firm belief, rather than just a set of testable hypotheses. This is absolutely the case with multimedia.
We can formulate the multimedia paradigm here as the dominant conviction that adding multimedia functionality to information systems (always) leads to improved information and knowledge transfer to people, e.g. learning, retention, understanding, knowledge acquisition, impact of commercial messages (Construct D2; see fig. 3).
Fig. 3. A scheme of the effects of multimedia functionality on multimedia processing, information and knowledge transfer, and entertainment value.
In general, two additional types of multimedia effects can be distinguished (Hoogeveen, 1996). It is believed that the inclusion of multimedia functionality (Construct I, the independent variable) in a system also leads to:
improved processing of multimedia objects (Construct D1), e.g. a multimedia document can not be produced without multimedia tools (this is not disputed here);
increased entertainment value of systems (Construct D3), i.e., it is more fun to use them, they are more exciting.
On the basis of the consistent responses of experts and other subjects in the field, it can be concluded that multimedia is perceived consistently to have added value for multimedia Business Catalogues, multimedia teleshopping and tele-ordering, multimedia promotion, Multimedia Assisted Instruction, and many other application domains (Hoogeveen, 1994). These perceptions are consistent with the multimedia paradigm. However, on the basis of these perceptions alone we can only conclude that the idea that a multimedia paradigm exists is supported, but we cannot conclude with certainty that multimedia really has added value.
A case against the multimedia paradigm can be made from the theoretical viewpoint that it seems inappropriate to assume causal relationships between adding multimedia functionality (the dependent variable) and the effects of multimedia systems on people (independent variables). Assuming causality is too optimistic. Therefore, a process theory (Markus & Robey, 1988) seems to be more appropriate, as it can be observed and reasoned that no guarantee can be given that an MS will be effective, however 'ideal' the multimedia technology that is used seems, and however 'ideal' the system development project is performed that results in the MS. Further, a mixed level of analysis (Markus & Robey, 1988) seems appropriate, assuming a dynamic interplay between individuals, technology, and larger social structures to explain the effectiveness of MSs.
If indeed the multimedia paradigm is too optimistic, in what direction should we look for a new multimedia paradigm? If we just look at the positive, general attitude with respect to multimedia, we can interpret this attitude in five alternative ways, leading to five alternative interpretations:
H0. Multimedia is believed to have a considerable added value, but this is definitely not true, multimedia has hardly any effect at all, it is only hype.
H1. Multimedia is believed to have a considerable added value, but this added value, insofar present, is based solely on the positive attitude with regard to multimedia, it is a self-fulfilling prophecy.
H2. Multimedia is believed to have a considerable added value, but this is only partially true: it is more than just a self-fulfilling prophecy, it has entertainment value, but nothing more.
H3. Multimedia is believed to have a considerable value added, but this is only partially true, because it has entertainment value, and if well applied, is about as effective as conventional means and thus may lead to efficiency and productiveness advantages when replacing those conventional means.
H4. Multimedia is believed to have a considerable added value, and this is true, because it can, if well applied, account also for improved information and knowledge transfer.
Let us look closer at these five alternative interpretations. If we follow the first interpretation (H0) sooner or later the multimedia hype will pass, as the results of multimedia projects fail completely to meet the high expectations. If we follow the second interpretation (H1), the minimum added value of multimedia is based on the paradigmatic belief that it is effective, which hence fosters system acceptance, and is therefore a critical success factor for systems, and has consequently a positive influence on system viability. If a paradigm shift occurs, sooner or later, the basis for adding multimedia disappears. If we follow the third, fourth and fifth interpretations (H2, H3 and H4), multimedia is no hype at all, it offers real measurable benefits for communication and learning; the added value of multimedia, if it is well applied, is based on significant improvements in information and knowledge transfer to people, suppliers and customers and/or on its entertainment value which make multimedia products and services more attractive for selected user groups, and/or its business value.
Why Multimedia is not Sufficient
To what degree is there support for the multimedia paradigm or the alternative interpretations? Let us first look at support for the multimedia paradigm and explanations about why multimedia is supposed to work. Second, we discuss if there is empirical evidence for this.
It can be argued in defence of the multimedia paradigm and statement H4, that a high level of multimedia eases the ability to reach a level of sensory stimulation, arousal and involvement necessary for a conscious processing of information and effective information transfer (see fig. 4). Furthermore, it can be reasoned that every medium has its own special characteristics, and that using these media in combination may lead to some synergy of effects. Speech is human, personal. Music is emotional. Sound in general is transient and public. Text is useful for conveying facts, detailed stories, sophisticated poetry, etc. Further advantages may be based on the use of a free sensory channel, when other sensory channels are busy. This is, for example, the case in the use of auditive warning signals for operator tasks (Alty et al., 1993). Another potential advantage of multimedia is that people with special needs can be addressed where that is not the case with more traditional means. For example, flexible multimedia interfaces can be developed to switch to good sound interfaces for visually handicapped persons.
Fig. 4. Some propositions about why a higher multimedia level is supposed to lead to increased entertainment value and improved information and knowledge transfer.
With regard to training, the general idea is that the more media included in a course, the more effective the course will be. Schadé (1993) puts forward a converging, neuropsychological, explanation for the relationship between multimedia and learning, and states that a story in pictures is picked up much faster and uses less memory capacity than a text story. Schadé argues further that our brains learn most from multimedia information flows, as our brain consists of two, relatively independent, problem solving systems, that are both addressed by multimedia information. According to Schadé, both language oriented representations and visual representations are necessary to fully utilize the cognitive capabilities of the mind.
In support of the multimedia paradigm and statement H4 are also notions from perceptual psychology. Perception psychologist Gibson (1966, 1979) argues that our senses are constructed to handle the very complex flow of information in natural environments, and that our senses are not constructed to handle simple stimuli. Gibson argues further that we are not passive receivers of information. Instead, our perceptual system is characterized by the pick up of information and by the integration of activities in the different senses. Although these arguments are directed at the experimental study of human perception they are, according to Marmolin (1991), also relevant to multimedia computing: information systems that need to support human information processing effectively should make full use of human perceptual and cognitive capabilities, and should represent natural information flows to users and offer support to process natural information flows. According to Marmolin complex, dynamic and integrated representations of information really are necessary to utilize all the capabilities of the mind. Marmolin does not make clear, however, why this is necessary, if this is always necessary, and if this is going to improve, for example, learning performance.
Marmolin (1991) argues further that our visual system is developed to handle continuously changing information rather than static pictures. This pleads in favor of the inclusion of time-based data, such as motion pictures. Faber et al. (1991) conducted several experiments to answer the question under which conditions media combinations including motion pictures, have substantial advantages in comparison to media combinations without motion pictures. The hypothesis was that motion pictures are superior for learning conditions in which spatial or temporal properties of motions must be learnt. With regard to criteria of efficiency, success and time required for successful learning, however, substantial superiority for motion picture presentations was found only with regard to learning a rather complex motion pattern, not for other learning tasks including simpler motion patterns.
In contrast to the multimedia paradigm and statement H4, many effectiveness studies find no significant improvements when comparing a MAI with a conventional course (Yildiz & Atkins, 1993; Van der Mast, 1995). That MAI may lead to positive learning effects is illustrated by the results of a study of Nicolson et al. (1991) examining the effectiveness of multimedia courseware for dyslexic children. The children were found to improve their spelling skills significantly by using the multimedia courseware. Unfortunately, the experimental design of this study is weak, as it did not include control groups. As a result, it is not possible to isolate experimentally the independent variables that were responsible for the learning effect, and it is not possible to draw conclusions about whether the multimedia courseware is an improvement in comparison with traditional courses or not.
The added value of multimedia in terms of office task performance is similarly controversial. In a study, Gale (1990) tried to determine the added value to an office system incorporating audio and video. His study did not show any significant differences between group performance on an information dissemination task, a creative co-operation task and a meeting scheduling task under three conditions: data sharing only, data sharing plus audio, and data sharing plus audio and video. It should be noted, however, that these tasks did not have an audio-visual component. So, we may argue that adding audio and video for these performed tasks is luxury, not a necessity.
Not only is evidence for the comparative effectiveness of multimedia very weak, it is also reported that incorrect use of multimedia can:
easily result in negative cognitive side effects (e.g., overstimulation, cognitive overload, distraction, fatigue (Heller, 1990)) and
thus reduce the effectiveness and efficiency of information and knowledge transfer.
This complies with the process theory viewpoint discussed before, and falsifies the multimedia paradigm.
Less controversial are findings with regard to multimedias entertainment value (H2). With regard to user satisfaction, ease vs. strain, enjoyability vs. boredom, Faber et al. (1991) found that learning by motion pictures is significantly easier and more enjoyable than learning without motion pictures. Nicolson et al. (1991) also found that their subjects enjoyed multimedia courseware. These findings support the idea that multimedia leads to positive arousal, which contributes positively to learning attitude and motivation. Stimulating learning motivation is often an objective in teaching situations, although we have to realize that a positive attitude to learning with new technologies and a genuine motivation to use a MAI are not necessarily a guarantee of enhanced conceptual learning (Yildiz & Atkins, 1993). The fun hypothesis (H2) is further supported by the observation that multimedia games, including audio and video, are eliciting a more enthusiastic responses from players, than non-multimedia predecessors. A finding of Gale (1990) in support of the entertainment hypothesis (see fig. 3) is that in his study audio-visual communication was socially appreciated as positive, 'personal' and 'informal'.
So, in contrast to the multimedia paradigm it can be observed that a higher multimedia level alone is not sufficient for a better task performance and information and knowledge transfer; only for some very specific learning tasks is an improved information transfer noted. Two possible explanations for not finding a clear relationship between multimedia level and system effectiveness are given by ergodynamics laws 1 and 2 (Venda & Venda, 1995), and are illustrated by fig. 5 and fig. 6.
Fig. 5. Applying ergodynamics law 1 (Venda & Venda, 1995) to multimedia offers one possible explanation for why increasing the level of multimedia does not necessarily lead to higher effectiveness. The basic assumption, which can be disputed, here, is that there is a relation between level of multimedia and system effectiveness. In the example it is illustrated that adding video may lead to no change in system effectiveness.
Fig. 6. Applying ergodynamics law 2 (Venda & Venda, 1995) to multimedia shows that increasing the multimedia level may lead to increased system effectiveness for one cognitive strategy, but decreased system effectiveness for another cognitive strategy.
Results indicating that multimedia is fun and enjoyable are far more consistent (H2). It can be also reasoned that in cases where MAI systems are about as effective as conventional courses they may contribute to efficiency gains (H3): teachers can be replaced or their productivity can be improved by improving the teacher/student ratio (Kustermans, 1991). In fact, the efficiency approach is followed in the application of simulators, such as high-fidelity simulation of batch air combats (Goodrich et al., 1992), and the lightweight cockpit in the human centrifuge at the Naval Air Development Center in the US (Cammarota, 1989). High-fidelity means here that the simulated behaviors of objects and environments matches their real behavior very well. The use of simulators for training of personnel or testing is far cheaper than setting up physical air combat situations or performing test flights with real tactical aircraft.
3. Independent Variables That May Facilitate Multimedia Effects
It has been made clear that rich communication (a high multimedia level) alone is not sufficient for improved information and knowledge transfer and the multimedia paradigm can be falsified. Which other variables facilitating multimedia effects can be identified in multimedia literature? The following subsections discuss the notion that in conjunction with rich communication, a high level of interactivity, a high level of congruence of used media, an adequate usage of reference models (e.g., by data visualization), an adequate quality of represented information and an adequate use of navigational structures are presumed to contribute, via a number of interactions, to improved information and knowledge transfer.
Fig. 7. Hypothesized relationships between 6 multimedia variables and information and knowledge transfer and entertainment value. This figure is based on
the literature review presented in this paper.
The interaction of these independent multimedia variables, depicted at the left side of fig. 7, is seen as leading to a number of intermediate psychological responses:
an increased level of stimulation of the senses, at least with regard to the auditory and visual perception systems;
an increased level of involvement, attention, concentration;
emotional arousal, e.g., fun; the word arousal is used in the psychophysiological sense of emotional, internal arousal, related to arousal of the nervous system;
improved recognition, when using mental reference models.
Next, these psychological responses are shown to interact in a complex way, leading to the hypothesized improved information and knowledge transfer (H4).
In the following subsections literature and evidence are reviewed with regard to the potential interaction effects with multimedia of the non-multimedia independent variables given in fig. 7.
Level of Human-Computer Interactivity
Human-computer interactivity is supposed to be one of the "secrets" of multimedia. The level of human-computer interactivity can be operationalized in terms of the degree to which a computer system is responsive to the users (explorative) behavior. A high level of interactivity, e.g., influence on the course of displayed events, manipulation of objects, and editing content data, is not new, but present in many computer systems. A high level of interactivity is, however, new for audio and video systems, for example the TV. It can be speculated that a low level of human-computer interactivity, e.g., switching a presentation on or off, may lead to less involvement and less arousal (see fig. 8) than systems with a high level of human-computer interactivity.
Fig. 8. Some propositions about why increased human-computer interactivity is supposed to lead to increased entertainment value and improved information and knowledge transfer.
As Gibson (1966, 1979) argues: we do not hear, we listen; we do not see, we look around. This means that we are actively exploring our environments. The zapping behavior of TV 'couch-potatoes' may be a current expression of our need for perceptual exploration instead of just passively watching a TV program. One of the visions of multimedia, based on this argument, is that the user of an MS should be enabled to explore natural multimedia information in an active way. Neither the author of the information nor the designer should decide how the information should be processed, the user should be in control (Marmolin, 1991). Schadé (1993) also argues that a high level of interaction in combination with multimedia improves sensory stimulation, and thus facilitates human information processing.
An indication of the role of interactivity is offered by the study of Marmie & Healy (1995) showing that procedural reinstatement results in superior retention. Procedural reinstatement means that durable retention results when the procedures, or operations, employed during acquisition are reinstated, or duplicated, at the time of the retention test. For example, Fendrich et al. (1995) found that motoric processing aided recognition memory when that processing was reinstated at test. So, procedures (interactivity) in combination with reinstatement are key ingredients for improved retention.
Interesting in this context, and in contrast with the fun hypothesis (H2), are the findings of Price (1995) who found that students were more satisfied with a passive medium (video) than with an interactive one (electronic distance education). It is a pity that the relation with learning effectiveness is not examined by Price.
Considering these arguments and findings it seems reasonable to assume that the level of interactivity of an MS is an important variable, in combination with multimedia, for effective information transfer (H4). Yet, we should be careful about drawing further conclusions since clear hard evidence for the hypothesized effects of interactive multimedia are lacking.
Level of Congruence
The level of congruence is the degree to which different media are used redundantly to express the same ideas. When reviewing literature, a high level of congruence of media seems to be far more effective than a low level of congruence for effective information transfer (learning) (H4).
Marmolin (1991) discusses, for example, the redundant use of colors having two effects. It facilitates the pick up and processing of information and it results in a more stimulating environment. This applies, for example, to the redundant use of sound. A study by Graver using sounding interfaces, where each event is characterized both visually and auditorily supports such a hypothesis (Marmolin, 1991). Also a study of Bradey & Henderson (1995) concerning voice-overs indicates that 75-100% of their students believe that audio sounds aided comprehension, helped them link new content to previous knowledge and generate new concepts. We must, however, keep in mind that although the students may believe that something is working, this does not prove that it is actually working!
With regard to the congruent use of synchronized video and audio, Hapeshi & Jones (1992) remark that the presence of moving images can serve to enhance comprehension and learning of spoken material.
An example of this is given by an experiment of Hayes, Kelly & Mandel (1986) comparing the effectiveness of TV presentations to that of radio presentations of narrative information. Generally, the inclusion, during recall, of inaccurate story content and the distortion of actual story details occurred more often in the auditory only condition than in the aural and visual condition. Hapeshi & Jones (1992) describe studies showing that the presence of an incongruent video presentation significantly reduces recognition memory of audio material, but that showing a congruent visual map results in better recall, particularly if the narrative structure is relatively simple. In the case of a monologue or a dialogue, visual display can facilitate processing of the auditory message if the speaker's face can be seen, because facial expressions, particularly lip movements enhance speech intelligibility. Visual display of text can also enhance speech intelligibility, e.g., to recognize the lyrics of a song.
Although sounds are generally thought to be useful to deliver warnings and for context switches, Alty et al. (1993) found the opposite. In their experiment subjects performed a process control task called 'the Crosman Waterbath' task. The Crosman Waterbath task involves the control of a simulated thermal hydraulic system that consists of a single tank. A valve on inflow and outflow pipes may be used to regulate the inflow and outflow. A heater, situated immediately underneath the tank, is the third control variable. In some experimental conditions sounds were used as warning signals, in others this was not the case. Alty et al. found in their experiment that such sounds have a detrimental, although not significant, effect on performance: task completion time, number of warning situations encountered, and number of user actions. It can be argued that the explanation for this is that the sounds were used too incongruently with the other information, and thus led to too much disruption. Disruption is one of the negative cognitive side effects MSs may have if media are used incongruently. In their review, Hapeshi & Jones (1992) also describe a number of studies that have demonstrated the attention grabbing, sometimes disrupting effect of audio, background speech and noise. Furthermore, a study measuring reaction times in reading tasks (Clawson et al., 1995) made clear that reactions were slower for test types involving incongruent stimuli (e.g., the word purple in combination with the color red) than for those that did not.
To summarize, all of this suggests that when the visual and auditory channels provide congruent messages, processing is easier than with incongruent messages (H4). The question remains why this does not help MAI to be more effective than traditional instruction. Perhaps the effects are too weak, or can be reached without using computers, e.g. land charts in a geography book may suffice in most cases, although 3D animations are more fun.
Adequate Usage of Reference Models
We can postulate that an adequate usage of reference models in presented information stimulates recognition and transfer of information to people. A mental reference model is the meaningful organization of information in our brains. Adequate usage of reference models means that we use, for example, meaningful sounds, pictures and movements to express ideas.
Schadé (1993) argues that if reference 'pictures' are added to text, people pick up and understand a story about 75% faster than if they are confronted with a text only story. Schadé hypothesizes that by using reference models, innate or acquired in our early childhood, text and picture stories are stored faster and more efficiently in our long term memory than text-only stories. He states that people tend to remember 25-35% more of a text and pictures story than of a text only story.
The use of reference models is a well accepted marketing practice to use basic reference pictures and sounds to improve advertisements and commercials. Often family scenes, status symbols, attractive women and responsible men etc. are weaved into the marketing material.
For these reasons, it seems reasonable to assume that adequate use of reference models is an important variable with regard to multimedia and effective information transfer (H4). This pleads for an improved use of auditive and visual ideas, models, etc., which is enhanced by using multimedia tools, but can in most cases also be realized with traditional means! Thus, the unanswered question is if and when multimedia models are better than non-multimedia ones.
Quality of Information Representation
The quality of the representation of multimedia information in an MS is an important variable. This is illustrated by the study of Brooks et al. (1991) who examined, in an experimental study, whether additional media added to a tutoring system would enhance effectiveness as measured by speed and accuracy of student performance. Student opinions on the suitability of the system and its media were also examined. Students received a tutorial incorporating either a) text and graphics, b) text, graphics and sound or c) text, photographs and sound. Significant differences were found between the test groups under conditions b and c. Group c performed better on a 'flower recognition' task and an 'object construction puzzle' task. For these tasks it was concluded that photographic representations were superior to simple graphic representations.
The importance of the quality of representation is also stressed by a number of classic experiments (Nielsen, 1990) with regard to reading efficiency in relation to text representation on paper and on screen. These comparative experiments indicate that subjects read 25-30% slower from computer screens than from paper and that these subjects also have significantly higher error rates. Nielsen (1990) reviews a comparative study of Wilkinson and Robinshaw regarding error frequencies of subjects in two test conditions: proof-reading from screen and proof-reading from paper. During the first ten minutes of the experiment, subjects had about the same error rates in the two conditions (25% vs. 22%), but after proof-reading for 50 minutes, the subjects using computer screens did significantly worse with an error rate of 39% vs. 25% for paper! Nielsen concluded that this experiment shows that users become tired fairly quickly when reading from the current generation of computer screens. According to Nielsen, it is only possible to achieve the same reading speed when the computer screen is high-resolution and uses anti-aliased proportional fonts. A side remark is that one can conclude from this that one should be careful with just replacing text books by electronic text books. In the case where one is using electronic text books, using multimedia is probably necessary to compensate for the loss of readability of text information using the current generation of computer screens!
Another example underlining the importance of quality of information representation concerns synthetic speech. Synthetic speech has a relatively bad quality compared to natural speech. Researchers have found that synthetic speech is hindering verbal learning (Hapeshi & Jones, 1992).
To summarize, it has been made clear that quality of information representation is an important multimedia variable for tasks including object recognition and/or reading. Here, we must admit that multimedia representations on average PCs are in many cases worse than commonly used analogue representations, e.g. text and pictures in a medical textbook. This pleads against interpretation H4 and this may be one of the explanations why MAI is rarely better than conventional instruction.
Use of Navigational Structures
Multimedia navigational structures are, in support of H4 believed to:
effectively support exploration using associative hyperlink mechanisms (e.g., explorative learning, exploring criminal relations, exploring ideas and structuring discussions) (Jonassen, 1989; Heller, 1990);
further improve search performance by a far better presentation of retrieved objects, e.g., in the form of Query By Visual browsing, in which a searcher selects or ignores retrieved database objects on the basis of associated pictures (Hoogeveen, 1996);
improve database management by offering facilities to manage multimedia databases (Hoogeveen, 1994).
However, it is a pity that the effectiveness of individual multimedia retrieval facilities in office situations is rarely addressed by research. One of the retrieval facilities that is strongly investigated and has a strong theoretic basis with regard to its cognitive effects is hypermedia. Hypermedia representations may correspond with the mental associative knowledge representation in the mental reference models of a user. Within a hypermedia document a user can trace associative references which correspond with associations with mental reference models in their memory. Discovering new hyperlinks stimulates the extension of these mental reference models. This is akin to Jonassen's (1989) concept of 'web learning', which assumes that information, when learned, is integrated with prior knowledge using a web structure rather than in a linear fashion.
Hyperlinking structures, however, are reported to have a number of negative cognitive side effects. The most important side effects are (Heller, 1990):
disorientation: users become lost in a hypermedia document;
cognitive overload: too much choice overstrains the cognitive capacity of a user;
inefficient searching: hypersearching is sometimes an inefficient way of searching purposefully.
Hypermedia textbooks, i.e., textbooks in the form of hyperlinked multimedia information fragments, have not yet proven to be a more effective learning tool than sequential, conventional textbooks. One of the alternative explanations for this can be that the current generation of computer screens is less effective than paper as a presentation medium for text (Nielsen, 1990).
4. Towards a New Multimedia Paradigm
Has multimedia added value for information systems or is it just hype and is the multimedia paradigm valid? In fact, we have seen that we can easily falsify the multimedia paradigm, that it is too optimistic: if multimedia is used wrongly, incongruently etc., it certainly has a negative added value, it certainly does not lead to improved information and knowledge transfer! Further we have seen that office tasks are not performed better when audio and video are added, and learning tasks are only in rare cases carried out better. Thus, we concluded that a paradigm shift is needed as the current multimedia paradigm is in too many cases clearly invalid, and we discussed five alternative interpretations about the effectiveness of multimedia systems.
Can we falsify some of the five interpretations on the basis of data gathered? The first interpretations (H0 and H1), that multimedia has definitely no real added value and that it is only a self-fulfilling prophecy, can be falsified on the basis of:
consistent research indicating that people experience multimedia as fun, as enjoyable, and that multimedia makes it more exciting to learn;
the observation that multimedia games with audio and video are experienced as more attractive to players, than non-multimedia games.
This brings us to interpretation three (H2). It can be the case that multimedia has significant business value (H3) although the psychological added value is limited. For example, with regard to training situations it can be reasoned that if MSs are not more, but just as effective as conventional courses, this can result in interesting efficiency gains: courses can be made cheaper by replacing teachers or improving their productivity by improving the teacher/student ratio. Examples are given of this.
This brings us to interpretation four (H3), which is more hard too falsify beyond any doubt, although some results point in the direction of interpretation five (H4):
learning experiments made clear that adding multimedia elements - if the media are used congruently, adequate use is made of mental reference models, and a sufficient quality is used for information representation - improves information and knowledge transfer to people for some very specific learning tasks;
multimedia information, presented congruently, is easier to recall than monomedia information;
tasks with an object recognition component are performed better when adequate visual representations are used.
On the basis of this argumentation, it can be concluded that multimedia is not just hype, it has all the elements of a self-fulfilling prophecy, which does not imply that it is ineffective, it has a distinct entertainment value, and, if carefully applied, has business added value and only sometimes seems to contribute to more effective information and knowledge transfer.
Careful application means that in combination with rich communication, a high level of interactivity, a high level of congruence of used media, an adequate use of reference models, an adequate quality of information representation, and adequate navigational structures are offered by MSs. MSs may have significant business value in cases where entertainment value needs to be added, multimedia data needs to be processed, or staff needs to be replaced or become more productive (e.g., sales, marketing and training). These interpretations may form the basis of a more moderate, more realistic new multimedia paradigm.
Alty, J. L., Bergan, M., Craufurd, P., & Dolphin, C. (1993). Experiments using Multimedia Interfaces in Process Control: Some Initial Results. Comput. & Graphics, 17(3), 205-218.
Brady, S. & Henderson, L. (1995). Voice-Overs and Auditory Cues: Their Impact and Role in Learning through IMM. In Maurer, H (Ed.). Educational Multimedia and Hypermedia 1995. Proceedings of ED-MEDIA95. (pp. 109-114). Charlottesville, VA: AACE.
Brooks, P., Schmeling, A. &, Byerley, P. F. (1991). Multimedia intelligent tutoring of human-computer interaction procedures: An experimental evaluation. In H.-J. Bullinger (Ed.). Human Aspects in Computing: Design and Use of Interactive Systems and Information Management. Amsterdam, the Netherlands: Elsevier Science Publishers (pp. 959-963).
Cammarota, J.P. (1989). Research in a High-Fidelity Acceleration Environment. IEEE Aerospace and Electronics Systems Magazine, 4 (9), 30-38.
Clawson, D.M., King, C.L., Healy, A.F. & Ericsson, K.A. (1995). Training and Retention of the Classic Stroop Task: Specificity of Practice Effects. In Healy, A.F. & Bourne, L.E. (Eds.). Learning and Memory of Knowledge and Skills: durability and specificity. (pp 30-65). Thousands Oaks, CA: Sage Publications.
Conklin, J. (1987). Hypertext: An Introduction and Survey. IEEE Computer, 2(9), 17-41.
Faber, J., Meiers, T., Ruschin, D., & Seyferth, A. (1991). The Motion Picture in Interactive Information Systems: A Necessary or Facilitating Component? In Bullinger, H.-J. (Ed.). Human Aspects in Computing: Design and Use of Interactive Systems and Work with Terminals. (pp. 485-490). Amsterdam, the Netherlands: Elsevier Science Publishers B.V.
Fendrich, D.W., Gesi, A.T., Healy, A.F. & Bourne, L.E. (1995).The Contribution of Procedural Reinstatement to Implicit and Explicit Memory Effects in a Motor Task. In Healy, A.F. & Bourne, L.E. (Eds.). Learning and Memory of Knowledge and Skills: durability and specificity. (pp 30-65). Thousands Oaks, CA: Sage Publications.
Frau, E., Midoro, V. & Pedemonte, G.M. (1991). Do Hypermedia Systems Really Enhance Learning? A Case Study on Earthquake Education. ETTI, 29(1), 42-50.
Gale, S. (1990). Human aspects of interactive multimedia communication. Interacting with Computers, 2(2), 175-189.
Gibson, J.J. (1966). The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin Co.
Gibson, J.J. (1979). The Ecological Approach to Visual Perception. Boston: Houghton Mifflin Co.
Goodrich, K.H., McManus, J.W., & Chapell, A.R. (1992). A High-Fidelity Batch Simulation Environment for Integrated Batch and Piloted Air Combat Simulation Analysis, 1992 AIAA Flight Simulation Conference, AIAA paper 92-4145.
Hapeshi, K., & Jones, D. (1992). Interactive Multimedia for Instruction: A Cognitive Analysis of the Role of Audition and Vision. International Journal of Human-Computer Interaction, 4(1), 79-99.
Hayes, D.S., Kelly, S.B., & Mandel, M. (1986). Incidental Processing of Speaker Characteristics: Voice as Connotative Information. Journal of Verbal Learning, 8, 28-45.
Heller, R. S. (1990). The Role of Hypermedia in Education: A Look at the Research Issues. Journal of Research on Computing in Education, 431-441.
Hoogeveen, M. J. (1994). The Viability of Multimedia Retrieval Systems for Marketing and Sales. Doctoral dissertation at the Delft University of Technology. Leidschendam, the Netherlands: KPN Research.
Hoogeveen, M.J. (1996). Multimedia Marketing Systems. Leidschendam, the Netherlands: Lansa Publishing.
Hoogeveen, M. J., & Van der Meer, K. (1994). Integration of Information Retrieval and Database Management in Support of Multimedia Police Work. Journal of Information Science, 20(2), 79-87.
Hooper Woolsey, K. (1991). Multimedia Scouting. IEEE Computer Graphics & Applications, 26-38.
Huber, G.P. (1990). A Theory of the Effects of Advanced Information Technologies on Organizational Design, Intelligence, and Decision Making. Academy of Management Review, 15(1), 47-71.
ISO/IEC (1992). MHEG WD S7. Information Technology - Coded Representation of Multimedia and Hypermedia Information Objects. Part 1: Base Notation. (Report No. N 235). Geneva, Switzerland: ISO/IEC.
Janda, K. (1992). Multimedia in Political Science: Sobering Lessons from a Teaching Experiment. Journal of Educational Multimedia and Hypermedia, 1, 341-354.
Jonassen, D. H. (1989). Hypertext/hypermedia. Englewood Cliffs, NJ: Educational Technology Publications, Inc.
Kustermans, L. (1991). Cost-effective Training on Medical Equipment through CD-i. In R. N. Tucker (Ed.). The Vision and Reality Reader. Den Haag, Netherlands: Interactive Learning Services Ltd.
Markus, M. L., & Robey, D. (1988). Information Technology and Organizational Change: Causal Structure in Theory and Research. Management Science, 34(5), 583-598.
Marmie, W.R., & Healy, A.F. (1995). The Long-Term Retention of a Complex Skill. In Healy, A.F. & Bourne, L.E. (Eds.). Learning and Memory of Knowledge and Skills: durability and specificity. (pp 30-65). Thousands Oaks, CA: Sage Publications.
Marmolin, H. (1991). Multimedia from the Perspectives of Psychology. In Kjelldahl, L. (Ed.). Multimedia. Systems, Interactions and Applications. 1st Eurographics Workshop, Stockholm, Sweden April 18-19, 1991. Berlin, Germany: Springer-Verlag.
Morariu, J. (1988, November). Hypermedia in Instruction and Training: The Power and the Promise. Educational Technology. 17-20.
Nicolson, R. I., Pickering, S., & Fawcett, A. J. (1991). A Hypercard Spelling Support Environment for Dyslexic Children. Computers Educ., 16(2), 203-209.
Nielsen, J. (1990). Evaluating Hypertext Usability. In D. H. Jonassen, & H. Mandl (Eds.). Designing Hypermedia for Learning (pp 147-168). Berlin, Germany: Springer-Verlag.
Price, M.L. (1995). Student Satisfaction with Distance Education and the Medium of Instruction: Is there a Relationship? In Maurer, H (Ed.). Educational Multimedia and Hypermedia 1995. Proceedings of ED-MEDIA95. (pp. 550-555). Charlottesville, VA: AACE.
Schadé, J. P. (September, 1993). Ons brein en CD-i [Our brain and CD-i]. Janssen Medisch-Wetenschappelijk Nieuws, 276-282.
Van der Mast, C. (1995). Developing Educational Software. Integrating Disciplines and Media. Doctoral dissertation. Delft, The Netherlands: Delft University of Technology.
Venda, V.F., & Venda, Y.V. (1995). Dynamics in Ergonomics, Psychology, and Decisions: Introduction to Ergodynamics. Norwood, NJ: Ablex Publishing Corporation.
Yildiz, R., & Atkins, M. (1993). Evaluating Multimedia Applications. Computers Educ., 21(1/2), 133-139.
© 1995-2002 Martijn Hoogeveen