By Ellen Isaacs, Trevor Morris, and Thomas K. Rodriguez

Published in 1995 in ACM's Proceedings of the Conference on Computer-Human Interaction, Denver, CO, pages 354-361. (1)

Abstract

Keywords: Distributed presentations, distance learning, computer-supported cooperative work (CSCW), video conferencing, multimedia, organizational communication.

Introduction

This combination of distributed workplaces and increased need for communication and training is leading many organizations to look to technology for support. In particular, the growth of networked computers can help them "shrink the workplace." So far, the CSCW community has explored two approaches to the problem. It has studied ways to use electronic mail [Sproull and Kiesler, 1988] and databases [Liberman and Rich, 1993] to help large groups communicate efficiently across time and distance. And it has experimented with the use of video and audio to provide a sense of teleproximity to small distributed groups who need to tightly coordinate their efforts [Olson and Bly, 1991; Tang and Rua, 1994]. There has been less exploration, however, of the use of video and audio to help support the communication needs of large distributed organizations.

There have been a few attempts to use multimedia to support large distributed group communication. For example, members of the Internet community have conducted live video presentations over the Multicast Backbone (MBone) [Macedonia, 1994]. Usually, video is multicast from a speaker and the audience participates through an audio channel. A shared whiteboard program may also be used. The growing use of MBone conferences demonstrates the need to connect large groups of people spanning across continents.

Our group has developed an application called Forum that broadcasts video-based presentations to audiences sitting at their workstations distributed across a network [Isaacs, et al., 1994]. Like MBone conferences, Forum multicasts live audio, video, and slides from a presenter, and it provides a mechanism for audiences to ask questions. Unlike the MBone, Forum enables the speaker to poll the audience anonymously, and it allows audience members to send written comments to the speaker and to each other. Forum is also distinguished by its emphasis on making the user interface easy and compelling.

As Forum and similar applications are used to improve communication and training, it is important to understand how technology-supported arenas differ from lecture halls and classrooms. Such an understanding will help us improve the design of these applications and it will help us set realistic expectations for successful use of the technology. To compare these two environments, we conducted a study comparing a set of presentations that were given both over Forum and face-to-face. In this paper, we describe the Forum application, how we conducted the study, and what we found. We also discuss ways to improve the design of distributed presentation technology and how people might adapt presentations to most effectively use the technology.

Forum

Figure 1: Forum's audience interface. Audience members watch the video and interact with the speaker in the upper left window. They view the slides in the lower window, using the thumbnails to view slides independently of the speaker. The Audience Windo shows a list of attendees and allows the audience members to send each other text messages..

Audience members interact with the speaker through a live poll, a spoken question, or a written comment. Speakers use the poll meter to ask the audience multiple-choice questions (shown as a yes/no poll in Figure 1). As each person clicks on a choice, everyone sees the poll meter update. To ask a question, audience members press the Ask to Speak button, which puts their name in the In Line to Speak list. When called on, they press the Speak button and speak into their microphones. Everyone else hears the question and sees a small picture of the person that appears while she speaks. In Figure 1, for example, Ellen Isaacs is asking a question and three others are waiting to speak. Audience members can also send the speaker written comments. The Comments button brings up a window in which they can type a note and then send it in. Only the speaker sees these comments, unless she chooses to present one to the audience as a slide. Everyone can see an audience list (upper right window), and audience members can send each other text messages. (See Isaacs et al., 1994 for a complete description of Forum.)

To give a presentation, speakers sit in front of a workstation and speak to a camera. Only one person, a member of the Forum development team, is usually in the room with the speaker to help manage any unexpected problems.

Method

The data consist of the following:

• Surveys asking both audience members and speakers to rate their reactions to the talks. (Audience survey response rate averaged 42% per talk.)
• Videotapes of all talks, in most cases from both the speakers' and the audiences' perspectives.
• Computer logs of the speakers' and audiences' Forum use.

Because the adoption of Forum will be determined by how people perceive it, we wanted to understand their reactions. For this reason, we rely heavily on the surveys. Since user perceptions do not necessarily reflect actual differences, we use the videotapes and logs to help explain users' reactions.

We attempted to observe a range of presentation topics and styles. Presentation topics included: a company executive's vision of the future, a new human resources policy, a career development workshop, a review of a project team's accomplishments, two presentations of experimental studies, and a description of a design evaluation technique. Three were designed to be given to large audiences and four were intended for small to medium-sized groups.

Although we did everything we could to make sure the two talks were comparable, we had to make certain compromises to gain cooperation with users. The talks did not always cover identical material explained in an identical way. One speaker combined material from a number of related talks on the subject and another adapted the style of presentation to Forum's capabilities. Four pairs of talks were given within a week or so of each other, but in the other three, between one and three months separated the two presentations. All but one of the talks were given first in a local setting and second over Forum.

Because we were working with real talks offered to the community, we could not control who attended in which presentation environment. In all but two cases, the talks were announced without indicating they would be presented in another format. Since users usually did not choose between formats, presentation environment played a role only to the extent that it influenced a person's decision to attend. Nonetheless, since audiences were self-selected, any results must be attributed to a combination of the technology and the type of people it attracted, not the technology alone.

Findings

Attendance and attention

These results are not surprising because Forum is one application in an environment where other activities on the computer and in the office compete for the user's attention. First, Forum makes it easy for audiences to attend talks. Audiences received an e-mail message announcing a talk, they clicked on an "invitation," and Forum appeared on their screens just before the talk began. Second, once Forum was running, audiences had available to them all their desktop applications as well as other resources (and distractions) in their offices. As one user commented, "having an on-line presentation like this is wonderful! I can choose to focus on areas of importance and tune out to get more worth-while work accomplished." When asked about their other activities, audiences most often reported reading e-mail, answering the phone, talking to officemates or co-workers, or doing other work. Some reported using other applications to help understand the material, for example by taking notes in a text editor or by browsing Mosaic when a speaker referred to information on the World Wide Web.

We were somewhat surprised that local audiences reported paying far less than full attention to the talk, especially since they had relatively few distractions. When asked what else they did, audiences most often said they thought about unfinished work and some even did work they had brought with them. (Other common answers were daydreaming and thinking about how the material applied to them.) So even those who decided to leave their offices still spent a portion of their energy thinking about the work they left behind.

Forum talks also tended to run longer than local talks, 1:10:31 vs. 1:01:33, but this difference was not significant (F(1,5) = 4.86, p<.079). Speakers did report losing track of time during Forum talks, perhaps because they could not see when audience members became restless or left as the scheduled end time approached. Also, some speakers said the Forum clock was not prominent enough.

Presentation Quality

Speakers perceived a wider difference in their performance than the audience, rating themselves 4.42 in local settings and 3.42 over Forum (F(1,5) = 15.00, p<.05). Although we cannot compare speakers' and audiences' ratings statistically, it is interesting to note that relative to audiences' ratings, speakers appeared to overestimate their performance in local settings and underestimate their performance over Forum.

Audiences had a slight tendency to find the material more interesting in a local setting (4.21) than over Forum (4.17) (F(1,415) =5.37, p<.05), but this difference may be due to the fact that people who made the effort to attend in person were more likely to be interested in the material. It is in fact encouraging that the difference in interest levels between Forum and local audiences was so small.

Speakers also felt that certain features of Forum enhanced their presentations compared with local talks, but other features detracted from their talks. Speakers said Forum enhanced their presentation more than the equipment available in local talks did, 3.00 vs. 2.00 (F(1,5) = 7.50, p<.05). They said it did so by providing the ability to annotate their slides, anonymously poll the audience, easily show videos, and reach a wider audience. However, they felt Forum detracted more than a local talk environment, 3.17 vs. 1.83 (F(1,5) = 10.00, p<.05) because it did not provide an image of the audience and it forced them to stay seated. "I like to move," one explained, "and my voice gets more interesting when I speak to a large room with lots of people." Some were distracted by the video preview of themselves and others said they had trouble referring to their notes, which they had to keep on their laps because Forum did not provide on-line support for speakers' notes.

Visual Materials

Audiences said they thought the visual materials helped them understand the information better over Forum (4.17) than in a local setting (3.58) (F(1,376) = 15.44, p<.001). This finding may be due to the fact that the slides window took up a large portion of the screen (see Figure 2) and that each person received a close-up and unobstructed view. The logs show that an average of 68% of each audience took advantage of its ability to view slides independently of the speaker, indicating that participants were using the slides to understand the material at their own pace. In addition, an average of 35% of each audience saved their own on-line copy of the slides. Audiences reported taking fewer notes during Forum presentations (F(1,406) = 28.02, p<.001), again probably because the slides were made available. Only two speakers handed out their slides to local audiences, and one of those did not provide a complete set.

The videotapes showed that speakers pointed to the slides during Forum talks an average of 26.8 times per presentation compared with 20.8 for local talks, but this difference was not significant (F(1,5) = 0.51, ns). (Figure 2 shows the same speaker pointing in each setting.) They also drew on the slides more often during Forum talks, 15.8 vs. 0.3 times per talk, but this difference was not significant because of large variances (F(1,5) = 3.83, p<.11).

Interactivity

The videotapes showed that significantly more spoken questions were asked in local talks (19.5) than Forum talks (2.8) (F(1,5) =12.05, p<.05). But when written questions are included (0.0 in local settings vs. 9.3 over Forum), there was no significant difference in the number of questions asked in local talks (19.5) vs. over Forum (12.2), (F(1,5) = 1.60, ns). In addition, more poll questions were asked over Forum (2.2) than in face-to-face talks (0.8) (F(1,5) = 7.27, p<.05).

However, interactions in local talks appeared to be richer in a number of ways. First, people who asked questions were more likely to ask follow-up questions or clarifications, which created more of a sense of a discussion rather than a lecture. Audiences asked follow-up questions 8.7 times per local talk compared with only 1.0 times per Forum talk. This difference was significant (F(1,5)=5.66, p<.05).

Second, speakers were more likely in local settings to incorporate other people into their presentation, most commonly drawing out audience members who had also worked on the material being discussed. During the six talks, there were 48 instances when other knowledgeable people volunteered information to follow up on the speaker's point or the speaker asked an audience member to help answer a question. One talk was in fact given by two people and it had a very different feel in the two settings. Over Forum, each speaker gave his part of the presentation separately, but in the conference room, each called out comments throughout the other person's section. Although it was possible over Forum for attendees to volunteer information or the speaker to draw out information, it happened only three times. Two of those times occurred through the written comments, which have a built-in delay. It was also more risky for a speaker to ask someone in the audience list to help them answer a question because they had no guarantee that the person was paying attention or that they had a microphone that would enable them to respond.

When asked to rate how well speakers handled questions, audiences rated Forum speakers at 3.93, compared with 4.29 for local talks (F(1,400) = 10.21, p<.01). Speakers also rated themselves lower for Forum talks (2.67) than for local talks (4.17) (F(1,5) = 45.0, p<.001). In this case, speakers appeared to underestimate their performance in both settings relative to audiences' perceptions, but they thought they did a much poorer job of answering questions over Forum than audiences did. One speaker mentioned a common problem in her response, "you don't get the immediate feedback of seeing the employee you are answering a question for. You can't see them shake their head, that yes they understand or no they need more info. I sometimes felt like I repeated myself too many times trying to answer the question because I could not see if I had made my point." Speakers also rated the quality of the questions poorer over Forum than in local talks (3.00 vs. 4.25), but this difference was not significant (F(1,5) = 4.75, ns).

Audience Awareness

The videotapes showed that the local talks included an average of 16.3 instances of spontaneous laughter or chuckling among the audience and an average of 1.2 instances of applause. Since Forum had no such mechanisms, there were no instances of either laughter or applause. Speakers said the lack of such feedback gave Forum a more sterile feeling. The videotapes showed that audience members sometimes did in fact applaud and laugh in their offices, but they had no means of conveying those emotions to the speaker or the rest of the audience. Some audience members were frustrated by the lack of such a channel and some went so far as to send in written comments explicitly praising the talk and thanking the speaker for doing a good job.

Both audiences and speakers reported having a lower sense of the audience's reaction to the talk. Audiences rated their awareness of each other at 3.08 at local talks but only 2.30 at Forum talks (F(1,397) = 57.91, p<.001). Speakers felt an even more dramatic gap, rating audience awareness at 4.50 at local talks and 1.83 over Forum (F(1,5) = 64.00, p<.001).

We also asked audiences to indicate how much they interacted with others before, during, or after talks in a way that related to the talk. Forum audiences rated their degree of interaction at 1.69 compared with 2.14 in local talks (F(1,398) = 38.59, p<.001). We found it interesting that although audiences interacted with each other more at local talks, they did not report interacting much in either setting. Speakers also said they interacted with audience members before and after the talk more often in a local setting than over Forum, 3.67 vs. 1.33 (F(1,5), p<.01).

The videotapes showed that audience interaction at the local talks consisted of participants chatting with each other before and after the talks as well as whispering to their neighbors during the talk. Some talks included more novel types of interaction. In one small-group presentation, each person briefly introduced himself or herself. In another talk, the person who had sponsored the presentation gave a pitch for her group's services before the talk began and then handed out material at the end. At the end of two other talks, the speaker passed around a prototype of a hand-held device and people discussed it as they passed it around.

We asked Forum audiences how they had interacted and they reported watching with others, talking with officemates or passersby, chatting in the halls with others who had also watched, having lunchtime discussions about the talk, and sending each other text messages. The logs indicated that an average of 13% of each audience sent an average of 2.9 messages per person. Although Forum audience members heard each others' questions and sometimes interacted with other attendees both near and far, those isolated interactions did not replace the sense of community that is conveyed among a large group in a single room.

Preferences

When speakers were asked which format they preferred, three chose a local setting, two said it depended on the material, and one chose Forum. The one who chose Forum did not actually prefer using it; instead she felt it was most important to reach a broad audience. (She described a new human resources policy, which had to be communicated to everyone in the company.) Speakers also said they enjoyed giving their talks in a local setting (4.25) more than they did over Forum (2.92) (F(1,5) = 10.00, p<.05). As one speaker reported, "face-to-face is still the most natural communication form and I feel that I can express myself better as a speaker when I can use my entire body and can see the audience." Nonetheless, most speakers made a point of saying that they enjoyed using Forum and appreciated being able to reach a wider audience. One explained, "I prefer face-to-face for giving a talk, but I would happily use Forum if I wanted to reach an audience who I didn't think would attend a real talk."

Audiences, on the other hand, strongly preferred Forum. When we asked audience members how they would attend next time, we found that Forum audiences preferred Forum by a wider margin than local audiences preferred local talks (c2 = 7.13, df = 1, p<.01). Ninety percent of Forum audiences preferred Forum, whereas 72% of local audiences said they would attend again in person. When we asked audiences to evaluate the extent to which the talk was worth the time and energy they spent to attend, they rated Forum and local talks almost equally (4.01 vs. 4.04) (F(1,410) = 2.45, ns). Since local talks take more time and energy to attend, this result shows that people who attended in person felt that the extra effort was worthwhile.

Those who preferred Forum offered many explanations. They liked the convenience of attending, the ability to do other work, the ability to tune in and out depending on their interest level, and the lack of travel time. Others also mentioned always getting "the best seat in the house," being able to talk with others, being able to leave talks politely, and being accessible to co-workers during the talk. Those who preferred local talks liked being able to interact and influence the discussion more easily, concentrate better, and see and hear the speaker with higher fidelity.

Effects of Presentation Topic

The HR policy talk provided information about a relatively important but mundane topic. It was the only one at which the Forum audience thought the material was more interesting (3.42 vs. 3.00) and presented better (3.73 vs. 3.54) than the local audience did (although these differences were not significant). It was also the only one in which the Forum audience thought they had made significantly better use of their time (3.46 vs. 2.62, F(1,38) = 7.77, p <.01). When asked which way they would attend in the future, even the local audience chose Forum by a 2:1 margin. Audience members commented that Forum was ideal for this kind of talk because they could get the information without devoting an uninterrupted hour to it.

The vision talk made good use of Forum because it reached a wide audience (250 people), whereas attendance at the local talk was restricted due to room size. People also appeared to feel more comfortable writing rather than speaking their questions to a high-level executive; he handled 24 (written) questions over Forum but only five (spoken) in person. Unlike at other pairs of talks, the Forum and local audiences' reported no significant difference in their sense of the audience (2.73 vs. 2.67), and the amount they interacted with each other (1.65 vs. 1.68). In most other talks, local audiences rated themselves significantly higher on these measures than Forum audiences.

At the other extreme, the experimental study talk did not seem to translate well to Forum. This talk was designed for a lunchtime discussion series among a small group of colleagues. Compared with the Forum audience, the local audience of this talk found the material significantly more interesting (4.44 vs. 3.33) (F(1,26) = 9.16, p<.01), thought the presenter did a much better job of presenting it (4.11 vs. 2.97) (F(1,6) = 17.49, p<.001), and was more satisfied that their time had been well spent (4.56 vs. 2.88) (F(1,25) = 19.30, p<.001). Audience members had a much greater awareness of each other (4.00 vs. 1.70) (F(1,25) = 54.94, p<.001) and interacted with each other to a far greater degree than they did at the Forum talk (3.22 vs. 1.61) (F (1,26) = 16.34, p<.001), probably because many of its members knew each other. Far more questions were asked (44 vs. 10) and there were more instances of laughter in this talk than any other (35).

This talk was also given by two presenters who frequently interacted and added to each other's explanations. It was very difficult for the speakers to translate this type of intimate, highly interactive, and informal presentation to Forum, and as a result, they strongly preferred the local setting over Forum (rating them 5 vs. 1). As one audience member said, "[This talk] series is as much of a social event as it is a technical talk. I would not want to miss the social aspects of a room full of interesting people."

Discussion

Although audiences also perceive lower quality interaction, they still prefer attending remotely because of the convenience, the ability to get other work done during the presentation, and the ability to easily view and save the slides. They generally believe they learn the material effectively and much more efficiently. As one person explained, "I actually liked the ability to look over the slides at my own pace, to continue doing work if I so choose. Great time savings and I got the information I needed." As a result, people attend a wider range of presentations remotely than they do when forced to attend in person. Several Forum audience members mentioned that they would not have attended the presentation if they had been required to travel to a conference room.

It is worth considering how Forum and other related tools might be designed to better support presentations. On the other hand, it is also reasonable to discuss how speakers might learn to adapt their presentations to better overcome the limitations of reaching remote audiences.

Improving the Design

When developing Forum, we decided not to provide video of the audience because of limited network bandwidth and a lack of commonly available video equipment. We were concerned that current networks would be overloaded if video were transmitted from even a portion of the audience. We also did not want to require audience members to have cameras because it would radically reduce the size of available audiences. We expect that network bandwidth will increase over time, however, and more people will have cameras built into their computers. At that point, it will be more practical to provide video of audience members.

Another approach is to allow a small group to attend locally as the talk is transmitted to a remote audience. We are in the process of building a studio to do just this. The local audience should provide a sense of at least part of the audience and it should create a more intimate atmosphere even when many people are attending. Speakers will be able to stand and move around, which some said would help them better express enthusiasm. And it will facilitate tight interaction among multiple presenters. One potential pitfall is that the remote audience may feel less involved in the talk. Care must be taken to carefully integrate the two audiences, especially during interactive portions of talks.

Less radical changes may also be possible. We found that much of Forum's sterile atmosphere was due to the lack of ongoing audio information from the audience. The lack of laughter and applause were notably missing, as were other spontaneous reactions, such as groaning at a bad joke and quick "quips" from the audience. Forum did not allow each audience member to have an "open mic" because doing so would have caused an unacceptable level of audio feedback. If improvements were made in echo cancellation and silence detection software, it would be possible to allow a more open environment in which spontaneous noises could be captured. Barring that, we could provide more overt mechanisms to convey applause and possibly amusement, although it would be important to make the interface natural and the sounds realistic. We might also enable audience members who asked a question to easily indicate when they were satisfied with the answer.

Another useful feature would be support for on-line speaker's notes. As mentioned, speakers often looked at notes on their laps because they were not available on line. Speakers could focus more of their attention on the material if they could prepare annotations that only they could view. Adapting to the Tool

Even if we could provide an ideal remote presentation tool, speakers would still have to learn to tailor their presentations to use it effectively. All seven speakers in this study gave their first-ever Forum talk, and only four had previously attended as a Forum audience member. Not surprisingly, all of them felt they would do a better job next time. Most of their proposed modifications were intended to improve interaction with the audience. In addition, we felt that over time speakers would learn to take advantage of the fact that audiences were watching from their desktops, with access to other desktop tools.

Many speakers said that if they gave another presentation, they would plan to use the poll more often or ask provocative questions. Two speakers, including the one who did adapt his presentation to Forum, posed a question and asked audience members to respond with written comments. After responses arrived, they read a series of answers to the audience. This technique seemed to work quite well because it gave them a sense of their audience, got the audience used to responding, and kept everyone interested.

One person said he would ask audience members to identify themselves, which would give him a better idea of who was attending and make the audience more comfortable using their microphones. A few said they would look at the camera more, hoping to create more sense of intimacy for the audience. And one person said he would put less on his slides to create more drama as he built to a conclusion. He disliked the audience peeking ahead at his punch lines.

Although none of the speakers mentioned using other computer desktop resources, we think this is a promising possibility, especially because it makes a virtue of the fact that audiences often split their attention during talks. During a presentation, audiences could be encouraged to explore more detailed on-line information, for example by using Mosaic. Doing so may encourage participants to ask more informed questions that raise the level of discussion. During a technical discussion or a training session, participants could be asked to retrieve examples from their own work to focus the discussion on issues most relevant to them.

Another approach is to keep users' attention focused by making Forum talks more compelling. Just as broadcast television uses attractive imagery and clever composition to capture viewers' attention, Forum speakers might draw in audiences by creating presentations with high production values. Of course, doing so requires more effort, but such investment may be appropriate for certain presentations.

Conclusions

With improved technology and designs, audience interaction will become easier and more natural, but even today, companies and universities can successfully use video-based presentation tools to supplement face-to-face instructional environments and text-based on-line tools. Companies can keep more people informed and synchronized in their efforts, and schools can reach out to those who cannot attend in person. It is important to recognize, however, that remote presentation environments have their advantages and disadvantages. Presenters should use the environment that best matches their purpose and adapt their presentation accordingly.

Acknowledgements

References

2. Liberman, K. and J.L. Rich, Lotus Notes Databases: The Foundation of a Virtual Library, Database, 1993, 16(3), pp. 33-46.

3. Macedonia, M.R. and D.P. Brutzman, MBone Provides Audio and Video Across the Internet, IEEE Computer, April, 1994, pp. 30-36.

4. Olson, M.H. and S. A. Bly, The Portland Experience: A Report on a Distributed Research Group, International Journal of Man[sic]-Machine Studies, 1991, 34 (2), pp. 211-228.

5. Peters, T., Thriving on Chaos: Handbook for a Management Revolution, 1987, New York: A. A. Knopf, Inc.

6. Smith, A.E., Innovative Employee Communication: New Approaches to Improving Trust, Teamwork, and Performance, 1991, Prentice Hall: Englewood Cliffs, NJ.

7. Sproull, L. and S. Kiesler, Reducing Social Context Cues: Electronic Mail in Organizational Communication, in Computer-Supported Cooperative Work: A Book of Readings, I. Greif (Ed.), 1988, Morgan Kaufmann: San Mateo, CA, pp. 683-712.

8. Tang, J.C. and M. Rua, Montage: Providing Teleproximity for Distributed Groups, Proceedings of the Conference on Computer Human Interaction (CHI `94), April, 1994, Boston, MA, pp. 37-43.

(1) Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post servers or to redistribute to lists, requires prior specific persmission and/or a fee.
Copyright 1995 ACM