Case Study 35: The COMET® Modules

Case Study 35: The comet Modules

Brent G. Wilson et al.

Audio by Paper2Audio.

The Interview

Brent Wilson and May Lowry stepped into Joe Lamos's corner office. Joe was head of distance learning in the comet Program, a cooperative agency funded principally by the National Weather Service, with support from the air force and navy, charged with linking forecasters with weather scientists through on-site and distance-based training programs. Joe had stepped out, so Brent and May parked their bags and sat down at the roundtable beside Joe's computer. Within a minute, he arrived.

"Thank you for waiting." After pleasantries, Joe initiated the conversation. "Let me give you some background. The comet ^{} Program was started here in Boulder in 1989. U.C.A.R-the University Center for Atmospheric Research has a cooperative agreement with its sponsoring agencies to provide education and training in the application of the latest atmospheric science to weather forecasting. The program uses teams of university professors. scientists, working forecasters, and instructional designers to design a program to provide professional development to weather forecasters. This education and training both supports the modernization activities of the weather services and provides a resource for the larger meteorological community.

"The program we want evaluated is our D.L--distance learning-program. The National Weather Service, air force, and navy have about 5,000 forecasters all over the world, including in small local offices and on overseas bases and ships. It is usually too expensive and time-consuming to send those folks to a training center, so we are asked to provide on-station learning materials. What we have tried to do with the distance learning is to provide high-quality science information and forecast training, using state-of-the-art multimedia, first on videodisc and now on C.D-ROM."

"You've been contracted to evaluate our series of multimedia modules. We have nine modules out now, distributed to all forecaster offices in the National Weather Service, air force, and navy. The modules run 4 to 12 hours in length and provide fairly comprehensive training on a variety of topics—the forecast process, marine meteorology, and so on. After turning out modules for five years, we're ready for a comprehensive evaluation of their quality. We want you to find out how they're being received and used out in the field."

May was listening intently. "Joe, tell us more about the module design."

"The modules are built around engaging scenarios depicting problems faced by forecasters," Joe responded with enthusiasm. "Learners are encouraged to describe how they would approach the problems. They're provided with help in the form of information, advice, quizzes, animation, and video clips of experts discussing the particular weather topic and forecasting techniques. We've put a lot of work into our designs; they've won awards and we think they're generally very sound."

Brent asked, "What leads you to look at evaluating the program now? Is there something in particular you're looking for?" Joe responded, "Our designers and staff meteorologists get out in the field occasionally.

They're expected to write up and share a short trip report for every site visit they make. Lately, these field reports have indicated a lack of use of the modules. We're hearing, word-of-mouth, that some offices don't have workstations operating correctly—or forecasters just aren't taking time to go through the material."

Joe continued, “That's why you need to get out in the offices to see how forecasters are making use of the training. I also think that a written survey will help you triangulate your office observations and interviews!”

"It'll be loads of work, but I think it'll be worth it," May agreed. "You don't do a comprehensive evaluation very often; it's best to do it right."

Joe pulled out a summary sheet containing some key information about the comet Program and the modules (see Figure 35 to 1). "Here is the basic information you requested earlier. I've tried to lay it out clearly and succinctly; ask me if you have further questions."

Brent took the sheet and glanced down at the information. "Looks great, Joe. We'll take it from here."

Field Interviews

Over a 30-day period, Brent and May traveled to various comers of the country, observing office conditions and interviewing forecasters. The following field notes present essential findings of those visits.

Brent's Field Notes: Southland Air Force Base

The weather office is located within the main flight terminal building. The forecasting work area is directly connected to the pilot briefing area—not divided or separated in any way. Eight or nine forecasters are assigned to the office, with three or so on-shift at a time. The comet modules have been moved alongside a wall in the hall just outside the forecasting area. close to traffic between rooms. There seems little room to view the modules without being in the way.

Stan, senior airman: The system keeps crashing. The main module that we use is for the Doppler radar. 1 just came out of school; I didn't know how to use Doppler. Pretty much, that's what we use. The other modules are not used.

Celia, master sergeant, forecaster, 13 years forecast experience: I did the Doppler series. It was interesting. They were good. To be honest, if I have to think about it, I cannot remember any other ones that I've done. About three-quarters of the time it doesn't work.

But Stan's in charge. Now we'll get it going. How do we get trained? Around here, it seems as if they just hand us new stuff to do, and it's kind of osmosis-I just learn to do it. You just pick everybody else's brain. Someone comes right out of school and says, "Hey, how come you're not using this?"

So I learn how to use a new tool or database by picking other people's brains. We share ideas.

Comet Program Summary Sheet

Mission. To provide training that shows forecasters how to apply the latest scientific findings to their everyday forecasting work.

Primary clients. Three government agencies: The National Weather Service (N.W.S) and the weather forecasting divisions of the air force (A.W.S) and navy (N.M.O.C). There are more N.W.S personnel to train; they tend to have more formal education than navy and air force forecasters.

comet modules. Nine modules have been developed and disseminated to forecast offices. The modules are funded through a cooperative agreement between the comet Program and its sponsoring government agencies and are automatically sent free of charge to all forecast offices. Each office has a comet workstation, comprised of a videodisc player attached to a computer.

Field support. The comet Program has a telephone helpline, which allows forecasters to call in with questions. Most questions have to do with setup and software problems. The workstations are supposed to be devoted to comet modules, but many offices install other applications, causing occasional conflicts and bugs in the running of the modules.

Figure 35-1

May's Field Notes: Westcoast Forecast Office.

National Weather Service

The Westcoast Forecast Office is located in a beautiful new facility. Like the other N.W.S offices, the hub is a ring of computer workstations, where the forecasters spend their time developing the weather products. This is a large office, with 40 forecasters and interns and a large support staff.

Group Interview

In an interview, the group talked about the N.W.S as a "fair weather service" --that is, they operate on the premise that there will never be a weather emergency and that no one will ever be out sick or take a vacation, so there is no slack at all in the staffing. This is a problem for training, because someone must be pulled off his or her shift to complete the training, which they are loathe to do. "Our ethic is that you shouldn't leave your station. You must maintain a presence at your desk."

Mark, a training officer, stated, "Training is something you do continuously. It does not fit for us to go down the hall and around the corner and carve out time for a module. Training for a forecaster is apprenticeship, and the problem always is a lack of time. Forecasters are individually trained-O.T.J."

Others in the group agreed that there wasn't enough of Mark to go around. A big help would be for the comet ^{} Program to help Mark.

The group made some comments on the modules:

- The advantages of the modules are that they are self-paced, one can go back to get information if reticent to ask someone in the office.

• The hardware is not sufficient.

- We need relatively small and discrete chunks of information (for example, 30 to 50 minutes).

- We really need instantaneous training on essential information.

- Shorter chunks: "We would love to have an hour on icing, an hour on turbulence, and so on."

- The modules need to be application-based-something we can use.

One group member commented: "comet ^{} can help us best by helping us frame local information, so that we can use it to train our own." He gave the example of the 1986 flood in their area. "It is an important weather event to us, and we have knowledge of it in our heads.

How can we best use it to teach our forecasters? We need an electronic three-ring binder."

Brent's Field Notes: Old Town Air Force Base

The forecasting office is located in one of the hangars near the airfield, presently used by a fleet of vehicles. The office appears to be in serious need of remodeling. A divider separates the forecast area from the briefing area. When a pilot comes in for a briefing, the forecaster leaves the forecast area, walks around the divider, and offers the briefing. Five forecasters are available for shift scheduling at this office.

That averages about one forecaster in the office at any given time. Professional interactions are usually limited to shift changes.

Pete, master sergeant, station chief/superintendent, 12 years forecast experience: We all know there's something in the modules to listen to; it's just different from what our focus is. We have never had the luxury of really forecasting the weather. What we've had to do is put together a forecast usually as fast as we can. We're talking to airborne air crews, and it's real easy to get into a reactive mode. We make the forecast quickly.

An hour later, the forecast is already blown. It doesn't take much for things to go downhill quickly—what happens is we have to issue amendments. We have to answer telephones, issue briefings, and deal with a number of clients at once; we don't really have time to do a thorough analysis, as we'd like to do. If it were just forecasting, we could do a better job.

Brent's Field Notes: South River Office.

National Weather Service

About a half-dozen forecasters are working. The comet ^{} workstation is in the main room, partially partitioned yet close to the circular forecasting area. I observe a good amount of collaboration in the office, with forecasters occasionally consulting together.

Jim, torecaster. 18 years forecast experience: I like being able to move around and interact with the material. I like the practices on the material. You're able to learn that way.

The material is good, good techniques. An index would be useful for the modules. That way, let's say I'm dealing with a recurring situation here that doesn't seem to fit any model.

I could try and look up similar cases in the modules or look for ideas that might also account for what's happening. I'd also like an opportunity to raise some of these problems to the scientists and get them to help us.

Blair, lead forecaster, 25 years forecast experience: The location of the comet ^{} workstation is separate from the work area. It's physically inconvenient for me to be walking over to the comet ^{} workstation, work a few minutes, then have to go back to answer a phone. I feel disconnected over there. I get an extra shift about once every two to three months where I can work on a comet ^{} module.

Pat, union steward. 33 years forecast experience on requiring completion of the modules: It's tough to get too regimental in an operational environment. You can't dictate requirements because you have to allow people to be fairly flexible in doing their jobs. From a union perspective, you probably could encourage individuals to complete certain modules, as part of their performance appraisals --you would want to negotiate that with them.

But I don't think it would work as a national policy. I'm sure a national policy like that would require some kind of union approval.

May's Field Notes: Windy Point Office, National Weather Service

The Windy Point office is located in a somewhat remote area near the water with a view of the mountains, The heart of the office is a ring of computer stations. The middle of the ring is a work area with phones and rows of three-ring binders and other kinds of well-thumbed notebooks. Around the perimeter of the office are smaller cubicles for the forecasters and support people, a conference/training room, a break room, storage rooms, and a cubicle devoted to the comet ^{} workstation.

I arrived during a shift change. It reminded me of shift changes in hospitals during morning report or night report. The shifts overlapped, so that the departing forecaster could brief the arriving forecaster. The weather had been quite mild, but the forecasters still took about 30 minutes to exchange information.

In addition to the formal briefing at the change of a shift, I was struck by the amount of informal talk and exchange. It seemed that everyone was talking to someone else most of the time. Forecasters would sit at a station for a few minutes but would inevitably call someone over to look at the screen or would go over to another forecaster's station to compare notes. The norm seems to be collaboration with others and coordination of the work almost constantly.

Later, when I asked them about this coordination, the marine forecaster told me that no "weather product" goes out of the office without several people checking it and concurring. In addition, the marine forecast must be compatible with the public forecast and the aviation forecast, so that the office forecasts are consistent. Several people reminded me that forecasting is inexact, and two good forecasters can look at the same data and come up with different conclusions.

They have to come to a consensus before the forecast goes public. I held an interview with a small group of forecasters; some comments included:

• "Give us something we can use at our workstations."

- "Smaller topics! enthusiastic agreement all around We have two hours to look at modules on a really good day. Thirty-minute to one-hour chunks would be best."

• "The index is missing; include a guide in the modules called 'Dr, Raindrop' office joke."

- "Give a time estimate on each section, so that we can plan when to use it"

- "Give me something I can finish; otherwise, I tend to go to sleep."

- "Give us something we can pull up--just in time-when we need it."

- "Good source of information. I like the self-paced strategy."

Jack, the lead forecaster, wasn't there for the group interview. Nevertheless, the same themes came up in conversation with him. Jack seems to be the grand old man of the office and everyone's mentor. This seems to be the norm in the forecast office-more examples of collaboration in forecasting and the apprenticeship style of training in the office.

One of Jack's main jobs is coaching. He spends about 30 to 45 minutes "on the fly" with a forecaster. His materials are in binders he has collected over the years, including materials from comet ^{} classes, which he thinks are great. He grabbed a binder and enthusiastically flipped through it. He said that every trainer has a binder like that one, full of their favorite maps, handouts, and so on. It is their personal cache, and they refer to them all the time --"the most useful training tool."

Results of the Written Survey

In addition to site visits, the research team administered written surveys to a sample of forecasters across all three agencies. The findings from the surveys were mostly consistent with reports from site visits, focus groups, and interviews. The modules generally have a good word-of-mouth reputation among both users and potential users but a low rate of use.

Module Use

Forecasters were asked which of the nine modules they had used or completed. Module use followed a distinct trend, with older modules used more heavily than newer modules. This declining use rate partly reflects less opportunity by the forecasters to review the modules. Use rates by the N.W.S are substantially higher than for either the air force or the navy.

Module Design

The survey asked forecasters to rate the modules on the appropriateness of the content and the effectiveness of the instructional strategies and media. Overall, the forecasters reported that the modules are well designed. They especially appreciated the multimedia format.

Interactive instructional strategies, and the self-paced learning. Eighty-one percent reported that the comet modules contain information that is useful to them in their jobs. A quarter of them felt that there is too much emphasis on scientific information and not enough on the everyday skills of forecasting. The most frequent complaint was not having enough time to complete the modules.

Pulling It All Together

After four months of gathering information through site visits and written surveys, it was time to make some sense of it. It was dear that the forecasters approved of the module design and sometimes used the modules but that they were not getting full use out of them. Why not?

Brent and May knew that there were issues of time, hardware/software, relevance to the job, accessibility, and compatibility with the routines of work and training. Now it was time to formulate recommendations and help the comet ^{} Program decide its course of action.

Preliminary Analysis Questions

1. Review the codes Brent and May used for categorizing the evaluation data, reflecting factors that seemed to have some effect on the use rate of the modules:

a) T: time issues

b) S/H: software/hardware issues

c) R: relevance to the forecasters' work

d) A: accessibility (Could the forecasters get to the modules to use, and, once they got to them, could they successfully navigate inside the lesson?)

e) C: compatibility (do the modules fit with the work and training routines of the office?) What other codes would you add to this list" Would you modify or delete any? Provide a rationale for your decision.

2. Make a list of barriers to the successful use of the modules. Which of these barriers seem most important? Develop a conceptual scheme for organizing these barriers into a meaningful framework (for example, individual versus group factors; motivational versus information; performance context versus training context) in order to understand the problem better and to develop appropriate interventions.

3. List some possible interventions or changes that would address the identified barriers. Classify interventions in terms of cost (expensive versus inexpensive) and time (short-term versus long-term).

4. Consider the value implications of various interventions. How radical should interventions be? For example, one intervention is to rethink the comet Program's fundamental mission, emphasizing work performance over the application of scientific knowledge. Such a change may have far-reaching consequences for the organization, including some unforeseen negative consequences. How do you decide a best approach?

5. Develop a set of recommendations to present to Joe and the comet Program's staff for improving the effective use of the modules in the field. Be prepared to defend your recommendations with data and reasoning that link your recommendations to likely improvements.

Implications for I.D Practice

1. Discuss the importance of contextual factors in the implementation of instructional design products.

2. What kinds of information can instructional designers collect to determine the level of use of instructional materials by the intended learners?

3. What can instructional designers do to optimize the chances that instructional materials will be used effectively by the intended learners?