Show or Tell, or Don't just do something, stand there!

 I don't think that you have to tell people everything. First off, of course, you can only tell people anything if they're interested in it (or you can tie it to things that they're interested in). But, even then, "show" is often better than "tell". And, sometimes you don't need to do anything because people will figure it out for themselves: all you could do is muddy the waters.

For instance, in the SOA course one of the important distinctions that needs to be made is between a "service" and an "operation." The course has an activity where participants have to come up with a list of services. I was concerned that, in this activity, participants would include operations among their services. I wanted to forestall that.

My first thought, of course, was to provide a formal definition of "service" and "operation." The problem was that I couldn't come up with any set of words that I thought would actually be helpful to participants when they were trying to make the distinction. So I decided to finish up the definition by providing a set of examples. After reviewing those examples, I would present a set services and operations mixed in together and ask participants to mark which were services and which were operations.

This is all standard instructional design: provide the training, demonstrate the training, let participants exercise the training while providing coaching to fix any misunderstandings that are exposed.

But, quite frankly, I didn't think my "training" was providing any value. This activity occurs very early in the course when participants' understanding of "what a service is" is still pretty hazy. Piling on more definitions of more abstract terms was going to make things muddier rather than clearer.

So I considered just going straight to the second list of examples and asking participants to pick out the services. Participants had seen some examples of services at this point in the course and I thought the success rate would be pretty high. This time, the problem was in my examples: They were all pretty lame. And, quite frankly, understanding the examples to determine which were services and which were operations required some business knowledge. It seemed to me that if I took examples from a part of a business that participants weren't familiar with....well, they'd have a hard time figuring out what I was talking about, let alone deciding if I was talking about a service.

Then it occurred to me that the activity I was obsessing about required participants to list services: The participants would be generating their own examples. I could just circulate among the teams during the activity (there would only be three to five teams) and do some personal coaching to resolve any problems. I could leverage examples that meant something to the participants to make the distinction. This was brilliant!

Except that the problem never came up. In class, for whatever reason (perhaps because of the example services shown earlier in the course), the participants never seem to have the problem that I worried so much about. Asked to generate a list of services (and not operations) the teams almost invariably generated a list of services (and not operations).

So far, there's been exactly one team in exactly one class that inadvertently put an operation on their list. I jumped on that opportunity and just coached the hell out of it. I think the team thought I'd lost my mind.

(the subtitle for this post comes from a wonderful book by the choreographer Doris Humphrey: The Art of Making Dances. She has a list of rules in that book that, while they apply to making dances, also apply to many other things, including technical writing and instructional design)
 
Reading or read

• Doghead: A Novel by Morten Ramshead
• Unto a Good Land by Vilhelm Moberg 
• Wellington's Smallest Victory by Peter Hofshröer 

Working on the SOA course

I'm continuing to revise the Service Oriented Architecture course that I'm working on for Learning Tree International. It's funny (and I think that I've said this before): When I write a course I'm convinced that I've created the definitive treatment of the topic and established new standards in course design. When I teach the course for the first time with actual participants, I'm always (!!always!!) astonished to discover just how bad my course is.

For instance, when I went to teach my first version of this course, I discovered that participants were keenly interested in how to integrate this process with their standard project management process--a topic that I hadn't considered but, in retrospect, seems obvious. They also found many of the activities I built into the course either pointless or incomprehensible. I discovered that, even to me, several of the slides were incoherent and that there were many (actually: many, many) places throughout the course where I was adding lecture material that wasn't represented in the course notes.

This is the major benefit of actually teaching a course: You find out (primarily from the participants) just what a jerk you are.

So that led to a pretty significant rewrite of the course. That version (with a few tweaks to correct errors/typos) is what Learning Tree is running now. That course is showing promise--an instructor who was new to the course taught it for the first time and the feedback was good. I've had the opportunity to teach the version once myself and thought it went well (something I didn't feel about any of my teaches of my first version).

Now, there are a couple of caveats here. First: This course, historically, has been tremendously dependent on who was in the room--the mixture of the participants. So a couple of successful teaches doesn't tell us much about how the course will do with all of the people who want to take it. Secondly, when I say the course went well when I taught it, it really only went well for the first two days. I thought the third day sagged.

So I'm working on a revision to address my perception of the issues with the course's third day (and to tidy up the first two days).

One of the other benefits of having a course actually taught is that you get feedback from the instructors. One of the instructors for the course pointed out several issues with the course, for instance, most of which revolved around material now concentrated in the third day. That input formed a lot of my strategy for this revision.

But it's still too early to actually commit to revising the third day. Mostly what I'm going on at this point is my perception of what's wrong. And, let's face it, as the course author I'm going to colour everything pretty rosily. We won't really know what the real issues are with the course until more instructors teach it and we get more feedback from participants.

I remain cautiously optimistic. My initial feeling, when I was offered the opportunity to be the author on this course, was that the task was doomed. I was going to thoroughly enjoy writing the course and would appreciate the money I would get. However, I considered the course fundamentally "unwriteable" and, as a result, would eventually be replaced by another author. After all, just because a topic exists, it doesn't mean that it can be turned into a three day course in the Learning Tree format (e.g. imagine a course on "driving things": cars, boats, trucks, planes, horses, submarines. Just because you can come up with a title--"Driving: A Comprehensive Hands-On Introduction"--it doesn't mean that you can write a course for it). And I had some evidence for this belief: I'm the fourth author to tackle this course.

But some evolution in the field, some changes in the constraints on the course, and--most importantly--the ability to draw on the  knowledge and experience of the previous authors and instructors gave me some help. Three months ago, I was convinced that there was no way to produce a version of this course that would make enough paying customers happy to make Learning Tree happy. Right now, I'm just not sure that it's impossible.Who knows? In a few more teaches, I might be convinced that it might be possible.

Reading or read:

• The Anthologist: A Novel by Nicholson Baker
• The Book of Good Love by Juan Ruiz 
• Robin Hood by James Clarke Holt 
• The Hunger Games by Suzanne Collins
• The Book of Irish Verse: Irish Poetry From the Sixth Century to the Present by John Montague
• Ancient Egypt (Travelers' Guide to the Ancient World)
• Ancient Greece (Travelers' Guide to the Ancient World)
• The Blue Fox by Sjon
• Why We Make Mistakes: How We Look Without Seeing, Forget Things in Seconds, and Are All Pretty Sure We Are Way Above Average by Josepth T. Hallinan
• Jar of Fools: A Picture Story by Jason Lutes

Reassuring the reader, or Doing it right on the wrong side of town

In an earlier post on writing commercial courses (Vogel's 12 Rules for Commercial Course Development) I listed that the three most important questions on the participant or reader's mind were "What do I do first?", "What do I do next?", and "What do I do NOW?". As I've been thinking about it, I think that the third question should really be "How do I know that I'm doing it right?"

When a reader is working in a new area--especially one where the reader has just picked up a lot of new knowledge--the reader is dealing with a lot of uncertainty. Not only is the reader unsure of whether or not they are doing the right thing (and what the correct next thing is), the reader is also concerned about what they just did. In a new field you find it difficult to assess success. Good technical writing provides support for letting the reader re-assure themselves that that they are, in fact, "getting it" and "getting it right."

The official term for this is feedback. However, that doesn't mean you should provide feedback at every step: Feedback, like anything else, is based on the reader's knowledge and expectations.

For instance, if your reader is a programmer and you're outlining some process that involves writing code, it's unnecessary to tell the reader that they must fix any syntax or compile time errors: Programmers know that syntax errors indicate an error. On the other hand, if the programmer is supposed to get an error (or even a warning) at some point in the process then you should advise the reader that that this particular warning or error doesn't represent a problem--precisely because of the reader's expectation that "no errors should occur."

While there is some flexibility in the feedback you provide during the process, it would be an unusual document that wouldn't provide the reader with information about what the final result is. Otherwise, how will the reader know if they got it right (assuming that the process doesn't end with a big flashing message that says "You've succeeded!")?

As always, if you're not sure what to include or what to leave out your first choice is to ask a few representatives of your typical reader. If your representatives are surprised at what they get when they are doing it right then you should include feedback to re-assure the reader; if your reader can easily determine whether they're succeeding or failing without your feedback then you should omit it.

Reading or read

• The Lexicographer's Dilemma: by Jack Lynch
• Intensity by Dean Koontz
• The Klee Universe by Christine Hopfengart, Olivier Berggruen, Peter-Klaus Schuster, and Dieter Scholz
• The Spirit: A Pop-up Graphic Novel by Will Eisner and Bruce Foster
• Every Book Its Reader by Nicholas A. Basbanes

Being Explicit, or I think we need to talk about this

Another busy week--so busy that I didn't get a chance to post last week. In addition to teaching last week, Jan and I drove to Ottawa (wonderful week there: ate at the Sweetgrass Bistro with some new friends and Serena Williamson who gave us a copy of the CD she based her one woman show around, prowled around Byward Market and brought home some unpasteurized Quebec cheese, cool sausages, and duck fat). Learning Tree's marketing arm wanted some input from me about the SOA course I wrote for them (running for the first time in New York near the end of September and then--possibly--in Chicago). In there, I was also commissioning and editing articles for Learning Tree's Management Insight's newsletter and trying to get caught up with my responsibilities with Visual Studio Magazine (who are being incredibly patient). I did a little bit of software development in there, also, but it will be this week before I can really get to write code again. Oh, I almost forgot: I edited an early draft of my son's book on Transformers (there's more than meets the eye, there).

I also was doing some course development work for one of my clients and one of the first reviewers brought out a key point that I hadn't discussed. The material I was working on centred around creating a community in a social networking tool. The reviewer pointed out that users didn't have to enter all the information required to set up the community at one time: They could enter part of the information, save their work, and return at a later date to complete setting up the community. If a user was going to employ this tactic they'll want to leave the community's access level at "private" so no one can see the community until the user does complete it.

I came up with what I thought was a great way to cover this material. I went through the whole "community setup screen" covering all of the options during the "lecture" part of the course material. Then, in the "exercise" part of the course material, I walked the course participant only through the initial steps of setting up their community. After the first few steps, I suggested that the participants save their community and finish it later--pointing out that they should only make the community public when they finished setting it up. This would let the user see the tactic in action rather than just being told about it.

A later reviewer didn't notice my clever way of addressing this tactic and asked if we intended to cover it. We decided to insert a new bullet in the lecture portion of the course that explicitly called out the tactic. And, you know, I'm not sure that's a bad thing. I think that allowing the course participant to see, discover, and experience things is very valuable. But I also think that "saying things out loud" is important to, either before or after the experience. I know that I don't really remember anything until I see it in print (and I also recognize that may just be my personal kink).

Looking back, I can see that I'm moving more towards calling out material after giving the participant an opportunity to use it: Let the participant have the experience and then step in afterwords to help the participant make sense of what just happened. But the calling out part is still important I suspect that I might be omitting that.

And, by the way: My client's client (who we're building this course material for) has an amazing style guide. Fortunately, I'm not expected to memorize it because there's a dedicated reviewer who either brings my written material in line with the style guide or points out where I have to modify my text to be in line with the guide. The guide covers everything (for instance, using "because" instead of "since" unless referring to an ordered set of events). I suspect that I could start implementing some of the style guide when I write but I'm not sure that having the reviewer apply it isn't the most efficient use of everyone's time.

Reading or read

• Letters to Juliet: Celebrating Shakespeare's Greatest Heroine, the Magical City of Verona, and the Power of Love by Lise Friedman and Ceil Friedman
• The Janissary Tree: A Novel by Jason Goodwin
• Jackson Pollock by Ellen G. Landau

Building a Visual Vocabulary, Or Drawing in readers

I'm presently rewriting a course with a large number of graphics in it...which I'm replacing. I'm not sure that if it isn't just ego driving me because, quite frankly, the original graphics were good. The original graphics were very structural: generally speaking, a set of nested boxes with lines--which sounds dismissive and I don't mean to be. Boxes-with-lines are excellent tools when you're trying, for instance, to show people the structure of concepts (e.g. organizational charts, which show a very abstract concept: relationships). And that's what these graphics did and did very well. There was at least one box-with-lines graphic that I left in. In fact, I repeated it three or four times showing it in bits and pieces (and I simplified its first appearance).

So why am I replacing the existing graphic? I looked at what I was replacing them with: By and large I was putting in pictures of people, computers, scrolls--any physical analog that I could come up with. I realized that I was trying to do two things.

First, I was providing physical representations of material that I was concerned was going to be too abstract for our participants. This is a business course around software architecture to I wanted to suggest that the course material is very practical, very realistic. In addition, many of the course participants are business/management types rather than programmers: I wanted to link the topics we were talking about to things they would recognize from their world.

But I also realized that I wanted to create distinctive graphics that participants would remember and associate with concepts. The problem with box-and-line graphics is that all boxes and lines look alike (only colour and size provide any variation). I was replacing labeled boxes with (I hope) memorable images...and then repeating those images on many pages. I realized that I was building a vocabulary of images that participants would come to recognize. With a vocabulary built in early parts of the course, I could mix and match those images later in the course to explain new things (very much the way we mix and match known words to create new explanations).

This vocabulary, in addition to providing a parallel visual explanation to the word-driven explanations, also helps tie the course together. When I introduced a concept on one page, I would also introduce a corresponding image. When the concept cropped up on a later page, I also added the image. To (mis)quote T.S. Eliot, I was creating a "imagictive correlative" for the ideas in the course that I could use to reinforce the course's points and coherence.

I had another more nefarious point to using these images. At the end of each chapter there is a chapter summary that repeats the key points from the course. On that slide I also repeat the items from my graphic vocabulary that were introduced in that chapter (in miniature). At the end of the course I have an (graphic only and unreadable) slide that repeats all of the graphic vocabulary from the whole course (still in miniature). If nothing else, this slide--put it up just before the participants fill out their evals--sends the message that we've sure covered a heck of a lot of material.

That still leaves, of course, the question of whether or not the participants think it's valuable material.

Reading or read:

• The Accordionist's Son: A Novel by Bernardo Atxaga
• Tomorrow in the Battle Think On Me by Javier Marias
• The Complete Poems, 1927-1979 by Elizabeth Bishop
• Astrophel And Stella by Philip Sidney
• Jonathan Strange & Mr Norrell by Susanna Clarke
  

Vogel's 12 Rules for Commercial Course Development, Or Trolling for evals

1. People come to courses to get solutions for their problems—not to learn some technique, tool, or technology. The technique/tool/technology is just a means to an end. Which means: First you have to know the audience's problems. Then you have to know the field.
   
   
2. Set expectations appropriately: At the start of the course tell participants what problems you're going to solve. Encourage people who don't have those problems to leave. Now. Use this time to also get the participants to tell you what they care about.
   
   
3. Start in Chapter 2: Participants aren't interested in the history of the field, the underlying principles. Skip that. Start solving their problems right after setting expectations. If there are principles that will help them organize their problem solving skills or develop new skills keep it to one slide now and then. And consider describing those principles only after you've shown them how to solve their problems.
   
   
4. If you're not telling the participants how to do something or how to do it better…why are you taking up the participant's time? If you're not telling them how to do something that the participants care about you'd better either make them care (by tying it to something that the participants do care about) or drop it.
   
   
5. JITT/TIA: Just-in-Time Training—Don't tell the participants anything until just before they need it/Training in Action—as soon as you tell the participants something, give them a chance to do it. Right now.
   
   
6. Examples matter. Make sure that the examples reflect the participant's world with their problems. An immersive case study that where the participants have a role is best. A consistent case study throughout the course is more than "good enough." But if there's a problem that you're going to address that won't fit into a case study with the other problems, don't force it: make it a standalone exercise.
   
   
7. Respect your audience: They know a lot of stuff. At the very least they know what their problems are (though they may need help codifying that knowledge) and what their expectations are (as before).
   
   
8. Don't do trade-offs: something for this group, something for that group. Solve the problems that everyone in the audience shares.
   
   
9. The three biggest questions on the audience's mind are: "What do I do first?", "What do I do next?", and "What do I do NOW?" Give them roadmaps, decision trees, guides—tell them what to do when and when to do it.
   
   
10. Use graphics to explain. If you're an instructor or course author then you're probably word oriented. Most of the population is picture oriented—talk to them with graphics. Don't use visual puns: a word appears on the slide so you put a picture of it on the page ("software bus" = picture of a double-decker bus). Use flowcharts, diagrams, arrows, whatever to say in pictures what the slide says in words.
    
    
11. Throughout the course show progress. Advertise what you all have accomplished—otherwise participants may not notice.
    
    
12. Have a finish that delivers on the audience's goal. Then hand out the evaluation forms.

Reading or read

• 52 Ways of Looking at a Poem: A Poem for Every Week of the Year by Ruth Padel
• Shortcomings by Adrian Tomine

Exercises count, or "Welcome home"

I figured it out: since April 23rd and May 23rd,  I slept in my own bed for seven nights--and all of those were in one continuous string between getting back from Spain (with a week in Toronto) and leaving on our "east coast road trip (New York to Alexandria, teaching the "Strategic Thinking for Operational Managers" course for the first time in the US for Learning Tree). Since I got back from Spain I've been trying to finish up the revision of my ASP.NET course and get back on deadline with Visual Studio Magazine.

Doing this revision of the course gave me a chance to look back at the way I've written exercises in the courses that I've written for Learning Tree. In the first course (way back in the late 90s) my goal in the exercises was to explain some technology...well, actually "to show off" some technology. Basically, the exercises proved that I hadn't lied in the lecture portion of the exercises. I bet that my courses were 75% lecture because, after all, if I said it then the participant's learned it, right? The exercises were also very "standalone": since all I was doing was demonstrating that I hadn't lied about the technology there was no need for the exercises to actually deliver a working application.

Learning Tree eventually gave me a book on Active Learning which emphasized that most people only really learned what they did, not what they were told. Learning Tree eventually advanced on this to say that it was silly to pack a course with knowledge that people only remembered 30% of it--you were better off to structure a course where people would remember 80% or 90% of the material even if you only covered 60% of the material (you do the math). About the same time, the company also started pushing us to have our exercises be built around a case study that reflected the real world: a "case study."

Here's my problem: In a technical course: I know the stuff in the course, the people taking the course didn't. What would "Active Learning" look like in this situation? I should talk, the participants should sit there listening and, eventually, they could do one of my exercises. The 'doing' part would be my exercises...which were, at best, 25% of the course time.

I eventually realized that what people did know was their problems and that was what was driving them into the course. One way I could incorporate activity into the course was to let participants talk about their problems and what they wanted to do. I could then base the course content around the problems of the people attending the course. I would drive the course forward by drawing a bright line between the participants' problems and the technology in the course.

If I wanted to do that, I also needed to provide more opportunities to do stuff in the course. The problem is that, with setting up to do a programming exercise and cleaning up afterwards, an exercise includes 5 minutes of unavoidable "overhead". I had the idea that, to make an exercise worthwhile, I needed an exercise to be "at least" 20 minutes in length. I was also still "lecture focused" so I felt that I needed to lecture through all the material that "hung together" before interrupting the lecture for an exercise (remember that 5 minutes of overhead?).

Putting those two things together resulted in 'death march' exercises that kept the participants banging away at their keyboards for 30 to 45 minutes at a stretch, often in exercises that did four or five different things because an exercise had to incorporate all of the topics covered in the previous lecture.

For the last couple of years, I've been moving in a different direction: JITT (Just-In-Time Training). My goal is a class where participants spend most (60% or better) of their time building the case study. The case study is interrupted by "just enough" training (lecture) to prepare the participants to do the next thing in the case study. Once we've covered enough material to do something in the case study, we implement that.

This sometimes means that the new version of the course may have one slide of lecture material followed by a slide with instructions on how to advance the case study. Sometimes the course has five or six slides that cover how to do the "next thing" and the participants do an exercise that takes 15 minutes (with overhead, I don't believe an exercise that requires more than a slide to describe will take less than 15 minutes). About once or twice a day, there may be an exercise that takes 20 minutes.

Each lecture portion takes somewhere between 3 and 15 minutes and leads immediately to extending the case study. Sometimes, following an exercise, the course covers material related to the stuff covered in the previous exercise that isn't part of the case study.

I like this new structure...but then, I would. It'll be interesting to see how (a) the paying customers and (b) the poor SOBs that have to teach the course feel.

Reading or read

• This Earth of Mankind (Buru Quartet) by Pramoedya Ananta Toer
• Twenty poems by Vicente Aleixandre 
• The Gospel According to Jesus Christ by Jose Saramgo
• Blindness by Jose Saramago 
• Mythology and Folklore in South-East Asia by Jan Knappert and Graham Saunders 
• Dragon's Blood: The Pit Dragon Chronicles by Jane Yolen
• The Shadow of Malabron: The Perilous Realm: Book One by Thomas Wharton
• Three-Cornered Hat (Everyman's Library (Paper)) by Pedro Antonio De Alarcon 
• Lazarillo de Tormes and The Swindler: Two Spanish Picaresque Novels by Anonymous and Michael Alpert

Learning from Instructional Design, Or Stealing from the best

While I'm here in Spain I'm grasping the opportunity to take some e-learning from Daryl L. Sink & Associates (they're the organization that Learning Tree partnered with to develop Learning Tree's Reality Plus courses). Since I do some instructional design for a couple of my clients, over the last 12 to 18 months I've been concentrating on increasing my knowledge in the field (my experience tends to take care of itself)--this training is part of that process.

The course that I'm taking from DSA is their "Course Development Workshop". I've been meaning to attend one of their "in-person" presentations of this course but work, so far, has always interfered. I finally decided that if I was going to do it at all, I'd best take advantage of their distance education, web-based version. I'm picking up lots of good stuff.

One of the tools that the course promotes is, I think, going to be especially useful to me and not only when I'm doing instructional design but also when I'm doing technical writing. In the course it's referred to as
   M - A = D: Master - Actual = Difference

The tool consists of a three column table that lists:

• The performance of a "Master" (not a "superstar" because it costs too much to train people to be superstars)
• The performance you currently have (the "actual")
• The difference

So, for technical writing, you might have this when comparing a new hire to a competent technical writier:

Master	Actual	Difference
Adjusts content and format for different audiences	Writes what he or she would want to read	Does not understand the demands of different audiences

It's actually a four column tool because DSA uses this in conjunction with a second table that lists the cause for each difference (e.g. knowledge, attitude, lack of essential tool).

In applying this to technical writing, a tweaked version of this table could be very useful in defining the knowledge domain for a document--one of the critical issues in designing a document. This table could go a long way to addressing the knowledge domain question: "What needs to be addressed in a particular document for a particular audience?" You only need, after all, to address the differences.

Of course, defining the knowledge domain is just the first step. Once you have defined the difference, the next step is determining how to address it--what content should be included to help the audience move from their current state to the required state?

It seems to me a small step to modify DSA's table to address the difference in knowledge/information between the required level of knowledge in the domain vs. the level of knowledge in document's audience. And it's an equally small step to extend the tool to include a tactic for addressing that difference. If I was writing a textbook on technical writing, I might create this version of the DSA table as part of designing the textbook's content:

Required	Actual	Tactic
Specify the different requirements of various audiences for the same knowledge domain	Treats all audiences as having the same needs as he or she does	Provide examples of different audience demands for the same knowledge domain. Contrast with what the author finds valuable

I'm finding this tool useful (I tried it out as part of revising a course proposal I put together for one of my clients) but obviously it needs work. Just because there is a difference and you have a way of addressing it, you don't necessarily have to address that difference. A prioritization step is also needed (DSA divides potential content up into three categories: Could, Should, and Must). It also seems to me that you need both tables: one to help you accurately define the difference and one to help you define a way of addressing it. Perhaps a five column version would work best: Master, Actual, Difference, Cause, Tactic? Too unwieldy?

And another critical point: DSA's original (M - A = D) spells out a word. My version (R - A = T) spells out an unfortunate word. And, as we all know, the success of any tool is dependent on its acronym.

Reading or read

• Stars in My Pocket Like Grains of Sand by Samuel R. Delany
• Soldiers of Salamis: A Novel by Javier Cercas

Writing Questions, or Stop me before I query again--Please!

I got swamped by work and haven't posted for two weeks (or read much, either). In between work, Jan and I also moved my youngest son home from college and celebrated Valentine's day (I know: whine, whine, whine). Over those two weeks, I did do some some software development but most of my time was taken up with some instructional design.

Actually, with a small part of instructional design: Writing multiple choice and short answer questions.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARGGGGGGGGGGGGH!!!

Don't get me wrong: It's not that I don't enjoy crafting multiple choice questions. But after generating three or four hundred questions my brain is starting to go numb. Fortunately, my client got the original deadline moved and I got a break over the weekend (hence, this post). But there's still about 100 questions left to generate and I'll be back at that tomorrow. You may be able to hear the screams.

But all of this work, of course, got me to thinking about effective testing. The first issue to ask of any test is whether the questions address the objectives of the training. It turns out that (at least for me) this is a considerably harder goal to achieve when generating questions after the content is created by someone else.

In the ideal instructional design process, you create your objectives for the training, create the tests that will prove that the participant has achieved those objectives, and (finally) write the content/create the experience that will allow the participant to pass the test.

In real life, I don't do it that way. When developing training material, as I generate the content I get smarter about what I'm writing. As a result, I often go back and modify, extend, or even drop some objectives. Rather than re-do the questions/tests after generating the content, based on the final set of objectives, I do most of the work on the questions/tests at the end of the process. I still do a pretty good job of generating tests in that process.

However, I don't do NEARLY as well when I come in at the end of the process when someone else has generated the content. Part of the problem, in generating questions for someone else's content, I feel obligated to provide questions for content, even if the content isn't tied into one of he objectives. My assumption is that students will feel obliged to study every page in the textbook so it's my obligation to provide a question to justify the strudent's effort. This is wrong-headed of course: just because it's on the page, it doesn't mean that i have to test for it unless it addresses the objectives of the course. I think that I identify too closely with the student.

The nice thing about working for other people is that I get feedback. My client's ultimate customer (the person my client is producing the instructional material for), it turns out, has a style guide. Some of the restrictions that the customer had in their style guide were:

• No negative questions ("which of the following is not")
• No "None of the above/All of the above"
• No questions where the answer finishes the question by providing the end of the sentence

The only one of these dictums that I had a problem adjusting to was the ban on "None/All of the above." My concern with "All/None of the above" are those tests when the only time this answer appears is when it's the right answer (i.e. when "All of the above" is always the right answer). Generally speaking, if I use "All of the above" in a test, I will ensure that the answer appears two or three more times as the wrong answer than as the right answer. Other than that concern, a blanket objection to "None/All of the above" seemed odd to me. But, hey, it's a style guide issue: I doubt that the test takers care one way or another.

I noticed that the customer's review of my questions sometimes fell into what I call the "knowledge fallacy." Often, when reviewing multiple choice questions where we know the answer we critique questions by simultaneously assuming that the student knows the answer and doesn't know the answer--that the student knows as much as we do while still being a student.

For instance, on one question with four potential answers, I had two of the answers in one format (e.g. "verb-noun") and two in a second format (just a noun). The correct answer was in the first group (answer "b"). The reviewer commented that, because the second set of answers ("c" and "d") were in a format different from the correct answer, the student could ignore the second set of answers.

This is true: but only true if you know that answer b is correct. If you don't know which answer is the correct one, the realization that the answers are in two different formats does you no good; if you do know the right answer....well, then, it doesn't matter if the answers fall into two groups.


Last note: In the same way that I think that the most important person on the writing team is a representitive of the audience, I often think that the real test of a multiple choice test would be to have someone completely ignorant of the subject take it. If that person does better than just guessing (e.g. if, on a multiple choice test with 4 answers for every question, the test-taker gets better than 25%) then you have a problem. But that leads to another question: If, under those circumstances, the test-taker does worse than 25% does that mean that you have a really well-designed test or a really unfair one?

Reading or read

• The Little Man: Short Strips, 1980-1995 by Chester Brown
• Get a Life by Philippe Dupuy and Charles Berberian
• The Book of Disquiet by Fernando Pessoa and Richard Zenith
• Second Space: New Poems by Czeslaw Milosz