Wiki Evaluation Methods

(Updates at the end of the post) I’m trying to put together some criteria for the summative evaluation of wikis as a learning technology in higher-ed courses. Perhaps you can take a look at what I have just brainstormed and provide some suggestions.

First, a quick search for materials on evaluating wikis in educational
settings produced only two substantive resources:

Do you know of any
others?

What I really want to do is to put together an instrument that learners
can respond to quickly and that will generate some useful data on how
the wiki was used in the classroom (without concern for the subject
matter of the class).

Here’s what I’ve come up with so far:

GENERAL

  • wiki type (use Wiki Tipster taxonomy)
  • number and type of users

QUANTITATIVE

  • Volume:
    1. how many pages were created?
    2. how many edits were made?
    3. how was the creation of pages and edits distributed throughout the semester (number of new pages and edits created per week)?
  • Page Activity:
    1. which pages were edited the most?
    2. which pages were edited the least?
    3. what was the average number of times a page was edited?
  • Collaboration Index:
    1. what was the average number of users that edited a page?
    2. which pages were edited by the most/least number of users?
  • Participation Index:
    • how
      many edits and new pages are attributable to n segment of
      the class?
  • Additional questions (Likert-scale questions):
    1. I have used wikis before.
    2. I feel I was an active contributor to the wiki.
    3. I feel that all members of the class contributed to the wiki proportionately.

QUALITATIVE:

  • what pages or sections of the wiki did you find most valuable? why?
  • what pages or sections of the wiki did you find least valuable? why?
  • what obstacles did you encounter during your participation in this wiki? were those obstacles overcome?
  • do you feel the wiki contributed to the learning experience? how so?

Sample data

The above would allow us to tell a story along these lines:

“The wiki in question can be classified as a Group – Terminal – Organize/Classify wiki. There were 30 users (25 students, 2 faculty and 3 TAs). In total, there were 67 pages created and 1,763 edits made (see attached chart for breakdown of page creation and edits by week). The most edited page was FinalAssignment . The least edited page was Pizza. Pages in this wiki were edited an average of 3.2 times. Each page was edited by an average of 0.86 users. The page edited by the most number of users was Pasta, edited by 15 users. The page edited by the least number of users was Pizza, edited by 2 users. Ten percent (10%) of the class was responsible for 60% of the edits
and 40% of the new pages. Only 5% of users said they had used wikis before. Eighty percent (80%) of users feel they were active contributors, but only 20% feel the class contributed to the wiki proportionately.” [a summary of the qualitative data could then follow]

Certainly, this would not be the whole story behind the use of a wiki, but it would provide a snapshot of the experience–specially when comparing different wikis across different courses. In other words, the purpose of the survey is to serve as the launching pad for more detailed research.

Here are my questions:

  • what other questions would you ask? (keeping in mind that we want the least number of questions but the most valuable data)
  • how easy or hard would it be to mine the quantitative data from the wiki’s logs?
  • has anybody else done similar things that I can look at?

Thanks.

UPDATE

This is what I’ve found in terms of tools to mine data in MediaWiki:

Other Resources

(many of the above can be found in Wikimedia’s Toolserver)

Also, see Jonah’s comment below.

2 thoughts on “Wiki Evaluation Methods

  1. Jonah

    Hi Ulises,

    We have been thinking about this a bit at the Center, and have even run a few surveys that partially address this question.

    My favorite questions on this survey are the ones about students showing their work to others outside the class, and visual diagram which asks the students to describe their impression of the connections within the wiki.

    Do you know about IBM’s history flow tool? We have been playing around w/ it a bit, but honestly no is sure what to make of the information it presents.

    Far more promising, I think, are tools which can help in visualizing the connections across the entire wiki, not just the texture of the conversation on a single page.

    I am thinking about tools like the delicious visualization tools, but constrained to a particular wiki instance. I know of one plugin like this for mediawiki, but its nothing special.

    Would be very interesting to brainstorm more on this – Curious to hear your thoughts on improving our surveys… what questions would you ask?

    Best,
    /Jonah

  2. Pingback: First steps to classification « Colligo: Reflections of a Learning Technologist

Leave a Reply