(Updates at the end of the post) I’m trying to put together some criteria for the summative evaluation of wikis as a learning technology in higher-ed courses. Perhaps you can take a look at what I have just brainstormed and provide some suggestions.
First, a quick search for materials on evaluating wikis in educational
settings produced only two substantive resources:
- Wiki Pedagogy, Dossiers technopédagogiques
- Wiki Brainstorming and Problems with Wiki Based Collaboration (I’m having some problems retrieving the PDF linked to in this page)
Do you know of any
What I really want to do is to put together an instrument that learners
can respond to quickly and that will generate some useful data on how
the wiki was used in the classroom (without concern for the subject
matter of the class).
Here’s what I’ve come up with so far:
- wiki type (use Wiki Tipster taxonomy)
- number and type of users
- how many pages were created?
- how many edits were made?
- how was the creation of pages and edits distributed throughout the semester (number of new pages and edits created per week)?
- Page Activity:
- which pages were edited the most?
- which pages were edited the least?
- what was the average number of times a page was edited?
- Collaboration Index:
- what was the average number of users that edited a page?
- which pages were edited by the most/least number of users?
- Participation Index:
many edits and new pages are attributable to n segment of
- Additional questions (Likert-scale questions):
- I have used wikis before.
- I feel I was an active contributor to the wiki.
- I feel that all members of the class contributed to the wiki proportionately.
- what pages or sections of the wiki did you find most valuable? why?
- what pages or sections of the wiki did you find least valuable? why?
- what obstacles did you encounter during your participation in this wiki? were those obstacles overcome?
- do you feel the wiki contributed to the learning experience? how so?
The above would allow us to tell a story along these lines:
“The wiki in question can be classified as a Group – Terminal – Organize/Classify wiki. There were 30 users (25 students, 2 faculty and 3 TAs). In total, there were 67 pages created and 1,763 edits made (see attached chart for breakdown of page creation and edits by week). The most edited page was FinalAssignment . The least edited page was Pizza. Pages in this wiki were edited an average of 3.2 times. Each page was edited by an average of 0.86 users. The page edited by the most number of users was Pasta, edited by 15 users. The page edited by the least number of users was Pizza, edited by 2 users. Ten percent (10%) of the class was responsible for 60% of the edits
and 40% of the new pages. Only 5% of users said they had used wikis before. Eighty percent (80%) of users feel they were active contributors, but only 20% feel the class contributed to the wiki proportionately.” [a summary of the qualitative data could then follow]
Certainly, this would not be the whole story behind the use of a wiki, but it would provide a snapshot of the experience–specially when comparing different wikis across different courses. In other words, the purpose of the survey is to serve as the launching pad for more detailed research.
Here are my questions:
- what other questions would you ask? (keeping in mind that we want the least number of questions but the most valuable data)
- how easy or hard would it be to mine the quantitative data from the wiki’s logs?
- has anybody else done similar things that I can look at?
This is what I’ve found in terms of tools to mine data in MediaWiki:
- Wikipedia User Edit Counter
- AKA’s wikipedia tools: look at RCHistStat (displays statistical overview of the recent changes in the selected Wikipedia)
- List of Wikipedians by number of edits : includes Python script
(many of the above can be found in Wikimedia’s Toolserver)
- MediaWiki special pages: see Statistics page, Newpages, Popularpages
- Wikimedia statistics
- Wikimedia wikistats
Also, see Jonah’s comment below.