First, a quick search for materials on evaluating wikis in educational
settings produced only two substantive resources:

• Wiki Pedagogy, Dossiers technopédagogiques
• Wiki Brainstorming and Problems with Wiki Based Collaboration (I’m having some problems retrieving the PDF linked to in this page)

Do you know of any
others?

What I really want to do is to put together an instrument that learners
can respond to quickly and that will generate some useful data on how
the wiki was used in the classroom (without concern for the subject
matter of the class).

Here’s what I’ve come up with so far:

GENERAL

• wiki type (use Wiki Tipster taxonomy)
• number and type of users

QUANTITATIVE

• Volume:
  1. how many pages were created?
  2. how many edits were made?
  3. how was the creation of pages and edits distributed throughout the semester (number of new pages and edits created per week)?
• Page Activity:
  1. which pages were edited the most?
  2. which pages were edited the least?
  3. what was the average number of times a page was edited?
• Collaboration Index:
  1. what was the average number of users that edited a page?
  2. which pages were edited by the most/least number of users?
• Participation Index:
  • how
    many edits and new pages are attributable to n segment of
    the class?
• Additional questions (Likert-scale questions):
  1. I have used wikis before.
  2. I feel I was an active contributor to the wiki.
  3. I feel that all members of the class contributed to the wiki proportionately.

QUALITATIVE:

• what pages or sections of the wiki did you find most valuable? why?
• what pages or sections of the wiki did you find least valuable? why?
• what obstacles did you encounter during your participation in this wiki? were those obstacles overcome?
• do you feel the wiki contributed to the learning experience? how so?

Sample data

The above would allow us to tell a story along these lines:

“The wiki in question can be classified as a Group – Terminal – Organize/Classify wiki. There were 30 users (25 students, 2 faculty and 3 TAs). In total, there were 67 pages created and 1,763 edits made (see attached chart for breakdown of page creation and edits by week). The most edited page was FinalAssignment . The least edited page was Pizza. Pages in this wiki were edited an average of 3.2 times. Each page was edited by an average of 0.86 users. The page edited by the most number of users was Pasta, edited by 15 users. The page edited by the least number of users was Pizza, edited by 2 users. Ten percent (10%) of the class was responsible for 60% of the edits
and 40% of the new pages. Only 5% of users said they had used wikis before. Eighty percent (80%) of users feel they were active contributors, but only 20% feel the class contributed to the wiki proportionately.” [a summary of the qualitative data could then follow]

Certainly, this would not be the whole story behind the use of a wiki, but it would provide a snapshot of the experience–specially when comparing different wikis across different courses. In other words, the purpose of the survey is to serve as the launching pad for more detailed research.

Here are my questions:

• what other questions would you ask? (keeping in mind that we want the least number of questions but the most valuable data)
• how easy or hard would it be to mine the quantitative data from the wiki’s logs?
• has anybody else done similar things that I can look at?

Thanks.

UPDATE

This is what I’ve found in terms of tools to mine data in MediaWiki:

Other Resources

• Wikipedia User Edit Counter
• AKA’s wikipedia tools: look at RCHistStat (displays statistical overview of the recent changes in the selected Wikipedia)
• List of Wikipedians by number of edits : includes Python script
• Interiot/Tool

(many of the above can be found in Wikimedia’s Toolserver)

• MediaWiki special pages: see Statistics page, Newpages, Popularpages
• Wikimedia statistics
• Wikimedia wikistats

Also, see Jonah’s comment below.

What goes on in Wikipedia? What is the “text” about? Which institutions determine this? Wikipedia strives to be an authentic encyclopedia. This means that the content consists of encyclopedic articles (alphabetical arrangement is inconsequential, given the non-linear access nature of the web). The text, in other words, is a compendium of human knowledge written in an accessible format. Although there is considerable buzz about the ‘openness’ of Wikipedia, how the text is written (its ‘tone,’ ‘voice’ or ‘style,’ in common terms) is established by the founders of Wikipedia, a non-profit organization called the Wikimedia Foundation. The Wikimedia Foundation has established policies regarding the ‘tone’ of Wikipedia: “Wikipedia requires that its contributors observe a “neutral point of view” when writing… If achieved, Wikipedia would not be written from a single “objective” point of view, but rather fairly present all views on an issue, attributed to their adherents in a neutral way.” (retrieved from http://en.wikipedia.org/wiki/Wikipedia on April 19, 2005). A detailed look at how ‘neutrality’ is constructed in this context, and to what degree Wikipedia articles reflect or fail to reflect this neutrality is beyond the scope of this paper, but an issue that deserves being raised nonetheless. Suffice it to say for the moment that the discourse in Wikipedia is anything but open or haphazard, but intentionally shaped by the Wikimedia Foundation’s goals of creating an ‘authentic’ encyclopedia, and social conventions of what an encyclopedia should ‘sound’ like.

While what goes on in Wikipedia is an interesting story, how it is happening is equally so, at least from a technological perspective. The device that allows anyone to manipulate the content of Wikipedia at any time is a web program called a wiki. What makes wikis innovative knowledge management tools is that they save and archive every change made to a web page, so that not only can anyone edit any page, but anyone can also revert back to a previous version if they wish. In this manner, users can create new pages, edit existing pages, completely re-write pages, or revert to older versions of pages at will. Additionally, users can discuss pages or suggest changes. And the technology behind this is fairly simple to use, so that most computer users can learn the skills required to edit Wikipedia articles in minutes.

Who participates in Wikipedia? What different roles do they play? In order to meet its goal of “creat[ing] and distribut[ing] a free encyclopedia of the highest possible quality to every single person on the planet in their own language” (Wales, 2005), Wikipedia relies not on professional editors or writers, but on volunteer authors. According to the Wikipedia site, “authors can be asked to defend or clarify their work, and disputes are readily seen” (retrieved from http://en.wikipedia.org/wiki/Wikipedia on April 19, 2005). The Wikipedia community believes that exposing all content to multiple authors will eventually increase, not decrease its quality (this sentiment is based on an open source philosophy of many programmers approaching the bugs of a software program).

While volunteer authors are responsible for writing all Wikipedia articles, they are by no means its only users. Apart from authors, there are other users that can be classified as browsers, vandals, and administrators. Browsers are users who go to Wikipedia to look up content, but who do not participate in any of the authoring activities. According to the Wikipedia website, the domain receives close to 50 million hits a day. Most of this visits are by browsers. Vandals are functionally similar to authors, except that in their case their intention is not to create useful knowledge, but to destroy it. As mentioned above, wikis make it easy to deal with vandalism by reverting to a previous version of a page. Some types of vandalism are blatant, while others are harder to detect, but both have negative repercussions for the Wikipedia community. Lastly, administrators are volunteers who have accumulated enough trust in the community to be given high-level functions, such as ‘locking’ a page (preventing further changes), deleting or moving pages, expelling users from the community, etc. The following quote encapsulates the distribution of power in Wikipedia: “Its articles are not controlled by any particular user or editorial group, and decision-making on the content and editorial policies of Wikipedia is instead done by consensus and occasionally majority vote, though Jimmy Wales [the founder of Wikipedia] retains final judgment” (retrieved from http://en.wikipedia.org/wiki/Wikipedia on April 19, 2005). As a result, at any given moment the text of a specific article indirectly reflects the application of these levels of decision-making, which become directly visible only in the log of changes to the page, found in the page’s History tab.

What material shape, or mode, does the text take, and how is this mode socially constructed? While visual and audio media are present, Wikipedia’s primary mode is written text, or to be more precise, hypertext. This has a number of important implications. For one, it means that a Wikipedia page, although composed mostly of words, shares many of the characteristics of a screen image identified by Kress (2003): its reading is approached in blocks (such as separate blocks for navigation, article content, discussion, and editing), and there is no single point of entry to the page. Kress’ argument about new forms of hypertextual reading applies perfectly: “Reading is the imposing of the reader’s order on this entity, and order which, while of course responding to what is there, derives from criteria of the reader’s interest, disposition and desire: (2003, p 138). In other words, a browser, an author, an administrator or a vandal will approach a Wikipedia page in very different ways, scanning and making sense of a page according to their needs and desires.

This brings us to another important implication of the hypertextual mode of Wikipedia: that every page is the result of social dynamics that are never completed (even if a page is locked at some point, it can be unlocked later). “Wikipedia does not declare any article finished” (retrieved from http://en.wikipedia.org/wiki/Wikipedia on April 19, 2005). This means that we are dealing with a ‘live’ mode, a mode that refuses to be fixed, and that is ever sensitive to social influences. Critics of Wikipedia claim that its content is worthless because it is not subjected to any form of authoritative review. For those who believe in the fixity of knowledge, this might be a reassuring argument. But I believe that the new mode of text embodied by Wikipedia can teach new generations about the responsibilities of social collaboration, the act of critical reading (applied even to Reference materials), and the permanently unfinished state of human knowledge.

References

Kress, G. R. (2003). Literacy in the new media age. London: Routledge.

Wales, J. (2005). Wikipedia is an encyclopedia. Retrieved from http://mail.wikipedia.org/pipermail/wikipedia-l/2005-March/038102.html on April 19, 2005.

In Literacy in the New Media Age, Gunther Kress (2003) argues that the image is displacing writing as the main resource for communication in Western societies. This does not mean, obviously, that writing is disappearing. But as Kress would put it, the world told is increasingly being replaced by the world shown—with all the social and cultural changes that this entails. An analysis of the considerations that go into how text is arranged and displayed on a web page, for example, suggests that Kress is correct in pointing out that writing is being treated more and more as a visual entity. No longer is the unbroken, uniform, left to right flow of text the norm. Instead, in the new media especially, text plays a secondary role to images, meandering around them, adjusting its visual properties (such as font, size and color) to fit the overall layout of the screen, and signaling different entry points into the non-linear flow of meaning. Think of a blog: although it is basically full of text, how you make sense of the blog has a lot to do with the visual arrangement of the text: how posts are organized, what information goes on the sidebars, where the link to the RSS feed is usually located, etc. Interacting with a blog has as much to do with the reading of the text as with the watching of the screen.

But is this ‘visual turn’ as totalizing as Kress suggests? Will all forms of writing in the new media age become increasingly organized according to the logic of the image? We must keep in mind that (because of bandwidth capacity) the internet was first and foremost a textual medium. This means that probably for a long time to come, conceptualizing innovative ways to do things online will involve the manipulation of textual symbols in some form or another. Consider the case of wikis. While their use and popularity remains relatively limited, they are a clear example of a new media technology grounded as much on the workings of text as on those of the image.

A wiki, for those unfamiliar with the tool, is a collection of web pages that can be edited by anyone using nothing more than a web browser. Technically, a wiki is a piece of software residing on a server that facilitates the creation, editing and hyperlinking of text files, which are displayed as online pages (it is also possible to embed images on the text). Two of the most salient features of a wiki are that the editing is open to anyone (hence the process is referred to as open editing), and that the creation, formatting and interlinking of pages requires very simple syntax. For example, to create a link to a new or an existing page, a user needs only to enclose the title of the page in brackets. While the tool itself is not very technologically sophisticated (hence part of its beauty), the collaborative and democratic affordances of wikis are giving us plenty to think about.

For one thing, wikis are challenging and redefining our notions of how text itself works. While hypertext changed our understanding of textual linearity and flow, wikis are changing our ideas about the ‘social’ life of text. Of course, in a way writing has always been social. Even when the writing is done by an isolated author, that author is nonetheless responding to social conventions and interests. No one writes in a vacuum. But wikis make writing social by allowing for the direct, continuous manipulation of a text by a group of people.

The idea of collaborative writing is not entirely new. Margaret Fleming’s work (1988), for example, recognizes that many professional occupations require that people cooperate in the writing of texts. However, such approaches have always assumed a group of people who come face-to-face to plan and discuss their actions. A wiki is different in that authors don’t have to be in the same physical space, don’t have to have a previous relationship with each other, and don’t need to plan their actions in any way. The wiki not only captures the content, but also the process; or rather, the wiki is the content and the process.

Wikis engender a new form of literacy: a social literacy. The word literacy is used loosely these days to define all sorts of competences (viz. visual literacy, musical literacy, computer literacy, and so on). Here, I am using Kress’ more exact definition: literacy as the “term which refers to (the knowledge of) the use of the resource of writing” (2003, p. 24). This definition makes it possible to separate literacy from other resources (such as speech), as well as other ‘metaphorical extensions’ of the concept (such as musical literacy, cultural literacy, etc.). The intention is to re-focus literacy exclusively on writing. Thus, social literacy (as I am using the term) is not a metaphorical extension of the concept and does not refer to the skills necessary to perform in society, but to the use of the resource of writing in social contexts. Social literacy amounts to the textual practices not (as has been true so far) of a single author, but of multiple and simultaneous authors. Wikis make social literacy apparent by allowing us to witness the evolution of text in time, and evolution that reflects the decisions not of a single individual, but of a community.

Brian Lamb (2004) summarizes some of the characteristics of wiki writing as follows:

Content is ego-less, time-less, and never finished. Anonymity is not required but is common. With open editing, a page can have multiple contributors, and notions of page “authorship” and “ownership” can be radically altered. Content “cloning” across wikis—sometimes referred to in non-wiki circles as “plagiarism”—is often acceptable. (This attitude toward authorship can make citations for articles such as this one a tricky exercise.) Unlike weblogs, wiki pages are rarely organized by chronology; instead they are organized by context, by links in and links out, and by whatever categories or concepts emerge in the authoring process. And for the most part, wikis are in a constant state of flux. Entries are often unpolished, and creators may deliberately leave gaps open, hoping that somebody else will come along to fill them in.

In wikis, the process becomes the product. What is important is not who changed a sentence in the text, but that the sentence has been changed and can be changed again, if someone doesn’t like it. As the following quote suggests, wikis significantly alter our ideas about the ownership and stability of text to an extent that not even earlier forms of electronic text achieve:

Concern with this openness – concern with the ability of others not only to read but to change what’s been written – is a measure of how closed we take writing to be. Even on the web, it seems, words are written in stone, and what’s more, we own the stones we write on, damnit! (WikiAsCulture, n.d., emphasis in original)

In a wiki, writing is so open that it ceases to be owned by any single individual. The surprising thing about wikis is that, although all the openness sounds like a recipe for disaster, committed communities seem to avoid chaos and actually manage to give shape to collectively shared meaning.

Which is not to say that wikis do not pose challenges to the articulation of meaning. Crystal (2001), while not writing about wikis specifically, enumerated some of the problems of social literacy. He argues that contrary to most traditional printed texts which have a single author, on the web:

[t]here are multi-authored pages where the style shifts unexpectedly from one part of a page to another. The more interactive a site becomes, the more likely it will contain language from different dialect backgrounds and operating at different stylistic levels—variations in formality are particularly common… People have more power to influence the language of the Web than in any other medium, because they operate on both sides of the communication divide, reception and production. They not only read a text, they can add to it. (Crystal, 2001, pp. 207, 208)

Which raises the question of how author/readers are able to write for and make sense of this new type of multi-styled text. Are authors/readers learning to ‘filter out’ the noise of multiple styles, and becoming more adept at developing a holistic understanding of the wiki text? In other words, are they becoming comfortable with textual bricolage, with a ‘genre’ of writing (cf. Kress, 2003, Chapter 6) characterized by the impermanence of genre? Or are authors/readers learning to write in a uniform, globalized form of wiki-speak? What are the social and cultural implications of each scenario?

Another area that merits attention is the educational affordances of the wiki. Wikis can facilitate a shift from an objectivist theory of learning to a constructivist or situated perspective. Instead of authors producing texts with ‘fixed’ meanings for the consumption of learners, groups of people collaborate in the production of the text, and meaning emerges out of these social interactions at the level of the text. This collective intelligence is something that can be encouraged both inside and outside the classroom. Learners can use wikis in the classroom for class projects, helping define the curriculum as they do so. But wikis can also be used outside the classroom for learners to pursue more personal interests and research agendas. It is important to keep in mind that, more than mere texts, wikis represent communities, online places where a group of people who share an interest come together to collaborate and learn. These communities can intersect the boundaries of school and non-school interests.

A final recommendation, based on personal observations, is that wikis in education should not be used to attempt to facilitate dialogue. There are plenty of other online tools better equipped to support an Initiation-Reply mode of conversation (such as discussion boards for collective dialogue, or blogs and email for more individualized forms of exchange). If appropriate, these tools can be used in conjunction with wikis. But the whole point of wikis is to de-prioritize the individual voice in favor of the collective voice, which dictates the structure and content of the text. This, of course, is a literacy which most individuals in our societies are unaccustomed to. Which is why scaffolding wikis with other technologies that support more traditional forms of communication might be an adequate strategy.

References

Crystal, D. (2001). Language and the internet. Cambridge, UK; New York: Cambridge University Press.

Fleming, M. B. (1988). Getting out of the writing vacuum. NCTE Committee on Classroom Practices in Teaching English, Focus on Collaborative Learning: Classroom Practices in Teaching English, 77-104. Urbana: NCTE.

Kress, G. R. (2003). Literacy in the new media age. London: Routledge.

Lamb, B. (2004). Wide open spaces: Wikis, ready or not. EDUCAUSE Review, 39, (5), 36–48. Retrieved on March 1, 2005 from  http://www.educause.edu/pub/er/erm04/erm0452.asp

WikiAsCulture. (n.d.) Retrieved on March 1, 2005 from http://199.17.178.148/%7Emorgan/cgi-bin/blogsAndWiki.pl?WikiAsCulture