Research problem

Tagging as a new information organization and retrieval tool, one greatest shortcoming is its inconsistency. The tagging consistency is threefold:

  • Personal consistency that one user are tagging similar contents using consistent tags;
  • Collaborative consistency, that a community of users tag similar contents using consistent tags;
  • Cross-platform consistency, which means users among different internet applications tag similar contents using consistent tags.

Above is a random-picked user’s tags on del.ici.us. The highlighted parts are tag inconsistencies: misspelling, plurals, different conjoint words, tenses, etc. For example, if she wants to find things related to “web2.0“, a lot will be neglected under the tag “web2.0,“. Therefore, inconsistency diminishes information retrieval efficiency from a tag system.

Many websites are using suggestive systems to improve tagging consistency. When users try to save a bookmark on del.icio.us, the system will show them what tags they used before, so they can click on tags that they needed for current content. The major setback of suggestive system is that it is difficult to find a sepcific tag in a tag list like the tag cloud, especially when the font size is small. Users will not try very hard to find it since tagging is supposed to be fast. Therefore they just type in what they are thinking at that moment, so web2.0 may become web2, web_2.0, etc.

So the problem can be reformulated as: how to increase the efficiency of visual searching in a set of tags? Visual search is nothing new, a lot of suggestions have been given.

20/80 rules in the tagging system

Source: Golder et al, 2005

In the long term, the tag distributions tend to converge to power law distributions, which suggests the emergence of a stable distribution. Some few tags tend to dominate the whole tag set. (Begelman, 2006)

Therefore, if we can help user to easily find frequently used tags, the tag consistency will be improved. Actually, this is the working mechanism of tag cloud.

Semantic clustering

Source: Montero et al, 2006

To facilitate visual browsing task in a list, we can cluster relevant items (Lin, 2002). And Montero et al suggested that visualization of inter-tag relevancy can improve tag consistency. The arguement is simple: if you can put similar tags together, then users can locate the category very quickly and then only do visual search in the category. However, no experiment has been conducted to verify this point.

Experiment design

We firstly conducted experiment to tackle tag consistency problem. It was a 2 by 2 design, 28 subjects participated:

Independent variables

  • visualization of tag frequency
  • visualization of inter-tag relevancy

Dependent variables

  • personal consistency

    • personal consistency is assessed by measuring the relative overlap between two tag sets.

  • mental workload in tagging process

    • Assessed by NASA-TLX measurement.

Tasks

  1. The subject signs consent form and register basic information.
  2. Warm up. The subject is asked to tag five pictures, to familiarize the experiment interface.
  3. Section 1, the subject is asked to tag 60 pictures.
  4. Question and interview. The subject fills a NASA-TLX questionnaire, and answer some open end questions
  5. Memory erasing. This stage is to erase subject’s short term memory about the tags. The subject is asked to add an arithmetic sequence to a 3-digit number, e.g. add 1, 2, 3, … successively to 200, and speak out the result for each addition. This ends when the subject comes to 500, then the subject is asked to subtract an arithmetic sequence from 800, this ends when the subject reaches 500.
  6. Section 2, the subject is again asked to tag 60 pictures, in which 20 pictures occurred in section 1. The order has been randomized.
  7. Debrief and payment.

Result

ANOVA showed the visualization of relevancy significantly improved personal consistency. (p=0.014)

Using NASA-TLX workload measurement, we found visualization of tag occurrence frequency (p=0.014) and the interaction between two independent variables (p=0.031) reduced user’s physical demand significantly.