Measuring success in the 2.0 world

I put this request up on the A&A listserv and so far have received several requests to share my findings and no findings whatsoever. I think I might have more luck here.

I am interested in learning how institutions are measuring the performance of blogs and other Web 2.0 projects (such as Facebook pages, Flickr and Twitter accounts, podcasts, etc). Have you developed metrics for measuring their impact or success? When you were planning your Web 2.0 use, did you define any measures before beginning? Has reporting on these efforts become part of your standard statistics gathering?

Some of you have been active in these areas for some time now, and I’m sure you must be measuring or reporting on your efforts. If so, please share, no matter how basic you think your efforts are.

Be Sociable, Share!
This entry was posted in Blogs, Outreach, Social networking, Web 2.0 & Archives. Bookmark the permalink.

13 Responses to Measuring success in the 2.0 world

  1. Judy Turner says:

    Sorry, this is anecdotal, not metric, and it’s about a growing pet peeve.

    There’s been an interesting discussion going on A&A List about (over)using that list to announce new blog posts. While I appreciate hearing about a new blog via a one-time announcement, if I’m interested I’ll follow them via an RSS feed. I don’t need to have my overly busy email account cluttered with repeated announcements of new postings followed by a round complaints, on the one hand, about those announcements and, on the other, requests to post the blog entry on A&A for discussion.

    There’s always been duplication of announcements across Listservs but wikis and blogs that repeat messages from listservs, tweets calling people’s attention to postings on any of the above and other essentially promotional message are a Web 2.0 phenomenon. The upshot is it’s increasingly difficult to separate the noise from the content.

    Wouldn’t it be grand if new applications were sufficiently different from each other that people could determine which to use for a specific purpose instead of trying to use all of them to accomplish the same thing?

    My sense is that there’s a lot of cannibalization of older resources going on. Web sites are being essentially simplified and duplicated on Facebook, Flickr with supplementary text showing up in the wikis and blog. Instead of being places for community-wide exchange of information and opinions listservs are being used as a way around SPAM filters.

    Back to Kate’s question, my sense is that we’ll see older web resources get less use while newer ones get more but the net effect may be a wash, not a real increase in usage.

  2. Kate T. says:

    First, to Judy:

    I know many people (including me) share your frustration with the recent bout of traffic on the A&A listserv. As you probably saw this morning, SAA’s A&A List Terms of Participation Task Force is currently reviewing the terms of participation and welcomes your comments. You can contact them at: list-terms-tf [at] forums.archivists.org. Anyone who has concerns about how the list works should share feedback with them.

    Your observations about cross-promotion and re-use of content with Web 2.0 tools are a result, in my opinion, of two factors. The first is that in many cases the audiences for these tools are different–many of people who find an archives on Flickr are not the same ones who will find the archives’ web site. Many people on the listserv probably don’t subscribe to blogs and I know many blog readers don’t bother with the listserv. Therefore people using these tools (both archival institutions and individual archivists) may feel that they need to make their presence known using many different tools in order to reach as many people as possible.

    For the most part, I think Web 2.0 tools do have different capabilities and strengths that make them well-suited for some types of outreach and not others. I think that, again for the most part, people understand this and use them appropriately. However, I think the
    second reason for the overlap you perceive is that some of these tools are still relatively new to many in the archival profession. Some people, in some cases our profession as a whole, do not yet understand how to use them appropriately. It may be, as is the case with the discussion on the listserv, that we have yet come to an agreement on what is appropriate use.

    Now, to everyone:

    I was trying to ask a very specific question targeted at institutions who have implemented 2.0 tools regarding the collection of metrics for those implementations. I admit that I’m a bit disappointed to have had no response to this query–both on the listserv and here. I understand that most implementations are probably somewhat experimental, but I know of many implementations that have surely moved beyond that stage now. Perhaps I’m too heavily influenced by my years in the Federal government, but I can’t believe that none of you have any requirements for reporting or metrics tracking. Really? What do you put in your annual reports?

    All I can say is that if it’s really true that no one is measuring anything, the chapter in my book on metrics for Web 2.0 should make it a best seller! But seriously, you have to collecting some data–what is it?

  3. Anne Foster says:

    I suspect that few are thoughtfully analyzing anything beyond basic webstats–if that. I’ve long been interested in measuring the effectiveness of outreach, but have yet to find any evidence that anything beyond anecdotal or “gut-feeling” analysis is occurring. Part of the problem is trying to define what the hoped for results are–are you successful only if you see a significant increase in researchers in a relatively short amount of time, or, does increasing general awareness count–and how do you count that without a major profession-wide research study (which I think we should be doing, but how–and with what funding?)

  4. Ben says:

    It would be surprising if metrics were not considered, especially when 2.0 apps make it so easy to measure use. Flickr and blogs, for example, have fairly robust built in tracking tools.

    I am sure you’ve already seen these, but (not specifically archival) institutions participating in Flickr have been documenting their work and including a lot of metrics. A bibliography can be found here: http://www.indicommons.org/about/bibliography/.

    I will begin my inaugural archival job next month, and I believe I’ll have the opportunity to explore 2.0 apps. I definitely intend on measuring their impact, and I look forward to reading your book on the matter.

  5. Amy Schindler says:

    I’ve asked similar questions of colleagues in the past and received mostly “yes, er, good idea that statistics gathering, but not sure how to approach it with Twitter/Facebook/etc.” responses. At present, the data we’re gathering (and really just beginning to analyze) is for two annual reports: my annual review by my supervisor and the use of the tools we are using. For my review (and this is a bit of conjecture on my part as I have not had to write my annual review yet for the current year), it will be a list of the tools implemented with a brief summary of numbers of images & videos loaded, blog posts/tweets, etc. Statistics on the number of views, comments, followers, contacts, members of a Facebook group, times gifts were sent using the Facebook gift application, links to these apps from others, etc. will also be tallied mostly using the built-in counters in the applications or Google Analytics in the case of the blog and wiki. Equally importantly, where possible we’re tracking how someone came to our 2.0 apps. Did someone come to our images on Flickr via a link from the blog or the finding aid in the collections database or via a search engine.

    I jokingly sent a scrapbook that I had recently rescued from the (now extinct) university archives backlog via the William and Mary gift app on Facebook to a bunch of archivists and librarians asking the question, can I count every time someone sends the image of this scrapbook to someone else on Facebook as a usage like we would someone who visited the reading room? Several folks replied with suggestions for rather ridiculous statistics to count, but there were a few who replied that quite simply it was a usage. Will I count each time someone sent that scrapbook as a use on the same level as someone who visits the reading room or is served through email, chat, or phone? Not this year, but we’re going to keep our eyes open and consider each tool on its own merits and on how well we use them.

  6. Ben, what is it we’re actually supposed to be measuring? I can’t say that archivists know with any certainty what the impact on their repositories will be if they aren’t entirely sure of what they’re expecting to get out of their 2.0-ish efforts. Without explicit connections and infrastructure to support them, web 2.0 initiatives in archives can be a mere flash in the pan, or at worst, a waste of precious resources to institutions. The reason that Flickr Commons was a successful project to the Library of Congress isn’t just because they happen to have people commenting and tagging their images. Instead, they’ve been able to revamp bibliographic data for their image collections – this is a strong and serious commitment on their part.

    I suspect archivists don’t really have a grip on metrics and web analytics – I know that the Archival Metrics Project and the upcoming SAA workshop on analyzing archival websites are a good start, but we need more.

  7. Ben says:

    What is it we’re supposed to be measuring? In a word: use! Whether the collection is online or on paper, isn’t that the point? In the archives profession, we have a chance to make unique materials that have traditionally been accessible only in person available to a much wider audience, while providing new ways to reach our existing users. We digitize to provide wider access, and 2.0 is just a tool for getting everyone’s attention. As the Powerhouse Museum noted in their report, no amount of interoperability, sharing, or Google indexing could generate the traffic they received from Flickr.

    You’re right that 2.0 can be a flash in the pan. New tools will come along, and the masses will flock there. It’s resource intensive to do these things, but is it more resource intensive than, say, purchasing and developing a CONTENTdm installation? Developing a robust online presence – 2.0 or otherwise – is resource intensive, period.

    Just off the top of my head, some items I would want to measure after embarking on a project of this nature would be: how many people accessed materials on the 2.0 app, did this increase the main website traffic, has it increased the use of the physical collections, or decreased it?, has there been an increase or decrease in reference inquiries about the shared materials, has there been an increase or decrease in duplication requests? I think such questions should suggest that metrics and analytics aren’t specifically a 2.0 endeavor. You would need baseline data on the use of physical collections, of the existing website traffic.

  8. Karen W. says:

    As an electronic resources statistician at Duquesne U. , I can attest re: metrics that lots of times counts of just “clicks” are just “hits” which could mean a number of random things, hits counted on a random google search by someone that retrieved it etc, and be of little value otherwise. Traffic is sometimes helpful but again you have to be able to tell what it is you are trying measure, are you counting your own views every day to your own blogs as well? blogs are a little different esp when they are “put out there” and get random hits, it may not reflect users always “coming to your site” but that you are putting it out there and gathering “hits”. I would imagine the content of the posts or not would be of more value than much else in terms of blogs. cheers, Karen

  9. Tiah says:

    As the program/project manager for the newest Flickr Commons member institution (and total stats groupie, read this as “I love the numbers, I have no skills for interpreting them), I can report this for Oregon State University Archives’ first week:
    1. No one commented on the historic images when they were in ContentDM, but people visited the digital collection page. I haven’t had much luck easily getting stats on individual photos, which says something…
    2. People looked at the historic images when they were on our regular Flickr page, though not *that* many people– and no one commented.
    3. Thousands of people have looked at our images in Flickr Commons (13K as of day 5), and we’ve received @50 comments. This is all on 116 images.
    I think there are possibilities for scientifically measuring 2.0 & “traditional.0″ success. I’ve dabbled with Google Metrics and my library’s Urchin stats, but I would have to call Flickr Commons a successful outreach tool– at least for this week– and one where gathering use data is amazingly easy.
    The true test will be what happens as we release more images *and* in the months ahead when we look at our plain old normal stats for digital collection downloads and reference inquiries. I think combining those “traditional” methods for measuring success with the more “gut feeling” methods (looking at comments, counting contacts, studying tags) might lead us to a different “level” of evaluation. And *maybe* that evaluation might make us look at how our users are using our images (and using them outside the boundaries of our institutional web pages and manicured digital collections). And *maybe* this will allow/force (you choose the word that best fits how you feel about this) us to change how we do things?

  10. Karen W. says:

    “I am interested in learning how institutions are measuring the performance of blogs and other Web 2.0 projects (such as Facebook pages, Flickr and Twitter accounts, podcasts, etc). Have you developed metrics for measuring their impact or success?”

    Additional thought process: Unlike measuring other electronic resources , blogs are often discreetly censored or politely lets say, ‘comment moderated’ so how are we also determining how ‘effective’ it is or ‘measuring impact” accurately ? Is it interactive or largely just that-an archive of one’s own writings ? So a question would go back to asking ‘what is our goal, aim, mission, objective’ i.e. what’s the point of it in the first place?

    is it self promotion for personal gain, or genuinely seeking to promote interaction and communication with others ?
    I agree also with many of the postings on the A & A list recently concerning blogs, and for most people who has time to read them all day long anyways, what are the standard forms of professional communication still for many on the job, etc lots of ways to look at it, btw I enjoy looking at Flickr albums , has anyone mentioned the recent usability study done at LC and Flickr –may be of some further interest, Karen

  11. Karen W. says:

    Somehow, I thought this was already posted from the Library of Congress Blog re: Flickr

    of further interest — a pdf of the related Web 2.0 study is linked on the information below
    ps Helena Zinkham is also a regular faculty at RBS Rare Book School teaching cataloging of image collections etc from what I recall –K.

    Library of Congress BLOG

    http://www.loc.gov/blog/?p=431

    Lincoln Photos Added to Library’s Flickr Stream
    http://www.loc.gov/rr/print/flickr_pilot.html

    View a webcast about the project, “Opening the Photo Vaults: A Web 2.0 Pilot Project to Enhance Discovery and Gather Input for the Library’s Photograph Collections” (presented Jan. 29, 2008)
    http://www.loc.gov/today/cyberlc/feature_wdesc.php?rec=4281

    TITLE: Opening the Photo Vaults: A Web 2.0 Pilot Project to Enhance Discovery and Gather Input for the Library’s Photograph Collections

    SPEAKER: Helena Zinkham, Justin Thorp, Barbara Natanson, Phil Michel, Dave Woodward, Michelle Springer, George Oates
    EVENT DATE: 01/29/2008
    RUNNING TIME: 76 minutes
    DESCRIPTION:
    Library of Congress staff from the Prints and Photographs Division, Office of Strategic Initiatives, and Information Technology Services — along with George Oates from Flickr, the popular photo-sharing site and Web 2.0 innovator — describe the pilot project in which the Library has mounted photographs from its Farm Security Administration and George Grantham Bain collections on the Flickr Web site.

    Speaker Biography: Helena Zinkham is acting chief of the Prints and Photographs Division at the Library of Congress.

    Speaker Biography: Justin Thorp is on staff in the Office of Strategic Initiatives at the Library of Congress.

    Speaker Biography: Barbara Natanson is head of the Prints & Photographs Division Reading Room at the Library of Congress.

    Assessment is a very hot topic everywhere now–yes !

  12. Amanda Hill says:

    Hi Kate

    I’m just taking a break from writing a presentation on this very topic for a conference on Archives 2.0 in Manchester (UK) next month. I think my conclusion is that basic usage metrics are generally pretty easy to generate for blogs and Flickr – they’re either built-in or can easily be bolted on with tools like Google Analytics. The difficult thing is going beyond those raw numbers and measuring the impact of those Web 2.0 tools.

    There I think you have to enter into a much more difficult-to-measure realm where you need direct feedback from users. Where comments are a feature of a service, those are easy to monitor and sample. Tracking incoming links is also fairly straightforward, but some of the best illustrations of the impact of Web 2.0 come from the contacts made directly by users with archives as a result of their initial interaction with an online service. These are the ones we need to be trumpeting from the rooftops if we want to demonstrate the real value of these tools.

  13. Susan says:

    I ran across this article on Twitter this morning about measuring online influence:
    http://mashable.com/2009/03/02/measuring-online-influence/

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>