User research: reading behavior on an online literary magazine

How might we encourage readers to curl up with a screen the way they used to with a printed book?

Project metadata
Project titleUser research for terrain.org: A Journal of the Built + Natural Environment.
Project taglineImproving engaged reading online through design
Project summaryUsability, interviews, and a heuristic review
Company/client namePublisher, Terrain.org
Project dates/time frameJanuary-April 2016
My major responsibilitiesUser research plan, In -person user interviews in context, literature review on readability
PlatformsWeb
Tools usedPages, Keynote
Key performance metricsIncreased readership, time on page, more donations
TeamUX team of one
Link to final projectTerrain.org

As a writer, you fight a war against indifference. You have to force people to care enough to click through to your story …they must answer a question… is this going to be worth my time?

Quincy Larson, Free Code Camp

Project summary

Terrain.org is an online literary magazine that publishes high quality essays, poetry, and creative nonfiction about the natural and urban environments. This project aimed at increasing readership,  encouraging more engaged reading of longer essays, and increasing donations.

I conducted usability interviews, and an expert review with other designers. My role was user researcher. I interviewed people in person and interpreted the results in collaboration with the publisher and site owner.

Terrain.org, home screens
People choose a story to read based on the featured image, title, and a short excerpt.

Executive Summary

At first glance, Terrain.org was attractive and looked like it was doing well. It had lots of visits (250,000 in 2015) and a large list of subscribers. But, when you went beneath the surface of visual design and images, there were some pain points. The number of subscribers had leveled off. Analytics showed the bounce rate was high (85%) and average time spent on site was low (less than a minute)–a major concern for a literary magazine featuring thoughtful essays, long articles, and poetry.

In addition, donations were low. The site does not show ads.

Pain Points

Session Time

Average duration is less than a minute

Returns

Only 20% of the total viewers were returning visitors.

Bounce Rate

85% of visitors exit from the home page.

Pages Viewed

The average number of pages viewed is 1.5.

In addition, the number of new subscribers has leveled off and few people use the “donations” link (less than 1% of site traffic). These numbers are worrisome for a literary magazine whose goal is to encourage long engaged reading.

Research overview

Since Terrain is an all online journal and there are a lot of challenges to reading longer articles on screens, we started comparing how people read online as compared to print magazines. After a review of the literature on readability and surveying similar text-heavy sites, we conducted an expert review of the site usability. Next we conducted live interviews and walk-throughs of the site with both current Terrain readers and potential readers.

Research questions

We wanted to compare the goals of new visitors browsing the site to newsletter subscribers who clicked through on an article.

  • How does site usability and typography influence engagement with longer literary texts?
  • How do Terrain readers arrive at the site?
  • What are their motivations for coming to the site?
  • What cues do readers use to search for content they want to read.

In addition, we asked people how they thought about supporting or subscribing to ad-free literary sites like this. Did they see the donate button? Were they aware that there were no ads? What would motivate them?

Data analysis

I created a matrix of results from the expert usability review (also called a heuristic evaluation) showing the scores, and comments for each check point, as well as an overall score for the site (54%).

In addition, the number of new subscribers has leveled off and few people use the “donations” link (less than 1% of site traffic). These numbers are worrisome for a literary magazine whose goal is to encourage long engaged reading.

Research overview

Since Terrain is an all online journal and there are a lot of challenges to reading longer articles on screens, we started comparing how people read online as compared to print magazines. After a review of the literature on readability and surveying similar text-heavy sites, we conducted an expert review of the site usability. Next we conducted live interviews and walk-throughs of the site with both current Terrain readers and potential readers.

Research questions

We wanted to understand the goals of new visitors and things like:

  • How does site usability and typography influence engagement with longer literary texts?
  • How do most Terrain readers arrive at the site?
  • What are their motivations for coming there?
  • How do they search for content they want to read.

In addition, we asked people how they thought about donations, to this and other sites. Is the Donate button too small to easily find?

Data analysis

I created a matrix of results from the expert usability review (also called a heuristic evaluation) showing the scores, and comments for each check point, as well as an overall score for the site (54%).

Expert review

A radar chart of the expert review, showing categories and detail questions asked.
A radar chart of the expert review, showing categories and detail questions asked.
Ideally you want your site to score around 75% on an expert review, or heuristic evaluation such as this (Balatbat, 2014). The lower scores are because we were very picky about readability. Note the low score on find-ability–we flagged find-ability as an issue.

User interviews

One of the main research questions was, “How do new visitors browse from the front page and find an article they want to read?” In other words, how do they stay on the site? To find out more, I combined the interview with a usability walk-through. I asked people to go to the front page, choose an article from the menus and read it to the end, speaking their thoughts aloud as they went.

I recorded the user interviews using audio and screen-capture videos. From our notes and transcripts, I organized the rich qualitative data obtained from the interviews by research question, using affinity mapping with sticky notes. I also noted new themes and ideas coming out of the research. I then organized this data into a Findings and Recommendations Matrix, featuring quotes, summaries and key insights from each participant.

Affinity mapping
Affinity mapping is a business tool used to organize free wheeling qualitative data like interviews. It allows large numbers of ideas to be sorted based on their natural relationships, for review and analysis.

Key findings

Most users commented positively on the site design, admired the images, and eventually read something interesting. Yet, the site’s overall usability score is below average. Users in the interviews were more forgiving than the experts, and combining both methods we found key design edits that would increase readability without changing the site’s WordPress theme.

Most people interviewed preferred print on paper for personal, literary, and in-depth reading over digital screens. Computers were seen as better for research, email, and finding facts. In keeping with this, participants all scanned the pages first, looking for cues to content before committing to an article. Everyone got stuck on the featured images, trying to use them to decide what to read, which was not reliable. The photos were either a beautiful distraction, or visual overload.

Everyone interviewed did finally enjoy some longer reading that engaged their interest. This supports our idea that providing the right visual cues, and clearing the critical path to interesting content would help people reach this desired state of engagement with text.

After watching people choose a story from the home page, I believe most of the page views come from links in the email newsletter to subscribers. People usually read only the page they clicked on, and don’t browse the rest of the site. That could be due to browsing usability. My theory is that a user is confronted with too many choices, however enticing, and has a hard time choosing just one to read. This is knowh as Hick’s Law: the more stimuli people have to choose from, the longer it takes the user to make a decision. In effect too much cognitive load and likely abandonment of the page.

 a drop-down secondary menu appears, (blue outline is mine) showing a curated view of articles in that category.
When a user clicks on a top menu item, like “Nonfiction,” a drop-down secondary menu appears, (blue outline is mine) showing a curated view of articles in that category.

Few people found the Donate and Subscribe links. People wanted to share articles, but there is no share link to make that easier.

Key recommendations. Usability improvements will help keep users on the site, but that alone will not increase readership. Since most people arrive at the site from external links to a particular article, Terrain should maximize its presence in the external places where likely readers readers would find the site, like links and banners on other relevant sites, content aggregators, social media, email newsletters, etc.

Given a longer-term trend away from home pages and towards apps and social sites, Terrain should look outward to publishing on other social sites like medium.com, Apple News app, and Facebook’s new Instant Articles feature which allows people to read without leaving that site. A longer-term goal might be to create a phone app, as more people are reading that way as well.

 


A Readability Reading List

Balatbat, C. (2014, January 27). A Beginner’s Guide to Heuristic Evaluation (Part 2). Retrieved April 24, 2016, from http://blog.foolprooflabs.com/2014/01/beginners-guide-heuristic-evaluation-part-2/

Jabr, F. (2013, April 11). The Reading Brain in the Digital Age: The Science of Paper versus Screens. Retrieved April 25, 2016, from http://www.scientificamerican.com/article/reading-paper-screens/

Goodman, Elizabeth; Kuniavsky, Mike; Moed, Andrea (2012–09–01). Chapter 11. Observing the User Experience: A Practitioner’s Guide to User Research (Kindle Locations 4818–4819). Elsevier Science.

Herrman, J. (2016, April 17). Media Websites Battle Faltering Ad Revenue and Traffic. Retrieved May 02, 2016, from http://www.nytimes.com/2016/04/18/business/media-websites-battle-falteringad-revenue-and-traffic.html smid=tw-nytimes

Holst, C. (2010, November 01). Readability: The Optimal Line Length. Retrieved May 21, 2016, from http://baymard.com/blog/line-length-readability

Krug, S. (2006). Don’t make me think!: A common sense approach to Web usability. Chapter 2. Berkeley, Calif: New Riders Pub.

Marjoo, F. (2013, June 6). You Won’t Finish This Article: Why people online don’t read to the end. Retrieved March 31, 2016, from http://www.slate.com/articles/technology/technology/2013/06/how_people_read_online_why_you_won_t_finish_this_article.single.html

Nielsen, J. (2003, November 10). The Ten Most Violated Homepage Design Guidelines. Retrieved May 09, 2016, from https://www.nngroup.com/articles/most-violated-homepage-guidelines/

Nielsen, J. (1995, January 1). 10 Heuristics for User Interface Design: Article by Jakob Nielsen. Retrieved April 01, 2016, from https://www.nngroup.com/articles/ten-usability-heuristics/

Nielsen, J. (1995, January 1). How to Conduct a Heuristic Evaluation. Retrieved April 2, 2016, from Nielsen Norman Group, https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

Nielsen, J. (2008, May 6). How Little Do Users Read? Retrieved March 30, 2016, from https://www.nngroup.com/articles/how-little-do-users-read/

Nielsen, J. (2007, July 9). Write Articles, Not Blog Postings. Retrieved March 30, 2016, from https://www.nngroup.com/articles/write-articles-not-blogs/

Nell, V. (1988). The Psychology of Reading for Pleasure: Needs and Gratifications. Reading Research Quarterly, 23(1), winter, 6–50.

O’Hara, K. (1997). A Comparison of Reading Paper and On-Line Documents. Retrieved July 26, 2016, from http://www.sigchi.org/chi97/proceedings/paper/koh.htm

Sharon, Tomer (2012–03–21). It’s Our Research: Getting Stakeholder Buy-in for User Experience Research Projects (p. 97). Elsevier Science. Chapter 3.

Travis, D. (2016, April 12). 247 web usability guidelines. Retrieved May 01, 2016, from http://www.userfocus.co.uk/resources/guidelines.html

uk.gov, Government Service Manual. (2014, June 4) Expert reviews: Getting input into products and services. Retrieved April 1, 2016, from https://www.gov.uk/service-manual/user-centred-design/user-research/expert-review.html

 

View the entire report

Leave A Comment

Your email address will not be published. Required fields are marked *