Subject guides: Text and data mining: 1. Is TDM right for you?

Is TDM right for you?

Increasingly, information is being collected and stored digitally in huge volumes. Older archival and print media is also being digitised. If you have a research question that would best be answered by large amounts of data or text, then TDM could be for you.

Text and data mining is an iterative process of inquiry and discovery; your question or perspective may change, and so might your methods.

There are some different places you might be starting from:

An interest in a particular area
A particular research question you want to answer
A particular set of data that you want to investigate further
Researching into text or data mining methodologies

Questions to ask yourself

Do you have a research question that would be best answered by TDM methods?

Example projects that have used TDM methods

Content Analysis of 150 years of British periodicals
The project analysed 28.6 billion words from 35.9 million articles contained in 120 UK regional newspapers over the period 1800-1950, which they calculate forms 14% or all regional newspapers published in the UK over the period in the British Library's collection.

This allowed the researchers to explore changes in culture and society, determined by changes in the language. Looking at changes in values, political interests, the rise of 'Britishness' as a concept, the spread of technological innovations, social changes and much more.

Lansdall-Welfare, Thomas, et.al., (2017) “Content analysis of 150 years of British periodicals”. Proceedings of the National Academy of Sciences of the United States of America, January 24, 114(4), p.E457-E465. https://doi.org/10.1073/pnas.1606380114.
Researchers mine online consumer reviews to identify unsafe toys
Virginia Tech used text mining on data in millions of consumer reviews posted online to identify toys with potential safety hazards.

The researchers developed a classification system for different types of safety and performance defects related to toys. The lists of “danger words” were then used to evaluate or score a large sample of the more than one million product reviews on Amazon.com in the “Toys and Games” category in the 1999-2014 period.

Winkler, Matt, Abrahams, Alan S., Gruss, Richard and Ehsani, Johnathan P. (2016). “Toy safety surveillance from online reviews.” Decision Support Systems, 90 (October), 23-32. https://doi.org/10.1016/j.dss.2016.06.016.
From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series
The researchers used sentiment analysis to measure surveys on consumer confidence and political opinion over the 2008 to 2009 period which were then compared with Twitter feeds of the same period. The correlation was found to be high, (80%) in some datasets, leading to the the potential of text streams as a substitute and supplement for traditional polling.

O' Connor, B., Balasubramanyan, R,, Routledge, B.R. and Smith, N.A. (2010) ‘From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series’. Paper presented at the International AAAI Conference on Weblogs and Social Media, Washington, DC, May 2010. Retrieved June 24, 2020, from http://homes.cs.washington.edu/~nasmith/papers/oconnor+balasubramanyan+routledge+smith.icwsm10.pdf

Library support

Support services

The Library provides a number of services to help you get started with text and data mining.

Consultations

Want to talk to someone about how to start your own text mining project? Chat to your Academic Liaison Librarian or email us for help!