Corpus frequencies: what exactly counts?

Senses, idioms and phrasal verbs

Following on from my last post prompted by Michael Rundell’s webinar: Tweets, blogs and corpora: How computer technology helps us make better dictionaries. There was one more question from a webinar participant which I think opens up a whole area of corpus research and word frequencies as shown in dictionaries that tends to get glossed over.

“Do these numbers [corpus frequencies] consider all the meanings of a word or only the common ones?”

Response from another participant:
“I’d guess that the counting program [the corpus software] doesn’t understand the meaning so it is for all meanings of the word.”

An astute question and a correct answer! Corpus software is very clever at number crunching and identifying patterns, but computers still fall down when it comes down to actually understanding language. When you do a corpus search, you can choose the part of speech you’re interested in (separating out noun and verb senses of a word like walk, for example) and you can search for a ‘lemma’ rather than just a string of letters (so searching for the verb walk will include walk, walks, walked and walking). When it comes down to differentiating between different senses or uses of a word though, that can still only be done “by hand” by a human being sorting through a sample of corpus lines one-by-one. Sometimes, where one sense is overwhelmingly more frequent, the sense frequencies are obvious at a glance. In other cases, especially with very polysemous words, it’s a trickier business. Thus, sense ordering by frequency is, to a degree, impressionistic and doesn’t involve exact statistics.

Does this matter? Again, my answer is “not really”, provided we’re only taking frequency information in a dictionary as a general guide. For many words, the most frequent sense(s) of a word will probably account for the majority of its occurrences, so it’s fair to say that overall its core meaning(s) will fall within a general frequency band. It’s unlikely that in many cases there will be lots of obscure senses of a word that significantly distort the frequency statistics.

Where caution may be required though is where it’s the less frequent senses you’re actually interested in. To take an example I came across recently working on EAP vocabulary, if you do a corpus search for chemist, physicist and biologist, chemist comes up as much more frequent – as reflected in most of the learner’s dictionaries. Now that isn’t because there are more scientists studying chemistry than there are physics or biology. But of course, in British English at least, a chemist can be a pharmacy or a pharmacist as well as a scientist in a lab with their test tubes.

And it’s not just the effect that different senses of a word might have on frequency that needs considering. Another big area to take into account is words that form part of a phrase of some kind. Going back to my example of walk, it crops up in various phrases or idioms – walk the walk, run before you can walk, walk free, etc. – and a whole list of phrasal verbs – walk away with, walk in on, walk off, walk out … In most learner’s dictionaries, these come at the end of the entry for the headword and in most of the major dictionaries (I think with the exception of Cambridge), they don’t have frequency information in their own right. Instead, they get lumped into the overall frequency for the whole entry. This has two consequences; firstly, it means that learners can’t see which phrases and phrasal verbs are most frequent and also, it further undermines the frequency information for some words as we can’t be certain what it’s referring to. Take the verb deal as an example, highlighted as frequent in most dictionaries. In fact, something like 85% of occurrences of the verb deal are actually instances of the phrasal verb deal with. Yet, in most dictionaries, it appears that the basic verb senses (giving out cards or drugs) are common, while the phrasal verb deal with has no highlighting at all.

It is possible to construct corpus searches to find particular phrases or phrasal verbs, even where their form varies slightly - for example, where phrasal verbs have moveable particles. So it is possible to get frequency information for them, albeit not as simply or reliably as for single words. And if you look in specialist dictionaries of phrasal verbs or idioms, you’ll often find the most common ones highlighted. So why don’t most general learner’s dictionaries include this information? Well, firstly, it’s very time-consuming to research and secondly, it isn’t easy within a traditional dictionary format to devise a system that encompasses frequency information for both whole words (with senses lumped together) and individual usages in the form of phrases and phrasal verbs.

So having completely ripped apart the frequency information in dictionaries, am I saying that it’s useless and should be ignored? No, far from it! I think as a broad guide to which words are generally more frequent (and so worth focusing on), I still think it’s an incredibly useful tool. But as in any area of life, statistics should always be approached critically and before you rely too much on them, you need to understand what’s behind them, how they’re compiled and what caveats you might need to take into consideration.

Corpus: gospel or guide?

A response to a webinar:

Lately, I’ve been starting to explore Twitter and linking through to various blogs and websites to see what folks are talking about in ELT at the moment, all in the name of “professional development”. Yesterday, I also ventured into the world of webinars for the first time. I started off with a recording from Macmillan’s Interactive webinars series. My first aim was really just to explore the medium, so I chose a familiar topic with Michael Rundell’s Tweets, blogs and corpora: How computer technology helps us make better dictionaries. The actual content wasn’t particularly exciting - not because there was anything wrong with Michael’s presentation, but as an experienced lexicographer, I clearly wasn’t his target audience. That meant though that I could focus more on what actually goes on in a webinar.

Rather unexpectedly, I got particularly caught up with the reactions of the participants as they appeared in the little text box on the side of the screen. Unfortunately, Michael ran out of time, so didn’t get to address any of the comments or questions that popped up. I, however, was itching to respond to them! So I thought I’d tackle some of the points here which I’ve been mulling over since. And in fact, I’ve got so much to say, I’m going to split this into two posts.

The part of the webinar that interested me most in terms of participant feedback was when Michael was talking about how we use frequency information from corpora to highlight the most common and so "useful" words in a dictionary. Below are some of the comments and questions and my reactions:

“Are there standard lists with the top 250 words?”
“But where can we find the list of words (to know if they are frequent or not)”

I always find interest in wordlists from teachers and students a little bit worrying. It seems to suggest that language learners are rather like computers and if we can just input the right list of words, then they’ll output English at a given level! Whilst I think frequency lists can have a role to play in helping prioritise what to focus on, my feeling is that generally checking frequency should be something that comes after you encounter new vocabulary. You look up a new word you’ve come across in the dictionary and you might use the information about frequency to decide whether it’s worth putting in your vocabulary notebook or whether it’s a word that you can naturally drop into conversation or not. Language is a wonderfully messy, organic, personal sort of a thing and what vocabulary you choose to teach or learn should be governed by all sorts of different factors - interests, needs, context, personality - not some (inevitably very dull) standard list of frequent words.

“Are the top 3000 words in Oxford the same top 3000 words in Macmillan?”

I haven’t researched the answer to this one, but I think I can fairly confidently say “more or less” if we’re just talking about frequency (more on that below). Each of the major dictionary publishers uses a different corpus – or rather a different collection of corpora, some of which overlap (like the BNC). In the early days, with relatively small corpora, you would have expected some variation, with different corpora slightly skewed towards particular types of language. Nowadays though, with all the big publishers using really huge and diverse collections of corpora, I think you’d probably expect a straightforward frequency list (at least at the most frequent end) to come out more or less the same, with only minor variations.

Having said that, each dictionary publisher has it’s own criteria for how it shows frequency information – where it sets it’s limits and how it puts words into frequency bands. The Oxford 3000™, for example, isn’t just based on frequency, but was put together using three criteria; frequency, range and familiarity (if you're interested, you can read more about it here). Does this variation matter? Personally, I don’t think so. How many students, or even teachers, ever read the blurb in the front (or back) of a dictionary that explains the frequency information? My feeling is that most students either don’t even notice it, or if they do, it’s just some general sense that a word is highlighted or has stars next to it, therefore it must be useful to learn. Of course, there’ll be times when some teachers (esp. vocab nerds like myself!) will make a point in class about frequent and more marked synonyms by pointing to the frequency information (and often register labels) in the dictionary. But my feeling is that’s the exception, not the rule. And that’s fine. It’s still worth the information being there as one more tool in the language learning toolbox.

Coming back to the title of this post, corpus information has been incredibly useful over the past couple of decades in understanding how language is actually used and in making teaching materials more natural, but it's still only a guide. Despite much of my work being in the area of corpus research, I'm still very wary about taking corpus data as gospel, partly for some of the reasons I'll talk about in my next post ...