Corpus insider #3: Corpus quirks

I love using corpus tools to research language. They throw up some fascinating results to feed into ELT materials in all kinds of ways. They can, however, also be infuriating at times! In this post, I thought I'd look at a couple of the quirks that it might help to be aware of when you're using corpus tools.

Apostrophes, contractions and negatives

Corpus tools are great when you're just searching for strings of letters, but throw an apostrophe into the mix and all kinds of confusion seems to ensue! That's particularly problematic when you want to start researching anything grammatical. Searching for contractions, including auxiliary verbs and negatives, gets tricky, as are possessives. Different corpus tools deal with these forms in different ways - it may be that you can just type in the whole word apostrophe and all, you may have to separate the contraction from the main part of the word so she [space] 's or the software may treat a negative as a separate entity, so you'd need to search for could [space] n't. 

The essential thing though is you don't miss out on these forms because you constructed a search that didn't include them. If you're looking into future forms and you're only getting examples of will in its full form, you're missing out on loads of 'll and won't forms that may well be significantly more common. That means you need to check how the corpus tools you're using deal with them - look out for any help or FAQs for guidance.

Some useful search tips from COCA (click to view larger)

Slippery parts of speech

Most corpora you're likely to use will be part of speech tagged. That means that the data's been automatically analysed to tag each word with a part of speech; noun, verb, adjective, etc. That allows you to do lots of things. It means you can search for a word like walk and decide you only want to see verb examples or noun examples, usually by selecting PoS from some kind of menu. It helps with searching for lemmas - so you just type in walk and select verb and you'll get walk, walks, walked and walking. It also allows collocate searches to categorize and display results by part of speech.

A collocate search from Sketch Engine

However, part of speech tagging isn't perfect. Sometimes - well, quite often actually! - the tech just gets it wrong. So in any search where a word has multiple parts of speech, you're likely to get a few odd examples cropping up, especially where the grammatical clues are misleading. So in one text I was looking at, the fact that the word weather was preceded by to in the sentence "Ice is very sensitive to weather" made it look like it might be a verb so the software incorrectly categorized it as one.

These occasional tagging errors aren't generally significant, but much more problematic are words that are just difficult to classify. That's particularly true of words that come before nouns. We tend to think of words that pre-modify nouns as adjectives, but actually lots of them aren't technically. So, we can have nouns before nouns; an evening dress, a window cleaner, the table decorations. These don't cause massive problems in corpus searches, but they're worth noting as they don't often get a look-in in ELT materials.

Far more difficult are verb participles. Think about these examples - which of the words in bold are adjectives?

a boring meeting

another satisfied customer

an increasing number

the desired effect 

a neatly folded piece of paper

It's just one of those features of English that's annoyingly problematic for anyone who's dealing with learners and doesn't want to get into over-technical jargon. If you look in a learner's dictionary, you'll likely find the common ones (boring/bored, tiring/tired, exciting/excited, etc.) listed as adjectives, but the less common ones (increasing, desired) may or may not be there. It's an editorial decision where the cut-off point is for these. It's unsurprising, then, that corpus tools often struggle to handle them. Often, they just don't recognize them as adjectives, instead, they get lumped in with the verb lemma. That has a number of consequences:

• You may not be able to search for increasing as an adjective. Of course, you can usually search for the exact word form instead of the lemma, but that will leave you with a mix of adjectival and verbal examples.
• You can't always do collocate searches for these words, because the tools don't recognize them as an adjective.
• When you do collocate searches, you need to take into account that some of the collocates might be in the 'wrong' places, so participle adjectives will often show up (in their base form) in the verb column. If I search for temperature, for example, I might get verb collocates that include lower, withstand and measure which are all genuine verb collocates, but then I get rise, operate and desire. When I click through, rise turns out to be a mix of rising temperatures and the temperature rose. Whereas desire and operate are always modifying the noun: the desired temperature and operating temperature.

Embracing quirkiness

So what does all this awkward quirkiness mean for the average corpus user? Well, it doesn’t mean you should give up using corpora or stop trusting corpus results. It just means you have to familiarize yourself with how your corpus tools deal with the odd stuff and be on the look-out for apparent anomalies, like mis-tagged parts of speech and unlikely collocates. And as you get more familiar with the tool you’re using, you’ll get to know its quirks and, more importantly, how to get around them or take them into account.

Sticking to your lexical guns: 4 principles for writing vocabulary materials

I was inspired to put together a talk for the recent MaWSIG/Oxford Brookes event by a number of posts by Katherine Bilsborough about materials writers’ principles. It’s a topic I’ve been pondering for a while and one I decided to give a vocabulary slant. After lots of thought, I came up with four broad principles. And with the title of the event being about challenges and opportunities, I combined my four principles with some of the challenges I face in sticking to them when they seem to conflict with the brief I’m working to.

Principle 1: Have a clear aim in mind for every activity

This may seem obvious and a bit of a universal truth when writing any kind of materials, but I think that all too often, vocab activities get tagged on – to a grammar syllabus, to reading lessons – without any real thought about what they aim to achieve other than “teach some words” ... which isn’t really a realistic aim, is it? Because vocabulary learning isn’t as simple as “doing” a word once and then it’s known. Yet it often gets pushed into a standard PPP model:

Pack some words into a text (whether they’re ever used together or not!)
Pop them into a gap-fill (because that’s what you do with vocab, isn’t it?)
Prod students into using them (because they ‘know’ them now, right?)

In fact, most research into vocab acquisition suggests that learning vocab is a gradual process in which students get to grips with words over a period of time via repeated exposures. Which suggests an approach to teaching vocab something like below might be more appropriate.

So, in an ideal world, at the first encounter with a word (in context), the focus of any activities would be on comprehension or receptive knowledge, that is recognizing the form of the word (spelling and pronunciation) and understanding its meaning in the current context ... quite enough for a first meeting. Then, as the same item pops up again and again, the focus shifts from reception and understanding how the word’s used in different contexts to controlled production. As a word becomes more familiar, students should be encouraged to look at how they can use it – what collocations is it used in, what register is it, what are its grammatical features and what patterns does it typically appear in? Then eventually, along the line, they’ll hopefully be ready to start using it for freer production. How long that process takes will depend on the type of word and also the stage the learner’s at. And crucially, at each stage, the aims of any vocab activities will be quite different.

The Challenge: “We certainly can’t have a student see a word twice!” – as Dorothy Zemach so succinctly highlighted in her IATEFL plenary earlier this year, within the world of ELT publishing, there’s very much of a focus on providing a constant stream of fresh, new vocabulary and repetition of items is actively discouraged – “you can’t have that word, it was covered at the previous level”. To be fair, this isn’t just coming from publishers. Students and teachers naturally want to feel like they’re making progress and in many people’s minds, understandably, that’s about increasing their vocab. They want to see new words to learn as they work their way through a course.

The Work-around: Exposing students to vocabulary doesn’t have to be confined to specific vocabulary sections. As a writer, if you keep a record of newly introduced vocab, you can sneak it in all over the place. A new word might pop up first in an explicit vocab activity, but it can easily be recycled in later units in different sections. If you’ve got a grammar activity to write, it’s relatively easy to look back at target vocab from previous units to include in example sentences. You could even slip some work on collocation into a revision of the present perfect, for example (this is a quickie, made-up exercise just to demo the point):

Complete the gaps using the best verb from the box in the present perfect.

do   give   make 

Jack ______ his homework, but he _____ a lot of mistakes. The teacher isn’t very happy and she _____ him a low mark. …

Principle 2: Create reasonable and memorable lexical sets

Following on from the pressure to constantly provide students with lots of juicy new vocab, a lot of ELT materials seem to regularly throw long lists of semantically-similar words at students in the hope that they’ll stick. The trickle-down of research into how useful (or not) it is to teach vocabulary in traditional lexical sets is patchy at best. From my understanding of the research (such as it is), things to avoid in materials include long lists of very similar words introduced as new vocabulary, including near synonyms, easily substitutable items and synforms (words which look very similar). The reason for this being that students easily get them confused and so find them more difficult to learn – an issue known in the literature as interference (Nation, 2000).

From a writing perspective, that means avoiding confusables in new vocab as much as possible. So, if you’re trying to write an activity and you’re struggling to come up with items that have unambiguous answers because more than one word in the set can fit in a gap, it probably means you have items that are too close together and you might want to consider tweaking your set.

Note, however, that the problem of interference is largely around new vocabulary. Researchers suggest that once words are fairly well-established, then bringing similar words together is actually beneficial. Arguably, at higher levels, it’s essential for students to understand how synonyms overlap and in what ways they differ, so bringing them together at this stage is a necessary part of learning.

The Challenge:  Especially if you’re working on a large, publisher-led project, you’re likely to have a scope and sequence document that includes vocab sets based largely on topics and often those will include sets of worryingly similar words.

The Work-around: Whilst sets of overly similar items can lead to confusion, that doesn’t mean you can’t have thematic sets which include a range of different vocab to talk about a topic. One really simple way to mix up a vocab set is to include different parts of speech. So, for example, if you’re doing an A2 unit on ‘feelings’ you might have a suggested vocab set that contains exclusively adjectives … *heart sinks*:

But of course, we don’t just use adjectives to describe our feelings and there are plenty of other on-level verbs and nouns you could include as well. This might involve bending the brief a little, but it’s quite possible without straying too far from the original plan. And not only does this make the items less easily confused by students, it also encourages more variety of expression – which has got to be a good thing.

Principle 3: Use research tools with a large dose of common sense

There are lots of tools that we use as writers to help us make choices about what vocabulary to include and what to leave out– wordlists (like the AWL), ‘vocab level’ lists (like EVP), text analysis tools (like Text Inspector), corpora, readability measures, etc. They all provide valuable input to complement our own experience and intuition, but they do need to be treated with care. Most importantly, I think it’s essential to fully understand any tool you use, to know what it’s based on, what it tells you and crucially, what its limitations are.

I won’t go into the ins and outs of wordlists again here, but will refer you back to my previous blog on the topic. A quick further note on text analysis tools though – whilst, they provide a really useful guide, they are automated and can’t always be relied on to get it right for every word in your text. Make sure you check for each word whether the tool has chosen the appropriate sense (the full version of Text Inspector allows you to choose the sense from a drop-down), whether it’s got the correct part of speech (a percentage of words in any text will usually be wrong here) and finally, check whether the words are part of a phrase or chunk as this won’t usually be recognized by the tool and may change the level significantly. When I put a short section of the abstract for my talk through Text Inspector, for example, I’d estimate that roughly 25% of the words were initially labelled incorrectly for level.

The Challenge: Your editor insists that you can’t use a B2 word in a reading text in a B1 book, because they’re sticking rigidly to a level list and, if they’re using EVP, have misunderstood (as many people do) that the level labels signal productive usage by students at that level which is very different from when a student might first encounter a word receptively. 

The Work-around: If you properly understand the list (or other tool) you’re being asked to use, then you’ll be in a much stronger position to argue your case – in this case, explaining that if a word is tagged as B2 for production, it’s perfectly reasonable to introduce it receptively in a B1 reading text. Of course, if you’re just one writer on a large multi-author project that’s already well in motion, then you’re not always going to win your case, but certainly for smaller projects, showing your understanding up-front might help steer things in a more informed direction.

Principle 4: Work beyond the level of the word

Research suggests that somewhere between 30% and 50% of any text is made up of phrases, idioms and other chunks of language (depending on the type of text and how you count). So understanding these chunks is vital for any language learner, as is getting to grips with how to use them themselves. One piece of research (Millar, 2011) found that atypical collocations used by learners slowed readers down significantly and made reading a text overall much harder work. Not something anyone wants when they’re trying to communicate. Yet, most vocab materials still focus on lists of individual words.

Again, students (and teachers) like to see lists of vocabulary and once you start trying to include more than the simplest of phrases, trying to compile a list gets messy. Short phrases – at least, as a result – work okay on lists, but in sb’s safe/capable, etc. hands or know better (than that/than to do sth) probably won’t fit neatly on a single line and let’s be honest, do look a bit confusing.

The Challenge: Your brief states that there needs to be a list of key vocab at the start of each unit/section and the vocab list you’re working from contains largely single words.

The Work-around: Just because you’re highlighting individual key words in your headline list doesn’t mean you can’t work phrases, collocations and other chunks of language into your activities. For example, a simple activity in which students have to match sentence halves can work for checking comprehension of the key words (one per sentence), but can also involve students noticing a collocation in the other half of the sentence. This might be explicit – mentioning the collocations in the rubric and even getting students to underline the pairs of words – or if your editor’s not keen, just leaving in the collocation element quietly for students to absorb implicitly.

So those are my four broad principles, some of the challenges I regularly face in trying to stick to them and just a few of the work-arounds I use to argue my case, to bend a brief or, if all else fails, to sneak things in under the radar.

A couple of references:

Martinez, R. & Schmitt, N. (2012) A Phrasal Expressions List, Applied Linguistics 33/3
Millar, N. (2011) The Processing of Malformed Formulaic Language, Applied Linguistics 32/2
Nation, P. (2000) Learning Vocabulary in Lexical Sets: Dangers and Guidelines, TESOL Journal