Text checkers: an overview

I’ve been mulling over a post about text analysis tools for ages but kept putting it off because I felt like I should research all the different tools out there thoroughly first. A recent post by Pete Clements though has forced my hand, so here’s my thoughts on the tools I have seen and tried. I should also say that I’m just focusing on the vocab aspect of the tools, not any other analysis features they have such as readability scores and the like.

So, what is a text analyser? Basically, it’s an online tool that allows you to cut and paste a text that you’d like to use with students into a box, you hit a button and it comes back with stats about the text. In particular, what most ELT materials writers are interested in is the level of the vocab. We’re usually looking for a breakdown by CEFR level to tell us whether the text is suitable for a particular class/level and which words might be “above level”.

READ THIS BIT FIRST
Before you use any kind of text analysis tool though, here are some basics to bear in mind:

WHICH WORD LIST?

It’s really important to understand how the tool you use is making those judgements about level. Most tools use some kind of word list that’s been developed to peg individual words to CEFR levels. It goes without saying that this in itself is fraught with problems – my blog post here looks at some of them. But if we’re accepting the basic premise of using a word list, then you need to know which one. If you can’t find out which list a tool is using, then I’d probably say, don’t use it because you can’t know what it’s showing you.

 A number of tools use Cambridge’s English Vocabulary Profile (EVP) list – the key thing to understand about EVP is it ranks words (largely) by productive level – so the level at which you might typically expect a student to be using a word themselves. Given the way we acquire vocab that might be a level (or two) after students recognize and can understand the same word receptively, i.e. if they read it. The Oxford 3000, on the other hand, ranks vocab more by receptive level, so the point at which students will typically be able to read and understand a word.

Text analysers use clever algorithms to analyse the text you input, but these have a number of shortcomings it’s really important to be aware of:

PART OF SPEECH

Starting at the most basic level, the tools don’t always correctly identify the part of speech of a word, especially words that have the same forms across parts of speech. So, weather is most frequently a noun, but can also be a verb, national is mostly an adjective, but can be a noun (a foreign national). Most tools will opt for the most common form and label its level accordingly. I put in the sentence:  Some people who contract this virus can feel very poorly for three to four weeks. And most of the tools identified contract here as a noun and labelled it as around B1, when in fact it’s a verb and EVP pegs it at C2.

Text Inspector: Out of all of the tools, the only one I’ve found to really deal with this issue is Text Inspector which allows you to click on any word in a text that looks like it’s been tagged incorrectly and choose the correct use (meaning and part of speech) from a drop-down menu.  Of course, that means you have to spot the incorrectly tagged words, but it’s better than most. 

[Click to enlarge the image].

Oxford Text Checker: If you hover over a word in your text with more than one possible part of speech, the Oxford Text Checker shows a box giving the CEFR level of each one, e.g. v = A2, n = B1. Although it only shows the level for the most basic meaning of the word (see multi-sense words below).

Many tools also fail to label certain words, especially function words. So, in the sentence - Others end up in hospital needing oxygen. – many tools left others without a level because they just weren’t sure what to do with it grammatically.  Contractions (they’ve, she’d, who’s) also tend to go unlabelled, but are rarely a big issue.

MULTI-SENSE WORDS

English is a highly polysemous language; lots of words have multiple meanings which students are likely to come across and recognize at different levels. Most published word lists take this into account and assign different level labels to different meanings. Most text analysers though just opt for the most frequent (and usually lowest level) sense. We’ve already seen that to an extent with contract above, but even without the part of speech issue, if you put in a sentence like - There are links in the table below. – table will be shown as A1 (the piece of furniture) rather than A2/B1 (a graphic).

Text Inspector: As we saw above, Text Inspector gets around this by offering drop-downs for any words you suspect may be used in a less obvious sense.

MULTI-WORD EXPRESSIONS

For me, the biggest issue to look out for with text analysers is that they mostly treat words individually and ignore the fact that a large proportion of most texts (30-50% by some estimates) is made up of chunks; phrasal verbs (end up, carry on), mundane phrases (of course, as usual, a lot of) and idioms (under the weather, have no idea). And of course, a phrase is often going to have a very different level from the sum of its parts.

Unlike with the more glaring mis-tags of part of speech and meaning, I think multi-word items are far more difficult to spot because as expert speakers, we tend to read through them without noticing, but for students, an unknown phrasal verb can be a real stumbling block. It takes a keen eye to spot every phrase and phrasal verb in a text when it hasn’t been tagged.

Text Inspector: Again, the only tool to rate a bit better here is Text Inspector. It does at least manage to identify some phrases. In the sample text that I’ve been using for this post, it correctly picked out the phrasal verbs end up and carry on and the phrase have no idea.

It didn’t recognize pass it on (presumably because of the object in-between), but you can click on pass and choose the phrasal verb sense from the drop-down. Similarly, it didn’t pick up under the weather, but again, you can click on weather and select the idiom and it changes weather from A1 to C2. It doesn’t allow you to neatly link up the whole phrase (I don’t think), but it’s a reasonable compromise.

 

WHICH TOOL?

You’ll probably have gathered by this point that Text Inspector is very clearly out in front when it comes to analysing vocab from an ELT perspective. I subscribe to the paid version which gives you full functionality. You’ll find a link to a free version in Pete’s post which does much the same, but I’m not going to reshare it because, well, I think we should be paying for the good stuff and it’s a very small amount to invest for a really useful resource.

Here’s a brief overview of some of what’s out there though:

Text Inspector

Free version has limited functionality and doesn’t give CEFR analysis. Sign up for the paid version to get everything via: https://textinspector.com/

Word lists: The paid version allows you to analyse the vocab in a text in terms of EVP, AWL (the Aacademic Word List), BNC and COCA – these last two are corpora and it shows you the frequency of words as they appear in each corpus – useful if you’re into corpora.

Comments: By far the best I’ve seen in terms of at least trying to take into account the factors above.

Oxford Text Checker

Free via: https://www.oxfordlearnersdictionaries.com/text-checker/ (If you get to the main dictionary home page, click on Resources to find it)

Word lists: Based on the Oxford 3000 & 5000.

Comments: Easy to use and colour codes words by CEFR level. However, it always opts for the most common form/meaning of a word and doesn’t recognize phrases. If you hover over a word, it does at least show different CEFR options for different parts of speech, e.g. hovering over feel, you get a box showing v=A1 n=B2. You can also double-click on any of the words in your text to go direct to the dictionary entry which is useful for quickly checking the CEFR label against different meanings. It also has options to create word lists and activities from texts, but given the shortcomings, I wouldn’t be inclined to use them without heavy editing.

 

VocabKitchen

Free via: https://www.vocabkitchen.com/profile

Word lists: It shows words on the AWL and NAWL (New Academic Word List). It also claims to show words by CEFR level, but I can’t find out what word list it’s using which for me is a bit of a red flag.

Comments: It’s intuitive and easy to use, but again doesn’t account for different meanings or phrases. I believe it has more options if you register and sign in which I haven’t tried out.

EDIA Papyrus

Free but you need to register via: https://papyrus.edia.nl/

Word lists: This site is based on a mix of experts’/teachers’ assessments of the level of texts and AI.

Comments: Quite a nice interface, but it seems to skip quite a few words in your input text - not just function words, but you can see below it completely ignores the phrasal verb end up. And as above, doesn’t deal with different meanings or phrases.

 

LexTutor

Free via: https://www.lextutor.ca/vp/

Word Lists: Originally designed for corpus geeks, the main focus for this tool is around corpus frequencies and the AWL. It does now include a CEFR option, but reading through the blurb, the CEFR levels seem to be based on some very old (1990) word lists published by Cambridge way back before this became a properly researched area, so I’m not sure how useful they are.

Comments: A horrible user interface, still really for geeks only. It's so messy, I couldn't even get a meaningful screenshot.

Pearson/GSE Text Analyzer

Free via: https://www.english.com/gse/teacher-toolkit/user/textanalyzer

Word lists: based on Pearson’s own Global Scale of English (GSE) lists

Comments: I hesitated to even include this as it’s just plain weird – unless I’ve missed something. It calculates an overall level for your text but doesn’t show the level of individual words. It does highlight words that it judges to be ‘above level’, but the choices seem to be a bit random. It pegged my sample text at B1+, then picked out poorly and passing as above level, ignoring asymptomatic.

 

 

Ludwig Guru: a review

Recently, a fellow ELT writer posted in a Facebook group about a new language tool they'd discovered. I hadn't come across it before, so couldn't resist checking it out.

It's called Ludwig Guru and it describes itself as:

"the first sentence search engine that helps you write better English by giving you contextualized examples taken from reliable sources." 

It's aimed at learners/non-expert users of English and the idea is you type in your best guess at an English sentence, or part of a sentence, and it comes back with examples of similar sentences from 'reliable sources'. Then you can see how well the examples match your own attempt. Presumably, if you find lots that are exactly the same, you know you're on the right track and if they're a bit different, you can adjust yours to sound more natural.

The post that had led me to it was from an ELT writer looking for ideas for how a slightly obscure tense (future continuous passive, yes, it's a thing!) is typically used. My first reaction was "Why not use a 'proper' corpus?" … but I am aware that corpus tools can be off-putting until you get used to them and this looked like a potentially more user-friendly alternative. I decided to test it out to see whether it might be a useful tool for ELT writers for checking intuitions or searching for ideas for authentic examples/contexts, as well as a tool to recommend to students.

As with many similar services, there's a free version with limited functionality and a premium version that gives you the full experience. I registered for the free version just to try it out. It's very restrictive! You only get 6 searches per day – and that's a 24-hour period, so if you hit your limit in the afternoon, you can't log back in the next morning – which made it very difficult to test out in any meaningful way. You also only get 15 results per search, which again made it difficult to know whether what I was seeing was a representative sample of what you'd get from a wider search.  You can sign up for a free 15-day trial of the premium version, but that requires you to enter your credit card details, which I wasn't prepared to do. So, to be honest, I didn't get as far as I'd like before I just gave up! But here's what I did find.

The data:

My first question was about what constituted 'reliable sources'. The site uses 22 sources including 8 news media sites (including the BBC, the Guardian, the New York Times, etc.), it has 5 academic science sources (mostly scientific journals), a couple of wikis, a couple of encyclopedias and a collection of other sources that it describes as 'Formal & Business' but are a bit of a mixed bunch, including documents from UNICEF and the European Parliament. You can (with the premium version) choose to filter your results by selecting which sources you want to include. 

My first thought is that it's actually not a bad spread. Many corpora depend heavily on news media sources because they're readily available and reasonably wide-ranging in terms of topics (and so spread of language/vocab). The encyclopedias and wikis will also provide a nice spread of topic vocab. The language of journalism though (and of reference materials too, I suspect) is quite a distinct genre, so isn't necessarily an ideal model for other contexts.

The academic content is made up of only science journals, so obviously doesn't help with other academic disciplines.  I also noticed that a lot of the unexpected results that came up showing examples that felt awkward (and in some cases positively incorrect) came from this section of the data. When I clicked through (as you can) to the original sources, they were papers that appeared to be written (at least judging very crudely by the names of the authors) by non-native speakers of English. That's unsurprising seeing as many academic papers in English language journals have a very international mix of authors. The judgment as to whether something that has managed to pass through the reviewing process for a journal (and in this case actually a small range of journals mostly from the same publisher) represents a good language model or not is up for debate.

The overall British/American split is difficult to determine, but the news media are 50/50 – which is just something to bear in mind as some searches will throw up clear differences between the two. For example, write me (with the person as the direct object) is standard in American English but sounds distinctly odd to a British English speaker (who'd use write to me).

For learners:

The tool has been designed for non-expert users of English to search for specific phrases, so this is where I started.  The results were kind of mixed and the main thing I took away was that they needed quite a degree of language awareness and analysis to be useful. Here a just a few of the searches I tried and the issues they threw up:

Just as a side note, I actually took some of the examples that sounded awkward/unlikely to me from interviews with the creators of the app itself … and interestingly, those exact examples often came up as the first result!

One obvious search is to check collocations, so I looked at a couple that seemed slightly odd to me; firmly think and obtain my goal. My intuition tells me (as do reference sources and other corpus evidence) that we'd be more likely to say firmly believe and attain my goal. Ludwig came back with the following results (click on the images to enlarge).

As a seasoned corpus researcher, I know that in a large body of data, you'll probably find some examples of almost any combination of words. Mostly though, with very small numbers like these, you'd discount them as untypical and unhelpful for a learner. (As I mentioned above, many of the results for obtain a goal, in fact, come from a handful of academic papers likely written by non-native speakers.) Corpus research is all about identifying frequent and typical patterns, not individual quirks of usage. For the student using a tool like this, I guess the question is how they make that judgment. Will they see that there are actually only a relatively small number of matches and instead click through to see the similar patterns? Or will they just see a first screen full of examples that appear to match their own, possibly slightly awkward, wording and stick with it?

If students do discount patterns with fewer hits, then the other tools available can be really helpful. The search for obtain a goal above shows suggestions for achieve/realize/attain a goal – all good, solid collocations. Another search for the slightly awkward a large part of them only turned up 17 exact matches, but Ludwig allows a search for synonyms (by putting an underscore before the word you want synonyms of) which offers some good alternatives shown in frequency order; a large percentage/proportion of them.

The other major issue, of course, is that learners need to feel that a construction might not be right in order to decide to check it in the first place. One review of the app which explicitly highlighted that it was written by a non-native English speaker using the app still contained a few clear language errors. That's not a criticism of the writer – or even of Ludwig, to be honest – but it goes to show that it's impossible to be conscious of all your own errors.

For language research:

Both the basic searches and some of the other tools available do have an appeal for the ELT writer wanting to check out typical usage or just search for ideas, but I think the limitations probably outweigh the benefits.

I was initially unsure whether the searches were lemmatised or not … by that I mean that if you search for take do you just get results for that exact form or do you also get takes/taking/took/taken? It's difficult to be certain with so few search results returned – many of my searches just seemed to come up with the exact form I'd typed in, but then some less frequent ones, like obtain my goal above, did seem to show other forms (obtaining) as 'similar' results. It seems though that exact matches always come up first and they are just that 'exact'. Which is not very helpful for researching most language patterns where you want to allow at least some variation. Even if you were searching for a particular tense, say present perfect, you'd want to allow for has done and have done. Certainly, if you were looking to compare collocations using the comparison tool, e.g. [take get] a bus, you wouldn't want to just look at the base form of the verb, you'd want to compare across all verb forms.

Another issue when it comes to searching for language patterns is allowing for variation. So taking that same example of take/get + bus, you want to see not just take/get a bus, but take the bus, take the airport bus, take the next bus, etc. too. Similarly with verb patterns, you want to allow for negatives have not done and possible adverbs have already done. It may sound a bit silly but by searching for exact matches, you only find what you were searching for … when actually what's often more useful, typical or interesting are the variations you hadn't thought of.

While investigating, I did come up against a number of unexpected results. So, for example, I searched for have * been which should have shown me the most common words that occur between have and been. A standard corpus search uncovers plenty of examples of have already/now/also/just/long/not, etc. been, so it was slightly surprising that Ludwig returned no matches at all. Oddly though, one of the suggested similar searches, the much less frequent, have * participated, came up with 5 matches (have also/never/already/not/consistently participated). This just planted a seed of doubt in my mind about consistency and reliability, but wasn't something I could really explore further within my limited searches.

Conclusions:

Overall, I think the idea behind the project is a good one and the app has some really nice features … but for me, the limitations of the free version make it fairly unusable and the limitations of the whole thing make it not worth paying for premium. Certainly in the case of an ELT writer, you'd be much better off investing a bit of time and your subscription money in learning to use a standard corpus tool which will give you much more flexibility and functionality.

Word Booster update

Last year, I wrote a review of Word Booster, an online tool that allows you to create an ELT lesson from an online text. It creates a (fully credited) pdf of the text that you can print out for students, along with definitions for key words and a follow-up vocab quiz. At the time, I was disappointed that an idea which seemed so promising fell short on a number of important details.

As soon as I’d posted the blog, the creator of Word Booster got in touch. He was really positive about my feedback and keen to improve the tool as quickly as time, manpower and finances would allow. I was really impressed by his commitment and even more impressed when he got in touch again recently about the latest updates to Word Booster.

• The latest version of the tool uses a learner’s dictionary (the Cambridge Advanced Learner’s Dictionary) which makes the definitions appropriate and accessible to the average learner.
• Whilst the tool makes suggestions about which words in a text to focus on, the user/teacher is now free to accept or reject these suggestions and to choose whichever words or phrases they feel are most appropriate for their learners or for the aims of the lesson.
• The tool suggests an appropriate definition for each word, but allows the user to check it’s the correct sense manually and change it if necessary. This is a massive improvement as automated sense selection can be a bit hit and miss. You can see in the example below, using one of my own blog posts, that when I click on 'folk' in the text, I'm able to select the appropriate sense for the context (the tool had automatically selected the more frequent, 'music' sense). [Click on the picture to view it more clearly full-screen.]

• There are also options to (de)select example sentences and to adjust the quiz activities slightly – although not to make edits beyond shuffling around which words appear in which activity type.

All of these changes make the tool far more usable. It still has a few minor technical glitches that I’ve passed back to the Word Booster team, but overall, it’s something I’d now happily recommend for teachers to try out.

I do, however, still have a few reservations.

Dictionary definitions in vocab activities

As a lexicographer myself, I’m a big fan of learner’s dictionaries, but I’m still slightly wary about the use of dictionary definitions in vocab practice activities. Research seems to show that using dictionary look-ups or referring to glosses while reading a text helps students’ incidental* vocabulary learning (Laufer & Hill, 2000). By actively looking up a word, focusing on the form and meaning, and relating it to the context, students are more likely to remember it later. And this is exactly what learner’s dictionaries are intended for. Definitions are written in the expectation that the students will come across a word and look it up to check the meaning – they’re intended for decoding. That’s what the first part of the Word Booster tool caters to perfectly.

Where I feel we get onto shakier ground is in the ‘reverse engineering’, if you like, where students are essentially given a definition and asked to guess the word. This is potentially a much more challenging task and isn’t something that dictionary definitions are designed for.  Without the target word alongside, a dictionary definition can seem vague, rather abstract and certainly very difficult to tell apart from definitions for similar words. 

That’s not a criticism of dictionary definitions, it’s just the nature of the beast. Definitions have to be concise, so there can’t be lots of detailed explanation to differentiate between similar words**. They have to be written within a defining vocabulary (a limited set of words that avoids the definitions being more difficult than the words they define), so they necessarily can’t be as subtle and nuanced as those in a dictionary for native speakers. They also have to cover all the possible uses of a word, which can make them a bit vague and sometimes slightly awkward. As a lexicographer, you split out clearly different senses, but you can’t just keep on splitting endlessly, you have to lump similar uses together at some point (e.g. this two-part definition from Cambridge Dictionaries – “option: one thing that can be chosen from a set of possibilities, or the freedom to make a choice”).

As a materials writer, if I want to create a practice activity around definitions (which, by the way, I’d do sparingly anyway), whilst I might start off by looking at a dictionary entry, I’d invariably edit the definition. I might change the wording, for example, losing slightly formal passives (e.g. “one thing that you can choose”). I might make it a bit more specific to the context at hand – so I’d choose just the relevant part of the above two-parter. And, if I was dealing with near synonyms, I’d probably add a bit more detail to help make the distinctions clearer.  

At the moment, many of the quiz questions generated by Word Booster are at best very tricky and at worst, downright confusing just because of the nature of the dictionary definitions. Being able to edit the defs in the quiz questions would undoubtedly help, but at the same time, it would add to the time required to create the material (which kind of negates one of the key selling points of the tool) and I guess, the ‘authority’ of the definitions would be lost somewhat. It seems to me that the key here is to use the activities sparingly and to choose items carefully, keeping an eye out for odd and confusing defs or combinations and deselecting them. Which brings me onto my main takeaway about this tool …

… it’s how you use it.

Just like any other tool, the success of what’s produced comes down not only to the features of the tool itself, but to how it’s used. As a novice teacher many years ago, I didn’t have fancy online tools like this, but I certainly fell into the trap of choosing a news article that I thought was interesting, photocopying it, picking out some random vocabulary and quite often writing out dictionary definitions for students to match to words from the text. The result was a bit of a confusing mess of a lesson, in which we’d invariably end up decoding the text as a class line-by-line because it was too hard for the students to manage. I’d have to give extra explanation of the definitions of above-level vocab and I’d often struggle to remember the correct answers to the questions that seemed obvious when I wrote them, but which, in the middle of a lesson with a load of confused students, suddenly didn’t make sense any more.

To use a tool like Word Booster effectively, the teacher needs to consider:

1. The choice of text – is it at the right level for the students both linguistically and cognitively? A few above-level words might provide challenge and interest, but too many will be confusing and demotivating. Is it the right length for the lesson?
2. The choice of vocab – in my last post, I wrote about the importance of choosing reasonable vocab sets to work with and about having a clear aim for vocab activities (Are you focusing on receptive or productive vocab? Do you want students just to decode the text or are these words useful to learn?). How many words is it reasonable to highlight and practise?
3. The choice of definitions – the option for the user to pick the correct definition is really useful, but it requires some skill. How is the word being used here and which def fits best? Is the word being used metaphorically? Is it actually part of a phrase, a phrasal verb or an idiom? Is the correct sense available in this learner's dictionary at all?
4. Activity selection – personally, I think one short-ish definition-based activity per text, using carefully-selected definitions that don’t cause confusion, is probably enough. I might stretch to a second that uses example sentences, but for me, any more than that and it’s becoming a bit mechanical and repetitive. I’d then want to supplement the material generated by Word Booster with some of my own content. Just mining a text for vocabulary doesn’t amount to a successful, engaging lesson. At a minimum, I’d want to add some kind of comprehension questions – whether those were traditional written questions about the text or looser points for discussion. I’d then want some kind of follow-up – a response to the content of the text, perhaps in the form of group discussions, maybe a writing task.

Overall, I’m really impressed with the improvements that Word Booster has made over the past year and I know the team have more upgrades in the pipeline to continue refining their algorithms and adding more features. I’d certainly say it’s worth trying out though. Whilst creating a usable lesson involves a bit of work in terms of choosing the vocab, checking definitions and selecting appropriate quiz questions, I think it does save time in creating a basis for a lesson that you can then build around.

*The term incidental vocabulary learning, doesn’t mean words that students just come across by accident. Incidental learning can be quite planned and intentional, but it just isn’t the main focus of the activity. So in a reading lesson, the main focus is on understanding the text, maybe for discussion or to answer some comprehension questions, but there can be a conscious focus on vocab too – this would be incidental learning.

**When I was working on the Oxford Learner’s Thesaurus, we often needed to add whole extra sentences to help differentiate between synonyms.

Reference:

Laufer, B. & Hill, M. (2000) ‘What lexical information do L2 learners select in a CALL dictionary and how does it affect word retention?’ Language Learning & Technology