The future of dictionaries (2): lexicographer versus computer

Some 20-odd years ago, as a young, Linguistics undergraduate, I became interested in the concept of computers ‘understanding’ human language. I did my undergraduate dissertation on Natural Language Processing (NLP), considering how far computers might go in really understanding language in all its subtle, complex, nuanced detail, and holding up the talking computer Hal, from Kubrick’s 1968 film 2001, as what I then suggested was an unobtainable goal. I went on to start a Master’s course in Computer Speech and Language Processing. I only lasted a term – mainly because I discovered I really hated all the computer programming involved, but also because I was disappointed to find that most of the course seemed to revolve around the speech processing side (i.e. voice recognition) and the language processing component came down to rather vague theoretical discussions that didn’t go much beyond my basic undergraduate research. Okay, that may be a bit of a distorted recollection of the actual course content, but I was only 21 and that’s how you see things when you’re barely out of your teens!

Obviously, in the intervening decades, technology has come on in leaps and bounds. Speech recognition has improved immeasurably - I'm actually dictating this blog post using speech recognition software and while it's not perfect, it's considerably more impressive than my early efforts are programming! I have to hold up my hands here and admit that I haven't kept up-to-date with developments in NLP, but I suspect progress has been much slower; we're still an incredibly long way from communicating with our technology in the same fluent way we can chat to our friends.

So, what’s any of this got to do with dictionaries? Well, let me try and explain my train of thought, triggered by the announcement by Macmillan back at the start of November that they are to stop printing paper dictionaries and focus on their online content:

• If publishers aren't actually selling paper dictionaries but are mostly focusing on a free online service, how much are they going to be prepared to spend on the time-consuming and labour-intensive work of lexicography?
• Of course, they'll be looking into other related income streams, selling dictionary data for other uses, and online advertising, but without a tangible, on-the-shelf product, will that justify quite the same budget?
• Reduced budgets often suggest a drive towards more automation, something we've already seen with the emergence of developments such as ‘TickBox lexicography’.
• Will more automation and “more efficient” ways of working inevitably lead to a drop in standards?

Clever developments in making the dictionary compilation process more automated do supposedly speed it up, for example, by automatically selecting ‘good’ dictionary examples from a corpus, to save a human lexicographer having to trawl through by hand. But any lexicographer who's worked with them will know that they only work to a degree and only speed things up to a certain extent … probably not quite compensating for the increased rate expected of said lexicographer without a drop in quality.

And then there's the whole established process for keeping dictionaries up-to-date. Currently, most dictionaries undergo a revision and a new edition every five years or so. This is a long, slow, and labour-intensive process that involves a team of lexicographers (mostly freelancers nowadays) going through the whole A-Z, looking at each entry and checking whether it needs updating. This doesn’t just involve adding trendy new buzzwords like ‘omnishambles’ or whatever – which are rarely of much use, or interest, to the average foreign learner anyway. There are all kinds of more subtle changes in the usage of existing words, sometimes due to linguistic trends and sometimes just as a result of changes in the real world. As one commenter on the Macmillan dictionaries blog pointed out, MED still contains an entry for Inland Revenue as the name of the UK tax authority, even though it changed its name to HMRC in 2005. And having done a quick search myself, I found it also has a couple of example sentences that rather unhelpfully in a digital age refer to cassettes (She slotted another tape into the cassette player. @ slot into, He quickly undid the screws that held the cassette together. @ undo).

 And I’m not just trying to pick holes in Macmillan here; all dictionaries naturally date as language and usage changes. Thus the need for new editions. And there are changes in style and presentation too as different aspects of language come to the fore within language research and teaching. More information about collocations has become de rigueur over recent years, for example. And whilst corpora are wonderful tools for researching collocational information, it still needs a team of lexicographers to trawl through each entry and decide where it’s worth adding a bolded collocate, or in some cases, whether a particularly strong collocation should actually be shown as a phrase or an idiom.

Which comes back to where I started … computer technology can do lots of wonderful things, but for me, when it comes to language, there still needs to be a human drudge working their way through that data to make intelligent decisions about what to present in a dictionary and how. In a world of online-only dictionaries, will dictionary departments have the clout to take on a team of lexicographers to do those regular sweeps through the database or will they just have a couple of people on the lookout for interesting, newsworthy nuggets that give the appearance of being “up-to-date”?

Spelling shame

Today, I had a chiropodist's appointment. When I arrived I was asked to fill in a form with a few bits of basic personal information. No problem, until I came to 'allergies'. I have an allergy to a common antibiotic, but I can't for the life of me ever remember how to spell it! My pen hovered for a moment, then I plumped for "penecillin" ... Sat in the chiropodist's chair, the polite conversation inevitably turned to:

"So what do you do?"
"Erm, I'm a writer. I mostly write dictionaries."
"Oh really, dictionaries?! You must be very good at English."
"Well, no, not really. My spelling's terrible actually. I've probably spelt penicillin wrong on that form, haven't I?"
(Looks at the form) "Yes. ... Oh well, we're all human, aren't we?"

I feel as if I ought to be able to spell better, but there's just something about certain words that I have a mental block with. I've always been the same - at school I was always being nagged by teachers about my spelling. I think it's something to do with spelling being something you just have to learn - I've never been good at learning things by heart - I have to have some kind of logic or reasoning behind something. I can explain complex grammatical rules, but I can't remember whether it's an e or an i in the middle of words like penicillin!

Thankfully, I know which words I can't spell, so in most circumstances, I just check - I've got a pile of dictionaries on my desk, after all! I've even got my most frequent problem word on a little post-it note on my monitor ...

Census

Last night, I was filling in my census form. I was ticking along happily, writing neatly in the boxes, until I came to the section about your job. When asked, I generally say that I'm a lexicographer; it's a nice neat description of a specific job and well, it's fun to have an interesting job title! I'm fairly certainly that I put down 'lexicographer' as my job on the 2001 census, but this year, I hesitated. I actually don't do an awful lot of lexicography nowadays. I write stuff, I edit things, I do corpus research, a bit of reviewing, a bit of teaching and only really occasionally work on a dictionary project. And there's something about a census form that compelled me to be accurate, so I finally plumped for the rather dull, all-encompassing "educational writer":

It's always a tricky question - on foreign visa forms, I usually go for the safe "teacher" option to avoid confusion and difficult questions from immigration officials. I once resorted to "clerical assistant" for a car insurance policy because anything that referred to "writing" seemed to put you in the same, ultra-high risk category as journalists and the like. I tried to explain to the girl that I just sat at a desk tapping stuff into a computer, thus she put me down as a clerical assistant and came up with a nice, cheap quote!

One out of thousands

At the end of last week I went for a biobank medical assessment. If you haven't come across it, biobank is a huge project to collect health and lifestyle information from thousands of people in the UK, who will then be tracked for years to come to see how their health develops and feed into various research projects. I got a letter a few weeks ago asking me to volunteer and last Thursday went along for a two-hour session at which I answered lots of questions about health, diet and lifestyle and had a whole range of medical checks.

Perhaps the most interesting part of the whole process for me was an interview with a nurse to add various bits of key personal data to a database. One of the first questions was "What's your job?". I was particularly impressed to find that lexicographer actually cropped up on the list of professions. Although the nurse admitted that I was the first one on the system!

More disappointing was the fact that RSI wasn't on the list of medical conditions and the nurse wanted to just ignore it until I insisted. It doesn't seem great for a study into the effect of modern lifestyles on health and adds to the frustration of having a controversial and often not recognised condition. Needless to say, I sent in my feedback at the end of the session and I'm awaiting a response.

A is for ...

As we bumped along in the back of the Landcruiser, trying to scan the horizon for giraffes or elephants at the same time as clinging on for dear life, I couldn't help but notice the huge pockmarked mounds of burnt orange soil dotted around the landscape. When we stopped, and I could be heard above the noise, I asked David, our guide, about them and he explained that they were termite mounds and the holes were made by aardvarks.

At the very mention of this singularly eccentric word, my lexicographer eyes lit up. Forget the cheetahs, forget the black rhino, I wanted to see the weird little creature that supposedly starts every dictionary; an aardvark! A little puzzled by my sudden overenthusiasm, David told me the termites' nemesis carries out his acts of murder and vandalism under cover of darkness. Undeterred, I talked him into taking us on a night drive.

Sadly, our usually attentive guide found the gossip of the friend he'd brought along to hold the spot lamp more interesting than our aardvark hunt and after a couple of hours of bouncing around in the dark trying to pick out shapes in the gloom while the spotlight roamed randomly across the landscape, we'd seen nothing more than a few startled antelope.

My lexico-zoological quest will go on though and I'm sure that on some future African adventure, I will come face-to-snout with this alphabetical oddity. At least, I suppose, I saw plenty of zebra ...