Clojure & lx

lx in core.logic #3: Finite State Transducers

2012-02-25T05:04:00.003-08:00

This is the third post in the series on using core.logic to implement basic constructs in computational linguistics. If you haven't already, you might want to have a look at the first two before you start:

Today, we're gonna look at finite state transducers, which are commonly used to model and implement translation. While sounding fancy and powerful, they are straightforward extensions of finite automata.

Counting words

2012-01-29T08:18:00.001-08:00

Zipf's law is a well-know word frequency distribution. Let's assume you are learning a foreign language and your teacher gives you books to read. You have to take exams that test if you acquired the vocabulary of the books. You have other commitments, and you prefer reading blogs and books on computational linguistics, so you'd like to determine the most frequent words of the texts and learn them by rote memorization right before the exam. You know that the higher the frequency of a word, the higher the probability it will be on the test. At first, it seems to be obvious that we have to count how many times each word occurs in a text, but it will get a bit complicated.
We need a text file, I'm using Austen's Persuasion from the NLTK corpora.
Warning! slurp reads the whole file into the memory! Counting the words is pretty straightforward.
Plot the text with (graph-words austen) (or your text) and you will see something like this.

Not an informative picture! Let's analyse our text before we modify our program. The raw text file contains a lots of "noise". E.g. it is full of punctuation marks, our program is case sensitive and etc. Another problem lies in the nature of language.
Function words like determiners and prepositions are high frequency words. We are interested in the so called content words like nouns and verbs.

Part of speech tagging consumes your resources, so instead of removing function words identified by their pos tag, we are going to use a stopword list, and a list of punctuation marks. I used the NLTK English stopword list and made my own list of punctuation marks.
The stop lists are stored in sets because we can filter the complement of a set (in Clojure, filter gives you the elements, doesn't remove them). It is a common practice to remove hapax legomena from the distribution and to use logarithmic scales on the axes of the chart.
Now we've got a nicer chart.

The chart shows you that you can get a decent score if you concentrate on the most frequent words.

lx in core.logic #2: Jumps, Flexible Transitions and Parsing

2012-01-27T04:55:00.000-08:00

This is a continuation of the post Finite State Machines in Clojure core.logic.

This current plan for this series is to follow the book Algorithms for Computational Linguistics using Clojure core.logic instead of Prolog.

Jumps, wildcard transitions and parsing are natural and useful ways to extend and leverage finite state machines for text analysis. This was an opportunity to introduce extensions of fact databases and non-deterministic matching. Here's the code:

Finite State Machines in core.logic

2012-01-25T05:28:00.000-08:00

This is an implementation of Finite State Machines in Clojure using core.logic. They are a good starting point for computational linguistics and illustrate several features of core.logic, such as various ways of defining new relations, pattern matching and also the invertibility of relations.

It is not an introduction to core.logic. To learn the basics, I would recommend the Logic Starter.

Beginning with Clojure

2012-01-24T13:31:00.000-08:00

I am, by heart, a linguist, not a computational linguist. I was trained in Edinburgh, which is theoretically heavy, although not in Chomskyan, traditional linguistics. What I learned of Python I essentially taught myself, and there's no limit to my ignorance with traditional programming languages. That doesn't mean I'm not willing to try something new - far from it.

So, here we go. Rather than sit and pretend I haven't been twiddling my thumbs or busy for the past few months, I'm going to come straight out and say that is exactly what has been happening. I like blogging as I go along, though.

Install Clojure from this site. And here is where I ran into my first problem. I downloaded Clojure 1.3.0, unzipped it into my 'code' folder, and then cd'd in there in the Terminal. (I run a Mac.) The site suggests running this:

java -cp clojure.jar clojure.main

Well, that didn't work. (Neither did posting code snippets on Blogger, it seems. update: nevermind.) Instead, I got this:

Exception in thread "main" java.lang.NoClassDefFoundError: clojure/main
Caused by: java.lang.ClassNotFoundException: clojure.main
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

This is because there's an issue where it is packaged, and you need to call it by the specific package name. Quick fix, and...:

java -cp clojure-1.3.0.jar clojure.main Clojure 1.3.0

We're properly off!

I've been messing around with clojure on and off for a while now, over here. I highly suggest the tutorial, it is great. (I also highly suggest checking out this post on why Clojure Con was great, but that's not really on topic.

Depending on your development style, you may also want

line editing and history at the REPL
a syntax-highlighting editor
package management
automated builds
a full IDE
a tutorial environment

The site suggests all of those. I'm not so sure. I generally do all of my code with the Terminal and with MacVim. I'll be relying heavily on Vim for Clojure.

That's enough for this post. I'll put more up tomorrow, I hope! I know I'm late, but I view this as a running project and not an end-based one. Again, I'm a linguist, not a coder. So this is a long process.

Next: Getting Started.

What makes Clojure different?

2012-01-13T09:46:00.001-08:00

A friend of mine asked me why Clojure matters and what makes it special and why I think it is good for linguists. This post is the edited version of my answer to my dear friend. Since there are very good books on the market (my favourite is Clojure in Action) and the internet is full of good tutorials (4Clojure is esp. good if you like the learning by doing method) my goal is only to give you a rough picture of functional programming.

An example

We are going to solve a "toy" problem stolen from the first chapter of Peter Norvig's seminal Paradigms of Artificial Intelligence. The question is how do you extract first and last names from someone’s full name. Before you think this is too simple and it doesn't worth dealing with, consider names like Robert Downey Jr, Admiral Grace Hopper, and what about Staff Sergeant William "Wild Bill" Guarnere (a character for the Band of Brothers series). Machines should be programmed to solve these problems, and even humans could have problems with names. It took me years to figure out that Martin "Boban" Doktor (a well known Czech Olympic champion sprint canoer) is not a real doctor...

First, we need some data to test our assumptions.

The function 'def' associates the symbol 'names' with names (oh, a vector of vectors). A first name is usually just the first word in a name.

And the last name is the last word in a name.

Let's test our functions. Calling first-name and last-name on my name gives the right answers.

We stored out test data in names, and now it's time to test our functions en mass. The higher order function map helps us in doing so. Map takes a function as its first argument and applies it to every member of its second argument.

Oooops, the program is having serious problems with "titles" or prefixes. Calling last-name on names gives interesting results too. Our program is not that bad, it captures the basic logic of identifying first and last names, but affixes cause problems. The first name should be the first word in a name if it is not a prefix. Let's store the affixes in vectors.

We want to test if the first word of the full name is a member of the titles. We need a function that tests membership.

The function member is recursive. First, it test if its second argument is a sequence. The second if gives us a terminating condition, if x and the first element of the second argument are equivalent, it returns the whole second argument. Otherwise it tests the membership again on the rest of the sequence (i.e. everything but the first element of the original sequence). Now, we can redefine our first name function. If the first word of the full name is in the list of prefixes, call first-name on the rest of the full name, otherwise return the first word of the full name.

Testing our new function shows it works correctly.

We can redefine last-name similarly.

Storing names in vectors of strings is very unnatural (at least for humans, I guess machines don't care about these issues). Wouldn’t it be nicer to type names like "Zoltán Varjú" instead of ["Zoltán" "Varjú"]?

First, we need new test data, which is a vector of strings.

We want to use our first-name and last-name functions. Can we split a name into individual words? clojure.string provides us a split function (that's why we put (:use [clojure.string :as str :only [split]] :reload) into ns) which splits a string into a vector of strings at a given point. The space character delimits the parts of a name. Our source code looks like this now:

Now we can test split from clojure.string.

Let's define a split-name function just to save ourself from repetitive strain injury caused by excessive typing.

Finally, we test if our functions work on splitted names.

Notes

I have to note, you can make the code more concise and idiomatic. I hope you can see 1) how can you solve a problem with functions and by combining them 2) you have a basic idea of what is recursion 3) how can you go from a basic problem to an acceptable solution.

What makes Clojure different?

Norvig lists eight features that make Lisp different:

built-in support for lists
automatic storage management
dynamic typing
first-class functions
uniform syntax
interactive environment
extensibility
history (see Paul Graham's essays, What Made Lisp Different and The Roots of Lisp)

Clojure is a Lisp on the JVM which makes it unique. The Java Virtual Machine makes it portable, reliable and secure, but there is a new JavaScript based version called ClojureScript. Slime is an excellent development environment, leiningen makes project automation easy. Java interoperability means Clojure has got a great collection of libraries for almost everything.

However Clojure is not for complete beginners. The Clojure community is very open and supportive, but asking the right question requires some sort of maturity. As this Reddit thread explains you shouldn't be a Java expert to pick up the language, even you can learn what you have to know on the go. But you should know at least one 'conventional' language like Python before you start learning Clojure. More propaganda in our Why Clojure lx? post.

Hello nlp!

2011-12-29T11:48:00.000-08:00

This post assumes you have already installed Leiningen and you can work with your choice of programmers' editor.

Starting a new project

This command creates a new directory (hello-nlp). Navigate into that new directory and open the file project.clj. You are going to see something like this:

We are going to use the Apache Foundation's OpenNLP library with the help of Lee Hinman's Clojure library interface (and this post is based on Hinman's tutorial). Searching for “opennlp” gives various results, so we picked up the first (ending with 0.1.7). The information page contains everything you might want to know, the location of the github repo and a short code snippet for leiningen users [clojure-opennlp "0.1.7"]. Copy and paste the code into project.clj as follows:

Now your project.clj knows everything and is ready to serve you. The command

downloads dependencies (e.g. the clojure-opennlp library) and puts them into your path. Have a look at the lib library in your project library and you'll see jar files.

The core
Now navigate into the hello-nlp/src/hello_nlp/ library. You'll find a core.clj file there. Open it in your editor.

You'll see something like this:

To “enable” OpenNLP, modify the file:

You need a few additional files. Make a models directory in hello-nlp and download the pre-trained models from here (http://opennlp.sourceforge.net/models-1.5/). In this post, we are using English models, but feel free to change to another one. You need the Sentence Detector (en-sent.bin), Tokenizer (en-token.bin) and the POS Tagger (en-pos-maxent.bin).

Now, we can add user defined functions to core.clj. In the example, we made a sentence detector (get-sentences), a tokenizer (tokenize) and a POS tagger (pos-tag) based on the downloaded models.

Get your hands dirty

You can try out the newly defined function on your own sentences!

Road-map – or n+1 steps to enlightenment (or loonybind)

2011-11-28T10:14:00.001-08:00

As we expressed in our previous post, we'd like to experiment with Clojure. Let us emphasis again, we are NOT developing a new library, we just believe that using Clojure in linguistic computing might be fruitful. In order to prove this assumption (or refute it), we are going to try some tools out, and summarize and share our experiences as blog posts. Here is our tentative road-map.

Topics

We don't want to cover everything since we are neither omniscient, nor experienced Clojure hackers – so take our words with a grain of salt. We've chosen a few “core” topics interesting to us. Naturally, the topics are divided into two categories; “classics” and “using Java power tools”. Please leave a comment; we'd appreciate your thoughts (even your critique!).

Classics

Algorithms for Computational Linguistics – we stole the title from the great (and open) book by Striegnitz, Blackburn, Erk, Walter, Burchardt and Tsovaltz. We'd like to approach finite state techniques from two perspectives: logical and functional.

Zipf's law – the well-known distribution – is the guinea pig of linguistic statistics. Inspired by the ZipfR package and Incanter, we examine some very basic stats about texts, like word length and frequency and we try out plotting our results.

Java power tools

OpenNLP and the Stanford parser are real power tools. Tagging, chunking and parsing are indispensable when we are working with data.

Latent semantics – using chisel to do LDA analysis with the MALLET package.

Hints for newbies

2011-11-25T01:27:00.001-08:00

We received emails from interested folks who are new to Clojure. We hope they can find enough information about setting up a convenient environment for working with us so that they can provide us feedback. Here we give them a few tips. Please share your experiences in the comments!

Installing Clojure requires some expertise, this means you should be comfortable with your operating system. The easiest way to run Clojure, is downloading the clojure.jar file, and using the java -cp clojure.jar clojure.main command from the command line. However, this isn't the most effective way. Finding information about how to install Clojure on your platform is not impossible with a search engine. Ubuntu users find everything in Clojure on Ubuntu, please note the clojure github repo has been moved to https://github.com/clojure/clojure and clojure-contrib also moved individual repos, so don't follow the description literally!

You'll also need to install Leiningen. Why? As you can read on its repo “Working on Clojure projects with tools designed for Java can be an exercise in frustration. With Leiningen, you just write Clojure”. We are going to use Java tools, and the Clojars community repository provides us with these tools. Although using Leiningen to include various Java libraries into our projects looks very tedious (have a look at the sample file), but taking some time before getting into coding can give us goodies like the Stanford parser, OpenNLP, WEKA.

If you haven't found your text editor of choice, emacs with SLIME is the stuff you need. The Clojure, Swank, and Leiningen with Emacs on Linux shows you how can you set up your development environment.

We won't speak about version control, but you using version control is good house keeping technique. If you are new to this theme, and haven't committed yourself to a tool yet, have a look at git, and github, and read the git community book.

Why Clojure lx?

2011-11-16T09:30:00.000-08:00

The NLTK is a natural choice for students of linguistics and computer science. It has matured into a stable project, its users are very active, and it is now used outside of academia. Those who are into functional programming can use the Scheme Natural Language Toolkit, or learn from the Natural Language Processing for the Working Programmer, and those who needs the JVM can turn to ScalaNLP. So why brother with Clojure?

First of all, we are NOT proposing a new framework/library here! Our main goal is to examine what Clojure offers to linguists. Although more and more linguistics departments offer courses in statistics and probability theory, the vast majority of students graduate with some background in discrete maths, mostly taught in an implicit way through a class in syntax and/or semantics (and the same is true for philosophy education). Using computer programs to test our scientific ideas is becoming a common practice in sciences, and this is true for linguists too. Stefan Th. Gries distinguishes linguistic computing from computational linguistics; following him, we think linguistic computing will become a common methodology used in the language sciences.

So, what's the difference between computational linguistics and linguistic computing? Well, there is no clear boundary! We'd say computational linguistics (or natural language processing) is a kind of applied science and engineering, and as such it is more “goal oriented”. Norvig's recent critique of Chomsky shows that commercial success is a measure of ideas, but despite the proliferation of statistical methods linguists are still doing research on rule based systems like HPSG, minimalism, etc., and new interdisciplinary research themes have emerged like Parikh's idea of the social software (and game theoretic semantics and dynamic epistemic logic, among others). But what is “pure” research today can become applied research tomorrow. To foster communication between pure and applied research, between linguistic computing and computational linguistics, we need a lingua franca.

As Clojure is the Lisp for the JVM, it is a convenient language for linguists. In the not-so-distant past, Touretzky wrote his Gentle Introduction to Symbolic Computation, an excellent book for beginners in the humanities. Gazdar and Mellish Natural Language Processing in X (where X stands for Prolog, Lisp or Pop11) is a good introduction to finite state techniques, grammars, parsing and it even has a chapter on question answering. We don't deny that these techniques are old, but they are still part of the well-educated linguists' body of knowledge. Also, although Norivig's PAIP is a real gem, one cannot argue against the “old” AI paradigm without seeing the past, and those ideas are still important for linguist, philosophers and cognitive scientists. Logic programming is a natural pair of functional programming. The basic techniques of computational linguistics can be expressed in logic programs, and although they have their computational limitations, these little programs has got unquestionable educational value.

Porting the classic into Clojure is not a novel idea, as some Google searching shows that people are turning the classic Lisp books like PAIP or the Structure and Interpretations of Computer Programs into modern Clojure. The core.logic library opens up the possibility to do the same with the Prolog literature.

The most common argument against NLTK is that you can't use mature, industry standard tools like the GATE framework, Stanford core, and openNLP. Clojure's Java interoperability solves this problem. If you are into machine learning, Weka, MALLET and etc. are at your service. The Incanter package provides an R-like statistical library.

With these tools in your hand, you can test your ideas in a language that's very close to what you learned about formal languages. Using Java libraries is like using rapid prototyping material when you are a marble sculptor. And as your works end result can be shared with the computational linguists, you can get more feedback, and even help from the greater community.

That's why we think that Clojure lx is an idea worths exploring. We'd like to test ourselves! Can we use Clojure to express our simple ideas? How easy is it to use Java libraries for a project? If you would like to join us, please send an email to zoltan.varju(at)gmail.com. We welcome everyone, linguists and Clojure hackers, philosphers, digital humanists, everyone who is interested!

About us
Zoltán Varjú – computational linguist at Weblib LLC, @zoltanvarju, Számítógépes nyelvészet
Richard Littauer – MSc computational linguistics student at the University of Saarland, @richlitt

Special thanks to
Neil Ashton - @nmashton