In a previous post, I introduced Clojure, a Lisp that runs in the JVM, and gave a simple example of using it for a slightly XNAT-related task. I’ve also been using Clojure as a client for the XNAT REST API, and I’ve written a small Clojure library for interacting remotely with XNAT.

There’s much more to set up in this example than in the previous one, because I use some third-party libraries, so there’s dependency downloading and classpath setting and all the complications that come from using libraries in the JVM. If I were writing this in Java, I’d wrap it all up in a Maven project. Maven can be used with Clojure, too, but I use Leiningen, a Clojure build tool that does Maven-style dependency management.

Here are detailed instructions for using Clojure as an XNAT REST client, adapted from the xnat4clj documentation:

1. Get a local copy of my Clojure XNAT code:

$ hg clone https://bitbucket.org/karchie/xnat4clj

2. Install Leiningen:

$ wget https://raw.github.com/technomancy/leiningen/stable/bin/lein
$ mv lein ~/bin     (or somewhere else in your $PATH)
$ chmod 755 ~/bin/lein    (make it executable)

If you’re a Mac OS X user, you might not have wget. curl will work, or you can use a web browser and save the file. If you’re a Windows user, I’m sorry. Really I am. You might be able to get this to work anyway.

3. Download the dependencies:

$ cd ~/xnat4clj
$ lein deps

4. Start the Clojure environment:

$ lein repl
REPL started; server listening on localhost port nnnn
user=>

Now you’re ready. I’ll start with another simple, not-quite-XNAT task: let’s use the DICOM C-ECHO service to make sure that the XNAT DICOM service is running. First, load the file where I’ve written some DICOM support code (all based on dcm4che):

user=> (load-file "src/nrg/dicom.clj")
#'nrg.dicom/verify-service
user=>

We loaded a Clojure source file into our REPL; Clojure compiled it and now the functions defined in it are JVM bytecode, ready to be evaluated. Clojure printed #’nrg.dicom/verify-service because the definition of that function is the last thing in the file, which Clojure treats as the result of the function call to load-file. Now let’s use that function:

user=> (nrg.dicom/verify-service "XNAT" :hostname "localhost" :port 8104)
ConnectException Connection refused  java.net.PlainSocketImpl.socketConnect (PlainSocketImpl.java:-2)
user=>


Oops. There’s no receiver to take the call — XNAT isn’t running. I’ll go start Tomcat and try again.

user=> (nrg.dicom/verify-service "XNAT" :hostname "localhost" :port 8104)
(2088)
user=>

Okay, that’s different, but a little cryptic. Let’s look at the documentation for verify-service to see what it’s doing:

user=> (require 'clojure.repl)
nil
user=> (doc nrg.dicom/verify-service)
-------------------------
nrg.dicom/verify-service
([scp-ae-title & {:keys [count device-name hostname interval port scu-ae-title], :or {count 1, device-name "cljecho", hostname "localhost", interval 0, port 104, scu-ae-title "xnat4clj"}}])
Use C-ECHO to ping an SCP n times. Returns a sequence of length count, representing the time in ms of each C-ECHO operation.
nil

The first line, (require ‘clojure.repl), loads the doc function; doc prints the documentation for any defined function. Looking at the documentation for verify-service, our result above was a sequence (surrounded in parentheses () ) of response times, measured in millseconds — so we did one C-ECHO and got a response in 2.1 seconds. Let’s do it five times instead:

user=> (nrg.dicom/verify-service "XNAT" :port 8104 :count 5)
(1 0 1 0 1)

Those C-ECHO times were a lot shorter, 1 millisecond or less each; there must have been some startup cost on the first C-ECHO call. (I took advantage this time of the fact that :hostname defaults to “localhost”. You can look at the documentation until this makes sense, or just trust me on it.)

Well, this is nice, but it’s only sort-of interacting with XNAT. Let’s try using the REST API.

user=> (load-file "src/nrg/xnat.clj")
#'nrg.xnat/with-xnat
user=> (in-ns 'nrg.xnat)
#<Namespace nrg.xnat>
nrg.xnat=>

This loads the file xnat.clj, where I’ve defined some REST API wrapper functions, and changes to the nrg.xnat namespace to save some typing. Let’s start by letting the environment know where to find our XNAT:

nrg.xnat=> (set-xnat! "http://my.xnat.org" "username" "password")
nil
nrg.xnat=>

This sets some internal variables so that all following calls know where to find XNAT and how to authenticate. Let’s try a few simple functions first:

user=> (get-project-ids)
("kaat")
user=>

I have access to just one project, named kaat. What subjects do I have defined in that project?

user=> (get-project-subjects "kaat")
("s001" "CP10048")
user=>

So there are two subjects. Let’s get a list of experiments for subject s001:

user=> (get-project-subject-experiments "kaat" "s001")
("s001_PIB_1" "s001_F-18_1")
user=>

There are about 30 functions defined in xnat.clj, all of them with documentation strings in the definition.

Let’s try something a little more complicated now: for an archived session, re-extract the DICOM metadata from all the DICOM files. We need to do this occasionally when we change the metadata translation rules, usually by adding new DICOM fields to the XNAT metadata; or when we use the REST API to add some more files to an existing session.

nrg.xnat=> (pull-data-from-headers! "project" "subject" "session")
project / subject / session : MyXNAT_E2015 -> 200

This prints the project, subject, and session, the site-unique experiment ID, and the HTTP response status (200 is the code for OK).

If the session label is omitted, re-extraction runs for all sessions for the provided project and subject. Similarly, if the subject and session labels are omitted, re-extraction runs for all subjects and sessions in the provided project.

In some cases, the REST connection may time out before the re-extraction is complete. The error in this case looks like:

nrg.xnat=> (pull-data-from-headers! "p" "s001" "s001_MR1"))
TimeoutException No response received after 60000 com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture.run (NettyAsyncHttpProvider.java:1922)
nrg.xnat=>

This error is usually harmless, since the XNAT server is still running the request. However, it does hide the HTTP response. The timeout (in milliseconds) can be increased via the optional :timeout argument to set-xnat! :

nrg.xnat=> (set-xnat! "http://my.xnat.org" "u" "p" :timeout 120000)
nil
nrg.xnat=> (pull-data-from-headers! "p" "s001" "s001_MR1"))
p / s001 / s001_MR1 : MyXNAT_E3117 -> 200
nrg.xnat=>

It’s possible to use pull-data-from-headers! to re-extract the metadata for all of a subject’s experiments, or for all experiments in a project, by passing it different parameters — call (doc pull-data-from-headers!) for more details.

xnat4clj is just a collection of functions I’ve written to solve specific problems, so it’s not as comprehensive as it could be — pyxnat is a great example of a more complete solution (in Python). So why use Clojure? I find the REPL a great tool for experimenting and doing incremental, interactive development. I can build up state as I go by saving results in variables:

nrg.xnat=> (def kaat-subjects (get-project-subjects "kaat"))
#'nrg.xnat/kaat-subjects
nrg.xnat=>

then use that value in later function calls:

nrg.xnat=> (map #(get-project-subject-experiments "kaat" %) kaat-subjects)
(("s001_PIB1" "s001_F-181") ("CP10048_v1"))
nrg.xnat=>

Any changes I need to make can be loaded into the running environment, either by typing in a new function definition, or by calling load-file to reload a modified source file. There’s no need to exit, recompile, and restart when I change the code.

In my next post, I’ll describe a way of getting a Clojure REPL inside a running XNAT, allowing you to interact with existing Java objects or create new ones interactively, at runtime.

Over the last two years, I’ve been using Clojure quite a bit for managing, testing, and exploratory development in XNAT. Clojure is a new member of the Lisp family of languages that runs in the Java Virtual Machine. Two features of Clojure that I’ve found particularly useful are seamless Java interoperability and good support for interactive development.

“Interactive development” is a term that may need some explanation: With many languages — Java, C, and C++ come to mind — you write your code, compile it, and then run your program to test. Most Lisps, including Clojure, have a different model: you start the environment, write some code, test a function, make changes, and rerun
your test with the new code. Any state necessary for the test stays in memory, so each write/compile/test iteration is fast. Developing in Clojure feels a lot like running an interpreted environment like Matlab, Mathematica, or R, but Clojure is a general-purpose language that compiles to JVM bytecode, with performance comparable to plain old Java.

One problem that comes up again and again on the XNAT discussion group and in our local XNAT support is that received DICOM files land in the unassigned prearchive rather than the intended project. Usually when this happens, there’s a custom rule for project identification where the regular expression doesn’t quite match what’s in the DICOM headers. Regular expressions are a wonderfully concise way of representing text patterns, but this sentence is equally true if you replace “wonderfully concise” with “maddeningly cryptic.”

Clojure has special syntax for regular expressions, so testing them is as easy as firing up your REPL (Read-Eval-Print-Loop, Lisp jargon for the interactive environment) and trying your regular expression against some test cases:

user=> (re-seq #"PN-(\d+)" "PN_371")
nil


What did I just do?

user=> is the prompt: Clojure is telling you that it’s ready for you to type something.

re-seq is the function we’re calling. It returns a sequence of regular expression matches, or nil if it fails.

#"PN-(\d+)" is the regular expression. The # in front tells the Clojure reader that the following string should be interpreted as a regular expression. This is analagous to the Java expression:

Pattern.compile("PN-(\\d+)");

"PN_371" is the string we’re testing against the regular expression.

nil is an empty result, indicating that there was no match. Let’s modify the regular expression to try to match the value.

user=> (re-seq #"PN[-_](\d+)" "PN_371")
(["PN_371" "371"])

That’s better! Now we get a sequence (surrounded by parentheses () ) containing one array (surrounded by square brackets [] ) of match groups. As with Java’s Matcher.group(i), group 0 is the text that matched the whole pattern, and each following group is the part of the text that matched the subpattern in a parenthesis pair in the original regular expression. By default, XNAT’s DICOM receiver (and the old
XNAT 1.4 DicomServer) use group 1 for extracting project titles, but that can be changed in dicom-project.rules (or the project_spec element of DicomServer.xml).

This is a very simple example, but it shows some advantages of a JVM-based, interactive environment: we’re using the same Java pattern matching machinery that XNAT uses, but we don’t have to build all the Java class boilerplate, write code to deal with input and output, and then compile and run the class. We can test a new regular expression or string as quickly as we can type it.

There are other JVM languages with interactive environments. Before Clojure came along, I often used BeanShell for testing regular expressions (but not for much else). Many people like Groovy. Scala has a REPL too. JRuby and Jython are JVM implementations of languages you may already know. I’m a long-time parenophile, so Clojure is a natural fit for me.

I’ve skipped a lot of details, most notably how to get and run Clojure. One really easy way is to go to Try Clojure and use their REPL in your browser. The example above runs there.  I’ve heard good things about Clooj as a starting point for beginners. There are plugins for the big Java IDEs — Enclojure for Netbeans, Counterclockwise for Eclipse, La Clojure for IntelliJ — but most Clojure power users use good old Emacs with SLIME and swank-clojure. Finally, the Clojure website has lots of documentation and links to all sorts of useful resources.