Clojure

Ahead-of-time Compilation and Class Generation

Clojure compiles all code you load on-the-fly into JVM bytecode, but sometimes it is advantageous to compile ahead-of-time (AOT). Some reasons to use AOT compilation are:

  • To deliver your application without source

  • To speed up application startup

  • To generate named classes for use by Java

  • To create an application that does not need runtime bytecode generation and custom classloaders

The Clojure compilation model preserves as much as possible the dynamic nature of Clojure, in spite of the code-reloading limitations of Java.

  • Source and classfile pathing follows Java classpath conventions.

  • The target of compile is a namespace

  • Each file, fn and gen-class will produce a .class file

  • Each file generates a loader class of the same name with "__init" appended.

  • The static initializer for a loader class produces the same effects as does loading its source file

    • You generally shouldn’t need to use these classes directly, as use, require and load will choose between them and more recent source

  • The loader class is generated for each file referenced when a namespace is compiled, when its loader .class file is older than its source.

  • A stand-alone gen-class facility is provided to create named classes for direct use as Java classes, with facilities for:

    • Naming the generated class

    • Selecting the superclass

    • Specifying any implemented interfaces

    • Specifying constructor signatures

    • Specifying state

    • Declaring additional methods

    • Generating static factory methods

    • Generating main

    • Controlling the mapping to an implementing namespace

    • Exposing inherited protected members

    • Generating more than one named class from a single file, with implementations in one or more namespaces

  • An optional :gen-class directive can be used in the ns declaration to generate a named class corresponding to a namespace. (:gen-class …​), when supplied, defaults to :name corresponding to the ns name, :main true, :impl-ns same as ns, and :init-impl-ns true. All options of gen-class are supported.

  • gen-class and the :gen-class directive are ignored when not compiling.

  • A stand-alone gen-interface facility is provided for generating named interface classes for direct use as Java interfaces, with facilities for:

    • Naming the generated interface

    • Specifying any superinterfaces

    • Declaring the signatures of interface methods

Compiling

To compile a lib, use the compile function, and supply the namespace name as a symbol. For some namespace my.domain.lib, defined in my/domain/lib.clj, in the classpath, the following should occur:

  • A loader classfile will be produced in my/domain/lib__init.class, under *compile-path*, which must be in the classpath

  • A set of classfiles will be produced, one per fn in the namespace, with names such as my/domain/lib$fnname__1234.class

  • For each gen-class:

    • A stub classfile will be produced with the specified name

Compiler options

The Clojure compiler can be controlled via the use of several compiler flags. At runtime these are stored in the dynamic var clojure.core/*compiler-options*, which is a map with the following optional keyword keys:

  • :disable-locals-clearing (boolean)

  • :elide-meta (vector of keywords)

  • :direct-linking (boolean)

These compiler options can be changed in a dynamic binding around a call to the compile function to change the compiler behavior.

Alternately, compiler options can also be set via the Java system properties at startup:

  • -Dclojure.compiler.disable-locals-clearing=true

  • "-Dclojure.compiler.elide-meta=[:doc :file :line :added]"

  • -Dclojure.compiler.direct-linking=true

See below for more info on each of these options.

Locals clearing

By default, the Clojure compiler produces code that eagerly clears GC references to local bindings. However, when using a debugger locals will appear as nulls, which makes debugging difficult. Setting disable-locals-clearing=true will prevent locals clearing. It is not recommended to disable locals clearing for production compilation.

Elide meta

Var meta (docstrings, file and line info, etc) will be compiled into strings in the constant pool of the compiled classes. To decrease class size and make classloading faster, meta can be elided. This option takes a vector of meta keywords that should be removed - some common ones include :doc, :file, :line, and :added. Note that eliding meta may make certain features inoperable (for example, doc cannot return docstrings if they have been elided).

Direct linking

Normally, invoking a function will cause a var to be dereferenced to find the function instance implementing it, then invoking that function. This indirection via the var is one of the ways that Clojure provides a dynamic runtime environment. However, it has long been observed that the majority of function invocations in a production environment are never redefined in this way, incurring unnecessary redirection.

Direct linking can be used to replace this indirection with a direct static invocation of the function instead. This will result in faster var invocation. Additionally, the compiler can remove unused vars from class initialization and direct linking will make many more vars unused. Typically this results in smaller class sizes and faster startup times.

One consequence of direct linking is that var redefinitions will not be seen by code that has been compiled with direct linking (because direct linking avoids dereferencing the var). Vars marked as ^:dynamic will never be direct linked. If you wish to mark a var as supporting redefinition (but not dynamic), mark it with ^:redef to avoid direct linking.

As of Clojure 1.8, the Clojure core library itself is compiled with direct linking.

Runtime

Classes generated by Clojure are highly dynamic. In particular, note that no method bodies or other implementation details are specified in gen-class - it specifies only a signature, and the class that it generates is only a stub. This stub class defers all implementation to functions defined in the implementing namespace. At runtime, a call to some method foo of the generated class will find the current value of the var implementing.namespace/prefixfoo and call it. If the var is not bound or nil, it will call the superclass method, or if an interface method, generate an UnsupportedOperationException.

gen-class Examples

In the simplest case, an empty :gen-class is supplied, and the compiled class has only main, which is implemented by defining -main in the namespace. The file should be saved in src/clojure/examples/hello.clj:

(ns clojure.examples.hello
    (:gen-class))

(defn -main
  [greetee]
  (println (str "Hello " greetee "!")))

To compile, ensure the target output directory classes exists:

mkdir classes

And create a deps.edn file describing your classpath:

{:paths ["src" "classes"]}

Then compile to generate the classes as follows:

$ clj
Clojure 1.10.1
user=> (compile 'clojure.examples.hello)
clojure.examples.hello

And can be run like an ordinary Java app like so (be sure to include the output classes directory):

java -cp `clj -Spath` clojure.examples.hello Fred
Hello Fred!

Here’s an example using both a more involved :gen-class, and stand-alone calls to gen-class and gen-interface. In this case we are creating classes we intend to create instances of. The clojure.examples.instance class will implement java.util.Iterator, a particularly nasty interface, in that it requires the implementation to be stateful. This class is going to take a String in its constructor and implement the Iterator interface in terms of delivering the characters from the string. The :init clause names the constructor function. The :constructors clause is a map of constructor signature to superclass constructor signature. In this case, the superclass defaults to Object, whose constructor takes no arguments. This object will have state, called state, and a main so we can test it.

:init functions (-init in this case) are unusual, in that they always return a vector, the first element of which is a vector of arguments for the superclass constructor - since our superclass takes no args, this vector is empty. The second element of the vector is the state for the instance. Since we are going to have to mutate the state (and the state is always final) we’ll use a ref to a map containing the string and the current index.

hasNext and next are implementations of methods in the Iterator interface. While the methods take no args, the implementation functions for instance methods will always take an additional first arg corresponding to the object the method is called upon, called by convention 'this' here. Note how the state can be obtained using an ordinary Java field access.

The gen-interface call will create an interface called clojure.examples.IBar, with a single method bar.

The stand-alone gen-class call will generate another named class, clojure.examples.impl, whose implementing namespace will default to the current namespace. It implements clojure.examples.IBar. The :prefix option causes the implementation of methods to bind to functions beginning with "impl-" rather than the default "-". The :methods option defines a new method foo not present in any superclass/interfaces.

Note in main how an instances of the classes can be created, and methods called, using ordinary Java interop. Using it would be similarly ordinary from Java.

(ns clojure.examples.instance
    (:gen-class
     :implements [java.util.Iterator]
     :init init
     :constructors {[String] []}
     :state state))

(defn -init [s]
  [[] (ref {:s s :index 0})])

(defn -hasNext [this]
  (let [{:keys [s index]} @(.state this)]
    (< index (count s))))

(defn -next [this]
  (let [{:keys [s index]} @(.state this)
        ch (.charAt s index)]
    (dosync (alter (.state this) assoc :index (inc index)))
    ch))

(gen-interface
 :name clojure.examples.IBar
 :methods [[bar [] String]])

(gen-class
 :name clojure.examples.impl
 :implements [clojure.examples.IBar]
 :prefix "impl-"
 :methods [[foo [] String]])

(defn impl-foo [this]
  (str (class this)))

(defn impl-bar [this]
  (str "I " (if (instance? clojure.examples.IBar this)
              "am"
              "am not")
       " an IBar"))

(defn -main [s]
  (let [x (new clojure.examples.instance s)
        y (new clojure.examples.impl)]
    (while (.hasNext x)
      (println (.next x)))
    (println (.foo y))
    (println (.bar y))))

Compile as above:

$ clj
Clojure 1.10.1
user=> (compile 'clojure.examples.instance)
clojure.examples.instance

And run like an ordinary Java app:

java -cp `clj -Spath` clojure.examples.instance asdf
a
s
d
f
class clojure.examples.impl
I am an IBar