Clojure Deref (Sept 17, 2021)

This week we released Clojure 1.11.0-alpha2 which pulls together several things the core team has been working on plus a variety of older bug fixes. I wanted to expand a bit more on some of the items in the release. In general, it’s worth digging into these jira tickets as we expend a fair amount of effort trying to making them good records of problem/alternatives/solution and I’m going to be heavily mining them here.

Add :as-alias option to require like :as but not load

This was the most voted open request on Ask Clojure and really stemmed from the big increase in qualified keyword usage from spec. In spec you might name a spec :my.cool.domain/account where "my.cool.domain" is not a "real" namespace that can be loaded, it’s just a useful qualifier. You can of course use the full qualifier every time but that can be tedious.

Clojure has long had namespace aliases, most commonly defined using the :require or :use clause of ns. However, both require and use bottom out in load-file, which loads the needed namespace before creating the alias. You can alternately use create-ns to create a runtime Namespace without loading, and then use alias to alias to it, and this has been the most common workaround (sometimes with some macro goo around it to make it easier).

We explored several different options for this, and they’re enumerated in CLJ-2123. We looked at changing alias to automatically create a runtime namespace, but this would change alias semantics, possibly in ways that would impact existing users. We looked at creating a new kind of keyword-only alias (I prototyped this - it was a terrible mess to retain backward portability). We looked at expanding ns to include a variant of alias, but this one would have been a breaking change for the spec (this is kind of a tangent) and we think would have been a bit harder to cover for existing ns analyzers than where we ended up, which was to add a new :as-alias clause to require, that is essentially like :as, but does not require a loaded namespace.

So you can then do:

The core.specs.alpha spec has been updated to include :as-alias and a new version of that library was built and is depended on by 1.11.0-alpha2. Important to note is that this was an additive (via keys*), not breaking, change.

update-keys and update-vals

Also, a highly voted old request, update-keys and update-vals are functions that have been rewritten many, many times in Clojure code bases and utility libraries (also often called map-keys/map-vals). When we look at stuff like this, we try to be clear about what challenges we believe are in or out of scope, how generic to make the implementation, what promises we should make in the docstring, and how to achieve the best performance given that. Other impls may make different choices than we did here, which is fine. For example, should this work on all associative types (like vectors?) or just maps? We decided covering the map case well was more important. There was a lot of perf testing done on implementation choices - that’s not all covered in the tickets.

IKVReduce - make old slow path (IPersistentMap) faster and extend to Object, removing impl ambiguity

This is kind of an old ticket, but had a direct impact on testing for update-keys and update-vals, which rely on the IKVReduce protocol. Protocols are an open extension mechanism where choices are made based on type matching (including Java inheritance). The IKVReduce protocol also has an associated Java interface IKVReduce which can be used by Java implementation classes needing to hook into the protocol (similar pattern is used in other places as well inside Clojure’s impl). To cover this, the IKVReduce protocol is extended to both the IKVReduce interface (fast impl) and to the IPersistentMap interface (slow impl). If a concrete impl (like PersistentHashMap) matches multiple types, the "closest" one will match. However, PHM implements both of these and it’s essentially a "tie". What we’ve seen over the years is that based on the ordering of things returned from reflection, it’s possible for PHM to route to either of the two protocol extensions, sometimes leading to much slower results for reduce-kv.

We looked at a bunch of ways to resolve this - tweaks to reduce-kv, special cases for the built-in impls (PHM, PAM), adding a protocol preference system like multimethods have, etc. There are tradeoffs in all these approaches, some pretty significant. In the end we decided that another way to remove the ambiguity is to define the "slow fallback" implementation on Object, rather than on IPersistentMap. This widens the scope of what reduce-kv can be applied to, essentially changing it from "maps" to "colls that seq to map entries". The Object case is always a last resort case, so there is no longer any ambiguity - PHM will always take the "fast" through IKVReduce interface (self reduction). Also, this means that reduce-kv now works on java.util.Maps (another very old jira request) and potentially other useful things. Needless to say, there was a lot of perf testing done on this and several of the alternatives. Along the way, it seemed clear that the "slow" path fallback was a lot slower than it needed to be, so that was also rewritten for better performance. The major tradeoff in the solution we landed on is that (satisfies? IKVReduce x) is now true for everything. As an internal protocol rarely used directly, there is very little code in the wild doing anything like this (xforms is the most notable example we found and we’ve been consulting with Christophe about that one).

The end result of all this is that reduce-kv is faster in the "slow" case, will predictably use the "fast" case when it can, and can be applied to more types of colls.

Other enhancements and bug fixes

We’ve been working on creating some new internal processes for working through jiras, and this is the first chunk of those. We’re hoping to get a mixture of highly voted (important to the community), important from core team perspective, and low-hanging fruit into each 1.11 alpha/beta. I don’t have a prediction for when we will "finish" 1.11 as it depends a lot on what we decide to include, and that is an ongoing discussion.

As always, it’s a huge help to us if you can swap it into your build and just run your test suite to see if anything breaks (or is faster!). We’d love to hear that feedback, even if it’s "all good".

New releases and tools this week: