Clojure Deref (Sept 17, 2021)

17 September 2021
Alex Miller

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem. (@ClojureDeref RSS)


This week we released Clojure 1.11.0-alpha2 which pulls together several things the core team has been working on plus a variety of older bug fixes. I wanted to expand a bit more on some of the items in the release. In general, it’s worth digging into these jira tickets as we expend a fair amount of effort trying to making them good records of problem/alternatives/solution and I’m going to be heavily mining them here.

This was the most voted open request on Ask Clojure and really stemmed from the big increase in qualified keyword usage from spec. In spec you might name a spec where "" is not a "real" namespace that can be loaded, it’s just a useful qualifier. You can of course use the full qualifier every time but that can be tedious.

Clojure has long had namespace aliases, most commonly defined using the :require or :use clause of ns. However, both require and use bottom out in load-file, which loads the needed namespace before creating the alias. You can alternately use create-ns to create a runtime Namespace without loading, and then use alias to alias to it, and this has been the most common workaround (sometimes with some macro goo around it to make it easier).

We explored several different options for this, and they’re enumerated in CLJ-2123. We looked at changing alias to automatically create a runtime namespace, but this would change alias semantics, possibly in ways that would impact existing users. We looked at creating a new kind of keyword-only alias (I prototyped this - it was a terrible mess to retain backward portability). We looked at expanding ns to include a variant of alias, but this one would have been a breaking change for the spec (this is kind of a tangent) and we think would have been a bit harder to cover for existing ns analyzers than where we ended up, which was to add a new :as-alias clause to require, that is essentially like :as, but does not require a loaded namespace.

So you can then do:

(ns foo
  (:require [ :as-alias d])) ;; works! (doesn't load)

::d/account ;; valid

The core.specs.alpha spec has been updated to include :as-alias and a new version of that library was built and is depended on by 1.11.0-alpha2. Important to note is that this was an additive (via keys*), not breaking, change.

Also, a highly voted old request, update-keys and update-vals are functions that have been rewritten many, many times in Clojure code bases and utility libraries (also often called map-keys/map-vals). When we look at stuff like this, we try to be clear about what challenges we believe are in or out of scope, how generic to make the implementation, what promises we should make in the docstring, and how to achieve the best performance given that. Other impls may make different choices than we did here, which is fine. For example, should this work on all associative types (like vectors?) or just maps? We decided covering the map case well was more important. There was a lot of perf testing done on implementation choices - that’s not all covered in the tickets.

This is kind of an old ticket, but had a direct impact on testing for update-keys and update-vals, which rely on the IKVReduce protocol. Protocols are an open extension mechanism where choices are made based on type matching (including Java inheritance). The IKVReduce protocol also has an associated Java interface IKVReduce which can be used by Java implementation classes needing to hook into the protocol (similar pattern is used in other places as well inside Clojure’s impl). To cover this, the IKVReduce protocol is extended to both the IKVReduce interface (fast impl) and to the IPersistentMap interface (slow impl). If a concrete impl (like PersistentHashMap) matches multiple types, the "closest" one will match. However, PHM implements both of these and it’s essentially a "tie". What we’ve seen over the years is that based on the ordering of things returned from reflection, it’s possible for PHM to route to either of the two protocol extensions, sometimes leading to much slower results for reduce-kv.

We looked at a bunch of ways to resolve this - tweaks to reduce-kv, special cases for the built-in impls (PHM, PAM), adding a protocol preference system like multimethods have, etc. There are tradeoffs in all these approaches, some pretty significant. In the end we decided that another way to remove the ambiguity is to define the "slow fallback" implementation on Object, rather than on IPersistentMap. This widens the scope of what reduce-kv can be applied to, essentially changing it from "maps" to "colls that seq to map entries". The Object case is always a last resort case, so there is no longer any ambiguity - PHM will always take the "fast" through IKVReduce interface (self reduction). Also, this means that reduce-kv now works on java.util.Maps (another very old jira request) and potentially other useful things. Needless to say, there was a lot of perf testing done on this and several of the alternatives. Along the way, it seemed clear that the "slow" path fallback was a lot slower than it needed to be, so that was also rewritten for better performance. The major tradeoff in the solution we landed on is that (satisfies? IKVReduce x) is now true for everything. As an internal protocol rarely used directly, there is very little code in the wild doing anything like this (xforms is the most notable example we found and we’ve been consulting with Christophe about that one).

The end result of all this is that reduce-kv is faster in the "slow" case, will predictably use the "fast" case when it can, and can be applied to more types of colls.

Other enhancements and bug fixes

  • CLJ-1908 Add clojure.test api run-test and run-test-var to run single test with fixtures and report

  • CLJ-2600 Don’t block realized? of delay on pending result

  • CLJ-2649 Fix order of checks in some-fn and every-pred for 3 predicate case to match other unrollings

  • CLJ-2636 Get rid of reflection on java.util.Properties when defining clojure-version

  • CLJ-2350 Improve keyword arity exception message

  • CLJ-2444 Fix typo in test-vars docstring

  • CLJ-1509 AOT compile more Clojure namespaces

  • CLJ-2387 Fix off-by-one in socket server port validation

We’ve been working on creating some new internal processes for working through jiras, and this is the first chunk of those. We’re hoping to get a mixture of highly voted (important to the community), important from core team perspective, and low-hanging fruit into each 1.11 alpha/beta. I don’t have a prediction for when we will "finish" 1.11 as it depends a lot on what we decide to include, and that is an ongoing discussion.

As always, it’s a huge help to us if you can swap it into your build and just run your test suite to see if anything breaks (or is faster!). We’d love to hear that feedback, even if it’s "all good".

Videos and podcasts

Libraries and Tools

New releases and tools this week:

  • Reveal Pro - Read Eval Visualize Loop for Clojure, Supercharged

  • secrets.clj 1.0.0 - A Clojure library designed to generate cryptographically strong random numbers.

  • build-uber-log4j2-handler v0.1.0 - A conflict handler for log4j2 plugins cache files for the uber task.

  • build-clj v0.3.0 - Common build tasks abstracted into a library

  • v0.5.0 - Library of functions to make Clojure builds

  • martian v0.1.18 - The HTTP abstraction library for Clojure/script, supporting Swagger, Schema, re-frame and more

  • clj-kondo 2021.09.15 - A linter for Clojure code that sparks joy

  • julian 1.0.0 - A Clojure(Script) library to convert between Julian Day Number and common time

  • fijit 1.0.7 - A Clojure library for Scala interop

  • holy-lambda 0.5.0 - The extraordinary simple, performant, and extensible custom AWS Lambda runtime for Clojure

  • clojure-exercism-template - Learn more Clojure and Interactive Programming with Exercism in the browser

  • clojure-lsp 2021.09.13-19.32.00 - Language Server (LSP) for Clojure

  • graal-build-time 0.0.11 - Library to initialize Clojure packages at build time with GraalVM native-image

  • - Convex is an open, decentralised, and efficient technology platform built in the spirit of the original Internet

  • Tutkain 0.10.0 - A Sublime Text package for interactive Clojure development

  • datalevin 0.5.13 - A simple, fast and versatile Datalog database

  • babashka 0.6.1 - Native, fast starting Clojure interpreter for scripting

  • trenchman v0.3.0 - A standalone nREPL/prepl client written in Go and heavily inspired by Grenchman