thread-local version of with-redefs

with-local-redefs.clj

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Commentary                                                            ;;
;;                                                                       ;;
;; The goal for writing this started with the idea to have tests run in  ;;
;; parallel using the leiningen plugin eftest                            ;;
;; https://github.com/weavejester/eftest.                                ;;
;;                                                                       ;;
;; With tests using with-redefs, it was not possible to run them in      ;;
;; parallel if they were changing the root binding of the same           ;;
;; vars. Here, we are binding the root of the var to one function that   ;;
;; respects per-thread rebindings, if any exist.                         ;;
;;                                                                       ;;
;; Known caveats:                                                        ;;
;; - This per-therad rebinding will only work with clojure concurrency   ;;
;;   primitives which copy per-thread bindings to newly spawned threads, ;;
;;   eg, using clojure futures. But will not work for, say a             ;;
;;   java.lang.Thread.                                                   ;;
;; - As of now this only supports functions being bound and not other    ;;
;;   vars which store values, say (def x 19) for example.                ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;



(def ^:dynamic local-redefinitions {})

(defn current->original-definition
  [v]
  (when (var? v)
    (get (meta v) ::original)))


(defn redefiniton-fn
  [a-var]
  (fn [& args]
    (let [current-f (get local-redefinitions
                         a-var
                         (current->original-definition a-var))]
      (apply current-f args))))


(defn dynamic-redefs
  [vars func]
  (let [un-redefs (remove #(::already-bound? (meta %)) vars)]
    (doseq [a-var un-redefs]
      (locking a-var
        (when-not (::already-bound? (meta a-var))
          (let [old-val (.getRawRoot ^clojure.lang.Var a-var)]
            (.bindRoot ^clojure.lang.Var a-var
                       (redefiniton-fn a-var))
            (alter-meta! a-var
                         (fn [m]
                           (assoc m
                                  ::already-bound? true
                                  ::original old-val))))))))
  (func))


(defn xs->map
  [xs]
  (reduce (fn [acc [k v]] (assoc acc `(var ~k) v))
          {}
          (partition 2 xs)))


(defmacro with-dynamic-redefs
  [bindings & body]
  ;; @TODO: Add support for non-functions
  (let [map-bindings (xs->map bindings)]
    `(let [old-rebindings# local-redefinitions]
       (binding [local-redefinitions (merge old-rebindings# ~map-bindings)]
         (dynamic-redefs ~(vec (keys map-bindings))
                         (fn [] ~@body))))))



(comment ;; for testing
  (defn funk [& args] {:original-args args})
  (dotimes [i 1000]
    (let [f1 (future (with-dynamic-redefs [funk (constantly -100)]
                       (Thread/sleep (rand-int 10))
                       {:100 (funk) :t (.getName (Thread/currentThread))}))

          f2 (future (with-dynamic-redefs [funk (constantly -200)]
                       (Thread/sleep (rand-int 1000))
                       {:200 (funk 9) :t (.getName (Thread/currentThread))}))
          f3 (future (do
                       (Thread/sleep (rand-int 1000))
                       {:orig (funk 9) :t (.getName (Thread/currentThread))}))]
      (when (or (not= (:100 @f1) -100)
                (not= (:200 @f2) -200)
                (not= (:orig @f3) {:original-args '(9)}))
        (println "FAIL")
        (prn @f1)
        (prn @f2)
        (println "----------------\n\n")))))

Hmm. Only case I can think of, currently, why it will revert to original definition is when it will use java native thread.
Or some custom future without doing binding-conveyor-fn

@mourjo

Other reason It might escape binding is due to laziness where body is lazy and body runs in different thread instead of thread which does binding or binding has been undone (see below). There is no solution to this problem either make code non lazy or use your solution (I am not sure if your solution avoids it though, at onset seems to me that it will faces same problem due to usage of local-redefinitions) .

For example

user=> (def ^:dynamic *x* 10)
#'user/*x*
user=> (def res (binding [*x* 20] (lazy-seq (repeat 10 *x*))))
#'user/res
user=> res
(10 10 10 10 10 10 10 10 10 10)

Lazy expressions wouldn't with my code either. For example:

(defn funk [& args] {:args args})

(with-dynamic-redefs [funk (constantly :redefined)]
  (map funk [1]))
=> ({:args (1)})

(with-dynamic-redefs [funk (constantly :redefined)]
  (doall (map funk [1])))
=> (:redefined)

Something else must be it, maybe some regression in my code. I'll get back to you on this in some time. I'd much rather prefer your solution.

@mourjo It is not regression. It is due to laziness https://cemerick.com/2009/11/03/be-mindful-of-clojures-binding/

@rdivyanshu I tried your approach but it doesn't seem to work if the Vars have already been compiled:

;; this is the source
(ns ptesttrial.core)

(defn funk [a]
  a)

;; this is the test
(ns ptesttrial.core-test
  (:require [clojure.test :refer :all]
            [ptesttrial.core :as pc]))


(deftest b-test
  (is (= 99 (pc/funk 99)))
  (.setDynamic #'pc/funk true)
  (binding [pc/funk (constantly -1)]
    (is (= -1 (pc/funk 99)))))

This test case fails:

expected: (= -1 (pc/funk 99))
  actual: (not (= -1 99))

Clojure docs say the following https://clojure.org/reference/vars#interning :

If a def expression does not find an interned entry in the current namespace for the symbol being def-ed, it creates one, otherwise it uses the existing Var. This find-or-create process is called interning. This means that, unless they have been unmap-ed, Var objects are stable references and need not be looked up every time.

I also tried using ns-unmap and it does not seem to work.

(ns ptesttrial.core-test
  (:require [clojure.test :refer :all]
            [ptesttrial.core :as pc]))


(deftest b-test
  (is (= 99 (pc/funk 99)))
  (.setDynamic #'pc/funk true)
  (ns-unmap 'ptesttrial.core 'funk)
  (binding [pc/funk (constantly -1)]
    (is (= -1 (pc/funk 99)))))

;; FAIL in (b-test) (core_test.clj:11)
;; expected: (= -1 (pc/funk 99))
;;   actual: (not (= -1 99))

This works fine.

(ns ptesttrial.core-test
  (:require [clojure.test :refer :all]
            [ptesttrial.core :as pc]))

(defmacro with-dynamic-redefs
  [bindings & body]  
  (let [vars (map (fn [x] `(var ~x)) (take-nth 2 bindings))]
    (doall (map #(.setDynamic (eval %)) vars))
    `(binding  ~bindings ~@body)))

(deftest b-test
  (is (= 99 (pc/funk 99)))
  (with-dynamic-redefs [pc/funk (constantly -1)]
    (is (= -1 (pc/funk 99)))))

Any reason for not doing this. Problem with your approach is setDynamic was not run during compilation so compiler emitted getRawRoot at call site (https://github.com/clojure/clojure/blob/b19b781b1f0f3f46aee5e951f415e0456a39cbcb/src/jvm/clojure/lang/Compiler.java#L5191). deftest put content of body in function which don't get invoked during compilation where as use of
defmacro make it run at compile.

@mourjo Just had this issue pop up on us. Thanks for sharing!

@rdivyanshu was trying to use your macro as a drop-in replacement to with-redefs, but think I've hit a problem.

The direct invocation usage works great:

(defn get-thing []
  :none)

;; fails, usually
(deftest with-redefs-not-thread-safe
  (let [results (pmap (fn [i]
                        (with-redefs [get-thing (constantly i)]
                          (get-thing)))
                      (range 100))]
    (is (= (range 100) results))))

(defmacro with-dynamic-redefs
  [bindings & body]
  (let [vars (map (fn [x] `(var ~x)) (take-nth 2 bindings))]
    (doall (map #(.setDynamic (eval %)) vars))
    `(binding  ~bindings ~@body)))

;; doesn't seem to fail?
(deftest with-dynamic-redefs-is-thread-safe
  (let [results (pmap (fn [i]
                        (with-dynamic-redefs [get-thing (constantly i)]
                          (get-thing)))
                      (range 100))]
    (is (= (range 100) results))))

But if I add indirection, it won't work until I re-evaluate the caller:

(defn get-thing []
  :none)

(defn get-indirect-thing []
  (get-thing))

;; fails, usually
(deftest with-redefs-not-thread-safe
  (let [results (pmap (fn [i]
                        (with-redefs [get-thing (constantly i)]
                          (get-indirect-thing)))
                      (range 100))]
    (is (= (range 100) results))))

(defmacro with-dynamic-redefs
  [bindings & body]
  (let [vars (map (fn [x] `(var ~x)) (take-nth 2 bindings))]
    (doall (map #(.setDynamic (eval %)) vars))
    `(binding  ~bindings ~@body)))

;; fails too
(deftest with-dynamic-redefs-is-thread-safe
  (let [results (pmap (fn [i]
                        (with-dynamic-redefs [get-thing (constantly i)]
                          (get-indirect-thing)))
                      (range 100))]
    (is (= (range 100) results))))

;; evaluating the caller again will make the test work
(defn get-indirect-thing []
  (get-thing))

;; now it doesn't fail
(deftest with-dynamic-redefs-is-thread-safe
  (let [results (pmap (fn [i]
                        (with-dynamic-redefs [get-thing (constantly i)]
                          (get-indirect-thing)))
                      (range 100))]
    (is (= (range 100) results))))

It looks like the same class of problem as before: callers that were defined before the macro runs get a non-dynamic version of the var.

For whoever finds this, the author actually made a lib that works with my example above: https://github.com/mourjo/dynamic-redef

@filipesilva It is happening due to there is no dynamic invocation of function get-thing. In get-indirect-thing you are invoking during definition. if you define get-indirect-thing according to below

(defn get-indirect-thing [f]
  (f))

And then call (get-indirect-thing get-thing) test passes. What you want in get-indirect-thing at run time is call current definition of get-thing but it is calling get-thing at time of definition. I am not sure if clojure provides such mechanism for functions with no arguments, it has been long since I wrote clojure code.

Hey thanks for getting back to me, wasn't expecting it at all :D

https://github.com/mourjo/dynamic-redef seems to handle the indirection case well so trying it out atm.

Thanks for this! Trying to understand this code - how are redefs reset when you go out of scope of the macro? looks like the var is permanently changed?

Yes the var is permanently changed. That is probably a good enough reason to only use this in tests. How it works is: the redefined var uses a dynamic (thread local) var to store local redefinitions.

As an aside: You might want to check out this repository (created based on this gist): https://github.com/mourjo/dynamic-redef

Ah okay. Would you accept a PR to have it remove itself once done, if I do that? (to the repo that is).
For my workflow permanently changing it means the changes leak out into my local dev env.

Absolutely, I would love your contribution!

Thanks a lot for this! This is really great, and we finally have something that can be used in a concurrent environment.
I have a small issue with the metadata, since some testing libraries are mutating them:

(def myfn ^:foo (fn []))
(meta myfn) => {:foo true}
(with-dynamic-redefs [myfn (fn [])]) 
(meta myfn) => {:already-bound? true, :original #function[.../myfn]}
(meta (:original (meta myfn))) => {:foo true}

The var invocation takes care of the local-redefinitions, but this is not the case for the metadata. [1]
I don't know if anything can be used to have a dynamic meta function that is going to call this kind of thing underneath:

(defn var->meta-fn
  [a-var]
  (fn [& args]
    (let [original-f (current->original-definition a-var)
          current-f (get local-redefinitions
                         a-var
                         original-f)]
      (meta current-f))))

Any idea there is welcome!
Thanks in advance!

[1] : https://github.com/mourjo/dynamic-redef/blob/main/src/dynamic_redef/core.clj#L35-L41

@arnaudgeiser I think https://github.com/filipesilva/with-dynamic-redefs/blob/d121f3b2d92d2b9a4d8538033dbf3a051f42d496/test/with_dynamic_redefs_test.clj#L27 is preserving the metadata, but haven't touch it in a while.

Thanks, @filipesilva, I will try to spend a bit of time on it, but I definitely encounter weird behaviour where with-dynamic-redefs is being used with libraries that are touching the metadata. Like https://cljdoc.org/d/tortue/spy/2.15.0/api/spy.core

@filipesilva : Wait... you are pointing to your implementation which apparently covers what I would like to do [1]. Let me give it a shot.

[1] : https://github.com/filipesilva/with-dynamic-redefs/blob/d121f3b2d92d2b9a4d8538033dbf3a051f42d496/src/with_dynamic_redefs.clj#L13-L15

EDITED: Ha, perfect, this is exactly the issue I'm encountering. https://github.com/filipesilva/with-dynamic-redefs?tab=readme-ov-file#with-dynamic-redefs. I wish we would be end up with a single with-dynamic-redefs library.

Developed at Metabase, but since we stopped needing it there I've preserved it in my github account.

Out of curiosity, what did you end up using?

We just removed it and did not do parallel tests. They were a bit tricky to get right, and weren't worth the complexity. If doing it again, I'd probably try sharding the test suite at the process level instead.

Got it, thank you!

This gist is a bit unfortunate since both of you (@filipesilva, @mourjo) have a way to fix a lot of complexity we are finding on the Clojure services created out there. The fact that with-redefs is not thread-safe influenced (for wrong reasons) the creation of a lot of indirections. I wish the Clojure community would embrace what you have created.

@KingMob wrote something great about it: https://modulolotus.net/blog/2022-06-22-tidd/

I wish all of this had a bit more visibility.

I wish all of this had a bit more visibility.

Me too 😄

Hey all! Sorry it took me so long to get to this - it would be really great to combine the two implementations
@filipesilva would you be okay if I blatantly copy how you are handling metadata in your implementation? Or if you prefer a pull request, that'd be great too.

I wish we would be end up with a single with-dynamic-redefs library.

@mourjo go for it :D