thread-local version of with-redefs

with-local-redefs.clj

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Commentary                                                            ;;
;;                                                                       ;;
;; The goal for writing this started with the idea to have tests run in  ;;
;; parallel using the leiningen plugin eftest                            ;;
;; https://github.com/weavejester/eftest.                                ;;
;;                                                                       ;;
;; With tests using with-redefs, it was not possible to run them in      ;;
;; parallel if they were changing the root binding of the same           ;;
;; vars. Here, we are binding the root of the var to one function that   ;;
;; respects per-thread rebindings, if any exist.                         ;;
;;                                                                       ;;
;; Known caveats:                                                        ;;
;; - This per-therad rebinding will only work with clojure concurrency   ;;
;;   primitives which copy per-thread bindings to newly spawned threads, ;;
;;   eg, using clojure futures. But will not work for, say a             ;;
;;   java.lang.Thread.                                                   ;;
;; - As of now this only supports functions being bound and not other    ;;
;;   vars which store values, say (def x 19) for example.                ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;



(def ^:dynamic local-redefinitions {})

(defn current->original-definition
  [v]
  (when (var? v)
    (get (meta v) ::original)))


(defn redefiniton-fn
  [a-var]
  (fn [& args]
    (let [current-f (get local-redefinitions
                         a-var
                         (current->original-definition a-var))]
      (apply current-f args))))


(defn dynamic-redefs
  [vars func]
  (let [un-redefs (remove #(::already-bound? (meta %)) vars)]
    (doseq [a-var un-redefs]
      (locking a-var
        (when-not (::already-bound? (meta a-var))
          (let [old-val (.getRawRoot ^clojure.lang.Var a-var)]
            (.bindRoot ^clojure.lang.Var a-var
                       (redefiniton-fn a-var))
            (alter-meta! a-var
                         (fn [m]
                           (assoc m
                                  ::already-bound? true
                                  ::original old-val))))))))
  (func))


(defn xs->map
  [xs]
  (reduce (fn [acc [k v]] (assoc acc `(var ~k) v))
          {}
          (partition 2 xs)))


(defmacro with-dynamic-redefs
  [bindings & body]
  ;; @TODO: Add support for non-functions
  (let [map-bindings (xs->map bindings)]
    `(let [old-rebindings# local-redefinitions]
       (binding [local-redefinitions (merge old-rebindings# ~map-bindings)]
         (dynamic-redefs ~(vec (keys map-bindings))
                         (fn [] ~@body))))))



(comment ;; for testing
  (defn funk [& args] {:original-args args})
  (dotimes [i 1000]
    (let [f1 (future (with-dynamic-redefs [funk (constantly -100)]
                       (Thread/sleep (rand-int 10))
                       {:100 (funk) :t (.getName (Thread/currentThread))}))

          f2 (future (with-dynamic-redefs [funk (constantly -200)]
                       (Thread/sleep (rand-int 1000))
                       {:200 (funk 9) :t (.getName (Thread/currentThread))}))
          f3 (future (do
                       (Thread/sleep (rand-int 1000))
                       {:orig (funk 9) :t (.getName (Thread/currentThread))}))]
      (when (or (not= (:100 @f1) -100)
                (not= (:200 @f2) -200)
                (not= (:orig @f3) {:original-args '(9)}))
        (println "FAIL")
        (prn @f1)
        (prn @f2)
        (println "----------------\n\n")))))

Ah okay. Would you accept a PR to have it remove itself once done, if I do that? (to the repo that is).
For my workflow permanently changing it means the changes leak out into my local dev env.

Absolutely, I would love your contribution!

Thanks a lot for this! This is really great, and we finally have something that can be used in a concurrent environment.
I have a small issue with the metadata, since some testing libraries are mutating them:

(def myfn ^:foo (fn []))
(meta myfn) => {:foo true}
(with-dynamic-redefs [myfn (fn [])]) 
(meta myfn) => {:already-bound? true, :original #function[.../myfn]}
(meta (:original (meta myfn))) => {:foo true}

The var invocation takes care of the local-redefinitions, but this is not the case for the metadata. [1]
I don't know if anything can be used to have a dynamic meta function that is going to call this kind of thing underneath:

(defn var->meta-fn
  [a-var]
  (fn [& args]
    (let [original-f (current->original-definition a-var)
          current-f (get local-redefinitions
                         a-var
                         original-f)]
      (meta current-f))))

Any idea there is welcome!
Thanks in advance!

[1] : https://github.com/mourjo/dynamic-redef/blob/main/src/dynamic_redef/core.clj#L35-L41

@arnaudgeiser I think https://github.com/filipesilva/with-dynamic-redefs/blob/d121f3b2d92d2b9a4d8538033dbf3a051f42d496/test/with_dynamic_redefs_test.clj#L27 is preserving the metadata, but haven't touch it in a while.

Thanks, @filipesilva, I will try to spend a bit of time on it, but I definitely encounter weird behaviour where with-dynamic-redefs is being used with libraries that are touching the metadata. Like https://cljdoc.org/d/tortue/spy/2.15.0/api/spy.core

@filipesilva : Wait... you are pointing to your implementation which apparently covers what I would like to do [1]. Let me give it a shot.

[1] : https://github.com/filipesilva/with-dynamic-redefs/blob/d121f3b2d92d2b9a4d8538033dbf3a051f42d496/src/with_dynamic_redefs.clj#L13-L15

EDITED: Ha, perfect, this is exactly the issue I'm encountering. https://github.com/filipesilva/with-dynamic-redefs?tab=readme-ov-file#with-dynamic-redefs. I wish we would be end up with a single with-dynamic-redefs library.

Developed at Metabase, but since we stopped needing it there I've preserved it in my github account.

Out of curiosity, what did you end up using?

We just removed it and did not do parallel tests. They were a bit tricky to get right, and weren't worth the complexity. If doing it again, I'd probably try sharding the test suite at the process level instead.

Got it, thank you!

This gist is a bit unfortunate since both of you (@filipesilva, @mourjo) have a way to fix a lot of complexity we are finding on the Clojure services created out there. The fact that with-redefs is not thread-safe influenced (for wrong reasons) the creation of a lot of indirections. I wish the Clojure community would embrace what you have created.

@KingMob wrote something great about it: https://modulolotus.net/blog/2022-06-22-tidd/

I wish all of this had a bit more visibility.

I wish all of this had a bit more visibility.

Me too 😄

Hey all! Sorry it took me so long to get to this - it would be really great to combine the two implementations
@filipesilva would you be okay if I blatantly copy how you are handling metadata in your implementation? Or if you prefer a pull request, that'd be great too.

I wish we would be end up with a single with-dynamic-redefs library.

@mourjo go for it :D