Skip to content

Instantly share code, notes, and snippets.

@vedang

vedang/clj-devtooling-inclojure2024.org

Created March 27, 2024 17:49
Show Gist options
  • Save vedang/d49e55b9d2dc5713d04256a487d90434 to your computer and use it in GitHub Desktop.
Save vedang/d49e55b9d2dc5713d04256a487d90434 to your computer and use it in GitHub Desktop.
Download ZIP
Slides for In/Clojure 2024 talk: Clojure Developer Tooling for Speed and Productivity
Raw
clj-devtooling-inclojure2024.org

title: Clojure Developer Tooling for Speed and Productivity in 2024

Why this talk?

So you’ll show me your cool workflows! (help me get better!)

I want to write a new library or application!

I need to figure out …

  • How to run my application
  • How to run all the tests in my project
  • How to build the artifacts of my app/lib
  • How to deploy these artifacts so others can use them

Speaker Notes

The Javascript ecosystem, for all the abuse we hurl at them, has done an incredible job. The templating that people have access to out of the box is something we should all aspire to!

I use deps-new!

https://github.com/seancorfield/deps-new

clojure -Tnew lib :name me.vedang/depsnew_lib
tree depsnew_lib

What deps-new gives me

  • All the boilerplate!
  • Clear Instructions in the README
  • Aliases:
    • clojure -X:run-x, clojure -X:run-m Run your app
    • clojure -T:build test Run tests
    • clojure -T:build ci Build artifacts
    • clojure -T:build deploy Deploy artifacts

You can even write your own deps-new templates!

AND YOU SHOULD!

(looking at you, framework authors!)

What can we as clojure devs do?

  • Does your favourite framework have a deps-new template? Add it!
  • Do you find yourself copying the same code everywhere? Try and extract it into a template!

References to learn deps-new

I want to add more aliases to my repository!

  • How do I connect to a REPL?
  • How do I setup logging?
  • …

I use neil!

https://github.com/babashka/neil

  • neil can add aliases for you!
  • neil can do so much more!

Examples of using neil!

neil add cider
:cider ;; added by neil
{:extra-deps {cider/cider-nrepl {:mvn/version "0.47.0"}
              djblue/portal {:mvn/version "0.52.2"}
              mx.cider/tools.deps.enrich-classpath {:mvn/version "1.19.0"}
              nrepl/nrepl {:mvn/version "1.1.1"}
              refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
 :main-opts  ["-m" "nrepl.cmdline"
              "--middleware" "[cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor,portal.nrepl/wrap-portal]"]}

Neil can do more than just add aliases!

neil dep search next.jdbc

Neil can do more than just add aliases!

neil dep add :lib com.github.seancorfield/next.jdbc
neil dep add :lib nextjournal/clerk :alias :cider

What can we as clojure devs do?

  • Make it easy to add aliases to neil!
  • I would like to add to existing aliases instead of creating new ones!

References for learning more about neil

Okay, I am now ready to spin up a REPL and connect to it!

clojure -M:test:logs-dev:cider

Speaker Notes

Wait, what’s the logging alias?

Logging is hard!

So once I figured it out, I made a template!

I’m happy I never have to figure this out again!

neil add logs-dev
:logs-dev ;; added by neil
 {:extra-deps {me.vedang/logger {:local/root "logger"}}
  :jvm-opts
   ["-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
    "-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.Slf4jLog"
    "-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"
    "-Dlog4j2.configurationFile=logger/log4j2-dev.xml"
    ;; Change logging.level to one of TRACE, DEBUG, INFO, WARN, ERROR
    ;; depending on requirement during development
    "-Dlogging.level=DEBUG"]}

Templates for the win

clojure -Sdeps '{:deps {io.github.vedang/clj-logging {:git/sha "e009d366c827705f513ef9018ffd920a49ce19da"}}}' -Tnew create :template me.vedang/logger :name me.vedang/logger
tree logger

7 directories, 6 files

I use structured logging everywhere!

http://pedestal.io/pedestal/0.7/reference/logging.html

(in-ns 'me.vedang.depsnew-lib)
(require '[me.vedang.logger.interface :as log])
(log/info :function :welcome
          :action :starting
          :args {:greeting "Hello World"})
{
  "instant": {
    "epochSecond": 1711161317,
    "nanoOfSecond": 787564000
  },
  "thread": "nREPL-session-d6c4a5db-e57c-4efe-906b-7e5f1d0ad6e6",
  "level": "INFO",
  "loggerName": "me.vedang.depsnew-lib",
  "message": "{\"function\":\"welcome\",\"action\":\"starting\",\"args\":{\"greeting\":\"Hello World\"},\"line\":3}",
  "endOfBatch": true,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {},
  "threadId": 41,
  "threadPriority": 5
}

Prefer emitting logs as JSON events. This lets us push them to Observability tools easily in production.

I want beautiful Documentation

Clojure is REPL-first! Where are my notebooks?

Speaker Notes

We’ve been writing notebooks and refactoring them into usable code-bases all our lives. So why do we not have beautiful looking notebooks like the Python people?

I use Clerk: write beautiful notebooks and beautiful code-bases!

https://github.com/nextjournal/clerk

Write your comments like you write Markdown, done!

Clerk: write beautiful notebooks and beautiful code-bases!

;;; # 📈 Sunday Morning Fun With Our Group Meditation Data

;; First, we need to parse the data from the Whatsapp Group. Whatsapp
;; provides a `export-chat` feature that we can use, but there are
;; some things we need to fix first:

;; ##  Whatsapp uses a ridiculous format for the date when the message was sent
^{:nextjournal.clerk/visibility {:code :show :result :hide}}
(def whatsapp-export-date-format
  "Frankly, this is a ridiculous format that no one should use."
  "MM/dd/yy, hh:mm a")

;; The problem isn't so much the format, as the fact that they make it
;; hard to parse. For example, they won't export the date as
;; `"05/01/23, 03:31 pm"`. They will export it as `"5/1/23, 3:31 PM"`.
;; Every part of this is meant to fail your parser. 🤷🏽‍♂️

What Clerk notebooks look like:

Speaker Notes

  • Talk about Separedit!

I use tagref: clear separation of code and documentation

https://github.com/stepchowfun/tagref

(require 'clojure.math)

;; [tag:polynomial_nonzero] This function never returns zero.
(defn polynomial [x]
  (+ (clojure.math/pow x 2) 1))

;;; in some other file
(defn inverse-polynomial [x]
  ;; This is safe due to [ref:polynomial_nonzero].
  (/ 1 (polynomial x)))

Documentation for the team: tagref

tagref list-tags | awk '{print $1}'

References for improving documentation

I want to write clean, maintainable code!

https://github.com/clj-kondo/clj-kondo https://github.com/weavejester/cljfmt https://github.com/kkinnear/zprint

I do automatic Linting and Formatting on the code base!

clj-kondo: My recommendation

Export clj-kondo config for libraries you use! (create ~.clj-kondo~ directory at root first)

clj-kondo --lint "$(clojure -A:dev:test:cider:build -Spath)" --copy-configs --skip-lint

(clojure-lsp will do this automatically)

tree .clj-kondo

12 directories, 6 files

clj-kondo: My recommendation

Commit your .clj-kondo folder!

cljfmt: My recommendation

(For Emacsen) Use Apheleia!

Runs code-formatter on the buffer “at the right time” with minimal disturbance.

What can we as clojure devs do?

Add clj-kondo configuration for your favourite libraries!

References for clj-kondo, cljfmt and zprint

Can we improve the REPL experience?

After all, this is where we are all day!

Speaker Notes

IF we can make the experience even 1% better, isn’t that something amazing?

Debugging like I’ve never experienced before: Flowstorm!

neil add cider-storm
:cider-storm ;; added by neil
 {:classpath-overrides
  ;; we need to disable the official compiler and use ClojureStorm
  {org.clojure/clojure nil}
  :extra-deps {cider/cider-nrepl {:mvn/version "0.47.0"}
               com.github.flow-storm/clojure {:mvn/version "1.11.2"}
               com.github.flow-storm/flow-storm-dbg {:mvn/version "3.13.1"}
               djblue/portal {:mvn/version "0.52.2"}
               mx.cider/tools.deps.enrich-classpath {:mvn/version "1.19.0"}
               nrepl/nrepl {:mvn/version "1.1.1"}
               org.openjfx/javafx-controls {:mvn/version "23-ea+3"}
               org.openjfx/javafx-base {:mvn/version "23-ea+3"}
               org.openjfx/javafx-graphics {:mvn/version "23-ea+3"}
               org.openjfx/javafx-swing {:mvn/version "23-ea+3"}
               refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}}
  :main-opts  ["-m" "nrepl.cmdline"
               "--middleware" "[flow-storm.nrepl.middleware/wrap-flow-storm,cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor,portal.nrepl/wrap-portal]"]
  :jvm-opts ["-Dclojure.storm.instrumentEnable=true"
             "-Dclojure.storm.instrumentOnlyPrefixes=me.vedang."]}

Debugging with Flowstorm

(in-ns 'me.vedang.depsnew-lib)

(defn foo [n]
  (->> (range n)
       (filter odd?)
       (partition-all 2)
       (map second)
       (drop 10)
       (reduce +)))

References for learning more about Flowstorm

Why Micro-services?

Smaller teams, clear ownership!

Why Micro-services?

  • Easier to test
  • Clear contracts (API)
  • Independent deployments

Why Monorepos?

  • No version hell
  • Easy refactoring across multiple services (in theory)
  • Easy collaboration and on-boarding (in theory)

So what goes wrong with Monorepos?

  • Contracts are unwritten
  • Testing becomes messy (specifically, test-suite time)
  • Refactoring, Collaboration, On-boarding Hell

Speaker Notes

Eventually, teams start building conventions and tooling and start refactoring.

The problem is, it’s too late by now.

I use Polylith from day one

https://github.com/polyfy/polylith

Polylith is a set of conventions and tooling that has thought through all these problems!

Let’s create our Polylith monorepo!

poly create workspace name:polylith top-ns:me.vedang :commit
tree -L 3

6 directories, 4 files

Speaker Notes

Addendum: you can also create a workspace in an existing repo: poly create workspace top-ns:me.vedang

The terminology of Polylith: project

projects/<name> : Interface with hosting platform

Let’s think through an example:

  1. Static Site : projects/weblog_static (Artifact: Public Folder, Deployment: Github Pages)
  2. Dynamic Site : projects/weblog_dynamic (Artifact: Uberjar, Deployment: Dockerfile, fly.toml)

The terminology of Polylith: base

bases/<name> : Interface with the external world

Continuing with our example:

  1. Static Site: bases/server_static
  2. Dynamic Site: bases/server_dynamic (server! routes)

Speaker Notes

This folder contains code that interacts with the external world

Think: CLI tool, Server, Lambda function, Worker

The terminology of Polylith: component

components/<name> : Interface with a module of business logic.

Continuing with our example:

  1. components/content: The markdown files I want to serve as posts
  2. components/render : Code to convert MD files to HTML
  3. components/readers: Logged-in visitors on my dynamic website

Speaker Notes

This folder contains re-usable, modular code that implements specific functionality.

How does all this tie together?

projects/weblog_static -> [bases/server_static components/content components/render]

projects/weblog_dynamic -> [bases/server_dynamic components/content components/render components/readers]

How Polylith promotes Modular architecture

Code outside a component can only refer to functions in the component’s interface namespace.

Speaker Notes

During development, you have access to the entire code-base! But there are rules around how you can access any code outside your own brick.

(so I don’t need to know how you implement the code, I’m only concerned with the interface of your component)

Interface functions pass-through the arguments to appropriate internal functions.

Polylith: getting the best of micro-services and mono-repos

  • Where does the Dockerfile go? projects/weblog_dynamic
  • Where do I put end-to-end tests? projects/weblog_dynamic/test
  • Where do I define the routes and the main class? bases/server_dynamic/core.clj
  • Which functions do I use to render HTML pages? components/render/interface.clj
  • Which functions do I write contract tests for? components/render/interface.clj
  • Which functions do I document thoroughly? components/render/interface.clj
  • Where do I hook observability? components/render/interface.clj, bases/server_dynamic/core.clj
  • Where does the actual rendering functionality live? render/posts.clj, render/tags.clj, render/index.clj …

What does the poly tool give me?

  • Enables running the right tests
  • Checks to ensure component code is used properly everywhere
  • Upgrades libraries across all your bricks

… and much more

References for learning more about Polylith

There is so much more …

  • CI/CD! (Use babashka!)
  • Observability (pedestal! or iapetos + clj-otel)
  • Web Development (martian makes talking to APIs a breeze!)

Come say hi!

█▀▀▀▀▀█ ▄ █ ▀▄▀▄█ █▀▀▀▀▀█ █ ███ █ ▄▄▄ █▄ █▀ █ ███ █ █ ▀▀▀ █ ▄█ ▄▄█▄▄▀ █ ▀▀▀ █ Vedang Manerikar ▀▀▀▀▀▀▀ ▀▄█▄▀ ▀▄▀ ▀▀▀▀▀▀▀ ▀▀█▀█▄▀▀█▀▄▀▄▀▀ ▄█ █ ▀ █ - @vedang on 🐘fosstodon, 🖥️github, 🐦twitter █▄▀█▀▀▀█▄ ▀ ▄ ▄█▄█▀█▀▀ ▀█ - https://www.salher.ai █▄▄▀▄▄▀ ▀▀ ▀█▀ ▄█▀█▀▄▀█▀ + We can help with your problems! █ ▄▀█▀▀ ▀▀██▀ ▄█▀█▀██▀ ▀█ + (design, product, engineering) ▀ ▀▀▀▀▀█▄▀▀▄▄ ██▀▀▀█▄▀ █▀▀▀▀▀█ ▀▄▄ ▄█▀ █ ▀ █▄▀██ THANK YOU! QUESTIONS? █ ███ █ ██▀ ▀ ▀▀█▀█▀█▄███ █ ▀▀▀ █ ██▄█▀ ▄▀▄▄ ▄▄█▀ █ ▀▀▀▀▀▀▀ ▀ ▀ ▀▀▀▀▀▀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment