First creation date: 2025-04-20
Latest update date: 2025-04-24
Changelog
- 2025-04-24: Updated solution for state-threading
If you have been following recent developments in the Scala ecosystem,
you might have heard about yaes (Yet Another Effect System): https://github.com/rcardin/yaes.
First of all, big thanks to its author Riccardo Cardin and others who have helped out here!
It leverages implicits (context parameters) to thread effects seamlessly throughout the call stack. You can have e.g. an effect for logging some data, without having to pass this effect around directly.
import in.rcard.yaes.Output.*
def program(name: String)(using Output): String = {
val result = s"Hello, $name!"
Output.printLn(result)
result
}
val program: Output ?=> Unit = program("Jane") // This does not print anything actually!
Output.run { program } // This does: Hello, Jane!However, any call to such an effect, e.g. to log some data on the logger effect, is not actually executed immediately. The execution of such code is deferred, until an effect handler is provided that takes care of running the effectful computation.
Thus, it results in code that is written in direct style without the need for for-comprehensions, while also offering staged execution of effectful program descriptions. Sounds great, but what's the catch?
If you look at the currently available effects https://github.com/rcardin/yaes?tab=readme-ov-file#effects-or-capabilities, and also at the plans for future effects https://github.com/rcardin/yaes/issues, you will notice the lack of a State effect. A State effect allows you to read values from the context, and more importantly, also write values to for future consumption by downstream code.
A very straightforward implementation for a plain (non-monadic) State effect would be a case class that contains a mutable current state value, and some additional helper code under the hood. Then, such capability could be passed around using implicits like the rest of these effects:
case class MutState[S](var value: S) {
def run[A](block: MutState[S] ?=> A) = block(using this)
}
object MutState {
def apply[S](using MutState[S]): MutState[S] = implicitly
}
def program(name: String)(using MutState[Int], Output): String = {
MutState[Int].value += 1
val time = MutState[Int].value
val result = s"Hello, $name, for the ${time}th time!"
Output.printLn(result)
result
}
val program: MutState[Int] ?=> Output ?=> Unit = program("Jane")
MutState(0).run { Output.run { program } } // prints "Hello, Jane, for the 1th time!"However, we lose referential transparency when updating an existing MutState instance:
val initalState = MutState(0)
initalState.run { Output.run { program } } // prints "Hello, Jane, for the 1th time!"
initalState.run { Output.run { program } } // prints "Hello, Jane, for the 2th time!"Let's convert it to an immutable state then, and create fresh instances when we need to update it, instead:
case class MutState[S](val value: S) {
def modify[SB](f: S => SB): MutState[SB] = copy(value = f(value))
def update[SB](newValue: SB): MutState[SB] = modify(_ => newValue)
}
object MutState {
def apply[S](using MutState[S]): MutState[S] = implicitly
def run[S, A](initial: S)(block: MutState[S] ?=> A) = block(using MutState(initial))
}
def program(name: String)(using MutState[Int], Output): String = {
given MutState[Int] = MutState[Int].modify(_ + 1)
val time = MutState[Int].value
val result = s"Hello, $name, for the ${time}th time!"
Output.printLn(result)
result
}
MutState.run(0) { Output.run { program("Jane") } }Great! However, if you try to update the state again and thus try to redeclare given MutState[Int] once more,
the compiler will complain about Ambiguous given instance.
There's a limitation in place that you can have only one given instance per scope.
We can work around this by assigning each MutState instance to its own variable:
def program3(name: String)(using m0: MutState[Int], O: Output): String = {
val m1 = m0.modify(_ + 1)
val result = s"Hello, $name, for the ${m1.value}th time!"
Output.printLn(result)
val m2 = MutState[Int](using m1).modify(_ - 1)
result
}This is quite error-prone, though. We can easily reference the wrong variable while updating state, or forget about passing the freshest state variable as the proper implicit parameter to other functions explicitly.
By sacrificing a bit of referential transparency, which affects the current scope only and does not spill over to the rest of the call stack (!), this can be cleaned up significantly though:
def program(name: String)(using M0: MutState[Int], O: Output): String = {
implicit var M: MutState[Int] = M0 // interestingly, we cannot use "given" syntax here for var
M = M.modify(_ + 1)
Output.printLn(s"Greeted for the ${M.value}th time")
M = M.modify(_ - 1)
Output.printLn(s"Processed index: ${M.value}")
s"Hello, $name!"
}
MutState.run(0) { Output.run { program("Jane") } }
// prints "Greeted for the 1th time"
// prints "Processed index: 0"We still might forget to re-assign the new state to the existing variable. Let's try to encapsulate this, to reduce the surface for human error:
case class MutStateVar[S](var state: MutState[S]) {
def value: S = state.value
def value_=(newValue: S): Unit = state = state.update(newValue)
}
object MutStateVar {
def apply[S](using MutState[S]): MutStateVar[S] = new MutStateVar[S](implicitly[MutState[S]])
}
def program(name: String)(using M0: MutState[Int], O: Output): String = {
val MutState = MutStateVar[Int](using M0)
implicit def mutState: MutState[Int] = MutState.state
MutState.value += 1
Output.printLn(s"Greeted for the ${MutState.value}th time")
val implicitMutState = implicitly[MutState[Int]]
val result = s"Hello, $name, greeting you for the ${implicitMutState.value}th time!"
MutState.value -= 1
Output.printLn(s"Processed index: ${MutState.value}")
result
}
MutState.run(0) { Output.run { program("Jane") } } // results in "Hello, Jane, greeting you for the 1th time!"
// prints "Greeted for the 1th time"
// prints "Processed index: 0"The state update looks much cleaner and safer now, and the implicit resolution works as expected, without having to pass the correct implicit instance manually!
We still need these two lines of "preamble" code at the start of the program method though,
which could maybe somehow be made even more concise.
Unfortunately, there's another issue with this approach we haven't examined yet -
how do we return the updated MutState to the caller?
We would like to know the final value for MutState at the top of the call stack,
thus we would like MutState.run to return a tuple (finalStateValue: Int, programResult: String).
If we make program return its final MutState as part of its return value
def program(name: String)(using M0: MutState[Int], O: Output): (MutState[Int, String)
we lose all the ergonomics and are missing the point of using effects seamlessly from the function's implicit environment.
Just like we can have implicit (context) arguments for a function, the idea crossed my mind to have implicit (context) return values for functions. Not sure whether that would be feasible or still result in a sound type system in Scala, but, anyways, that's not an option right now.
Update:
Previously, I thought the only way to deal with this was to thread a single MutStateVar all the way
from the beginning to the end of the call stack where the MutState effect is needed.
This would have resulted in losing referential transparency across the whole call stack.
However, with a new approach we can achieve partial referential transparency though, after all.
Namely, we have to adapt the implicit resolution via MutState.apply to return the most up-to-date MutState at all times.
The MutState instances themselves can remain immutable, though.
By having the MutState.run handler keep track of the most up-to-date state S, the whole mechanism works even across different call stacks!
trait MutState[S] {
def value: S
def value_=(s: S): Unit
private[MutState] def implicitValue: S
}
object MutState {
def run[S, A](initialValue: S)(block: MutState[S] ?=> A): (S, A) = {
var internalValue: S = initialValue
val mutState = new MutState[S] {
val value = initialValue
def value_=(s: S) = internalValue = s
def implicitValue = internalValue
}
given MutState[S] = mutState
val a = block
val s = internalValue
(s, a)
}
def apply[S](using M: MutState[S]): MutState[S] = new MutState[S] {
val value = M.implicitValue
def value_=(s: S) = M.value = s
def implicitValue = M.implicitValue
}
}
def program(name: String)(using MutState[(Int, Int)]): String = {
var result = s"Processing $name:\n"
val initialState = MutState[(Int, Int)]
result += getGreeting(name)
result += getFarewell(name)
result += getGreeting(name)
result += getFarewell(name)
val finalState = MutState[(Int, Int)]
val diff = (finalState.value._1 + finalState.value._2) - (initialState.value._1 + initialState.value._2)
result += s"Processed statements for $name: $diff\n"
result
}
def getGreeting(name: String)(using MutState[(Int, Int)]): String = {
val (greetings, farewells) = MutState[(Int, Int)].value
MutState[(Int, Int)].value = (greetings + 1, farewells)
s"Greetings, $name!\n"
}
def getFarewell(name: String)(using MutState[(Int, Int)]): String = {
val (greetings, farewells) = MutState[(Int, Int)].value
MutState[(Int, Int)].value = (greetings, farewells + 1)
s"Farewell, $name!\n"
}
MutState.run((0,0)) { program("Jane") }
/* returns ( (2,2), "
Processing Jane:
Greetings, Jane!
Farewell, Jane!
Greetings, Jane!
Farewell, Jane!
Processed statements for Jane: 4
") */As the library's name clearly states, it's yet another effect system. Even if this effect system's approach was the ultimate solution to all our problems and wishes for safe yet lean FP programming in Scala, no application already in production will be migrating its whole codebase to this new approach. You can have a state-of-the-art solution for a problem, if it's hard to apply and integrate in practice it won't be used unfortunately.
Supermonad effect systems may tell you otherwise, but tagless final is a powerful abstraction that allows you to be more flexible in what actual effect system implementation you use under the hood, at least in theory. In practice, compared to Typelevel (cats & co), Monix, Izumi (BIO) and tofu-tf, I have yet to see ZIO and kyo ecosystem offer typeclass instances for common effect types, if that even is feasible and makes sense on a technical level.
Prior to putting the cart before the horse though, what is Tagless Final (TF) and how do you structure your application with it? I'd like to thank and refer to existing resources on the Web here, that explain that better than I ever could, for example:
- Tagless Final for Humans | Noel Welsh | Scalar Conference 2025
- Tagless Final in Scala Quickly Explained | Daniel Ciocîrlan | Rock the JVM
How does an example TF application look like then? Let's use the following contrived example, which uses a couple of TF typeclass abstractions:
//> using scala 3.3.5
//> using dep org.typelevel::cats-mtl:1.5.0
//> using dep org.typelevel::cats-effect:3.6.1
//> using options -deprecation -encoding UTF-8 -feature -unchecked
import cats.Monad
import cats.mtl.Stateful
import cats.mtl.Raise
import cats.effect.Unique
type Cache = Map[String, String]
case class Error(message: String)
def statement[F[_]](id: String)(using M: Monad[F], U: Unique[F], S: Stateful[F, Cache]): F[String] = {
import cats.syntax.all._
for {
cache <- S.get
result <- cache.get(id) match {
case Some(result) => M.pure(result)
case None => U.unique.map(t => t.hash.toString())
}
_ <- S.set(cache.updated(id, result))
} yield result
}
def program[F[_]](using Monad[F], Unique[F], Raise[F, Error], Stateful[F, Cache]): F[String] = {
import cats.syntax.all._
import cats.mtl.syntax.all._
for {
res1 <- statement("123")
res2 <- statement("456")
_ <- if (res2.toInt % 2 != 0) then
Error("Cannot use odd res2 for some reason!").raise
else
Error("").pure
res3 <- statement("123")
res = s"[$res1, $res2, $res3]"
} yield res
}To make the integration between TF and YAES work and to have YAES thus drive the execution of this code, we need to provide typeclass instances for these various effects that are used:
cats.effect.Uniquecats.mtl.Raisecats.mtl.Stateful
Plus we need a cats.Monad instance for our effect container F[_],
so that we can construct a monadic for-comprehension, as seen above.
What could be our effect container F[_]?
- we need to be able to access our capability implementations from it
- we need the effect container to be a monad
- we need to be able to make it a type constructor of kind
* -> *
The cats.data.Reader monad perfectly fits the above requirements, so let's use that!
Thus, we can "substitute" F[_] with [A] =>> Reader[EF, A].
A is the type of the effectful computation, the result of our above program (in our case String),
wrapped in our effect container Reader.
EF is the type of some read-only value we can get from the environment, from our effect container Reader.
In our case, ef: EF needs to have some methods like ef.unique, ef.raise and ef.stateful defined.
With a general plan laid out, let's start by providing the typeclass instance for (arguably) the most straightforward effect:
trait Unique[F[_]] {
def unique: F[Token]
}We need to create a new YAES capability that will match the cats effect for generating an unique token,
since YAES capabilities perform their effect directly, "outside" the effect container.
YAES capabilities are non-monadic, their computation does not depend on previous operations from the effect container,
nor do they alter the behavior of the effect container for follow-up computations.
To avoid name-clashes with the existing cats trait Unique, we'll call our YAES trait TokenGen.
Putting above points into action, creating a effect typeclass instance for Unique by delegating the work to a TokenGen capability,
which is part of environment context EF via HasTokenGen:
trait TokenGen {
def generate: Token
}
trait HasTokenGen {
def tokenGen: TokenGen
}
implicit def contextUnique[EF <: HasTokenGen]: Unique[[A] =>> Reader[EF, A]] =
new Unique[[A] =>> Reader[EF, A]] {
def unique: Reader[EF, Token] =
Ask.ask[[A] =>> Reader[EF, A], EF].map(_.tokenGen.generate)
def applicative: Applicative[[A] =>> Reader[EF, A]] = implicitly
}Let's construct a simple program, substituting F[_] with our Reader monad.
This is just a description of a program, it's not run until we give it the required capabilities:
def tokenGenTest[F[_]](using M: Monad[F], U: Unique[F]): F[String] = {
import cats.syntax.all._
U.unique.map(t => t.hash.toString())
}
val monad /*: Reader[HasTokenGen, String]*/ = tokenGenTest[[A] =>> Reader[HasTokenGen, A]]We now need to give it the implementation of the capability TokenGen, wrapped in our environment context HasTokenGen,
in order to run this program description:
val hasTokenGen = new HasTokenGen {
def tokenGen = new TokenGen {
def generate = new Token
}
}
val result: String = program.run(hasTokenGen)
result // returns some random hash, e.g. "974592379"However, this is not entirely in the spirit of YAES, and its idiomatic way to run capabilities by providing context parameters.
Let's try to rewrite this in YAES's spirit then:
object TokenGen {
def run[A](block: TokenGen ?=> A): A = {
given TokenGen = new TokenGen {
def generate = new Token
}
block
}
}
val result: String = TokenGen.run {
val hasTokenGen = new HasTokenGen {
def tokenGen = implicitly
}
program.run(hasTokenGen)
}
result // returns some random hash, e.g. "1049174687"We have successfully crossed the bridge between YAES-like capabilities and TF-style applications, at least for this simple program, yay!!
Let's attempt a similar approach for the capability MutState[S] that will be used as the counterpart for the effect Stateful[F[_], S]:
- Instead of
Reader[EF, A]we will be usingState[SF, A]as our effect container forF[_]sf: SFstays the same asef: EFbefore, bar the name change
- This allows us to make
MutStateimmutable, and update it using theStatemonad instead - We need this in order to satisfy the laws for the
Statefuleffect - We'll use a local variable inside
MutState.run, in order to keep track of the finalMutState[S].value- refer to the end of the [#state-threading](state-threading section) for more details on this approach
- we are adding an additional private method
MutState.withValue, to make the creation of a newMutStateatomic
trait MutState[S] {
def value: S
def value_=(s: S): Unit
private[MutState] def implicitValue: S
private[MutState] def withValue(s: S): MutState[S]
}
object MutState {
def run[S, A](initialValue: S)(block: MutState[S] ?=> A): (S, A) = {
var internalValue: S = initialValue
case class MutStateImpl(value: S) extends MutState[S] {
def value_=(s: S) = internalValue = s
def implicitValue = internalValue
def withValue(s: S) = {
value = s
copy(value = s)
}
}
given MutState[S] = MutStateImpl(initialValue)
val a = block
val s = internalValue
(s, a)
}
def apply[S](using M: MutState[S]): MutState[S] = new MutState[S] {
val value = M.implicitValue
def value_=(s: S) = M.value = s
def implicitValue = M.implicitValue
def withValue(s: S) = M.withValue(s)
}
def updated[S](s: S)(using M: MutState[S]) = M.withValue(s)
}
trait HasMutState[SF, S] {
def state: MutState[S]
def withState(newState: MutState[S]): SF
}
implicit def contextStateful[S, SF <: HasMutState[SF, S]]: Stateful[[A] =>> State[SF, A], S] =
new Stateful[[A] =>> State[SF, A], S] {
def get: State[SF, S] =
Stateful.get[[A] =>> State[SF, A], SF].map(_.state.value)
def set(s: S): State[SF, Unit] =
Stateful.modify[[A] =>> State[SF, A], SF](sf => sf.withState(
MutState.updated(s)(using sf.state)
))
def monad: cats.Monad[[A] =>> State[SF, A]] = implicitly
}Testing the new MutState capability:
def mutStateTest[F[_]](using M: Monad[F], S: Stateful[F, Int]): F[String] = {
import cats.syntax.all._
S.set(1).map(_ => "Hello!")
}
class SF(using val state: MutState[Int]) extends HasMutState[SF, Int] {
def withState(newState: MutState[Int]): SF = new SF(using newState)
}
val program = mutStateTest[[A] =>> State[SF, A]]
val result = MutState.run(0) {
program.runA(SF()).value
}
result // returns (1, "Hello")After migrating TokenGen to the State monad, and following this pattern going forward, we can create similar bridges between other capabilities and effects.
You can find the whole runnable code here in the other file of this gist.
Scastie: https://scastie.scala-lang.org/uDlRcQOSS2SZOa06RrnGyg
One more interesting thing is how you can build and run the program from the start of this chapter,
after putting everything together:
TokenGen.run {
RaiseError.run[Error] {
MutState.run(Map.empty) {
class SF(using val state: MutState[Cache], val raiseError: RaiseError[Error], val tokenGen: TokenGen)
extends HasMutState[SF, Cache] with HasTokenGen with HasRaiseError[Error] {
def withState(newState: MutState[Cache]): SF = new SF(using newState, implicitly, implicitly)
}
val p = program[[A] =>> State[SF, A]]
val result = p.runA(SF()).value
result
}
}
}
// sometimes prints e.g. "(Map(123 -> 833957828, 456 -> 559994164),[833957828, 559994164, 833957828])"
// sometimes prints "Error(Cannot use odd res2 for some reason!)" insteadI hope my thoughts and examples here have given at least some food for thought on using and integrating YAES into existing applications. It seems like a lightweight alternative to monadic effect system, and a step towards more direct-style effect handling.
One of the advantages here of using YAES instead of cats for the program above is
that we have gotten rid of the whole monad transfomer stack that would have been used for the effect container F[_]!
We are using a simple State monad for F[_] instead.
There have been reports of performance degradation due to deeply nested monad stacks.
Furthermore, the compiler's type inference is getting more tricky, and may need additional type hints the more complex the stack gets.
Moreover, Monads to not compose in all scenarios, you cannot make a monad transformer for all combinations of all kinds of monads. However, I'm also quite not sure whether YAES capabilities can compose properly either, in all scenarios. This would be an awesome boon if true.
Furthermore, as we have seen above, some compromises have to be made in terms of referential transparency when trying to thread state context throughout the call stack.
Do note that I may have missed a critical aspect for state threading or while integrating YAES into TF-style code. I would greatly appreciate any further input and discussion on this topic!
Hey, thanks for the hard work! You've pushed the YAES library further than I've ever done, and I really appreciate it.
You were right when you said that you cannot find many plans for the future of the library. I'm trying to evolve it one step at a time, but I'm the only contributor, and my spare time is minimal.
I hope you'll join the list of contributors very soon.