Programming with Effects

I recently ran some workshops at work around getting to the heart of functional programming:

• pure/total functions
• referential transparency
• local reasoning
• type driven development
• effects

Much of the inspiration came from Rob Norris' excellent talk on Functional programming with effects . This is not the talk I presented at work; however, I did put together an addendum to that talk that walked through some of the same effects highlighted as Rob but run through the same parametric function that was generic on the effect. I did not look very hard but I have not seen anything else quite like it so here it goes for a blog post. This is also a blog about monadic and applicative properties.

SETUP

The example we will use is a generic function that takes in three of the same context / effect / container (use whatever term works best in your brain), combines them in order to create a User , and then returns that User wrapped in the same context / effect / whatever. The F[_]: Monad means give me any context / effect / whatever that implements Monad (flatMap, bind, whatever).

case class User(name: String, id: Long, age: Double)

object Thing {

  def doThing[F[_]: Monad](a: F[String], b: F[Long], c: F[Double]): F[User] =
    for {
      aa <- a
      bb <- b
      cc <- c
    } yield User(aa, bb, cc)
}

We will run doThing over and over by supplying values wrapped in different effects: Option, Either, Future, etc. Remember that the above for comprehension is not a for loop and desugars into:

def doThing[F[_]: Monad](a: F[String], b: F[Long], c: F[Double]): F[User] =
  a.flatMap(aa => b.flatMap(bb => c.map(cc => User(aa, bb, cc))))
}

Also remember that Monad is just:

trait Monad[F[_]] {
  // some way to put a value into a monad
  def pure[A](value: A): F[A]

  // some way to collapse the nested monad down
  // e.g. I am mapping some fn over my context but that fn also returns the context
  def flatMap[A, B](value: F[A])(func: A => F[B]): F[B]
}

Effects

Option

Intuition:

• handling partial functions where other languages may throw
• gives us back total functions when something can go wrong

// option
object ValidOptionThing {
    val oName: Option[String] = Some("mat")
    val oId: Option[Long] = Some(17828382L)
    val oAge: Option[Double] = Some(1.3)
}

object InvalidOptionThing {
    val oName: Option[String] = None
    val oId: Option[Long] = ValidOptionThing.oId
    val oAge: Option[Double] = ValidOptionThing.oAge
}

// Some(User(mat, 1782382, 1.3))
object validOThing extends App {
    val res = Thing.doThingS(ValidOptionThing.oName, ValidOptionThing.oId, ValidOptionThing.oAge)
    println(s"res: $res")
}
// None
object invalidOThing extends App {
    val res = Thing.doThing(InvalidOptionThing.oName, InvalidOptionThing.oId, InvalidOptionThing.oAge)
    println("we short circuit since aa is none, b and c never evaluated")
    println(s"res: $res")

Some(User..)
None

The compiler turns the generic doThing function into something like the following at compile time:

def doThing(a: Option[String], b: Option[Long], c: Option[Double]): Option[User] = for {
    aa <- a
    bb <- b
    cc <- c
} yield User(aa, bb, cc)

I will skip pointing out this translation step in the remainder of the examples.

Either

Intuition:

• handling partial functions where other languages may throw
• gives us back total functions when something can go wrong
• gives us back exceptions where we can say what went wrong

object ValidEitherThing {
    type VEThing[A] = Either[Throwable, A]
    val oName: VEThing[String] = "mat".asRight
    val oId: VEThing[Long] = 17828382L.asRight
    val oAge: VEThing[Double] = 1.3.asRight
}

object InvalidEitherThing {
    type VEThing[A] = Either[Throwable, A]
    val oName: VEThing[String] = new java.lang.NoSuchFieldError("nope").asLeft
    val oId: VEThing[Long] = 17828382L.asRight
    val oAge: VEThing[Double] = 1.3.asRight
}

 // Right(User(mat,17828382,1.3))
object validEThing extends App {
    val res = Thing.doThing(ValidEitherThing.oName, ValidEitherThing.oId, ValidEitherThing.oAge)
    println(s"res: $res")
}

// Left(NoSuchFieldError("nope"))
object invalidEThing extends App {
    val res = Thing.doThing(InvalidEitherThing.oName, InvalidEitherThing.oId, InvalidEitherThing.oAge)
    println("we short circuit since aa is none, b and c never evaluated")
    println(s"res: $res")
}

• When we have three Right values we get back Right(User..)
• When we have one or more Left values we short-circuit and get Left(what went wrong)

Future

Intuition:

• something that is happening concurrently, possibly on another thread, like a network call
• also has a failure channel

Note: Future is not referentially transparent, prefer to use IO or Task

// pretend these are actually long running results running elsewhere

object ValidFutureThing {
    val name: Future[String] = Future.successful("mat")
    val id: Future[Long] = Future.successful(17828382L)
    val age: Future[Double] = Future.successful(1.3)
}

object InvalidFutureThing { object applicativeInvalidTThing extends App {
    val name: Future[String] = Future.failed(new TimeoutException("nope"))
    val id: Future[Long] = Future.successful(17828382L)
    val age: Future[Double] = Future.successful(1.3)
}

object validFThing extends App {
    import ExecutionContext.Implicits.global
    val res = Thing.doThing(ValidFutureThing.name, ValidFutureThing.id, ValidFutureThing.age)
    val r = Await.result(res, Duration.Inf)
    println(s"res: $r")
}

object invalidFThing extends App {
    import ExecutionContext.Implicits.global
    val res = Thing.doThing(InvalidFutureThing.name, InvalidFutureThing.id, InvalidFutureThing.age)
    val r = Await.result(res, Duration.Inf)
    println(s"res: $r")

• When we run this with 3 successul futures, we get Future[User(..)]
• When we run this with one failed future, say one of those futures like aa times out we short circuit since aa failed

Task

Intuition: The same as Future

// Task
object ValidTaskThing {
    val name: Task[String] = Task.now("mat")
    val id: Task[Long] = Task.now(17828382L)
    val age: Task[Double] = Task.now(1.3)
}

object InvalidTaskThing {
    val name: Task[String] = Task.raiseError(new TimeoutException("nope"))
    val id: Task[Long] = Task.now(17828382L)
    val age: Task[Double] = Task.now(1.3)
}

object validTThing extends App {
    val res = Thing.doThing(ValidTaskThing.name, ValidTaskThing.id, ValidTaskThing.age)
    val task = res.runAsync
    val r = Await.result(task, Duration.Inf)
    println(s"res: $r")
}

object invalidTThing extends App {
    val res = Thing.doThing(InvalidTaskThing.name, InvalidTaskThing.id, InvalidTaskThing.age)
    val task = res.runAsync
    val r = Await.result(task, Duration.Inf)
    println(s"res: $r")
}

• we get the same result as the Future case
• note since a times out, b and c are never evaluated

Aside - Applicative

In the above examples, a , b , c don't depend on each other however, we have sequenced them due to flatMap. Since they have nothing to do with each other, we can use applicative rather than monadic behavior here. E.g. we want to run a sequence of independent computations and combine the result. Sadly, in Scala, no more For syntax; however, we can write something very similar to the original doThing method replacing the Monad constraint with an Applicative constraint:

object ApplicativeThing {
    def doThingA[F[_]: Applicative](a: F[String], b: F[Long], c: F[Double]): F[User] =
     (a, b, c).mapN {
         case (aa, bb, cc) => User(aa, bb, cc)
     }

If we run something like the failing task example, we get much the same answer as before:

object ApplicativeThing {
object applicativeInvalidTThing extends App {
    val res = ApplicativeThing.doThingA(InvalidTaskThing.name, InvalidTaskThing.id, InvalidTaskThing.age)
    val task = res.runAsync
    val r = Await.result(task, Duration.Inf)
    println(s"res: $r")

We still blow up with a timed out task; The big difference here is that unlike the monadic case where b and c are never evaluated, b and c do get evaluted here. It's just the combination that fails at the end.

Aside - Applicative Validation

If we have independent effects, we can combine them like an Either but accumulate everything that went wrong rather than just the first thing that went wrong.

object ValidValidationThing {
    type NelThing[A] = ValidatedNel[String, A]
    val oName: NelThing[String] = "mat".validNel
    val oId: NelThing[Long] = 17828382L.validNel
    val oAge: NelThing[Double] = 1.3.validNel
}

object InvalidValidationThing {
    type NelThing[A] = ValidatedNel[String, A]
    val oName: NelThing[String] = "username invalid".invalidNel
    val oId: NelThing[Long] = 17828382L.validNel
    val oAge: NelThing[Double] = "age invalid too".invalidNel
}

If you try to use this monadically it is a compiler error, since ValidationNel has no Monad.

object validVThing extends App {
    val res = Thing.doThing(ValidValidationThing.oName, ValidValidationThing.oId, ValidValidationThing.oAge)
    println(s"res: $res")
}

// :( compiler error)

But ValidationNel does have an Applicative :

object validVThing extends App {
    val res = ApplicativeThing.doThingA(ValidValidationThing.oName, ValidValidationThing.oId, ValidValidationThing.oAge)
    println(s"res: $res")
}

object invalidVThing extends App {
    val res = ApplicativeThing.doThingA(InvalidValidationThing.oName, InvalidValidationThing.oId, InvalidValidationThing.oAge)
    println("we short circuit since aa is none, b and c never evaluated")
    println(s"res: $res")
}

• When we have 3 valid elements, we get Valid(User..)
• When we have 2 of three elements as invalid we get:
  • Invalid(NonEmptyList(username invalid, age invalid too))
  • we all all the errors with no short-circuiting
  • if we had used Either we would only see the username error

Aside - Nested Task with Either

What if we end up with a Task[Either[Throwable, User]] ? - thing.doThing doesn't work! it's a compiler error

We would have to write something like this with two for comprehensions:

object nestedTaskEither {
    type NestedTask[A] = Task[Either[Throwable, A]]

      def doThingNested(
        a: NestedTask[String],
        b: NestedTask[Long],
        c: NestedTask[Double]): NestedTask[User] = for {
        aa <- a
        bb <- b
        cc <- c
    } yield for {
            aaa <- aa
            bbb <- bb
            ccc <- cc
    } yield User(aaa, bbb, ccc)

This sucks but it does work:

object Nested extends App {

    import nestedTaskEither._
    val teName: NestedTask[String] = Task.now("mat".asRight)
    val teId: NestedTask[Long] = Task.now(17828382L.asRight)
    val teAge: NestedTask[Double] = Task.now(1.3.asRight)


    val res = doThingNested(teName, teId, teAge)
    val task = res.runAsync
    val r = Await.result(task, Duration.Inf)
    println(s"res: $r")

How do we make this work with the original DoThing without this annoying double unpacking of an effect inside an effect? We can use a monad transformer like EitherT to get us back to where we want to be. EitherT knows about the task inbetween:

object monadTransformer extends App {

    type TE[A] = EitherT[Task, Throwable, A]

    val mtName: TE[String] = EitherT(Task.now("mat".asRight))
    val mtId: TE[Long] = EitherT(Task.now(17828382L.asRight))
    val mtAge: TE[Double] = EitherT(Task.now(1.3.asRight))

    // call our original method at the top of the file that only
    // has the single for comprehension
    val res = Thing.doThing(mtName, mtId, mtAge)

    val task = res.value.runAsync
    val r = Await.result(task, Duration.Inf)
    println(s"res: $r")

}

aaaaaaaaaaaaaaaaaaand it all works again and we get back Right(User(...))

Back to Monads and Effects

The previous examples were all about failure. Which is cool. This is super useful for us. But there is way more to Monads than that, including an entire book

List

This one doesn't really make that much sense with our running example. List imbues the effect of multiple possible results. We get all the possible results with List.

object listThing extends App {

    val userNames = List("mat", "steve", "jim")
    val ids = List(1L)
    val ages = List(1.3, 2.7, 99.9)

    val res = Thing.doThing(userNames, ids, ages)
    println(s"res: $res")
}

In the previous example we just got one result. In this case, we get all the combinations of our inputs:

User(mat,1,1.3)
User(mat,1,2.7)
User(mat,1,99.9)
User(steve,1,1.3)
User(steve,1,2.7)
User(steve,1,99.9)
User(jim,1,1.3)
User(jim,1,2.7)
User(jim,1,99.9)

Reader

Intuition: dependency injection

It's true that we can just use constructors for dependency injection in scala. This is often the right idea, but not the only way to do dependency injection. See pros and cons here and here

Reader (and it's Monad) let us sequence operations that depend on some input:

object readerThing extends App {

    // some config or whatever class we are injecting
    case class Injected(version: String, idShift: Long, ageShift: Double)

    // two different versions of the thing we want to inject
    val injected = Injected("3.2", 200000L, 37.2)
    val someOtherInjected = Injected("9.9", 382973238L, 99.9)


    // the inputs to doThing
    type Config[A] = Reader[Injected, A]
    val rName: Config[String] = Reader(in => s"${in.version}:mat")
    val rId: Config[Long] = Reader(in => 17828382L + in.idShift)
    val rAge: Config[Double] = Reader(in => in.ageShift + 1.3)


    // this is the "program" returned from running doThing
    // it hasn't done anything yet, it's really more like
    // a function that is waiting for an input before the result
    // e.g. program is a function of Injected => User
    val program = Thing.doThing(rName, rId, rAge)

    // this just gives us the composed "program" waiting for an input
    // we need to supply the input

    val runRes = progam.run(injected)
    println(s"res: $runRes")

    // run it with some other config injected
    val runRes2 = program.run(someOtherInjected)
    println(s"res: $runRes2")

• now this is super interesting

  User(3.2:mat, 180....)
  User(9.9:mat, ...)
  

• notice how making the program Thing.doThing(rName, rId, rAge) didn't do anything
  • it just guves us back Reader[Injected, User] but not the actual user
• we need to supply it with the config we want for it to work
• we can re-use the same program by applying different configs and getting different users

Writer

Intuition: we want to run some computation and while we are doing that we want to annoate that computation.

• feels like tracing and logging (but doesn't have to be, and don't actually use it for logging)

object writerThing extends App {

    case class Computation(notes: String, money: Int)

    type Trace[A] = Writer[List[Computation], A]

    val wtName: Trace[String] = for {
        a <- "mat".pure[Trace]
        _ <- List(Computation("fetched user", 100)).tell
    } yield a

    val wtId: Trace[Long] = for {
        a <- 17827382L.pure[Trace]
        _ <- List(Computation("fetched id", 1000)).tell
    } yield a

    val wtAge: Trace[Double] = for {
        a <- 1.3.pure[Trace]
        _ <- List(Computation("fetched age", 10000)).tell
    } yield a

    val program = Thing.doThing(wtName, wtId, wtAge)
    val (notes, user) = program.run
    println(s"trace: $notes \n ----------\n")
    println(s"user: $user")

}

this results in:

notes: List(Computation(fetched user,100), Computation(fetched id,1000), Computation(fetched age,10000))
 ----------

user: User(mat,17827382,1.3)

doThing

Conclusions

Phheww. We looked at a bunch of effects. Some of those effects short-circuited, some of those effects did some work somewhere else (possibly on another thread), and some of those effects were very different from the rest:

• we ran some computation in F resulting in an F[User]
• the effect is whatever differentiates User from F[User]

We can keep going with these building blocks. For example, in the Reader example we would keep composing more readers and only at the end of the world (edge of the program) do we supply the initial config value. This is very cool. This is a very different way to think about programs and putting together programs in a referentially transparent way.