scalax-2016-day-1.md

Ammonite for Scala Scripting (@li_haoyi)

can do amm foo.sc, but also scala foo.sc. So why use it? Because I can import dependency on the fly. With scala I have to use -cp ... => Blah! And I can't import other scripts

####amm scripting

import $file.foo.bar` //import ./foo/bar.sc
foo.bar.methodOfBar(argument)

import dependencies: $ivy.group:artifact:version example: import jsoup to scan html sites

=> Takeaway: We can use dependencies, isolate features, use Python style to hack things up. When something is too small for sbt, but too big for REPL, use Scala scripting with Ammonite.

Even write CI files

To invoke shell command, use %git or %ls etc...

TODO: Pretty printer using Shapeless!

Using Docker for better software

50 uservices.

Docker compose: declare containers in a compose.yaml, start them e.g.

mongo:
  image: mongo: xxx
  expose:
    - "27107"

qotd:
  image: ubuntu
  expose:
    - "8081"
  links: mongo

Used to produce consistent integration tests.

Typicallly:

• run Database
• cleanup
• Stub of different versions of legacy systems
• Check that versions of our services are compatible
• Dependencies on other services: Shall we stub another service when another team is responsible for it. And we can have Service B slow at startup that makes my integration test to blow up.

With Scala:

• Generate Docker image with native packager. Can add params such as dockerExposePorts += 9000 and then sbt docker:publish. This makes a Dockerfile in the target directory. For testing this file is ok most of the times.
• Use a container with sbt, configure userv we are going to need, define the sbt tasks that define the docker files, and run the docker-compose.yml and then the integration tests
• N.B. need to mount also the repository and ivy cache from localhost.

Problems:

• The slow service might still cause flaky tests to exist

Approache - Docker-it-scala:

• If we have only 1 dependency, Docker-it-scala defines a simple DSL, we can mix in DockerTestKit or DockerKitSpotify to our test suites. This has a StartAllOrFail mathod that brings up all the dependent containers. => sbt dist:test
• Can enable also parallel execution, but watch out that the parallel tests should not share containers otherwise we have shared mutable state
• Exists also a withReadyChecker that checks when the encapsulated app within the container is ready. The test blocks until all conditions are met, and only at that point they run.

Check github kasia-kittel

Risking everything with Akka-Streams (@johofer)

Zalando becoming a platform for fashion services. Connect mus via event streams. 3 Billion EUR / year. Use case: automaticallly decide which prods should be shown and hidden (because of stock, price, image quality, anything). Required latency: seconds. Team: Java developers, no Scala ninja. TODO: Check blog: "Which shoe fits you?" from zblog.

Lessons learned

Streams dsl is probably easier to understand than the graph dsl, which is unavoidable in some cases. Difficult concepts: Materialization

Failures: caused by onError: this completes the stream, wo one has to recover or recoverWithRetries. Errors: caused by exceptions but they get escalated to failures by default. Recover via SupervisionStrategy. One can use by default the Resume strategy. Does this make sense? Mostly not. E.g. if event comes in, and rule is required for that event, and the rule should be evaluated. If rule cannot be found, error gets Resumed but this blows up because we end up evaluating events with wrong rule, and this causes the Evaluate stage to backpressure. Solution: Encapsulate errors as \/ and then no need to involve the SupervisionStrategy.

Other errors: akka-http client. What happens if I don't consume the response (e.g. because the response status != 2xx), but we should consume the response anyway! Solution: discardEntityBytes. Other errors/2: no response from target server. There is no default timeout, you should use the xxxTimeout stage instead! Other errors/3: Internal buffers for stages. Default size = 16. Normally increaasing the buffer doesn't buy that much. If necessary, put an explicit buffer stage.

Configuration in etcd servers. How to inject it? Same configuration could be used by different stages. How to deal with it? Open point. Configuration source + events crossing. To be solved.

Shining points

• Backpressure handled automatically
• Thinking in streams => Thinking statelessly => Can be tested easily
• Built in stages help a lot in tweaking the asynchronous behavior. groupedWithin(maxNr, maxTime), mapAsync(f: A => Future[B]). Throttling stages (useful for new deployments where we want to test if it works first with small numbers)

OPS - Monitoring:

• Monitoring from OPS perspective. monitor stage doesn't help that much. How to monitor the buffer size?
• Easy to instert any custom monitoring stage

OPS - Tuning:

Easy to tune, very efficient and performant. Easy to scale because stateless. Distribute over multiple nodes: GearpumpMaterializer. (TODO: Check!!!)

Conclusion:

It's magic, but it requires studying. It's efficient and powerful and performant. Latency very low.

If it blows up during the processing, how do we recover the unprocessed events? At the end we should acknowledge events, and whatever hasn't been acknowledged should end up in a queue and be reprocesed.

Practical Eff Monad (@etorreborre)

How to frame service as components that can evolve? How do we deal with DI? To be radical, use constructor injection (forget about frameworks and cake) and further, make it a Reader monad. See Grafter [https://github.com/zalando/grafter] project for Zalando. it uses Shapeless to derive a reader from the Config object. At the end of the day, we can even have a Reader from Config to server, and inject final configuration only in the main.

Build a route as a case class with a val. Service that the route needs is just a constructor parameter (or a reader input). All dependencies work as long as they know how to looof for each other. Important: always provide an implementation for default.

Can I put a Reader[Config, Route] in a library? yes, as long aas I add a way to extract the specific component from the generic one. => Reader all the way down! => Unit testing easier. Integration testing? It can be hard to replace some components => I want to rewrite the tree of dependencies. In Scala there is Kiama. We can define a strategy to rework the nodes of the tree, e.g. replace all nodes that match a given condition with something else. => easy to replace mocks of target services with mocked ones. Tree must become a graph in some cases, e.g. when one dependency is a ThreadPool. Kiama allows to do this.

How to stay lazy: use Eval from Cats Takeaways: Use case classes, interfaces, Reader instances, Tree rewriting, use lazyness

---

#####Eff monad

def get[R: _async :_logged :_flowId](a, b, c): Eff[R, Price]

R means all the possible effects that the computatoin will produce, in this case async, logged, and with flowId. Then I have to provide the actual effects, and the interpreters for this.

• FlowId: When request comes in, I want to pass this flowId as a correlation ID. I don't want to pass it along to every input, therefore I can define a Reader[FlowId, ?]. If I need the flowId I can use the ask method.
• Logged effect. Use the send(_) method. And then I define the interpreter. USeful for testing. In testing I can check that log statements are actually sent, or just ignore them on the other hand.
• Eff is also applicative, in that we can run stuff in parallel and I want to make this transparent. The _async effet does exactly this. Use |@| combinator as map2
• Can interpret Async how I want, as scala.Future or scalaz.Future, or Monix
• Drawback: Can't use for comprehensions

Json Path (@JulienTruffaut)

Dsl to simplify manipulation of Json documents. Recommendation: Use tut for documentation.

Javascript is completely untyped,so we don't know until last moment. Also Json is mutable in Js. Json is a coproduct of different types. For any coproduct we can define a prism. It is a mix of pattern matching and a constructor. Prism safe cast: A Double can be seen as an int or any double minus all the ints. Check doubleToInt in monocle, a prism Double -> Int Prisms compose, provided both prisms are happy.

monocle.Optional

We use this to modify a structure. Optionals are also composing. Plus Optional compose Prism or Prism compose Optional gives us an Optional This works equally for Map. Result: we can define in 1 shot how we are zooming in the structure, and modify it. This is only too verbose ==> JsonPath defines alias for composePrism and composeOption To make things better, use Scala Dynamic extension (this is great stuff!) => I can write root.foo.bar.json.getOption("blabla")

Traversal

We can build a traversal through selectDynamic This way we can deep and select like in xslt for Json.

Pimpathon

Alternative approach is using Pimpathon which uses strings instead of DSL.

The whole thing is about generic ADT, not just about JSON! So we can use it for any manipulation. Check also the Plated feature of monocle, that does an action no matter what