Ruby Conf San Diego 2014

#Day 1

Matz Keynote

Static Type vs. Dynamic Type.

Static Typing is against DRY.
Soft typing:
- Type is represented by set of methods e.g name and # of arguments
- Class (as set of methods)

Ruby 3.0. Matz believes "the future is bright"

Promises in Ruby - by Dinshaw Goghani

Project had to do with automating adds. Used state machines to manage the work flow; issues with call backs and a lot of spaghetti code. Read reference article: https://blog.domenic.me/youre-missing-the-point-of-promises. Promises have 3 states:

Pending state
Acceptance or Rejection state.
Fullfilled state

How would you handle concurrency? A promise is a specification. No matter how long you wait as long as you have a handler on these promises you get them all back. Use All to return everuything. When its on the last success it will let us know that all promises are fulfilled. Resolution can be either be fullfilling and rejection. Value is to distinguish betweem asyncrhonous (via Thread.new) and synchronous stuff.

Good read Adam Waxman Blocks, Procs and Lambdas.

Key concept: should return a promise immediately and either fullfil or rejected.

Gems: Sequential Workflow, celluloid.

Letting concurrency help you today

get more out of hardware
better user experience

Its confused with parallelism. Concurrency is about dealing with a lot of things at once while Parallelism is doing a lot of things at once. Eg. Batch processing background jobs vs. Threaded or forking web server multicore (concurrency vs. parallelism).

You want the benefits of parallelism but you want it in a clean way. What this talk is about. Basically THREADS

Celluloid: almost 20 years old. You start off async but you have explicity mention to be synchronous.

Sidekiq: multi threaded - uses Celluloid. Luncher (actor) -> Fetcher (actor) -> Manager (actors!) -> Processors (actors!) (all running concurrently).

Another background job: sucker_punch. Same API as sidekiq. Its all in memory, so not for very important jobs. Mostly for prototyping.

Puma: Multi-threaded, forking, evented

Concurrent Deisgn.

Strong Duck Type Driven Development

A compiler telling you when you break something along with what needs to be fixed.

Gem lawyer, you telling what methods much be implemented i.e a contract via :confirmed

We check the type before run time. You tried to call a method that does not exist and you want to know a lot earlier that this occurred. Focus on the messages between parts and system. Don't put the focus too much on objects; put it on messages. Reduce accidental coupling. Avoids ball-of-mud code.

Ruby Idioms You're Not Using Yet

Difference between an idiom and pattern. Good read: http://www.amazon.com/Smalltalk-Best-Practice-Patterns-Kent/dp/013476904X

Why use Idioms?

Memoization
- e.g @yesterday || Date.today - 1
- for expensive operations
Fetch with Default Block .fetch instead of [] so you can handle nil values
- alternative: In case of Hash, new it up with a different default Hash.new[:not_found]. Can't do this with Arrays :(
Tag Module
Page Object: good if using capybara or cucumber (Selenium Developers; more for Java)
Test Spies

Module Factory

 module Virtus
 
   def self.model(options={})
     Module.new.tap do |mod| 
       if options.fetch(:construction) { true }
         mod.send(:include, Model::Constructor)
       end
     end
   end
   
 end

Circuit Breaker: stop trying to access an inaccessible resource. Slow or unresponsive server etc.
- What do you do when a circuit is broken:
  - Drop the request
  - Raise an exception
  - Use a promise for a return result
  - Use logging
Blue-Green Deployment
- 2 complete production environments. 1 is live and the other one is either 'previous' or 'next'. When you deploy, you deploy to the non live one and then you run your tests and swap the live with next. Very easy to switch back to 'previous'.
- Trickiest part is data base integrity
Other Idioms: Config blocks, DSLs, Flip-Flops, Block or value, Enumerable/Enumerator, Keyword arguments (2.0+), Soft Typing
References: http://martinfowler.com/bliki/
Slide show software - remark

Ruby Deoptimization: http://www.chrisseaton.com/rubytruffle/deoptimizing/ READ ME.

Your Bright Metaprogramming Future: Mistakes You'll Make (and How to Fix Them)

method_missing? when you send a bag of args to a method, Ruby looks up the look-up chain up to Object and when it does not find anything it looks for method_missing. Its good to cache the method with ```find``
pretty dynamic DSLs: Hat.find_by_yarn_and_size instead of options hash
respond_to_missing?
define_method
Hooks: .included, .inherited and .extended - DON'T. Use a service object instead.
Keep in mind: grep, extend and debug the code while metaprogramming.

#Day 2

Harnessing other languages to make Ruby better

Vectorisation: LAPACK, BLAS, etc. It is very solid but it allows to optimize particular pathway.
Looked into R, Scala but Python ended up working for big data related algorithms.
Why Python? Size of scientific community, depth of libraries, similar(ish) attitude of usability, vectorization for performance and easy transition.
What does Python have to offer?
- Numpy: all about Vectorization starting from 1995
- Pandas: built on Numpy. Pots the best bits of R into Python and is very fast. Higher level abstraction tools.
Focus on the scientific communit not the differences b/w Python & Ruby.
So how can we get the flexibility of Pandas, speed of Numpy in Ruby. The inspiration was Ruby ActiveRecord API. ActiveRecord speaks SQL.
Getting to Ruby to run Python === Getting Python to run Ruby. Ruby => Data => Python*
Lessons from LISP: immutibilty is important (ActiveRecord is doing it; you can't change the previous scopes) & S expressions
- S-Expressions e.g: (function arg1 arg2 arg3.....), (+ 1 1) => 2, (+ 1(* 2 2)) => 5 etc.
- Limitations: pmap pythons parallel map which Ruby doesn't support.
Added benefits: Compilers do optimization
- tree rewrites
- target multiple backends (like python)

mruby Debugger

A lot of Ruby developers are not familiar with debugger. Targets of mruby - microcontrollers
Pros: support C & Ruby
Cons: need GDB not standalone, a bit complicated.
Whats a debugger?
- Control Execution: break point, step into etc
  - suspend execution, resume execution and show internal status
  - GDB: machine intervace protocol for machines not humans.
- Show Variables
- Show Stacktraces
How mrubyVM works: instruction pipeline fetch -> code_fetch_hook -> decode -> _ -> _
Bytecode

Elixir

Thread for some reason seems like an incorrect way to establish concurrency in Ruby as it is not the strongest feature of Ruby. Elixir is a functional and concurrent language.
Exilir is an improvement of Erlang.
Mix: allows you create Elixir programs quickly, allows you to create dependencies etc
Pipe Operator:
```
1..10
|> map()
|> filter()
```
Actor concurrency model:
- is a process - performs a specific task - can send and receive a message - responds to specific types of messages
Processes: the basic concurrency
Ackermann: concurrency
```
w1 = spawn(Ack, :loop, [])
```
Fault tolerance:
Supervisor Hierarchy: Elixir programmers don't mostly write defensive programs
Coordinator process
- tasks: execute in a seperate process
"There is a word for people not doing functional programming 5 years from now, its called maintenance programming." So get into functional programming.

Functional Programming with Haskell

All variables in Haskell are immutable. Some code:

f :: Int -> Int
f x = 2 * x

(put spaces between arguments)

quicksort :: Ord a => [a] -> [a]
quicksort [] = [] #pattern matching
quicksort (p:xs) = (quicksort lesser) ++ [p] ++ (quicksort greater)
  where
    lesser = filter (< p) xs #defining the context of leseser & greater
    greater = filter (>= p) xs

Haskell is also lazy language.... eg take 4 [0..] take first 4 elements from an infinite list
:t nameOfFunction returns the types this function takes
if you wrap an infix function into a parantheses it becomes a prefix function eg. (*) 3 5
Higher order function: is a function that receives a function as an argument .fold is almost the same thing as .inject in Ruby eg. foldl (+) 0 [1,2,3,4] #=> 10
Simulating Annealing: annealing is a physical and chemical process
hUnit - testing framework for Haskell
no side effects
parallelization
Ruby-processing: gem runs with jruby.

#Day 3

Benchmarking in Ruby

Terraformer: a geometric toolkit for dealing with geometry, geography. Convex Hall algorithm: convex's set of geo data. First algorithm that was implemented was Jarvis March: easy to read; however, some performance issues. Another algorithm is Monotone Chain which in comparison has better performance.

Most important part is to verify your assumptions such as state of your machine, the code, the algorithm you have chosen. You learn a lot about Ruby language it self besides about your own code.

Just use Benchmark IPS vs. Benchmark gem and try to use it all the time. Good for quick spot checks and expansive analysis.

Benchmark Togo (10 times longer than IPS) If the input data you are trying to test has a range of sizes i.e you can grow or shrink or you can invision it into a chart.

It can; however, take some time. Writing Fast Ruby by Erik Michaels-Ober (check it out) + Fast Ruby repo. For Rails check out Derailed Benchmark which has various analysis tools including RAM size usage.

Ruby performan secrets and how to uncover them

https://www.airpair.com/ruby-on-rails/performance

What can go wrong when your code is slow 'sometimes'? It smells like GC

Lets profile memory allocations: Ruby by default will not give you information of memory allocation etc. --patch railsexpress have memory profile patches. gem install ruby-prof

brew install qcahcegrid visual profiling.

```ruby-prof -p call_tree --mode=allocations .. > ...``

Example #1 while loops vs. inject of eg, 10,000 array allocations: while will allocate 10,000 vs. inject will allocate 30,000. Why? Inject creates an object memo. After running it inside GDB r -e '[1,2,3].inject() after putting a breakpoint in inject source code. So the second object was converting the block into the function so 2 T_NODE objects per inject call. You can easily pass the GC threshold!! Eek.

Lesson learned:

Use profile to understand why your code is slow
Use C debugger to understand Ruby behavior

Example 2 Everything with ! has better performance..supposedly. eg. gsub vs. gsub! did not save any memory, it just copied the memory.

Lesson learned:

Profile not only CPU, but Memory. Do not profile # of GC calls (no need to profile second order of things)

Number of memory allocation:

Iterator	Enumerable	Array	Range
all?	3	3	3
any>	2	2	2
collect	0	1	1
each	0	0	0
each_index	0	-	-
each_with_index	2	2	2
find	2	2	2
find_all	1	1	1
inject	2	2	2
map	0	1	1
select	0	1	0

Real World Ruby Performance at Scale

Therapy Steps:

Its my fault.
Diagnosing the problem: metrics, measurements, mmmnumbers, milliseconds matter!
Treatement. Optimizing something vertically (different application layers) and horizontally (hardware)

Being fast means being staple (code and deployment process)

Nothing that runs locally will ever run the same in production. Check out StackProf + StackProf-remote which can be run on production (not limitlessly but decent) and you can profile your live application.

If you can't scale you stuff out due to concurrency then buy new hardware?? Don't be cheap :p

Starting with a hitlist: eg. take numbers from graphite every 2 hours for a given controller action * 90th percentile response time = Total (hypethotical) Time. Stackprof flamegraphs.

Ruby core team is now focusing on scalability tools considering now a lot of large applications are running on Ruby.

Q&A with Matz

Q. What will be removed from Ruby 3.0? A. Getting rid of pluralism but not sure yet.

Q. Thread absctraction in 3.0? A. Has not been concluded yet

Q. What do you think of Gil? A. Gil is there to protect you but a lot of people complained about it. You will have the ruby interpreter without Gil and you will see the difference e,g File I/O. We need a higher level of abstraction.

Q. will Ruby 3.0 have macros? A. No and no way.

Q. How do you use Github on your own machine? A. StackGit like Quail.

rafiamafia/ruby_conf_2014.md