producer consumer in go explained

producer-consumer-go.md

Producer consumer pattern

Question: Can you write code implementing the consumer and producer pattern?

This is a classic concurrency problem where we have threads generating data to be consumed (producers) by other threads (consumers).

The implementation with POSIX threads can be a pain in the ass but it is quite straight forward in golang thanks to its concurrency constructs.

Here is some code, heavily commeted for those coming from other programming languages.

package main // Tell go runtime your main lives here

import ("fmt") // We need the fmt package to print to the stdout

/*
These next two lines create two new channels, which are 
one of the concurrency constructs golang offers.
The first channel is a boolean channel the second one is
an int channel. We can read or write data to the channels.
We will see how shortly.
*/
var done = make(chan bool)
var msgs = make(chan int)

/*
In our main function, we spawn two go routines. go routines
are cheap functions that will be run concurrently by go runtime. 
They are just regular functions, but using the keyword go before
a function call we can run them as go routines.
So those two functions will run concurrently.
Finally we block the main thread by reading from the done channel.
As soon something comes down the channel, we read it, toss it and
the main thread continues executing. In this case we exit the 
program.
*/
func main () {
   go produce()
   go consume()
   <- done
}

/*
The produce go routine (or function) loops 10 times
and writes an integer (0..10) in the msg channel.
Notice that we will block until someone (the consumer)
reads form the other side of the channel. 
Once we are done, we send a boolean on the done channel
to let the main go routine that we are done.
*/
func produce() {
    for i := 0; i < 10; i++ {
        msgs <- i
    }
    done <- true
}

/*
The consume go routine loops infinitely and reads 
on the msgs channel. It will block until something 
comes in the channel. 
The syntax can be a little bit strange for people 
coming from other languages.
':=' creates a variable assigning it the type of the value
coming on the right of the assignation. An int in this 
case.
'<-' is the go way to read from a channel.
Once we have the msg (int) we dump it in the stdout.
*/
func consume() {
    for {
      msg := <-msgs
      fmt.Println(msg)
   }
}

I tested it with multiple producers, and looks like it is having some concurrency problems. here is my main:
func main () {
go produce()
go produce()
go consume()
<- done
}
I expected that consumer would print out 20 integers in random order, instead I got this:
$/cmd/channels$ ./channels
0
1
0
2
3
4
5
6
7
8
The number of printed messages varies from 10 to 20.

Yup... It demonstrates a producer-consumer pattern that is imperfect because not all of the produced items will be consumed before the application terminates. The issue is that not all of the produced items will be consumed because main() is waiting for a signal on the done channel, which producer sends when it is done. Producer will be expected to terminate sooner than the consumer, so all of the items may not be consumed. So, it is expected that each execution will have different consumption results. Hope this is helpful...

Here is my solution, i implement pipeline and unbuffered channel so it will block producer to create data until consumer consume its data first and at the same time it will block consumer to consume data until there is data from producer:

package main

import (
	"fmt"
	"testing"
)

func Producer() <-chan int {
	output := make(chan int)

	go func() {
		for i := 1; i <= 10 ; i++ {
			fmt.Println("Producting data ke-",i)
			output <- i
		}
		close(output)
	}()
	return output
}

func Consumer(in <-chan int) <-chan int {
	output := make(chan int)
	go func() {
		for n := range in {
			output <- n
		}
		close(output)
	}()
	return output
}

func Test_producerConsumer(t *testing.T)  {
	dataChan := Producer()
	dataConsume := Consumer(dataChan)

	for n := range dataConsume {
		fmt.Println("Consume data ke-",n)
	}
}