Ultimate Go Programming

Based on this Video

gotraining/README.md at master · ardanlabs/gotraining · GitHub

Design Guidelines

Prepare your mind

Don’t get impress by programs with a large number of LoC. this is a big problem.
Don’t use large layers of abstraction, we don’t need them to decouple.
The hardware is the platform. Many programming languages are based on VMs but Go is based on a real machine.
Every decision you make comes with a cost. There is not good or bad.
Constantly self-review your code and your learning
Adopting new ways of thinking is a lot slower than adopting a new technology.

Performance vs Productivity

If performance is your #1 priority you should use C or Assembly, however, developer productivity is more important in the long run. You lose performance in order to be more productive. So the question is “is it fast enough?” The hardware industry is not going to give us more CPU cores if we, the software side, are not going to use them. We can write programs in Go that take advantage of all CPU cores in a machine.

Correctness vs Performance

We should optimize for correctness not performance. Once the program is correct then improve performance.

"The correctness of the implementation is the most important concern, but there is no royal road to correctness. It involves diverse tasks such as thinking of invariants, testing and code reviews. Optimization should be done, but not prematurely." - Al Aho (inventor of AWK)

Language Mechanics

Variables

Type is life. They tell us 2 things: what is their memory consumption and what data they represent. We should use int instead of int32 or int64 in order to get mechanical sympathy. what is the size of int? In 64 bit architectures, we use 64 bits addresses, the WORDSIZE in go will match the size of our addresses then the int is 64 bits.

Zero value is an important concept, all variables must be initialized by the developer or Go will do that, Go will initialized variables to their zero value which depends on their data type. Zero value doesn’t mean null or nil, it means it will set all its bits to zero, memory is allocated. e.g 0, “” or false. This is for the sake of Integrity.

// Declare variables that are set to their zero value
var a int
var b string
var c float64
var d bool
// Declare and initialize variables
aa := 10
bb := "Hello"
cc := 2.332
dd := true
// Go use data coversion over casting
aaa := int32(10)

Strings in Go are a 2-word size data structure. The first is a pointer to an array of bytes and the other is the length of bytes of the current value.
In Go, you don’t create objects or instances, you create values!
In Go, data conversions seems like we are making a function call. e.g. string(value), int32(value).
In Go, we prefer short variable names over long variable names. The farther away a variable is being declared from where it’s being used, the longer the name it needs to be, because the context is missing.

Struct Types

The following type requires 7 bytes, however, in Go, this struct uses 8 bytes. There is one byte Go uses for alignment. It’s more efficient for the hardware to read memory using these alignment boundaries. Since the int16 is a 2-bytes values it should fall in a 2 bytes boundary, which means, it should start in the byte # 2, 4, 6, 8, etc. In this case, the int16 starts in the second byte so the padding is just 1 byte. If it were a 4 bytes value it should fall in a 4 bytes boundary, which means, it should start in the byte # 4, 8, 12 which will make a padding of 3 bytes. The greater the padding then more memory would be allocated to a value of this struct.

Due to the alignment, it’s recommended you list the fields that will use more memory first so the padding is at the end. The alignment boundary is dictated by the larger field, in this case, float32, 4 bytes.

type example struct {
	flag		bool
	counter	int16
	pi			float32
}

// Declare a variable of type example and set it to its zero value
var e1 example
e2 := example {
				flag: true,
				counetr: 10,
				pi: 3.1213,
}
fmt.Println("Flag:", e2.flag)

Pointers

The main goal of pointers is to share value across program boundaries, the most common is between function calls. Every Goroutine (you can think of this as a thread) in Go has its own stack (memory), it starts at 2k (it could grow and when this happens the stack is copied over to a larger one). In most OSs a thread is giving 1 Mb of memory. Each function is giving a stack frame (piece of the stack) to work with and execute its code. If at compile time the compiler knows about the size of memory required for the function, the size of the stack frame, then it goes to the stack otherwise it will go to the heap. Escape analysis is the only one that can determine if a value stays in the function’s stack frame or it’s moved to the heap where is shared (this requires the garbage collector to free that memory). The Garbage collector in Go is managed by the pacing algorithm which determines when the GC needs to run to keep the heap within the desired size. When the GC goroutines are running, reducing the size of the heap, all other goroutines will take a performance hit of 25%. A goroutine stack cannot have a pointer to another goroutine’s stack. The stop the world to reduce heap is reported to be about 100 microseconds, almost irrelevant, starting in Go 1.8.

count := 10;
// count is the value, &count is the memory address where that value is stored in memory
fmt.Println("count:", count, &count)
increment(count)
incrementByRef(&count)

function increment(inc int) {
	inc++
	fmt.Println("inc:", inc, &inc)
}

function incrementByRef(inc *int) {
	*inc++
	fmt.Println("inc:", *inc, &inc, inc)
}

Values in Go are always pass by value. The * operator declares a pointer variable (*int the type is important because it defines what values are going to be stored in the memory address of the pointer so we can read/write values properly, keeping integrity) and the "Value that the pointer points to" (*inc). Start with value semantics and then, if you need to share the value, move to pointer semantics.

Constants

Constants in Go have their of type system. They only exists at compile time.

// untyped constants, they could be more precise when dealing with numbers. Go converts untyped constants in order to do arithmetic opertaions with them (implicit conversions) 
const ui = 12345
const uf = 3.141592
// typed constants - precision is restricted. Conversion must be explicit.
const ti int = 12345
const tf float64 = 3.141592

Functions

Functions in Go can return more than one value. Most of the time, the second value is an Error, if your function returns more than 2 values that could be a code smell. The _, read blank identifier, identifier serves for those cases where you don’t need to use a value returned from a function and you don’t want to declare a variable. If you declare a variable, Go will make you use it.

_, error := updateUser(user)
if error != nil {
	// do things with the negative path
	...
}
// continue doing things in the happy path
...

In functions, there is always the happy path and negative path. The negative path is managed within if statement.

Data Structures

Data-oriented design

Performance comes from the hardware. If performance matters then mechanical sympathy for how the hardware and operating system work should be important. We should write code that minimizes cache misses (L1 private,L2 private,L3 shared) and avoid going to main memory. Cores do not access main memory directly. They tend to only have access their local caches. Main memory is built on relatively fast cheap memory. Caches are built on very fast expensive memory. The prefetches job is to predict what data is going to be needed and move it from main memory to one of the 3 caches before is needed. It requires predictable access patterns to data. How? by allocating contiguous memory, what data structure allow us to do that? The Array!

If you don't understand the data, you don't understand the problem.
Data transformations are at the heart of solving problems. Each function, method and work-flow must focus on implementing the specific data transformations required to solve the problems.
Changing data layouts can yield more significant performance improvements than changing just the algorithms.
Efficiency is obtained through algorithms but performance is obtained through data structures and layouts.

Arrays

Hardware loves Arrays! Arrays are a special data structure in Go that allow us to allocate contiguous blocks of fixed size memory.

Arrays are fixed length data structures that can't change.
Arrays of different sizes are considered to be of different types since An array’s length is part of its type.

// Declare string arrays to hold names.
var names [5]string
// Declare an array pre-populated with friend's names.
friends := [5]string{"Joe", "Ed", "Jim", "Erick", "Bill"}
// Assign the array of friends to the names array.
names = friends
// Display each string value and address index in names.
for i, name := range names {
	fmt.Println(name, &names[i])
}

Slices

It’s the most important data structure in Go. It is incredibly important for all Go programmers to learn how to uses slices.

Slices are like dynamic arrays with special and built-in functionality. e.g slice => []string vs array => [5]string
Slices is a 3-word data structure: pointer to the Array behind the slice, length(number of elements I can read/write) and capacity (total number of elements of the backend array I can use later).
There is a difference between a slices length (what we have access to) and capacity and they each serve a purpose.
Slices allow for multiple "views" of the same underlying array taking an slice of an existing slice. since both slices are sharing the backend array, any change you make to the backend array using any of those 2 slices is going to be available in both of them.
Slices can grow through the use of the built-in function append. When the length and the capacity of the slice are the same, the append function creates a backend array of 2x its current length (after 1024 elements, it just grow 25% of its current size), capacity is now 2x but length is just length+1. When capacity is greater than length, the functions just add a value to the backend array and updates the slice’s length property.

// Create a slice with a length of 5 elements.
slice := make([]string, 5)
slice[0] = "Apple"
slice[1] = "Orange"
// Create a slice with a length of 5 elements and a capacity of 8.
slice := make([]string, 5, 8)
// Slice length and capacity built-in functions
fmt.Printf("Length[%d] Capacity[%d]\n", len(slice), cap(slice))
// Declare a nil slice of strings.
var data []string
// Use the built-in function append to add elements to the slice.
data = append(data, fmt.Sprintf("Rec: %d", record))
// Take a slice of slice1. We want just indexes 2 and 3.
// Parameters are [starting_index : (starting_index + length)]
slice2 := slice1[2:4]
// Using the value semantic form of the for range.
five := []string{"Annie", "Betty", "Charley", "Doug", "Edward"}
for _, v := range five {
	five = five[:2]
	fmt.Printf("v[%s]\n", v)
}
// Using the pointer semantic form of the for range.
five = []string{"Annie", "Betty", "Charley", "Doug", "Edward"}
for i := range five {
	five = five[:2]
	fmt.Printf("v[%s]\n", five[i])
}

The make built-in function only works with slices, maps and channels.

Maps

Maps are key-value pair data structures. e.g used are in-memory cache.

type user struct {
	firstname string
	lastname string
}
// Declare and make a map that stores values of type user with a key of type string
users := make(map[string]user)
// Add key/value pairs to the map
users["roy"] = user{"Rob", "Roy"}
users["lucia"] = user{"Lucia", "Santana"}
// Iterate over the map. Order is random.
for key, value := range users {
	fmt.Println(key, value)
}
// Delete a map key
delete(users, "roy")
// Find a key
u, found := users["roy"]

Decoupling

Methods

Using function receivers you can add methods to custom types/structs in Go. It’s a way for data to expose a capability.

Value semantics. Used with built-in types. If you’re using reference types except when you need to pass an slice to an un-marshal function. We operate in our own copy of the data.
Pointer semantics. If you’re not sure you can default to pointer mechanics. We share data access. Only concrete types can have methods. Only user-defined types can have methods.

// user defines a user in the program.
type user struct {
	name  string
	email string
}
// notify implements a method with a value receiver.
func (u user) notify() {
	fmt.Printf("Sending User Email To %s<%s>\n",
		u.name,
		u.email)
}
// changeEmail implements a method with a pointer receiver.
func (u *user) changeEmail(email string) {
	u.email = email
}

// Values of type user can be used to call methods
// declared with both value and pointer receivers.
bill := user{"Bill", "[email protected]"}
bill.notify()
// this is what Go is doing underneath
user.notify(bill)
bill.changeEmail("[email protected]")
// this is what Go is doing underneath
(*user).changeEmail(&bill)

Interfaces

Interfaces provide a way to declare types that define only behavior. This behavior can be implemented by concrete types, such as struct or named types, via methods. When a concrete type implements the set of methods for an interface, values of the concrete type can be assigned to variables of the interface type. Then method calls against the interface value actually call into the equivalent method of the concrete value. Since any concrete type can implement any interface, method calls against an interface value are polymorphic in nature. * Interfaces are reference types, don't share with a pointer. * Interfaces are a 2-word size data structure: the first stores the type of data that implements the interface and the second is a copy of the value so it can call the concrete implementations of the behavior declared in the interface. The concrete type needs to implement all the functionality declared in the interface type in order to be in compliant with the interface. * The method set for a value, only includes methods implemented with a value receiver. * The method set for a pointer, includes methods implemented with both pointer and value receivers. * Methods declared with a pointer receiver, only implement the interface with pointer values. * Methods declared with a value receiver, implement the interface with both a value and pointer receiver.

// reader is an interface that defines the act of reading data.
type reader interface {
	read(b []byte) (int, error)
}

// file defines a system file.
type file struct {
	name string
}

// read implements the reader interface for a file.
func (file) read(b []byte) (int, error) {
	s := "<rss><channel><title>Going Go Programming</title></channel></rss>"
	copy(b, s)
	return len(s), nil
}

// retrieve can read any device and process the data.
// This function aceppts any value/pointers that implements the reader interface
func retrieve(r reader) error {
	data := make([]byte, 100)

	len, err := r.read(data)
	if err != nil {
		return err
	}

	fmt.Println(string(data[:len]))
	return nil
}

// Call the retrieve funcion using the concrete type.
f := file{"data.json"}
retrieve(f)

Embedding

Through the use of inner type promotion, an inner type's fields and methods can be directly accessed by references of the outer type

Embedding types allows us to share state or behavior between types.
This is not inheritance.
The outer type can override the inner type's behavior.

// user defines a user in the program.
type user struct {
	name  string
	email string
}

// notify implements a method notifies users
// of different events.
func (u *user) notify() {
	fmt.Printf("Sending user email To %s<%s>\n",
		u.name,
		u.email)
}

// adminTest represents an admin user with privileges.
type adminTest struct {
	person user // NOT Embedding
	level  string
}

// admin represents an admin user with privileges.
type admin struct {
	user  // Embedded Type
	level string
}

func main() {

	// Create an admin user.
	ad := admin{
		user: user{
			name:  "john smith",
			email: "[email protected]",
		},
		level: "super",
	}

	// We can access the inner type's method directly.
	ad.user.notify()

	// The inner type's method is promoted.
	ad.notify()
}

Exporting

Packages are self-contained units of code. Packages contain the basic unit of compiled code. They define a scope for the identifiers that are declared within them. Exporting is not the same as public and private semantics in other languages. But exporting is how we provide encapsulation in Go.

Code in go is compiled into packages and then linked together.
Identifiers are exported (or remain unexported) based on letter-case. If their names start with a Capital letter they are exported. If their names start with lowercase then they are not exported. Exported identifiers can be used from outside the package.
We import packages to access exported identifiers.
Packages are imported in others using their relative path to the GOPATH variable.

Composition

Grouping Types

Go promotes grouping by behavior and not by state. Grouping by what you can do and not what/who you are.

Don't group types by a common DNA but by a common behavior.
Everyone can work together when we focus when we do and not what we are.
Declare types that represent something new or unique.
Question types whose sole purpose is to share common state.
Validate that a value of any type is created or used on its own.
Embed types to reuse existing behaviors you need to satisfy.
Interfaces provide the highest form of composition.

Decoupling

Decoupling means reducing the dependencies between components and the types they use. How do we know we’re done with a piece of code?

80% of code coverage, at least the happy path.
Solve one problem at a time!
Is it fast enough? if so, you’re done. No need to worry about performance.
Ship to production so it solves the problem it’s meant to solve!
Start with concrete logic then work your way up to decouple using interfaces.
What changes are coming from the biz, the tech stack that I should start planning for? refactoring! Refactoring should be constant!

Conversion and Assertions

Interfaces in Go can be converted to other interfaces as long as the interface value implements all the methods of the target interface.

// Mover provides support for moving things.
type Mover interface {
	Move()
}

// Locker provides support for locking and unlocking things.
type Locker interface {
	Lock()
	Unlock()
}

// MoveLocker provides support for moving and locking things.
type MoveLocker interface {
	Mover
	Locker
}

In the example above, a value of interface type MoveLocker can be converted to interface Mover but no the other way around.

Interface Pollution

Use an interface:

When users of the API need to provide an implementation detail.
When API’s have multiple implementations that need to be maintained.
When parts of the API that can change have been identified and require decoupling.

Question an interface:

When its only purpose is for writing testable API’s (write usable API’s first).
When it’s not providing support for the API to decouple from change.
When it's not clear how the interface makes the code better.

Mocking

When doing tests for your Go code, you can define interfaces for the functionality you want to mock, then create a mock struct that satisfy that interface.

Design Guidelines

Interfaces encourage design by composition.

Error Handling

Error handling is critical for making your programs reliable, trustworthy and respectful to those who depend on them. A proper error value is both specific and informative. It must allow the caller to make an informed decision about the error that has occurred. There are several ways in Go to create error values. This depends on the amount of context that needs to be provided.

Use the default error value for static and simple formatted messages.
Create and return error variables to help the caller identify specific errors.
Create custom error types when the context of the error is more complex.
Error Values in Go aren't special, they are just values like any other, and so you have the entire language at your disposal.

// http://golang.org/pkg/builtin/#error
type error interface {
	Error() string
}
// http://golang.org/src/pkg/errors/errors.go
type errorString struct {
	s string
}
// http://golang.org/src/pkg/errors/errors.go
func (e *errorString) Error() string {
	return e.s
}
// http://golang.org/src/pkg/errors/errors.go
// New returns an error that formats as the given text.
func New(text string) error {
	return &errorString{text}
}
func main() {
	if err := webCall(); err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println("Life is good")
}
// webCall performs a web operation.
func webCall() error {
	return New("Bad Request")
}

Using error variables and returning multiple error values from a function. The client needs to be able to make decisions based on the type of error received.

var (
	// ErrBadRequest is returned when there are problems with the request.
	ErrBadRequest = errors.New("Bad Request")

	// ErrMovedPermanently is returned when a 301/302 is returned.
	ErrMovedPermanently = errors.New("Moved Permanently")
)

// webCall performs a web operation.
func webCall(b bool) error {
	if b {
		return ErrBadRequest
	}
	return ErrMovedPermanently
}

func main() {
	if err := webCall(true); err != nil {
		switch err {
		case ErrBadRequest:
			fmt.Println("Bad Request Occurred")
			return

		case ErrMovedPermanently:
			fmt.Println("The URL moved, check it again")
			return

		default:
			fmt.Println(err)
			return
		}
	}

	fmt.Println("Life is good")
}

When error variables are not enough, probably because you need to add more information about the error itself (more context), then there is the option to use custom concrete types for the error. Use this only when you really need it. Here is an example of a custom error type: src/encoding/json/decode.go - The Go Programming Language When using custom concrete types for the error, you should ALWAYS use the error interface as the return type from your functions.

// An UnmarshalTypeError describes a JSON value that was
// not appropriate for a value of a specific Go type.
type UnmarshalTypeError struct {
	Value string       // description of JSON value
	Type  reflect.Type // type of Go value it could not be assigned to
}
// Error implements the error interface.
func (e *UnmarshalTypeError) Error() string {
	return "json: cannot unmarshal " + e.Value + " into Go value of type " + e.Type.String()
}
// Unmarshal simulates an unmarshal call that always fails.
func Unmarshal(data []byte, v interface{}) error {
	rv := reflect.ValueOf(v)
	if rv.Kind() != reflect.Ptr || rv.IsNil() {
		return &InvalidUnmarshalError{reflect.TypeOf(v)}
	}
	return &UnmarshalTypeError{"string", reflect.TypeOf(v)}
}

func main() {
	var u user
	err := Unmarshal([]byte(`{"name":"bill"}`), u) // Run with a value and pointer.
	if err != nil {
		switch e := err.(type) {
		case *UnmarshalTypeError:
			fmt.Printf("UnmarshalTypeError: Value[%s] Type[%v]\n", e.Value, e.Type)
		case *InvalidUnmarshalError:
			fmt.Printf("InvalidUnmarshalError: Type[%v]\n", e.Type)
		default:
			fmt.Println(err)
		}
		return
	}

	fmt.Println("Name:", u.Name)
}

You can also declare interface types that group errors by functionality and deal with them in a more general, less concrete way.

We only need to log those things that are actionable by a person and use for debugging. Some information we’re putting into our logs could be better suited for a metrics dashboard. When dealing with errors you have 2 options:

You deal with the error and log it
You just pass the error up the stack, in this case, you add more information to the error by wrapping it. The Go Playground

import (
	"fmt"
	"github.com/pkg/errors"
)
// AppError represents a custom error type.
type AppError struct {
	State int
}
// Error implements the error interface.
func (c *AppError) Error() string {
	return fmt.Sprintf("App Error, State: %d", c.State)
}

func main() {
	// Make the function call and validate the error.
	if err := firstCall(10); err != nil {
		// Use type as context to determine cause.
		switch v := errors.Cause(err).(type) {
		case *AppError:
			// We got our custom error type.
			fmt.Println("Custom App Error:", v.State)
		default:
			// We did not get any specific error type.
			fmt.Println("Default Error")
		}
		// Display the stack trace for the error.
		fmt.Println("\nStack Trace\n********************************")
		fmt.Printf("%+v", err)
	}
}
// firstCall makes a call to a second function and wraps any error.
func firstCall(i int) error {
	if err := secondCall(i); err != nil {
		return errors.Wrapf(err, "secondCall: %d", i)
	}
	return nil
}
// secondCall makes a call to a third function and wraps any error.
func secondCall(i int) error {
	if err := thirdCall(); err != nil {
		return errors.Wrapf(err, "thirdCall")
	}
	return nil
}
// thirdCall create an error value we will validate.
func thirdCall() error {
	return &AppError{99}
}

Packaging

Package Oriented Design Package Oriented Design allows a developer to identify where a package belongs inside a Go project and the design guidelines the package must respect. It defines what a Go project is and how a Go project is structured. Finally, it improves communication between team members and promotes clean package design and project architecture that is discussable.

Language Mechanics

Packaging directly conflicts with how we have been taught to organize source code in other languages. Each folder contains a package (several files) and the compiler will create a static library per folder, independently of each other.
In other languages, packaging is a feature that you can choose to use or ignore.
You can think of packaging as applying the idea of microservices on a source tree. Decoupled design.
All packages are "first class," and the only hierarchy is what you define in the source tree for your project. There is not sub-packages.
There needs to be a way to “open” parts of the package to the outside world. Exporting => open up the firewall to be able to access identifiers/functionality within the package.
Two packages can’t cross-import each other. Imports are a one way street. e.g Package A imports package B but package B cannot import package A ta the same time.

Design Guidelines

To be purposeful, packages must provide, not contain.

Packages must be named with the intent to describe what it provides. e.g http, io, json
Packages must not become a dumping ground of disparate concerns. e.g common, utils

To be usable, packages must be designed with the user as their focus. Packages must be intuitive and simple to use. e.g readability, flexibility

Packages must respect their impact on resources and performance.
Packages must protect the user’s application from cascading changes.
Packages must prevent the need for type assertions to the concrete.
Packages must reduce, minimize and simplify its code base.

To be portable, packages must be designed with reusability in mind.

Packages must aspire for the highest level of portability.
Packages must reduce setting policy when it’s reasonable and practical.
Packages must not become a single point of dependency.

Package-Oriented Design

Recommended project structure for Go, there are 2 types of projects:

Kit projects. Standard library for you or your company, single repo. All packages that are reusable across all companies (Go Vendor tool). Packages that provide foundational support for the different Application projects that exist. e.g logging, configuration or web functionality
Application projects. Single repo for all your binaries that should be deployed together. cmd is where all the binaries of the project live, one folder per binary, each folder could have its own sub-folder structure, all packages under this folder have one purpose and that is building the binaries (start, shutdown and configuration). Cannot import package from other binary folder. Use the letter d at the end of a program folder to denote it as a daemon. internal is for packages that are reusable in multiple binaries that belong to the same application project in the cmd folder. Packages in this folder cannot import packages at the same level, it’s preferred that you have multiple declarations of the same custom types than couple packages together so you can use a type definition. You import packages by the functionality they provide not by what they contain. No package outside of this project can import packages from inside of internal. e.g. CRUD, services or business logic. platform here is the place for foundational packages within a project. These would be packages that provide support for things like databases, authentication or even marshaling. vendor this is where all third-party packages are stored, including your kit project’s packages.

Kit                     Application

├── CONTRIBUTORS        ├── cmd/
├── LICENSE             ├── internal/
├── README.md           │   └── platform/
├── cfg/                └── vendor/
├── examples/
├── log/
├── pool/
├── tcp/
├── timezone/
├── udp/
└── web/

Goroutines

Goroutines (path of executions) are methods/functions that are created and scheduled to be run independently by the Go scheduler. The Go scheduler is responsible for the management and execution of goroutines. * We must always maintain an account of running goroutines and shutdown cleanly. * Concurrency is not parallelism. - Concurrency is about dealing with lots of things at once. - Parallelism is about doing lots of things at once. The Go scheduler have 4 opportunities to make a scheduling decision to rebalance load among all its logical processors (global run queue or local run queues): * When it finds the keyword go which creates a new goroutine * when the program makes a system call * when executes a channel operation * garbage collection (GC). it always force itself to run at some point. Take into account that multiple goroutines are running at the same time and if we’re not careful we can create chaos in the form of data race errors. This could put integrity at risk. * Data race errors. multiple goroutines one trying to read and the other trying to write to the same memory location.

How the scheduler works

[image:26B63229-3F90-4216-A099-43482F999ACF-8499-000262E4FAAA0C4C/3666D267-EF28-47AA-BD32-59C89CAC15F0.png]

Difference between concurrency and parallelism

[image:8F73D8D1-007B-4878-ADC7-8D29ED273C17-8499-000262E9509216B3/26506A4F-CA22-486C-83CE-F0CA018DBDF1.png]

Language Mechanics

You can manage how many logical processors your Go application can use, by default, Go uses all cores from version 1.5 up.

import (
	"fmt"
	"runtime"
	"sync"
)

func init() {
	// Allocate one logical processor for the scheduler to use.
	runtime.GOMAXPROCS(1)
}

Everytime you create a goroutine using the keyword go you need to know when and how this new goroutine is going to terminate, this is how you manage concurrency. Otherwise, the main goroutine will terminate while there are other goroutines running which will kill your application. We should also start goroutines cleanly and shutdown them cleanly.

package main

import (
	"fmt"
	"runtime"
	"sync"
)

func init() {
	// Allocate one logical processor for the scheduler to use.
	runtime.GOMAXPROCS(1)
}
func main() {
	// wg is used to manage concurrency.
	var wg sync.WaitGroup
	wg.Add(2)
	fmt.Println("Start Goroutines")
	// Create a goroutine from the lowercase function.
	go func() {
		lowercase()
		wg.Done()
	}()
	// Create a goroutine from the uppercase function.
	go func() {
		uppercase()
		wg.Done()
	}()
	// Wait for the goroutines to finish.
	fmt.Println("Waiting To Finish")
	wg.Wait()
	fmt.Println("\nTerminating Program")
}
// lowercase displays the set of lowercase letters three times.
func lowercase() {
	// Display the alphabet three times
	for count := 0; count < 3; count++ {
		for rune := 'a'; rune < 'a'+26; rune++ {
			fmt.Printf("%c ", rune)
		}
	}
}
// uppercase displays the set of uppercase letters three times.
func uppercase() {
	// Display the alphabet three times
	for count := 0; count < 3; count++ {
		for rune := 'A'; rune < 'A'+26; rune++ {
			fmt.Printf("%c ", rune)
		}
	}
}

Less is more

You have to know that adding more goroutines to your application it doesn’t necessarily mean your application will run faster. Concurrency adds complexity and you need to use it carefully. Ideally, you should try to use as many goroutines as you have cores in your machine. Adding more goroutines than you have CPU cores will add more load to the Go scheduler. Optimize for correctness first, benchmark the solution and if it is not fast enough then and only then try to make it faster by adding the complexity that concurrency provides.

Data Races

A data race is when two or more goroutines attempt to read and write to the same resource at the same time. Race conditions can create bugs that appear totally random or can never surface as they corrupt data. Atomic functions and mutexes are a way to synchronize the access of shared resources between goroutines.

[image:1916E446-2847-443F-9C8A-8FF5228CBFA7-8499-000284EE024345CC/C338771E-48FC-4629-8D91-D08A090CA8E9.png]

Atomic Functions

Atomic functions were a way to orchestrate read/write access to share memory among multiple goroutines. Channels make this a lot easier. Atomic functions are for managing 4 to 8 bytes of memory. If your shared state is more than 8 bytes of memory, you should use mutexes or channels.

Mutexes

Mutexes are used to define a critical section of code.

package main

import (
	"fmt"
	"sync"
)
// counter is a variable incremented by all goroutines.
var counter int
// mutex is used to define a critical section of code.
var mutex sync.Mutex
func main() {
	// Number of goroutines to use.
	const grs = 2
	// wg is used to manage concurrency.
	var wg sync.WaitGroup
	wg.Add(grs)
	// Create two goroutines.
	for i := 0; i < grs; i++ {
		go func() {
			for count := 0; count < 2; count++ {
				// Only allow one goroutine through this critical section at a time.
				mutex.Lock()
				{
					// Capture the value of counter.
					value := counter
					// Increment our local value of counter.
					value++
					// Store the value back into counter.
					counter = value
				}
				mutex.Unlock()
				// Release the lock and allow any waiting goroutine through.
			}
			wg.Done()
		}()
	}
	// Wait for the goroutines to finish.
	wg.Wait()
	fmt.Printf("Final Counter: %d\n", counter)
}

Channels

Channel Design Channels allow goroutines to communicate with each other through the use of signaling semantics. Channels accomplish this signaling (with or without data) through the use of sending/receiving data or by identifying state changes on individual channels. Don't architect software with the idea of channels being a queue, focus on signaling and the semantics that simplify the orchestration required. * Buffer channels. We don’t get the guarantee that the signal was received. Reduced latency. The “Send” happens first. The send is not blocked if there are free spots in the channel, the capacity. If there are free spots, that means, there is another goroutine receiving the signals you’re sending. When the send is blocked, it means the other goroutine is probably having issues. * Unbuffer channels. Signaling with a guarantee that the signal was received. Higher latency. The send of data is blocked until the other goroutine receives the data. The “receive” part happens first.

Channels has 3 states: nil, open and closed. When a channel is open you can send/receives signals. You cannot close a channel twice, the system will panic. When you close a channel, all goroutines that were waiting on the channel will continue their work. Concurrent Software Design Channel mechanics - Sample Code

func waitForTask() {
	ch := make(chan string)

	go func() {
		p := <-ch
		fmt.Println("employee : recv'd signal :", p)
	}()

	time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)
	ch <- "paper"
	fmt.Println("manager : sent signal")

	time.Sleep(time.Second)
	fmt.Println("------------------------------------------------")
}

func fanOut() {
	grs := 20
	ch := make(chan string, grs)

	for g := 0; g < grs; g++ {
		go func(g int) {
			time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
			ch <- "paper"
			fmt.Println("employee : sent signal :", g)
		}(g)
	}

	for grs > 0 {
		p := <-ch
		fmt.Println(p)
		fmt.Println("manager : recv'd signal :", grs)
		grs--
	}

	time.Sleep(time.Second)
	fmt.Println("--------------------------------------------")
}

Concurrency Patterns

Context

The package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes. * Incoming requests to a server should create a Context. * Outgoing calls to servers should accept a Context. * The chain of function calls between them must propagate the Context. * Replace a Context using WithCancel, WithDeadline, WithTimeout, or WithValue. * When a Context is canceled, all Contexts derived from it are also canceled. * Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. * Do not pass a nil Context, even if a function permits it. Pass context.TODO if you are unsure about which Context to use. * Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions. * The same Context may be passed to functions running in different goroutines; Contexts are safe for simultaneous use by multiple goroutines.

package main

import (
	"context"
	"fmt"
)
// TraceID is represents the trace id.
type TraceID string
// TraceIDKey is the type of value to use for the key. The key is
// type specific and only values of the same type will match.
type TraceIDKey int

func main() {
	// Create a traceID for this request.
	traceID := TraceID("f47ac10b-58cc-0372-8567-0e02b2c3d479")
	// Declare a key with the value of zero of type userKey.
	const traceIDKey TraceIDKey = 0
	// Store the traceID value inside the context with a value of
	// zero for the key type.
	ctx := context.WithValue(context.Background(), traceIDKey, traceID)
	// Retrieve that traceID value from the Context value bag.
	if uuid, ok := ctx.Value(traceIDKey).(TraceID); ok {
		fmt.Println("TraceID:", uuid)
	}
	// Retrieve that traceID value from the Context value bag not
	// using the proper key type.
	if _, ok := ctx.Value(0).(TraceID); !ok {
		fmt.Println("TraceID Not Found")
	}
}

Context with cancel function

package main

import (
	"context"
	"fmt"
	"time"
)

func main() {
	// Create a context that is cancellable only manually.
	// The cancel function must be called regardless of the outcome.
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	// Ask the goroutine to do some work for us.
	go func() {
		// Wait for the work to finish. If it takes too long move on.
		select {
		case <-time.After(100 * time.Millisecond):
			fmt.Println("moving on")
		case <-ctx.Done():
			fmt.Println("work complete")
		}
	}()
	// Simulate work.
	time.Sleep(50 * time.Millisecond)
	// Report the work is done.
	cancel()
	// Just hold the program to see the output.
	time.Sleep(time.Second)
}

Context with deadline

package main

import (
	"context"
	"fmt"
	"time"
)

type data struct {
	UserID string
}

func main() {
	// Set a deadline.
	deadline := time.Now().Add(150 * time.Millisecond)
	// Create a context that is both manually cancellable and will signal a cancel at the specified date/time.
	ctx, cancel := context.WithDeadline(context.Background(), deadline)
	defer cancel()
	// Create a channel to received a signal that work is done.
	ch := make(chan data, 1)
	// Ask the goroutine to do some work for us.
	go func() {
		// Simulate work.
		time.Sleep(200 * time.Millisecond)
		// Report the work is done.
		ch <- data{"123"}
	}()
	// Wait for the work to finish. If it takes too long move on.
	select {
	case d := <-ch:
		fmt.Println("work complete", d)
	case <-ctx.Done():
		fmt.Println("work cancelled")
	}
}

Context with timeout

// Set a duration.
	duration := 150 * time.Millisecond
	// Create a context that is both manually cancellable and will signal a cancel at the specified duration.
	ctx, cancel := context.WithTimeout(context.Background(), duration)
	defer cancel()

Task

Sample code Using this pattern, you create a set of goroutines you later use to execute any given work. e.g a pool of workers waiting for tasks to be assigned.

Logger

Sample code using this pattern, you create a buffer channel with N number of slots, and when the buffer channel gets full, it means the logger is having issues writing data, the program will start getting blocked while sending data to the channel and that in turn will cause the message to be dropped to the floor. Useful when you want to overload your system when it’s having issues, you can only handle N number of request at any given time.

Testing

Testing is built right into the go tools and the standard library. Testing needs to be a vital part of the development process because it can save you a tremendous amount of time throughout the life cycle of the project. Benchmarking is also a very powerful tool tied to the testing functionality.

Basic Testing

All your testing files should end with _test.go for the Go testing tool to find them. * Write tests in tandem with development. * Example code serve as both a test and documentation. * It’s a good idea to group tests under his own package. e.g package_name_test * The functions we write to test our code should start with the word Test followed by a descriptive name (first letter is capitalized). e.g TestDownload * Testing code should be of the same quality as production code. * You run tests with go test The standard library provides a testing framework which should be enough for our needs. This is the core API: * Log and Logf. Log messages. * Fatal and Fatalf. Log error, fail test and continue with next function. * Error and Errorf. Log error and fail test.

Basic unit test

package example1

import (
	"net/http"
	"testing"
)

const succeed = "\u2713"
const failed = "\u2717"
// TestDownload validates the http Get function can download content.
func TestDownload(t *testing.T) {
	url := "https://www.goinggo.net/post/index.xml"
	statusCode := 200
	t.Log("Given the need to test downloading content.")
	{
		t.Logf("\tTest 0:\tWhen checking %q for status code %d", url, statusCode)
		{
			resp, err := http.Get(url)
			if err != nil {
				t.Fatalf("\t%s\tShould be able to make the Get call : %v", failed, err)
			}
			t.Logf("\t%s\tShould be able to make the Get call.", succeed)
			defer resp.Body.Close()
			if resp.StatusCode == statusCode {
				t.Logf("\t%s\tShould receive a %d status code.", succeed, statusCode)
			} else {
				t.Errorf("\t%s\tShould receive a %d status code : %d", failed, statusCode, resp.StatusCode)
			}
		}
	}
}

Table Unit Test

package example2

import (
	"net/http"
	"testing"
)

const succeed = "\u2713"
const failed = "\u2717"

// TestDownload validates the http Get function can download content and
// handles different status conditions properly.
func TestDownload(t *testing.T) {
	tests := []struct {
		url        string
		statusCode int
	}{
		{"https://www.goinggo.net/post/index.xml", http.StatusOK},
		{"http://rss.cnn.com/rss/cnn_topstorie.rss", http.StatusNotFound},
	}
	t.Log("Given the need to test downloading different content.")
	{
		for i, tt := range tests {
			t.Logf("\tTest: %d\tWhen checking %q for status code %d", i, tt.url, tt.statusCode)
			{
				resp, err := http.Get(tt.url)
				if err != nil {
					t.Fatalf("\t%s\tShould be able to make the Get call : %v", failed, err)
				}
				t.Logf("\t%s\tShould be able to make the Get call.", succeed)
				defer resp.Body.Close()
				if resp.StatusCode == tt.statusCode {
					t.Logf("\t%s\tShould receive a %d status code.", succeed, tt.statusCode)
				} else {
					t.Errorf("\t%s\tShould receive a %d status code : %v", failed, tt.statusCode, resp.StatusCode)
				}
			}
		}
	}
}

Web Testing

Mocking a web server response Testing internal endpoints

Example Test

Example tests serve 2 purposes: testing and documentation. If you start the program godoc in your machine, you’ll see the documentation for all the local packages you have under your GOPATH directory.

godoc -http :4000

* It’s a good idea to have a `doc.go` file where you add documentation for your package overview
* If in your `_test.go` you add a function that starts with `ExampleFunctionName` then its code will be displayed in the documentation of the function `FunctionName`.
* If you want to test your `Example` code, you need to add as comments what you expect as the result of the function. e.g See below

// Sample to show how to write a basic example.
package handlers_test

import (
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
)
// ExampleSendJSON provides a basic example example.
func ExampleSendJSON() {
	r := httptest.NewRequest("GET", "/sendjson", nil)
	w := httptest.NewRecorder()
	http.DefaultServeMux.ServeHTTP(w, r)
	var u struct {
		Name  string
		Email string
	}
	if err := json.NewDecoder(w.Body).Decode(&u); err != nil {
		log.Println("ERROR:", err)
	}
	fmt.Println(u)
	// Output:
	// {Bill [email protected]}
}

Sub Tests

Subtests are anonymous functions we create within the test functions. By doing this, we can run those tests in parallel or execute any given subtest specifically from the command line.

// TestDownload validates the http Get function can download content and handles different status conditions properly.
func TestDownload(t *testing.T) {
	tests := []struct {
		name       string
		url        string
		statusCode int
	}{
		{"statusok", "https://www.goinggo.net/post/index.xml", http.StatusOK},
		{"statusnotfound", "http://rss.cnn.com/rss/cnn_topstorie.rss", http.StatusNotFound},
	}
	t.Log("Given the need to test downloading different content.")
	{
		for i, tt := range tests {
			tf := func(t *testing.T) {
				t.Logf("\tTest: %d\tWhen checking %q for status code %d", i, tt.url, tt.statusCode)
				{
					resp, err := http.Get(tt.url)
					if err != nil {
						t.Fatalf("\t%s\tShould be able to make the Get call : %v", failed, err)
					}
					t.Logf("\t%s\tShould be able to make the Get call.", succeed)
					defer resp.Body.Close()
					if resp.StatusCode == tt.statusCode {
						t.Logf("\t%s\tShould receive a %d status code.", succeed, tt.statusCode)
					} else {
						t.Errorf("\t%s\tShould receive a %d status code : %v", failed, tt.statusCode, resp.StatusCode)
					}
				}
			}
			t.Run(tt.name, tf)
		}
	}
}

Test Coverage

Test coverage of 80% overall and 100% of the happy path. In the coverage profile report, lines in green are lines that were hit (covered) by our tests and lines in red the ones that don’t.

go test -cover
go test -coverprofile cover.out
go tool cover -html cover.out

Benchmarking

Benchmarks should be in a _test.go file and instead of having a TestSomething function, you have a BenchmarkSomething function. You should never guest about performance, just run the benchmarks and let Go tell you what option is more performant.

go test -run none -bench . -benchmem -benchtime 3s
go test -run none -bench Sprint/format

Basic Benchmarking

// Basic benchmark test.
package basic

import (
	"fmt"
	"testing"
)
var gs string
// BenchmarkSprint tests the performance of using Sprint.
func BenchmarkSprint(b *testing.B) {
	var s string
	for i := 0; i < b.N; i++ {
		s = fmt.Sprint("hello")
	}
	gs = s
}
// BenchmarkSprint tests the performance of using Sprintf.
func BenchmarkSprintf(b *testing.B) {
	var s string
	for i := 0; i < b.N; i++ {
		s = fmt.Sprintf("hello")
	}
	gs = s
}

Sub-benchmarks

go test -run none -bench . -benchtime 3s -benchmem
go test -run none -bench BenchmarkSprint/none -benchtime 3s -benchmem
go test -run none -bench BenchmarkSprint/format -benchtime 3s -benchmem

package basic

import (
	"fmt"
	"testing"
)
var gs string
// BenchmarkSprint tests all the Sprint related benchmarks as
// sub benchmarks.
func BenchmarkSprint(b *testing.B) {
	b.Run("none", benchSprint)
	b.Run("format", benchSprintf)
}
// benchSprint tests the performance of using Sprint.
func benchSprint(b *testing.B) {
	var s string
	for i := 0; i < b.N; i++ {
		s = fmt.Sprint("hello")
	}
	gs = s
}
// benchSprintf tests the performance of using Sprintf.
func benchSprintf(b *testing.B) {
	var s string
	for i := 0; i < b.N; i++ {
		s = fmt.Sprintf("hello")
	}
	gs = s
}

Fuzzing

Fuzzing Go-fuzz is a coverage-guided fuzzing solution for testing of Go packages. Fuzzing is mainly applicable to packages that parse complex inputs (both text and binary), and is especially useful for hardening of systems that parse inputs from potentially malicious users (e.g. anything accepted over a network). * Fuzzing allows you to find cases where your code panics. * Once you identify data inputs that causes panics, code can be corrected and tests created. * Table tests are an excellent choice for these input data panics.

First thing is to install the Go fuzz tooling:

go get github.com/dvyukov/go-fuzz/go-fuzz
go get github.com/dvyukov/go-fuzz/go-fuzz-build

Profiling

Profiling Go programs Seven ways to profile a Go program We can use the go tooling to inspect and profile our programs. Profiling is more of a journey and detective work. It requires some understanding about the application and expectations. The profiling data in and of itself is just raw numbers. We have to give it meaning and understanding.

How does a profiler work? A profiler runs your program and configures the operating system to interrupt it at regular intervals. This is done by sending SIGPROF to the program being profiled, which suspends and transfers execution to the profiler. The profiler then grabs the program counter for each executing thread and restarts the program.

Types of profiling: * CPU * Memory * Block. It records the amount of time a goroutine spent waiting for a shared resource.

Tools

hey

hey is a modern HTTP benchmarking tool capable of generating the load you need to run tests. It's built using the Go language and leverages goroutines for behind the scenes async IO and concurrency.

go get -u github.com/rakyll/hey

ngrok

ngrok: Introspected tunnels to localhost

Memory Tracing

Memory and CPU Profiling If you think your program has a memory leak, this is the thing you want to do:

GODEBUG=gctrace=1 ./programName

Learn the basics of using GODEBUG

Scheduler Tracing

This helps you find if your program is leaking goroutines

GOMAXPROCS=2 GODEBUG=schedtrace=1000 ./programName
hey -m POST -c 8 -n 100000000 "http://localhost:port/endpoint"

pprof Profiling

Learn the basics of using http/pprof Using the http/pprof support you can profile your web applications and services to see exactly where your performance or memory is being taken.

import _ "net/http/pprof"

Look at the basic profiling stats from the new endpoint

http://localhost:4000/debug/pprof
hey -m POST -c 8 -n 1000000 "http://localhost:4000/sendjson"

Blocking Profiling

Learn the basics of Blocking Profiling

federico-garcia/ultimate-go-programming.md

Ultimate Go Programming

Design Guidelines

Prepare your mind

Performance vs Productivity

Correctness vs Performance

Language Mechanics

Variables

Struct Types

Pointers

Constants

Functions

Data Structures

Data-oriented design

Arrays

Slices

Maps

Decoupling

Methods

Interfaces

Embedding

Exporting

Composition

Grouping Types

Decoupling

Conversion and Assertions

Interface Pollution

Mocking

Design Guidelines

Error Handling

Packaging

Language Mechanics

Design Guidelines

Package-Oriented Design

Goroutines

How the scheduler works

Difference between concurrency and parallelism

Language Mechanics

Less is more

Data Races

Atomic Functions

Mutexes

Channels

Concurrency Patterns

Context

Task

Logger

Testing

Basic Testing

Web Testing

Example Test

Sub Tests

Test Coverage

Benchmarking

Fuzzing

Profiling

Tools

hey

ngrok

Memory Tracing

Scheduler Tracing

pprof Profiling

Blocking Profiling