Skip to content

Instantly share code, notes, and snippets.

@blockspacer
Last active July 23, 2020 13:41
Show Gist options
  • Save blockspacer/211a932cb0a535a1b098ec0ac562eeb8 to your computer and use it in GitHub Desktop.
Save blockspacer/211a932cb0a535a1b098ec0ac562eeb8 to your computer and use it in GitHub Desktop.
net

grpc + folly fibers https://github.com/leftyio/tortuga/search?q=fibers&unscoped_q=fibers

Istio grpc C++ https://github.com/Toeplitz/pwave_cloud/commit/fc5ae5a786fef80ad4afee7a98cb21db94a47ad2#diff-a055b85bab3694cc2083e4ae4a7d2b50 https://github.com/NVIDIA/tensorrt-laboratory/blob/f76e4b445f6a2a63485dbc4d8668159328868fcf/examples/12_FlatBuffers/server.cc#L58

Istio tensorrt-laboratory https://github.com/NVIDIA/tensorrt-laboratory/blob/2916998333aab2492da0d0c004fbd076d858b1ba/examples/90_Kubernetes/istio/README.md

Datadog OpenTracing C++ Client istio

Lwan in Performance and Scale @ Istio Service Mesh https://github.com/lpereira/lwan/tree/4bb64e2a7b46992249bff2afc8b582a5ce11b464

falcoAAupdate https://github.com/And03380/falcoAAupdate/search?q=Istio&unscoped_q=Istio

A social network with unidirectional follow relationships, implemented with loosely-coupled microservices, communicating with each other via Thrift RPCs.

wrk load generator

Jaeger C+

cpprest

Boost & 0mq & thrift

thrift

Seastar

boost.Fiber

Wangle

folly/fibers

libcoro coroutines

continuable

Obiew C++ & Java

Nakama C++ server

istio ws / socketio

TODO: read

Istio HTTPS https://www.reddit.com/r/docker/comments/b2ojr6/istio_enduser_authentication_for_kubernetes_using/

envoy/Istio for JWT verification

TODO:

Istio rate limit (requestcountquota)

Envoy rate limit

http_filters:
          - name: envoy.rate_limit # enable the Rate Limit filter

https://venilnoronha.io/envoy-grpc-and-rate-limiting

Istio load balance

WxLogin auth server grpc C++

pitaya

A login demo using grpc.

foobar-social - grpc - Colossus - Example Microservices

grpc

C++ google cloud

C++ k8s

Continuously build, test, and deploy.

Cloud Bigtable

Istio

Refactoring to microservices

Best practices for using Cloud Spanner as a gaming database

docker

minikube

agones

gcloud compute firewall-rules allow udp

C++ Continuous Integration

Sanitizer

Reflection & Serialization

Plugin management library with static and dynamic plugins

Load testing

Statistics

TODO:

TODO:

TODO

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

having a monster in an MMO game remember which players have tried to attack it before so it can be hostile when it sees them again.
https://blog.demofox.org/2015/02/08/estimating-set-membership-with-a-bloom-filter/

@blockspacer
Copy link
Author

Nebula Graph database based on the relations between different vertices
https://github.com/vesoft-inc/nebula/blob/master/docs/manual-EN/1.overview/2.quick-start/1.get-started.md

@blockspacer
Copy link
Author

blockspacer commented Jan 16, 2020

Build network priority system:

  1. Closer to camera
  2. Within camera frustum
  3. Custom object importance (grenade, vehicle, etc)
  4. Is entity visible or behind solid object. If an entity is in our sight, but behind a mountain, I can possibly ignore it.
    the most distant entity will receive updates less frequently

@blockspacer
Copy link
Author

blockspacer commented Jan 21, 2020

# see https://www.jfrog.com/confluence/display/JFROG/Installing+Artifactory
sudo mkdir -p /data/artifactory
sudo chmod 0755 /data/artifactory
# Artifactory runs as user 1030:1030 by default. When passing a volume to the Artifactory container, this directory (on the host) must be writable by the Artifactory user.
sudo touch /data/artifactory/etc//system.yaml
sudo chown -R 1030:1030 /data/artifactory
#  artifactory-cpp-ce - JFrog Artifactory Community Edition for C/C++, a completely free of charge server for Conan repositories. 
sudo docker pull docker.bintray.io/jfrog/artifactory-cpp-ce
 
#    -e EXTRA_JAVA_OPTIONS='-Xms1024m -Xmx2g -Xss256k -XX:+UseG1GC' \
sudo docker run -d --restart=always --name artifactory \
    -v /data/artifactory:/var/opt/jfrog/artifactory \
    -p 8081:8081 -p 8082:8082 \
    --memory="2048m" \
    -e EXTRA_JAVA_OPTIONS='-server -Xms512m -Xmx4g -Xss256k -XX:+UseG1GC' \
    docker.bintray.io/jfrog/artifactory-cpp-ce
docker logs -f artifactory

visit http://localhost:8081 to open configuration wizard User:admin. Password:password
NOTE: you can skip Configure a Proxy Server step, read https://www.jfrog.com/confluence/display/RTF6X/Configuring+a+Reverse+Proxy
turn off anonymous access in the Admin> Security Configuration page http://localhost:8081/artifactory/webapp/#/admin/security/general
open http://localhost:8081/artifactory/webapp/#/artifacts/browse/tree/General/conan and click the Set Me Up button as in https://www.jfrog.com/confluence/display/RTF/Using+Artifactory#UsingArtifactory-SetMeUp

export CONAN_REVISIONS_ENABLED=1
conan remote add conan-local http://localhost:8081/artifactory/api/conan/conan False
conan remote list
# NOTE: change password
conan user -p password -r conan-local admin
# --all                 Upload both package recipe and packages, see https://docs.conan.io/en/latest/reference/commands/creator/upload.html
# conan upload libhello/stable --all -r=conan-local -c --retry 3 --retry-wait 10
# conan search -r conan-local libhello

Follows
http://chu-studio.com/posts/2019/%E4%BB%8E%E9%9B%B6%E5%BC%80%E5%A7%8B%E7%9A%84C++%E5%8C%85%E7%AE%A1%E7%90%86%E5%99%A8CONAN%E4%B8%8A%E6%89%8B%E6%8C%87%E5%8D%97
https://qiita.com/navitime_tech/items/c1c878b6d22e911f7fb6
https://www.cnblogs.com/rongfengliang/p/9229462.html
https://www.cnblogs.com/xl2432/p/11896466.html
https://www.baeldung.com/java-ee-oauth2-implementation
https://medium.com/@robert.broeckelmann/openid-connect-authorization-code-flow-with-red-hat-sso-d141dde4ed3f

Setting up Jenkins and Artifactory in the CI machine

https://dzone.com/articles/dockerizing-jenkins-part-2-deployment-with-maven-a

docker pull conanio/conan_server

OR

# from https://www.jfrog.com/confluence/display/RTF/Installing+with+Docker
docker pull docker.bintray.io/jfrog/artifactory-cpp-ce
docker run --name artifactory -d -p 8081:8081 -p 8082:8082 --memory="512m" docker.bintray.io/jfrog/artifactory-cpp-ce:latest

https://blog.conan.io/2018/04/25/Continuous-integration-for-C-C++embedded-devices-with-jenkins-docker-and-conan.html
https://docs.conan.io/en/latest/howtos/generic_ci_artifactory.html
https://www.reddit.com/r/devops/comments/8eunov/continuous_integration_for_cc_embedded_devices/
https://cloud.google.com/blog/products/application-development/integrating-google-cloud-build-with-jfrog-artifactory
https://jfrog.com/blog/use-conan-to-simplfy-your-c-containerized-application-build-and-deployment-experience/
https://jfrog.com/blog/container-optimized-workflow-for-coreos-now-red-hat-tectonic/
https://devops.com/devops-challenges-c-c-projects/
https://www.slideshare.net/openstackil/kubernetes-is-hard-lessons-learned-taking-our-apps-to-kubernetes-eldad-assis-jfrog-cloud-native-day-tel-aviv-2018
image

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

blockspacer commented Jan 23, 2020

Kubernites ResourceQuotas
https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits

Kubernetes Failure Stories
https://k8s.af/

grpc timeouts
https://about.sourcegraph.com/go/grpc-in-production-alan-shreve

grpc memory arena allocation
https://developers.google.com/protocol-buffers/docs/reference/arenas

gRPC Compression
https://github.com/grpc/grpc/blob/master/doc/compression.md

SSL in gRPC and performance
https://github.com/grpc/grpc/blob/master/doc/ssl-performance.md

Optimize gRPC
One way to speed up programs is to make sure the CPU is not idling. To do this, we issue work concurrently.
https://grpc.io/blog/optimizing-grpc-part-1/
We need some way of allowing multiple operations to happen safely at the same time. Otherwise, the program can’t take advantage of all the available processors.
https://grpc.io/blog/optimizing-grpc-part-2/

@blockspacer
Copy link
Author

What is the difference between a Kubernetes cluster using 100x n1-standard-1 (1 vCPU) VMs VS having 1x n1-standard-96 (vCPU 96), or 6x n1-standard-16 VMs (vCPU 16)?
https://dev.to/gajus/mistake-that-cost-thousands-kubernetes-gke-46ij

@blockspacer
Copy link
Author

Preemptible VMs are up to 80% cheaper than regular instances. Pricing is fixed so you will always get low cost and financial predictability, without taking the risk of gambling on variable market pricing.

https://cloud.google.com/preemptible-vms/

@blockspacer
Copy link
Author

Speed up docker build with cache in Kubernetes environment
https://medium.com/swlh/fast-docker-build-in-kubernetes-f52088854f45

@blockspacer
Copy link
Author

blockspacer commented Jan 24, 2020

Microservices blog series

===

Go Microservices blog series, part 13 - data consistency, gorm and CockroachDB.
https://callistaenterprise.se/blogg/teknik/2018/02/14/go-blog-series-part13/

Create Versatile Microservices in Golang — Part 4 (Authentication With JWT)
https://dzone.com/articles/create-versatile-microservices-in-golang-part-4-au

Nodejs rest JWT
https://crosp.net/blog/software-development/web/nodejs/secure-rest-api-using-nodejs-part-2-express-jwt-mongoose/

@blockspacer
Copy link
Author

blockspacer commented Jan 24, 2020

TODO: time series database ???

InfluxDB open source time series database, purpose-built by InfluxData for monitoring metrics and events, provides real-time visibility into stacks, sensors, and ...
https://habr.com/ru/post/449028/

@blockspacer
Copy link
Author

TODO: HashiCorp Nomad ??? Why it's simpler than Kubernetes???

@blockspacer
Copy link
Author

blockspacer commented Jan 24, 2020

HOT roblox Microservices
https://portworx.com/architects-corner-roblox-runs-platform-70-million-gamers-hashicorp-nomad/

In my previous job at another company, the biggest issue we had was that we didn’t want to use a storage layer because we felt that per-node storage or local volumes were the best because of cost and flexibility. The truth is, you can probably make it work, but if I have a database that I need to make sure never fails, it just seems way too risky to run it like that. The cost-saving and flexibility you think you are going to get evaporate in the extra management time you have to put in.

Portworx Enterprise is the Kubernetes storage platform trusted in production by the world’s leading enterprises.

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

https://github.com/sourabhsahay/gameserver

Game Server 

Creating a Game Server is challenging on many levels.Services are stateful (different than conventional E-commerce model), scale is high and write:read ratio may be as much as 3:1.
Also there are issues of network latency which may require not only a faster protocol, but also data compression.

While part of scalability can be achieved through horizontal scaling, it is not as simple as scaling many servers. Inter-server communication between services will increase latency, so that also has to be kept in mind. 
 :
This document is divided in two parts : -
Game Server Architecture and implementation (As per problem)
Scalability and Future Scope

Game Server Architecture And Implementation:

Networking : 
       Since Game requires fast transport between different services, conventional TCP may not be able to perform at the frame rate required. For this, UDP has been used for the implementation for most of the game fast exchanges with server. For conventional services like Login, TCP or WebSocket is fine. For sending chats to game, the data needs to be consistent (chats need to appear in Sequence in which they arrive, so that has been handled in TCP/Websocket). For full duplex communication, Websocket is preferable to conventional HTTP. 

Data Marshalling/UnMarshalling:
It is important to marshall/unmarshall the data for communication over the network for maximum efficiency. Sometimes, the packets do not arrive at one go and are collated together, so marshalling scheme will help packing as much data as possible. For illustration purpose, in the solution Json format has been used. But it can be readily replaced by binary formats : 
For Flash based clients, AMF protocol can be used.
Unity based clients can be made to work with protobuf. The implementation contains import of protostuff library which helps in annotation driven protobuff generation.

To that effect in the codebase, two servers are implemented.
com.junglee.task.server.impl.TCPServer (Used for Login and Chat Server)
com.junglee.task.server.impl.UDPServer (Used for game session communication)

Database Handling:
Database can be decided depending upon storage and access pattern and consistency/availability guarantees. Stats are past data for user which can be stored in NoSQL solution like MongoDB. State/Session for user can be stored in fast distributed caches like Redis or Mem-cache. Both the game data and chat tend to be write-heavy, so Nosql wins in this regard over conventional DB. Nosql these days come with eventual consistency guarantees 
And implementation contains (LSM tree instead of conventional B Trees) which results in high throughput. 
Chat :The difference is that chat requires data to be highly consistent which can be achieved with a bit of latency (All chats connected to one channel can be partitioned in one partition, so latency becomes an issue). Using read replicas and write master, chats can be taken care of.Writes can be written to majority of nodes and masters keeps track of which nodes have this value. Failure and recovery can be managed by another machine with master which takes over if master fails.

Game session writes : This needs to be highly available with eventual consistency. Also failure and recovery can be taken care by consistent hashing.

In Memory storage of data : Kyro can be used for storing the data in memory. This gives a very high compression compared to conventional Java.

Thread Management : Since a game server maintains state and has high write requirement at the same time, it makes sense to not block one thread for each incoming connection. Using Java NIO Channel  and event loops helps in this regard. Netty has been used in the project for building up things.
Moreover, the internal communication between services is asynchronous and event-driven which scales better than conventional threading scheme. This has been proven time and again in nginx and new systems like AKKA.
That said, there are places for thread synchronization (creating new tables, adding new players to Table) which have been handled in (PokerTable and PokerTableManager).

Event Dispatcher : com.junglee.task.event.impl.ExecutorEventDispatcher is used for handling events. For good scalability, as mentioned in the previous point, it is necessary that communication is asynchronous in nature and event dispatcher helps in that. Also, as mentioned in the problem document, it helps in keeping the services loosely coupled.
 For example, 
 Incoming packets are regarded as events for which handlers are used for processing. Even the game logic is contained in an eventHandler.
For pushing data for Stats, event handler is used.
For chat processing, event handler is used.
The communication between TableManager(PokerTableManager) and PokerTable(at which a particular game instance is being played) is through event Dispatcher.
In this manner the same Event Dispatcher  pool can be used across multiple services.

Authentication, Player Creation and Player Management :  Authentication is done in TCPUpStreamHandler. Currently the DBLookUpServiceImpl is used for user and gameChannel(explaned later). Chats are contained in ChatServiceImpl. For illustration purposes, in-memory Java HashMap is used, but it can be readily replaced by DB.

Game Table Management/Player Management/Game Logic.
This will require a bit explanation
Game state is managed by GameStateManagerService (get or Set State, serialize/deserialize game states etc)
For maintaining user actions  for a game, PlayerSession is used. A Player can play mutiple games together and has a set of sessions for each of these games.
A player session is linked to a GameChannel where multiple players collaborate. For instance, 
A poker table could be one game channel, a chessboard can be another.
Game Logic is handled in a “handler” in PokerTable (which extends GameChannel).

So for a game Channel, A Player Session is proxy for remote client. 
A SessionManager  is used for maintaining Session. First incoming request is Login, upon which PlaySession is created and linked to a PokerTable instance. Also SessionManager stores
<remoteAddress, PlaySession> key-value pair which can be subsequently used.

Data Flow : - Incoming Message (Session events)
Network => PlayerSession => PlayerSession Handler(attached in Poker Table) => Game Channel( Or Poker Table) => PokerSessionHandler



Outgoing event (NetworkEvent) : Used for broad-cast and sending individually
For sending individually, PlayerSession is directly used, which uses TCP/UDPSender to send.
For broadcast, PokerTable is used, which calls event dispatcher, but all PlayerSession(s) are registered, so they get the event. Exactly in the reverse order

Sessions can be stored in a key-value store like RIAK or cached in Redis/Memcached.

GAME LOGIC : Game Logic is contained in PokerSessionHandler. As it is event driven, it is handled whenever there is an input from player. This is in continuation of even-driven design given in problem statement.


Project Setup :
All different parts are wired together using spring DI framework. Starting point is GameServer.


Extensibility:
Any other game can be developed extending GameChannelSession and developing custom-logic. At Server level, new/custom protocol can be registered by implementing com.junglee.task.server.Server.
As the various services are wired through interfaces, the implementation can be changed.

Not covered : If all players leave, destroy the table. (Made it more consistent with if everyone moves, return to pool, like in case of game over, though other can be implemented easily).

Future Scope :
EventsDispatcher can be extended to make sure that it is more cognizant of sending events not only based on events +sessions.
The event handling is based on handler and source/handlers are separated. There maybe cases where callbacks are required. Akka frame work provides this along with consistent hashing routing, so can be used to scale this further.

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

@blockspacer
Copy link
Author

TODO
Hands-on with DevOps
https://github.com/juliogomez/devops

@blockspacer
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment