thread pools

gistfile1.txt

```
This document goes over Cattle executor services, their use and its thread pools configurations.
1) Core executor service
Default executor service. Examples of use:
* API requests dispatching
* Agent events execution (example: sending agent.ping, replying to agent.ping)
* Reporting resource state.change events to subscribers (UI is an example of such subscriber)
Thread pool settings (configurable): 
=====================================
pool.coreexecutorservice.core.size=15
pool.coreexecutorservice.max.size=15
pool.coreexecutorservice.keep.alive=120
pool.coreexecutorservice.queue.size=-1 (infinite)
2) Event executor service
Used for:
* Events dispatching. Example: agent.ping event gets dispatched by this service, and then forwarded for execution to Core executor service.
Thread pool settings (configurable): 
=====================================
pool.eventexecutorservice.core.size=5
pool.eventexecutorservice.max.size=5
pool.eventexecutorservice.keep.alive=120
pool.eventexecutorservice.queue.size=-1 (infinite)
3) ScheduledThreadPool executor service
Used for schedulable periodic tasks like:
* config updates publishing
* purging removed resources
* cleaning up purged data from the database
Thread pool settings (not configurable): 
=====================================
core.size=10
max.size=10
keep.alive=120
queue.size=-1 (infinite)
4) Process thread pools executor services
The pools below used for process executions for cattle resources (examples: service.activate, instance.create, host.activate, etc)
4.1) ProcessEvent executor service
Default executor having 0 queue size. Once the thread count reaches max.size (10), the process gets rejected and delegated to one of 4.2-4.5 executors
Thread pool settings (configurable): 
=====================================
pool.processeventexecutorservice.core.size=3
pool.processeventexecutorservice.max.size=10
pool.processeventexecutorservice.keep.alive=120
pool.processeventexecutorservice.queue.size=0
4.2) ProcessNonBlocking executor service
Handles processes which are:
* not blocked waiting on other processes completion
* do not require any backend operations
Example: project.activate, serviceIndex.create
Thread pool settings (configurable): 
=====================================
pool.processnonblockingexecutorservice.core.size=3
pool.processnonblockingexecutorservice.max.size=5
pool.processnonblockingexecutorservice.keep.alive=1200
pool.processnonblockingexecutorservice.queue.size=1000
pool.processnonblockingexecutorservice.priority.queue=true
4.3) ProcessBlocking executor service
Used by processes than can be blocked either waiting for other processes to finish, or backend calls to complete. Example: instance.start waiting for container being started on the docker host.
Following resources' operations are blocking:
* containerevent
* instance
* volumestoragepool
* agent
* host
Thread pool settings (configurable): 
=====================================
pool.processblockingexecutorservice.core.size=15
pool.processblockingexecutorservice.max.size=35
pool.processblockingexecutorservice.keep.alive=1200
pool.processblockingexecutorservice.queue.size=1000
pool.processblockingexecutorservice.priority.queue=true
4.4) ProcessBlockingExtra executor service
Handles services' processes (service.update, service.activate). The reason services get dedicated thread pool is - their processes can be blocked for a prolonged time by waiting on instance.create/instance.start to finish.
Example: service.activate doesn't finish till all instances in the service are Running.
Thread pool settings (configurable): 
=====================================
pool.processblockingextraexecutorservice.core.size=15
pool.processblockingextraexecutorservice.max.size=35
pool.processblockingextraexecutorservice.keep.alive=1200
pool.processblockingextraexecutorservice.queue.size=1000
pool.processblockingextraexecutorservice.priority.queue=true
4.5) ProcessBlockingSystem executor service
Dedicated to handling processes for system containers. Example: ipsec container start. 
Having a dedicated thread pool will ensure that the system containers operations are prioritized, and regular instances starts won't fail because ipsec container hasn't been started yet.
Thread pool settings (configurable): 
=====================================
pool.processblockingsystemexecutorservice.core.size=15
pool.processblockingsystemexecutorservice.max.size=35
pool.processblockingsystemexecutorservice.keep.alive=1200
pool.processblockingsystemexecutorservice.queue.size=1000
pool.processblockingsystemexecutorservice.priority.queue=true

```