Observations from basic correctness tests done for the non blocking router

1_non_blocking_router_correctness_test_observations

We tested out the new NonBlockingRouter for correctness with small to sufficiently large blobs.
Blobs were put and got and diffed and verified for content equality.

Preparation
===========
- New MessageFormat was enabled locally and built.
- frontend-nb Restserver config was modified: s/CoordinatorBackedRouterFactory/NonBlockingRouterFactory

Setup
=====
The setup involed three servers and one frontend - all run locally (so not a good setup for performance, only correctness).
See attached HardwareLayout.json and PartitionLayout.json files used.

Testing
=======
Did a few tests with varying blob sizes below. The default MAX_CHUNK_SIZE is 4MB, and so for the following objects, the first
two resulted in simple blobs, rest ended up as composite blobs.
[pnarayan@pnarayan-ld1 target]$ ls -l a_48_KB b_2_MB c_25_MB d_125_MB e_12_GB
-rw-r--r-- 1 pnarayan eng       48967 May 28 00:32 a_48_KB
-rw-r--r-- 1 pnarayan eng     2232561 May 28 00:34 b_2_MB
-rw-r--r-- 1 pnarayan eng    25474878 May 28 00:33 c_25_MB
-rw-r--r-- 1 pnarayan eng   130426880 May 28 00:33 d_125_MB
-rw-r--r-- 1 pnarayan eng 12280070144 May 24 12:43 e_12_GB

Observation
-----------
The composite blob for c_25_MB resulted in 7 chunks, the one for e_12_GB resulted in 2928 chunks. Details attached.
Puts, Gets and Deletes succeeded for all kinds of blobs, and contents were verified. No real issue related to the core flow was observed.
We had to tweak the configs for timeouts and netty idle timeout etc. Since all the servers and the frontend were run locally, things were
slow, especially puts. Have included the user perceived latency nevertheless. We can have another such document put up after performance 
tests are done in the perf cluster, which should also include things like memory usage.

2_HardwareLayout.json

{
    "clusterName": "LocalSetUp_ThreeNodeOneDisk_OnePartitionThreeReplicas",
    "version": 1,
    "datacenters": [
        {
            "dataNodes": [
                {
                    "disks": [
                        {
                            "capacityInBytes": 21474836480,
                            "hardwareState": "AVAILABLE",
                            "mountPath": "/home/pnarayan/workspace/ambry/target/0"
                        },
                    ],
                    "hardwareState": "AVAILABLE",
                    "hostname": "localhost",
                    "port": 6667,
                },
                {
                    "disks": [
                        {
                            "capacityInBytes": 21474836480,
                            "hardwareState": "AVAILABLE",
                            "mountPath": "/home/pnarayan/workspace/ambry/target/1"
                        },
                    ],
                    "hardwareState": "AVAILABLE",
                    "hostname": "localhost",
                    "port": 6668,
                },
                {
                    "disks": [
                        {
                            "capacityInBytes": 21474836480,
                            "hardwareState": "AVAILABLE",
                            "mountPath": "/home/pnarayan/workspace/ambry/target/2"
                        },
                    ],
                    "hardwareState": "AVAILABLE",
                    "hostname": "localhost",
                    "port": 6669,
                },
            ],
            "name": "Datacenter"
        }
    ]
}

3_PartitionLayout.json

{
    "clusterName": "LocalSetUp_ThreeNodeOneDisk_OnePartitionThreeReplicas",
    "version": 1,
    "partitions": [
        {
            "id": 0,
            "partitionState": "READ_WRITE",
            "replicaCapacityInBytes": 20737418240,
            "replicas": [
                {
                    "hostname": "localhost",
                    "mountPath": "/home/pnarayan/workspace/ambry/target/0",
                    "port": 6667
                },
                {
                    "hostname": "localhost",
                    "mountPath": "/home/pnarayan/workspace/ambry/target/1",
                    "port": 6668
                },
                {
                    "hostname": "localhost",
                    "mountPath": "/home/pnarayan/workspace/ambry/target/2",
                    "port": 6669
                }
            ]
        }
    ]
}

4_server_and_frontend_startup_commands

servers:
nohup java -Dlog4j.configuration=file:../config/log4j.properties -jar ambry.jar --serverPropsFilePath ../config/server1.properties --hardwareLayoutFilePath ../config/h.json --partitionLayoutFilePath ../config/p.json > logs/server1.log & tail -f logs/server1.log
nohup java -Dlog4j.configuration=file:../config/log4j.properties -jar ambry.jar --serverPropsFilePath ../config/server2.properties --hardwareLayoutFilePath ../config/h.json --partitionLayoutFilePath ../config/p.json > logs/server2.log & tail -f logs/server2.log
nohup java -Dlog4j.configuration=file:../config/log4j.properties -jar ambry.jar --serverPropsFilePath ../config/server3.properties --hardwareLayoutFilePath ../config/h.json --partitionLayoutFilePath ../config/p.json > logs/server3.log & tail -f logs/server3.log

frontend:
nohup java -Dlog4j.configuration=file:../config/frontend.log4j.properties -cp "*" com.github.ambry.frontend.AmbryFrontendMain --serverPropsFilePath ../config/frontend.properties --hardwareLayoutFilePath ../config/h.json --partitionLayoutFilePath ../config/p.json > logs/frontend.log & tail -f logs/frontend.log

5_POST

[pnarayan@pnarayan-ld1 target]$ time curl -i -H "x-ambry-blob-size : `wc -c ./a_48_KB | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : a_48_KB" http://localhost:1174/ --data-binary @./a_48_KB
HTTP/1.1 201 Created
Date: Sat, 28 May 2016 07:43:32 GMT
Location: /AAEAAQAAAAAAAAAAAAAAJGM4ZGVmZjQzLTE5YWUtNDIwNy04ODA3LTQ3ZDJjZmRhNzUyNA
Content-Length: 0
x-ambry-creation-time: Sat, 28 May 2016 07:43:31 GMT


real    0m1.027s
user    0m0.005s
sys     0m0.004s
[pnarayan@pnarayan-ld1 target]$ time curl -i -H "x-ambry-blob-size : `wc -c ./b_2_MB | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : b_2_MB" http://localhost:1174/ --data-binary @./b_2_MB
HTTP/1.1 201 Created
Date: Sat, 28 May 2016 07:44:20 GMT
Location: /AAEAAQAAAAAAAAAAAAAAJGJjZmE5MTI1LTRmZmUtNDEwZi05NGMyLTEzMDJhMDkxNjY3ZA
Content-Length: 0
x-ambry-creation-time: Sat, 28 May 2016 07:44:19 GMT


real    0m1.168s
user    0m0.002s
sys     0m0.012s
[pnarayan@pnarayan-ld1 target]$ time curl -i -H "x-ambry-blob-size : `wc -c ./c_25_MB | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : c_25_MB" http://localhost:1174/ --data-binary @./c_25_MB
HTTP/1.1 201 Created
Date: Sat, 28 May 2016 07:45:03 GMT
Location: /AAEAAQAAAAAAAAAAAAAAJDgyZTc2OWExLWRiMzItNGYxOS04NGVjLWU4YjZlYjhkMzM4Yw
Content-Length: 0
x-ambry-creation-time: Sat, 28 May 2016 07:45:01 GMT


real    0m2.403s
user    0m0.012s
sys     0m0.049s
[pnarayan@pnarayan-ld1 target]$ time curl -i -H "x-ambry-blob-size : `wc -c ./d_125_MB | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : d_125_MB" http://localhost:1174/ --data-binary @./d_125_MB
HTTP/1.1 201 Created
Date: Sat, 28 May 2016 07:46:12 GMT
Location: /AAEAAQAAAAAAAAAAAAAAJGE0M2IyMzBiLTc0NDEtNGEyMi04NmQ0LTVlZjVlNjExZGEzYQ
Content-Length: 0
x-ambry-creation-time: Sat, 28 May 2016 07:46:03 GMT


real    0m9.276s
user    0m0.047s
sys     0m0.135s
[pnarayan@pnarayan-ld1 target]$ time curl -i -H "x-ambry-blob-size : `wc -c ./e_12_GB | xargs | cut -d" " -f1`" -H "x-ambry-service-id : CUrlUpload"  -H "x-ambry-owner-id : `whoami`" -H "x-ambry-content-type : image/gif" -H "x-ambry-um-description : e_12_GB" http://localhost:1174/ --data-binary @./e_12_GB
HTTP/1.1 201 Created
Date: Sat, 28 May 2016 08:01:01 GMT
Location: /AAEAAQAAAAAAAAAAAAAAJGFmNGM4NDhlLTVhMTYtNGRiOC1hZGY4LWQ0MTdmYTkwNmQwNA
Content-Length: 0
x-ambry-creation-time: Sat, 28 May 2016 07:47:40 GMT


real    14m11.904s
user    0m2.508s
sys     0m13.403s

6_GET

[pnarayan@pnarayan-ld1 target]$ time curl http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGM4ZGVmZjQzLTE5YWUtNDIwNy04ODA3LTQ3ZDJjZmRhNzUyNA -o a                                                                                                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48967    0 48967    0     0   651k      0 --:--:-- --:--:-- --:--:--  664k

real    0m0.180s
user    0m0.000s
sys     0m0.008s
[pnarayan@pnarayan-ld1 target]$ diff a_48_KB a
[pnarayan@pnarayan-ld1 target]$ time curl http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGJjZmE5MTI1LTRmZmUtNDEwZi05NGMyLTEzMDJhMDkxNjY3ZA -o b
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2180k    0 2180k    0     0  34.6M      0 --:--:-- --:--:-- --:--:-- 35.4M

real    0m0.065s
user    0m0.002s
sys     0m0.007s
[pnarayan@pnarayan-ld1 target]$ diff b_2_MB b
[pnarayan@pnarayan-ld1 target]$ time curl http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJDgyZTc2OWExLWRiMzItNGYxOS04NGVjLWU4YjZlYjhkMzM4Yw -o c
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 24.2M    0 24.2M    0     0  59.8M      0 --:--:-- --:--:-- --:--:-- 59.8M

real    0m0.410s
user    0m0.004s
sys     0m0.040s
[pnarayan@pnarayan-ld1 target]$ diff c_25_MB c
[pnarayan@pnarayan-ld1 target]$ time curl http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGE0M2IyMzBiLTc0NDEtNGEyMi04NmQ0LTVlZjVlNjExZGEzYQ -o d
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  124M    0  124M    0     0  58.5M      0 --:--:--  0:00:02 --:--:-- 58.5M

real    0m2.129s
user    0m0.023s
sys     0m0.201s
[pnarayan@pnarayan-ld1 target]$ diff d_125_MB d
[pnarayan@pnarayan-ld1 target]$ time curl http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGFmNGM4NDhlLTVhMTYtNGRiOC1hZGY4LWQ0MTdmYTkwNmQwNA -o e
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.4G    0 11.4G    0     0  78.7M      0 --:--:--  0:02:28 --:--:--  237M

real    2m28.681s
user    0m1.507s
sys     0m20.238s
[pnarayan@pnarayan-ld1 target]$ diff e_12_GB e
[pnarayan@pnarayan-ld1 target]$

7_GET_BlobInfo

GetBlobInfo of a 12G object only results in one (newly implemented) BlobInfo request to the servers for the metadata chunk:
[pnarayan@pnarayan-ld1 target]$ time curl -i http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGFmNGM4NDhlLTVhMTYtNGRiOC1hZGY4LWQ0MTdmYTkwNmQwNA/BlobInfo
HTTP/1.1 200 OK
Date: Sat, 28 May 2016 08:20:04 GMT
Last-Modified: Sat, 28 May 2016 07:47:40 GMT
x-ambry-blob-size: 12280070144
x-ambry-service-id: CUrlUpload
x-ambry-creation-time: Sat, 28 May 2016 07:47:40 GMT
x-ambry-private: false
x-ambry-content-type: image/gif
x-ambry-owner-id: pnarayan
x-ambry-um-description: e_12_GB
Content-Length: 0


real    0m0.011s
user    0m0.001s
sys     0m0.002s

8_DELETE

[pnarayan@pnarayan-ld1 target]$ time curl -i -X DELETE http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGE0M2IyMzBiLTc0NDEtNGEyMi04NmQ0LTVlZjVlNjExZGEzYQ
HTTP/1.1 202 Accepted
Date: Sat, 28 May 2016 08:24:22 GMT
Content-Length: 0


real    0m0.027s
user    0m0.003s
sys     0m0.002s
[pnarayan@pnarayan-ld1 target]$ time curl -i http://localhost:1174/AAEAAQAAAAAAAAAAAAAAJGE0M2IyMzBiLTc0NDEtNGEyMi04NmQ0LTVlZjVlNjExZGEzYQ/BlobInfo
HTTP/1.1 410 Gone
Date: Sat, 28 May 2016 08:24:44 GMT
Content-Length: 17
Content-Type: text/plain; charset=UTF-8

Failure: 410 Gone
real    0m0.021s
user    0m0.001s
sys     0m0.003s

9_Miscellaneous

Chunk puts happening in different order (but in groups of 4, 4 being the MAX_IN_MEM_CHUNKS):

...
[2016-05-28 00:48:28,491] INFO Completed putting chunk 142 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:28,492] INFO Completed putting chunk 143 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:28,492] INFO Completed putting chunk 140 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:28,492] INFO Completed putting chunk 141 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:29,507] INFO Completed putting chunk 147 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:29,507] INFO Completed putting chunk 146 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:29,507] INFO Completed putting chunk 144 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:29,508] INFO Completed putting chunk 145 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:30,562] INFO Completed putting chunk 148 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:30,562] INFO Completed putting chunk 151 (com.github.ambry.router.PutOperation)
[2016-05-28 00:48:30,562] INFO Completed putting chunk 150 (com.github.ambry.router.PutOperation)
...

Was the fix for PutRequest (buffer size) already accommodated in this testing or not yet?