Nginx settings and jQuery File Uploader chunk sizes

results.txt

jQuery File Uploader (blueimp) and nginx benchmark notes.

Hypothesis:

Assigning the chunk size to be greater than the client_body_buffer_size will result in slower uploads and corrupt files.

Table 1: Stability
------------------

The table below shows the number of successful file uploads after a random number of pause and resume operations.
Note that these tests include nginx tweaking of the "client_body_buffer_size".

File size: 43 MB to 730 MB videos

Chunk size                 nginx client_body_buffer_size            files successfully uploaded
-----------------------------------------------------------------------------------------------
1024 * 128  (128 KB)	               128 KB                                  7 / 7


1024 * 128  (128 KB)                   256 KB                                  7 / 7
1024 * 256  (256 KB)                   256 KB                                  7 / 7	
                                 
1024 * 128  (128 KB)                   512 KB                                  7 / 7
1024 * 256  (256 KB)                   512 KB                                  7 / 7            
1024 * 512  (512 KB)                   512 KB                                  7 / 7

1024 * 128  (128 KB)                  1024 KB                                  7 / 7
1024 * 256  (256 KB)                  1024 KB                                  7 / 7
1024 * 512  (512 KB)                  1024 KB                                    *
1024 * 1024   (1 MB)                  1024 KB                                    *

* = skipped


Note: In most cases where the chunk size was higher than the client_body_buffer_size, several of
the uploads ended up corrupt. We concluded that the tests were not valuable in cases where the chunk 
size was greater than the buffer size. 


Table 2: Performance
--------------------

The table below shows the performance of uninterrupted chunk uploading at various chunk sizes
against different nginx client_body_buffer_size.

Number of simultaneous uploads: 5
File size: ~730 MB

Chunk size                       nginx client_body_buffer_size         time to complete
------------------------------------------------------------------------------------------------
1024 * 64   (64 KB)                         128 KB                          6m 28s
1024 * 128 (128 KB)                         128 KB                          3m 48s

1024 * 128 (128 KB)                         256 KB                          3m 58s
1024 * 256 (256 KB)                         256 KB                          3m 10s, 3m 15s

1024 * 128 (128 KB)                         512 KB                          4m 52s
1024 * 256 (256 KB)                         512 KB                          3m 34s
1024 * 512 (512 KB)                         512 KB                         31m 20s

1024 * 128 (128 KB)                        1024 KB                          4m 12s
1024 * 256 (256 KB)                        1024 KB                          3m 20s
1024 * 512 (512 KB)                        1024 KB                            *
1024 * 1024  (1 MB)                        1024 KB                            *

* = skipped


Note: Specifying a chunk size of 512 KB or higher resulted in a drastically slower upload.



Conclusion:

Hypothesis was correct. Based on the data collected, it seems our most optimal configuration is
to use a maxChunkSize of 256 KB and a client_body_buffer_size of 256 KB because it performed the
best out of all the other configurations. 

It is possible that the other configurations did not work as well due to how the test environment 
was configured. These settings did work a bit differently in a much more realistic setting, but that
could be related to the difference between hardware. The important takeaway from this documentation
is to realize that your chunk size must be less than or equal to your client_body_buffer_size.

Hi,

I'm interested on the script you used for the benchmark. Is it possible to share how you do the benchmark or perhaps some scripts used?

I'd like to perform same tests for my nginx server and see which configuration best suited for my setup.

Thanks

+1

👍