ESP8266 Webserver: send or download huge files

esp8266webserver_download_hack.md

The Arduino webserver library from the ESP8266 is very simple and you can get to its limits pretty fast!

So that beeing said I first want to recommend you this library: ESPAsyncWebServer.
It's a very good alternative with a lot of features and it's easy to use.

However I wanna show you an easy way to hack around the standard webserver library for projects which can't make use of the ESPAsyncWebserver library.

So what we want to do is making the ESP8266 serve files via its webserver that are bigger than the RAM size we have left.
A picture for example. You don't want to hold that in the RAM, it's limited and way to valuable to be used for this.

So normally you would write something like this: server.send(200, "text/plain", "some cool website content yey!");.
Of course this will work because the string which is sent is small.

So let's create a random and way to long string:

int fileSize = 80000;
String content;
  
for(int i=0;i<fileSize;i++){
  content += random(256);
}
  
server.send(200, "text/html", content);

This should let your ESP8266 crash when executing, because we set the fileSize to 80,000 characters which is 80KB and the RAM has only 80KB.

So to send such a long string anyway we can use a little trick.

//first create a fixed buffer
const int bufferSize = 6000;
uint8_t _buffer[6000];

//a little counter to know at which position we are in our buffer
int bufferCounter = 0;
  
int fileSize = 80000;

//now send an empty string but with the header field Content-Length to let the browser know how much data it should expect
server.sendHeader("Content-Length", (String)fileSize);
server.send(200, "text/html", "");

for(int i=0;i<fileSize;i++){
  _buffer[bufferCounter] = random(256); //write some random bytes into our buffer
  bufferCounter++;
  
  if(bufferCounter >= bufferSize){ //when the buffer is full...
    server.sendContent_P(_buffer, bufferCounter); //send the current buffer
    bufferCounter = 0; //reset the counter
  }
  
}

//send the rest bytes if there are some
if(bufferCounter > 0){
  server.sendContent_P(_buffer, bufferCounter);
  bufferCounter = 0;
}

TL;DR:

use server.sendHeader("Content-Length", <your file size>); to add a header and then send it via server.send( <code> , <type> , "");. Then send your content via server.sendContent_P( <contentArr> , <size> );.
The reason why you need to send an empty string first is simple. If the string is not empty, the library would count the length of the string and set this as Content-Length in the http header. And then the browser wouldn't wait for more packets, because it thinks it got all data and close the connection after the first packet.

I would also like to learn about if there is a similar hack so that I can send a post request to the esp8266webswerver with a very big Json file as the payload. In the library it seems that the handleclient() method will call some other methods to read the byte into a string and it looks like it would crash when the string becomes bigger than the ram.

in order for this to work I had to add

server.setContentLength(CONTENT_LENGTH_UNKNOWN);

before

server.sendHeader("Content-Length", (String)fileSize);
server.send(200, "text/html", "");```

otherwise the library was adding a Content-Length: 0 header