with tls enabled im getting errors in httpd-freertos.c line 427 and 441, looks like its trying to call continue from function, just replacing continue with return fixed the issue

"The MPLv2 restricts freedom in that it requires modifications to be given back to the community. This license establishes the agreement that in exchange for using this great library that users are required to give back their changes to let others benefit."

So how about pushing back upstream already?

It would be great if users will be able to replace espfs with the custom file system, e.g., FatFS on MicroSD, spiffs, etc.

Hi,

I'm using this component on ESP32 withc basic http auth. But the problem is that usre can call cgi scripts even it is not authorized. For example my configuration is:
...
ROUTE_CGI("/admin/example.cgi", cgiExample),
ROUTE_AUTH("/admin/*", webBasicAuth),
...
And user can call cgi dirrectly when enter full url.
Is this OK or this is a BUG ? If this is OK, I think I need to verify autorization in CGI script, but is there any function/macro providing infromation that current request comes from authorized browser ?

BR,
Marcin.

I have libesphttpd working find on ESP32 + esp-idf 3.3. Today I tried to enable SSL support. I got logs and everything seems to work fine but SSL_accept takes about 11 second...(see logs below). This is not the latest version, but I didn't see any commits related with this. My current version is 51084e0. About 140kb RAM free before I call https request.

Any ideas to solve this problem ?
``

I (10618) httpd-freertos: address 0.0.0.0, port 443, maxConnections 16, mode ssl
I (10618) httpd-freertos: SSL server context create ......
I (10618) httpd-freertos: OK
I (10618) httpd-freertos: SSL server context setting ca certificate......
I (10628) httpd-freertos: OK
I (10628) httpd-freertos: SSL server context setting private key......
I (10958) httpd-freertos: init
I (11018) httpd-freertos: esphttpd: active and listening to connections on 0.0.0.0
D (11018) httpd-freertos: Sel add listen 54
I (11018) httpd-freertos: listening for new connections on '0.0.0.0'
D (13948) httpd-freertos: select ret
D (13948) httpd-freertos: SSL server create .....
D (13948) httpd-freertos: OK
D (13948) httpd-freertos: SSL server accept client .....
D (24388) httpd-freertos: OK
D (24388) httpd-freertos: Sel add listen 54
D (24498) httpd-freertos: select ret

``
Thanks Marcin.

I'm finally upgrading my project to ESP-IDF 5.x and it looks like this library will need some changes to support it.

The first issue I encounter is

Failed to resolve component 'openssl'.

Looking at the 4.4 to 5.0 migration guide:

OpenSSL-API component is no longer supported. It is not available in the IDF Component Registry, either. Please use ESP-TLS or mbedtls API directly.

I see that @jkent has already worked on removing openssl on his fork: jkent@4da0313 (I wish he had contributed his fixes back to this project.)

I assume that's not the only issue that I will have. I'll document them here as I go...

printf() has sometimes been used rather than httpd_printf() in (but not limited to) the following files:

• httpd-freetos.c
• httpd.c
• espfs.c

this causes unwanted messages to be output even when httpd_printf() has silenced.

P.S. what's the best practice for silencing httpd_printf()?

I place the libesphttpd repository into the components directory of your esp-idf folder.
ran menuconfig and enabled it in menuconfig.
I then copied the esphttpd-freertos, and ran make on it. I get the following error.

C:/msys32/home/JIm/esp/esp-idf/components/libesphttpd-master/espfs/heatshrink_decoder.c:18:50: fatal error: ../lib/heatshrink/heatshrink_decoder.c: No such file or directory compilation terminated.

There are no files in the /lib folder.

When it receives a websockets frame that spans multiple TCP segments, the receive callback gets called once with each segment, with no obvious way to reassemble them.

https://github.com/chmorgan/libesphttpd/blob/master/util/cgiwebsocket.c#L174

https://github.com/chmorgan/libesphttpd/blob/master/util/cgiwebsocket.c#L254
^ this should get called once, with the whole frame

console log

I (1420127) user_main: UartWs: connect
D (1420147) cgiwebsocket: Upgrade: websocket
I (1420147) user_main: McuWs: connect
D (1420167) cgiwebsocket: Upgrade: websocket
I (1420167) user_main: EchoWs: connect
D (1420187) cgiwebsocket: Frame payload. wasHeaderByte 1 fr.len 9 sl 9 cmd 0x82
I (1420187) user_main: McuWs: len=9, more=0, bin=2
I (1420187) user_main: set time: 1543542461
I (1420197) RTC: >> writeValue: 1543542461
I (1420197) RTC:  - Fri Nov 30 01:47:41 2018

D (1437817) cgiwebsocket: Frame payload. wasHeaderByte 1 fr.len 1538 sl 1428 cmd 0x82
I (1437817) user_main: McuWs: len=1428, more=0, bin=2
I (1437817) user_main: invalid schedule len: 1428; expected 1538
D (1437827) cgiwebsocket: Frame payload. wasHeaderByte 0 fr.len 110 sl 110 cmd 0x82
I (1437827) user_main: McuWs: len=110, more=0, bin=2
I (1437837) user_main: invalid set time len: 110

browser log

wireshark decode

Hi, thank you so much for maintaining this library! It'll save me a ton of work.

I noticed that the init functions for the flash and tcpip adapter are not called. Without those no socket can be allocated and the wifi will not work.

tcpip_adapter_init();    
nvs_flash_init();

Is this new to ESP-IDF?

I realize this isn't the right place for this, but I didn't see any contact information. I was going to open an issue against esphttpd-freertos, but there's no option for that. Was that an oversight, or intentional?

Thanks!

libesphttpd/espfs/mkespfsimage//main.c:339:18: error: 'Z_BEST_COMPRESSION'undeclared
int compLvl = Z_BEST_COMPRESSION;
^~~~~~~~~~~~~~~~~~
A known issue you have caught, when your don't use any compression.
Viele Gruesse
Axel

We can add this library to the new IDF Component Registry https://components.espressif.com/

Some info about what is required:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/tools/idf-component-manager.html

Hi,

I followed the instructions in the readme but I notice the sdkconfig option CONFIG_ESPHTTPD_SSL_SUPPORT is not detected thus the build defaults to mode non-ssl.

I have also tried to add "sdkconfig.h" to the respective files including httpd-freertos, and also to the app_main file in the example usage repo.

Any thoughts are appreciated.

@jkent was absolutely right complaining about strcpy #91 (comment)

It causes damaging conn->priv.sendBuffLen in httpdFlushSendBuffer().

I'm going to submit a PR fixing that issue soon.

Hello, (sigh) well, this is not really an issue any more, or maybe not yet again since I tried to use this code to give my web server a bit of asynchronicity but using it turned out to be more problematic than I expected so I left this idea alone for now, but.
I thought that the idea behind serving static files should have been in... well, not serving them as dynamic ones rather than letting them to be "cached forever" (like that was ever any good). By that I mean, rather send (initial) response with E-Tag or Last-Modified with max-age let's say 2-5min (I'd rather send "304 Not Modified" more often than let the file be cached for a year and keep pressing F5 on every page I switch to) just enough time till I re-upload a new image :-). And then there is no need in static file distinction since with this strategy dynamic content can be served the same way as long as dynamic stuff hasn't expired.
E-Tag is some sort of CRC/MD5 and rather identifies the file content rather than the file properties (name, location, mtime, etc). Kinda heavy to evaluate on the fly and storing it with the file would mean that ESPFS would need either extra flag and field to save it after the header or trail after the file content similar to NTFS streams which I guess a bit cleaner way of having filename.md5 files around.
Last-Modified requires just file MTIME (modified time) which is, guess what, not in ESPFS either. It's a shame really, reserving 256 bytes for a file name using only like a third of it and not sacrificing 4 bytes for time_t. One could use app image compile time instead but that only works if ESPFS is linked with the image. And again, adding more fields in the header without breaking backward compatibility would probably require as much code as rather using dynamic headers (like TLV encoding).
And couldn't really send 304 Not Modified since everything is send with "OK" (304 OK in this case) and buried deep down, more copy-paste-modify code, making original one... well, left out.
Someone really have to remove ESP8266 tag from the page, this code won't build on v3.3 8266 SDK. Mostly because... yep, guessed right, ESPFS, no similar mmap, no pyhton3, no heatshrink2 that currently works there.

I recently started using libesphttpd, and am getting a couple warnings with the latest esp-idf pull.

CC build/libesphttpd/core/httpd-nonos.o
CC build/libesphttpd/core/httpd-freertos.o
CC build/libesphttpd/core/sha1.o
CC build/libesphttpd/core/httpdespfs.o
CC build/libesphttpd/core/auth.o
CC build/libesphttpd/core/base64.o
CC build/libesphttpd/core/httpd.o
CC build/libesphttpd/espfs/espfs.o
CC build/libesphttpd/espfs/heatshrink_decoder.o
CC build/libesphttpd/util/cgiwebsocket.o
CC build/libesphttpd/util/cgiflash.o
CC build/libesphttpd/util/esp32_httpd_vfs.o
CC build/libesphttpd/util/esp32_flash.o
CC build/libesphttpd/util/cgiredirect.o
CC build/libesphttpd/util/captdns.o
CC build/libesphttpd/util/cgiwifi.o
/home/jcw/Projects/test/components/libesphttpd/util/cgiwifi.c: In function 'startSta':
/home/jcw/Projects/test/components/libesphttpd/util/cgiwifi.c:343:2: warning: 'esp_wifi_set_auto_connect' is deprecated [-Wdeprecated-declarations]
  esp_wifi_set_auto_connect(1);
  ^
In file included from /home/jcw/Projects/test/components/libesphttpd/util/cgiwifi.c:14:0:
/home/jcw/Projects/esp-idf/components/esp32/include/esp_wifi.h:804:11: note: declared here
 esp_err_t esp_wifi_set_auto_connect(bool en) __attribute__ ((deprecated));
           ^
/home/jcw/Projects/test/components/libesphttpd/util/cgiwifi.c: In function 'tplWlan':
/home/jcw/Projects/test/components/libesphttpd/util/cgiwifi.c:517:12: warning: unused variable 'st' [-Wunused-variable]
  esp_err_t st = esp_wifi_sta_get_ap_info(&stconf);
            ^
AR build/libesphttpd/liblibesphttpd.a

I identified errors that are not propagated to the client via JSON. Will fix with a PR.

1. If try to AP scan when not in STA mode, UI does not show an error and the command is retried indefinitely.

2. If STA is trying to connect (but some error is preventing it from connecting), then scan fails, but UI is not aware (keeps starting scan). This condition should be handled somehow (timeout connecting if not connected after xx s? abort connecting as soon as scan-APs commanded?)

Unrelated change I propose: don't scan APs automatically from UI because it interrupts ongoing WiFi communications. This is causing me a headache when one may leave the WiFi page open in another browser tab and then complain that service is intermittent.
I'm adding a warning and a "Start Scanning" and "Stop Scanning" and "Scan Once" buttons to the UI.

Another unrelated change I propose: I will extend the cgiWiFiConnStatus function to be able to give info for all interfaces (Eth, STA, AP). URL argument can specify which interface, but default will send all interfaces. JSON response something like :

{ 
  "AP": { 
    "status": "Connected",
    "SSID": "poco",
    "encryption": "WPA2",
    "IP": "192.168.4.1",
    "MASK": "255.255.255.0",
    "GW": "0.0.0.0",
    "clients": "2"
  },
  "STA": { 
    "status": "Connected",
    "SSID": "janie2",
    "encryption": "WPA2",
    "IP": "192.168.2.108",
    "MASK": "255.255.255.0",
    "GW": "102.168.2.1"
  },
  "Eth": { 
    "status": "Connected",
    "IP": "192.168.2.102",
    "MASK": "255.255.255.0",
    "GW": "102.168.2.1"
  }
}

Where status could be one of
"Failed",
"Connected",
"Idle",
"Update",
"WPS Start",
"WPS Active",
"Connecting",
"Fall Back"

Some modern webservers will automatically ignore a double-slash
in the URL. (i.e. http://192.168.1.79//min.json is equivalent to http://192.168.1.79/min.json)
(see https://webmasters.stackexchange.com/questions/8354/what-does-the-double-slash-mean-in-urls )

How can I have the esphttpd server ignore the double-slash?
(without sending a redirect to the browser)

I've installed it as directed into the current esp-idf library. This is using a windows install as directed in the expressif instructions. esp-idf examples and such build and work correctly so I don't suspect it is the toolchain per se.

/home/miker/esp/esp-idf/components/libesphttpd/espfs/mkespfsimage//main.c:30:18: fatal error: zlib.h: No such file or directory

Changes in embedded html/js files are ignored by esp-idf make system, which results in incomplete OTA update files. Attached patch fixes this problem.
patch.txt

Consider adding some GitHub "Topics" to libesphttpd and esphttpd-freertos to get some more exposure and help.
Currently libesphttpd is only tagged with: https-server and http-server.

I searched and found these popular Topics that seem relevant:

• esp32
• esp8266
• esp-idf
• esp8266-webserver
• webserver
• websocket
• websocket-server

@chmorgan seems only repo owner can add these topics

I'm developing a product using libesphttpd. I'm serving 5~15 static files (html, js, css) on the browser initial load. I'm also serving a couple of simple cgi functions.

I've got most everything working for a demonstration in a couple days, but I have a critical bug. Please help!

The httpd server seems to hang occasionally for several minutes. It most often happens during an initial load of the static files.

The ESP still responds to PING while the httpd is frozen.

Sometimes, After several minutes (>20min) of hanging, I see error messages on the console like this: (but sometimes it seems to hang indefinitely)

E (6791432) httpd: send buf tried to write 1032 bytes, wrote -1
E (6791432) httpd-freertos: accept failed
accept: Socket is not connected
E (6791482) httpd: send buf tried to write 1032 bytes, wrote -1

Wireshark shows that the /static/plugins/jquery.min.js file was starting to transfer, but never finishes. There are just a bunch of TCP Keep-Alives after the last TCP segment is received.

TCPView shows that there are still 7 TCP connections from firefox to the ESP. All are ESTABLISHED, except one is FIN_WAIT2. (but the connection that seems to have stalled is listed as ESTABLISHED)

I've hooked-up the JTAG debugger and see that it is stuck in a function. I'll post that info shortly.

I've tried many things to isolate the problem, but it persists:

• I served static files using the built-in espfs.

• I served static files using vfs-fat filesystem. (it seems to hang even more often with this method)

• I compressed the static files using uglifyjs and yui-compressor.

• I just served a single html file as a test. The hang happens much less often, but still occurred.

• I tried with Ethernet and Wifi STA - seems the same occurrence of hangs. Wifi AP mode seems to cause it to hang every time.

• I have adjusted the following compile options (I've tried several combinations of these settings, to no avail):
  (especially the Wifi and ethernet buffer sizes)
  sdkconfig.txt

CONFIG_OPTIMIZATION_LEVEL_RELEASE=y
CONFIG_STACK_CHECK=y
CONFIG_STACK_CHECK_NORM=y
CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=240
CONFIG_SPIRAM_BOOT_INIT=y
CONFIG_SPIRAM_IGNORE_NOTFOUND=y
CONFIG_SPIRAM_USE_MEMMAP=
CONFIG_SPIRAM_USE_CAPS_ALLOC=
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_TYPE_ESPPSRAM32=y
CONFIG_SPIRAM_SIZE=4194304
CONFIG_SPIRAM_SPEED_40M=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_CACHE_WORKAROUND=
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384
CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST=y
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768
CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY=y
CONFIG_TASK_WDT_TIMEOUT_S=15
CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=16
CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=16
CONFIG_ESP32_WIFI_STATIC_TX_BUFFER=y
CONFIG_ESP32_WIFI_TX_BUFFER_TYPE=0
CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM=16
CONFIG_EMAC_L2_TO_L3_RX_BUF_MODE=y

CONFIG_FREERTOS_THREAD_LOCAL_STORAGE_POINTERS=4
CONFIG_FREERTOS_ISR_STACKSIZE=4096
CONFIG_SUPPORT_STATIC_ALLOCATION=y

CONFIG_ESPHTTPD_STACK_SIZE=8192
CONFIG_ESPHTTPD_SO_REUSEADDR=y
CONFIG_ESPHTTPD_SANITIZE_URLS=y

LWIP

CONFIG_L2_TO_L3_COPY=y
CONFIG_LWIP_MAX_SOCKETS=16
CONFIG_LWIP_IP_REASSEMBLY=y
CONFIG_LWIP_STATS=y
CONFIG_ESP_GRATUITOUS_ARP=y
CONFIG_GARP_TMR_INTERVAL=60

UDP

CONFIG_TCPIP_TASK_STACK_SIZE=4096

Hi, I have developed a module to provide two additional cgi functions that I have named

• cgiEspVfsHook
• cgiEspVfsUpload
  as the names suggest and the title indicates, these provide for serving static files from an ESP32 VFS, typically FAT FS mounted from an attached SD card, and for HTTP file uploads to the ESP32 VFS.

The are used, for example, like:
{"/store", cgiEspVfsUpload, "/sdcard/www/store", NULL},
{"/public/", cgiEspVfsHook, "/sdcard", NULL},
{"", cgiEspVfsHook, "/sdcard/www", NULL},

Note:

• the VFS filesystem needs to already be mounted and available, that isn't done in these modules
• the static files can be manually gzipped on the filesystem and will be matched and served with the correct encoding header without having to include the .gz extension in the URL
• code for supporting templates is included, but has never been tested. Its modelled off an older version of httpdespfs.c
• some things might be specific to my requirements (like the returned content at the completion of a file upload)

They work for me, so I am unlikely to put any more effort into maintaining them or improving them. I am happy to provide them "as is" for you or others to review/rework and better generalise into libesphttpd if anyone finds them useful.

Is you want me to provide the code please tell me the best way to send it. I could do it as a pull-request, or set it up as a gist, or anything else you want to suggest.

I found there was a change in commit 21fb779 where you removed two character overrides with '-'. I am a little confused about both states.

We are searching for "boundary=" ->b and later set the pointer to b+7. The length of our search subject is 9 so our buffer will contain for example "y=----WebKitFormBoundarywrAqgDbLLFmgN4Zn". Is there a reason the y= have previously been overwritten with hyphens? Should we exchange the 7 for strlen("boundary=")?

EDIT: I have found in this specification https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html that the two hyphens are mandatory. While the original solution was not well designed it correctly "prepended" two hyphens

Seems I got unlucky and stumbled on this bug. I guess my minimized and gzip compressed javascript is an exact multiple of the chunk size defined in serveStaticFile() = 1024.

My browser hangs forever trying to read that file. Wireshark shows that the file is indeed sent, but there is no \r\n at the end and the TCP connection is never closed (I'm using header: Connection: close, so I expect the connection closed after each file).

https://github.com/chmorgan/libesphttpd/blob/master/core/httpd-espfs.c#L202
(this code is mostly unchanged from the original author @Spritetm)

	len=espFsRead(file, buff, FILE_CHUNK_LEN);
	if (len>0) httpdSend(connData, buff, len);
	if (len!=FILE_CHUNK_LEN) {
		//We're done.
		espFsClose(file);
		return HTTPD_CGI_DONE;
	} else {
		//Ok, till next time.
		return HTTPD_CGI_MORE;
	}

It seems if the file is an exact multiple of FILE_CHUNK_LEN = 1024, then the final call will be with len==0. That should close the file and return HTTPD_CGI_DONE, but without sending any data.
It seems like that should work, but maybe httpd doesn't handle that case.

I hacked it for now by sending a dummy line-break if len==0 and it is now working for me:

	len=espFsRead(file, buff, FILE_CHUNK_LEN);
	if (len>0) httpdSend(connData, buff, len);
	// workaround if file length was exact multiple of FILE_CHUNK_LEN then we should send some dummy data in the last chunk
	if (len == 0){
		httpdSend(connData,"\r\n",2);
	}
	if (len!=FILE_CHUNK_LEN) {
		//We're done.
		espFsClose(file);
		return HTTPD_CGI_DONE;
	} else {
		//Ok, till next time.
		return HTTPD_CGI_MORE;
	}

But I think it needs a proper fix in httpd.

I have pushed security fix to original repo Spritetm#44
Guess it will be useful for your fork too.

Currently I am searching for a webserver for ESP32. There are different solutions out there - each with specific features and problems. With your solution I like that it is based on ESP-IDF and that it supports websockets and SSL.
But does it also allow real parallel access from several clients being seved simultaneously? In other words: does it work asynchronously?

I'm trying to upload a file via HTTP PUT using the supplied cgi function cgiEspVfsUpload().

(file downloading via the accompanying function cgiEspVfsHook() is working)

I added upload handler:

HttpdBuiltInUrl builtInUrls[]={
...
	ROUTE_CGI_ARG("/upload/*", cgiEspVfsUpload, FS_BASE_PATH "/html"), // FATfs filesystem
	ROUTE_CGI_ARG("*", cgiEspVfsHook, FS_BASE_PATH "/html"), // FATfs filesystem
	ROUTE_FILESYSTEM(), // espFs filesystem

	ROUTE_END()
};

And I test upload with cURL:

$ curl -X PUT -T test.txt "http://192.168.33.173/upload/test.txt"
curl: (56) Recv failure: Connection reset by peer

The ESP then crashes Guru Meditation Error: Core 1 panic'ed (LoadProhibited) at pc=40099419. Setting bp and returning..

I trapped it with debugger. It broke at strncat() at strncat.c:78 0x40099419
(I don't have the source code for that function. Where can I find it?)

The calling function in the stack is cgiEspVfsUpload() at esp32_httpd_vfs.c:243 0x40138589

Some clues:

• Seems cgiEspVfsUpload() wants to append string connData->getArgs to filename, but that string is empty! -- but it doesn't seem that should cause a crash?!
• Seems to me the correct string to append would be connData->url, since that is where the rest of the file path is specified.

If I send POST data to the server and there is no cgi function configured to handle it, it crashes horribly.

To repeat: Send post data with cURL

$ curl --progress-bar -X POST -T bigfile.txt "http://192.168.33.173/newdir/test.txt" | tee /dev/null
#                                                                          2.0%
404 File not found.

E (6827) httpd: url = NULL
E (6827) httpd: Unexpected data from client. blah blah blah........
... pages of error...
E (6827) httpd: Unexpected data from client. blah blah blah........
... pages of error...
... repeats indefinitely ...

I fixed it by adding a check to cgiNotFound() that continues if post data has not been completely received.

//Used to spit out a 404 error
static CgiStatus ICACHE_FLASH_ATTR cgiNotFound(HttpdConnData *connData) {
    if (connData->isConnectionClosed) return HTTPD_CGI_DONE;
    if (connData->post.received == connData->post.len)
    {
        httpdStartResponse(connData, 404);
        httpdEndHeaders(connData);
        httpdSend(connData, "404 File not found.", -1);
        return HTTPD_CGI_DONE;
    }
    return HTTPD_CGI_MORE; // make sure to eat-up all the post data that the client may be sending!
}

Also, I prevented that (and other?) error from repeating indefintely by adding a break; after it.

Philosophically, should the server continue to eat-up all of the post data if it can't do anything with it? The file could be gigabytes and used as a DoS attack. Is there a way to gracefully refuse the post?

Stay tuned for pull-request.

Hi,

I've ran into some bugs while using the cgiWebsocket implementation in a project on an esp32:

• Error "send buf tried to write %d bytes, wrote %d" in httpd.c got triggered sometimes. But allways with more bytes written than it said it should have (e.g. "tried to write 0 bytes, wrote 19").
• Some messages from server to client were sent twice.
• Frameheaders got corrupted (Chrome throws errors and closes connection). Some could not be decoded as valid utf-8 and some had reserved bits turned on.

A recording with wireshark revealed that in such cases, all bytes get sent sort of "correctly" but out of sync (e.g. in the TCP-Stream the WebSocked-Header should start but some random other bytes were sent before the correct header is finally sent). This lead to the wrong decoding in the browser as the header was not where it should have been.

So I've dug inside the code and noticed that inside all cgi-functions the sendBuff is guarded with
httpdPlatLock() and httpdPlatUnlock(). This works great if the server only ever reacts to requests. But with a WebSocket the server is allowed to send at any time. With functions cgiWebsocketSend() and cgiWebsocketClose() the server sends data to the client without a request and the httpd instance and sendBuff is unguarded.

I've added the locks and tested extensively. With the following code none of the mentioned bugs appeared:

int ICACHE_FLASH_ATTR cgiWebsocketSend(HttpdInstance *pInstance, Websock *ws, const char *data, int len, int flags) {
	int r=0;
	int fl=0;

	// Continuation frame has opcode 0
	if (!(flags&WEBSOCK_FLAG_CONT)) {
		if (flags & WEBSOCK_FLAG_BIN)
			fl = OPCODE_BINARY;
		else
			fl = OPCODE_TEXT;
	}
	// add FIN to last frame
	if (!(flags&WEBSOCK_FLAG_MORE)) fl|=FLAG_FIN;

    if (ws->conn->isConnectionClosed) {
        ESP_LOGE(TAG, "Websocket closed, cannot send");
        return WEBSOCK_CLOSED;
    }

	httpdPlatLock(pInstance);
	sendFrameHead(ws, fl, len);
	if (len!=0) r=httpdSend(ws->conn, data, len);
	httpdFlushSendBuffer(pInstance, ws->conn);
	httpdPlatUnlock(pInstance);
	return r;
}

void ICACHE_FLASH_ATTR cgiWebsocketClose(HttpdInstance *pInstance, Websock *ws, int reason) {
	char rs[2]={reason>>8, reason&0xff};
	httpdPlatLock(pInstance);
	sendFrameHead(ws, FLAG_FIN|OPCODE_CLOSE, 2);
	httpdSend(ws->conn, rs, 2);
	ws->priv->closedHere=1;
	httpdFlushSendBuffer(pInstance, ws->conn);
	httpdPlatUnlock(pInstance);
}

Only problem now is, that the lock possibly blocks the calling task without a timeout. In my usecase no problem, but I don't know if it is the best solution.

I hope this helps as I find this lib really great and I want to give a little bit back.

Cheers
Nik

The WiFi code in this httpd repo is outside the scope of an http server. It requires maintenance as the ESP-IDF SDK changes and it is application dependent.

Anyone object if I factor out the wifi code and move it to the example repo: https://github.com/chmorgan/esphttpd-freertos ?
Or should the wifi code be moved to another git submodule (like we did with espfs)?
Or it could be replaced with a newer solution like https://github.com/farpatch/esp32-wifi-manager ?

BTW cgiflash and vfs could also be factored out... Thoughts?

Hi, and thank you for this fork, along with some of @jkent's cmake stuff I'm up and running!

I'm wondering about concurrency and access to the sendBuf.

I noticed some concurrency lock stuff around websocket broadcast, and it looks like the other sending functions don't have that, do they assume to only be called from inside a cgi handler?

Also poking around a bit I noticed the static send buffer changes. It can consume quite a lot of memory (maxConnections * HTTPD_MAX_SENDBUFF_LEN), but I like that it's not going to fragment memory.

If responses are assumed to be composed in callbacks only, is it possible for more than 1 sendBuf to be written to simultaneously from the server task? It doesn't look like there's an intermediate action that could cause writes to sendBuf that don't also get flushed.

It looks like every write to the sendBuf is protected by httpdPlatLock (even the async websocket broadcast). It looks like it is fully synchronous, and you won't have other tasks trying to mutate it simultaneously).

I'm thinking about trying a single static sendbuf per HttpdInstance.

Does this sound like it could work? Am I missing something?

Good Day.
Thank you for the great code you have written. I am having an issue with flashing example code on to my esp32.
Any assistance will be greatly appreciated.
The following error pops up:
LD build/user.elf
C:/msys32/home/wdoug/esp/esphttpd-freertos/main/build/esp32\libesp32.a(cpu_start.o):(.literal.main _task+0x20): undefined reference to app_main' C:/msys32/home/wdoug/esp/esphttpd-freertos/main/build/esp32\libesp32.a(cpu_start.o): In function main_task':
C:/msys32/home/wdoug/esp/esp-idf/components/esp32/cpu_start.c:452: undefined reference to `app_mai n'
collect2.exe: error: ld returned 1 exit status
make: *** [C:/msys32/home/wdoug/esp/esp-idf/make/project.mk:388: /home/wdoug/esp/esphttpd-freertos /main/build/user.elf] Error 1

Hi,

If I reload a page that has an active web socket connection then there is a crash at:
https://github.com/chmorgan/libesphttpd/blob/master/util/cgiwebsocket.c#L149

Presumably the previous page's web socket is no longer valid and there should have been a LL update or guard here?

I'm testing the OTA upgrade function cgiUploadFirmware. My .bin file size is ~1.3Meg.
The OTA partition size is 4Meg. (16Meg total flash size)

The OTA process starts and seems to repeat mid-way. Then the ESP32 reboots without any warning message.

Then it appears to be running the new OTA image after rebooting, but I'm afraid it didn't complete flashing it.

Is there a size limit to the HTTP POST data? Is there a watchdog timer expiring?
Shouldn't there be some console message indicating that the process completed before reboot?

I (521305) ota: Running partition type 0 subtype 16 (offset 0x00420000)
checkBinHeader: e9 4008 3f400020
I (521305) ota: Writing to partition subtype 17 at offset 0x830000
I (521305) ota: esp_ota_begin succeeded
E (521305) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521305) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521305) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521305) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
E (521315) wifi: esf_buf: t=8 l=1600 max:16, alloc:16 no eb
I (523585) esp_image: segment 0: paddr=0x00830020 vaddr=0x3f400020 size=0x855c0
(546240) map
I (523795) esp_image: segment 1: paddr=0x008b55e8 vaddr=0x3ffb0000 size=0x04ca8
( 19624)
I (523805) esp_image: segment 2: paddr=0x008ba298 vaddr=0x3ffb4ca8 size=0x00000
(     0)
I (523805) esp_image: segment 3: paddr=0x008ba2a0 vaddr=0x40080000 size=0x00400
(  1024)
0x40080000: _iram_start at C:/msys32/home/labview/To
ols/esp-idf/components/freertos/xtensa_vectors.S:1685

I (523805) esp_image: segment 4: paddr=0x008ba6a8 vaddr=0x40080400 size=0x05968
( 22888)
I (523825) esp_image: segment 5: paddr=0x008c0018 vaddr=0x400d0018 size=0x7f6cc
(521932) map
0x400d0018: _stext at ??:?

I (524015) esp_image: segment 6: paddr=0x0093f6ec vaddr=0x40085d68 size=0x0f7f8
( 63480)
0x40085d68: ieee80211_output_process at ??:?

I (524035) esp_image: segment 7: paddr=0x0094eeec vaddr=0x400c0000 size=0x00000
(     0)
I (524035) esp_image: segment 8: paddr=0x0094eef4 vaddr=0x50000000 size=0x00000
(     0)
I (524035) esp_image: segment 0: paddr=0x00830020 vaddr=0x3f400020 size=0x855c0
(546240) map
I (524245) esp_image: segment 1: paddr=0x008b55e8 vaddr=0x3ffb0000 size=0x04ca8
( 19624)
I (524255) esp_image: segment 2: paddr=0x008ba298 vaddr=0x3ffb4ca8 size=0x00000
(     0)
I (524255) esp_image: segment 3: paddr=0x008ba2a0 vaddr=0x40080000 size=0x00400
(  1024)

I (524255) esp_image: segment 4: paddr=0x008ba6a8 vaddr=0x40080400 size=0x05968
( 22888)
I (524265) esp_image: segment 5: paddr=0x008c0018 vaddr=0x400d0018 size=0x7f6cc
(521932) map
0x400d0018: _stext at ??:?

I (524465) esp_image: segment 6: paddr=0x0093f6ec vaddr=0x40085d68 size=0x0f7f8
( 63480)
0x40085d68: ieee80211_output_process at ??:?

I (524495) esp_image: segment 7: paddr=0x0094eeec vaddr=0x400c0000 size=0x00000
(     0)
I (524495) esp_image: segment 8: paddr=0x0094eef4 vaddr=0x50000000 size=0x00000
(     0)
ets Jun  8 2016 00:22:57

rst:0x3 (SW_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)

Also, where are these "I () esp_image: segment x:..." message coming from?

Here's my parition table:

# -------------------------------------------------------
# -    Partition layout generated by BUILD.sh script    -
# -------------------------------------------------------
# Name,         Type, SubType, Offset,  Size,       Flags
# -------------------------------------------------------
nvs,            data, nvs,     0x9000,  16K,
otadata,        data, ota,     0xd000,  8K,
phy_init,       data, phy,     0xf000,  4K,
MicroPython,    app,  factory, 0x10000, 4160K,
MicroPython_1,  app,  ota_0,   ,        4160K,
MicroPython_2,  app,  ota_1,   ,        4160K,
internalfs,     data, fat,     ,        3840K,

I thought that these options might be incorrect, but it seems that most of them are not even used for ESP32?

#define OTA_FLASH_SIZE_K 4096
#define OTA_TAGNAME "generic" // not used in ESP32

CgiUploadFlashDef uploadParams={
	.type=CGIFLASH_TYPE_FW,
	.fw1Pos=0x1000,
	.fw2Pos=((OTA_FLASH_SIZE_K*1024)/2)+0x1000, // not used in ESP32
	.fwSize=((OTA_FLASH_SIZE_K*1024)/2)-0x1000, // not used in ESP32
	.tagName=OTA_TAGNAME
};

Hello,

Have you tested SSL on ESP8266 as well as on ESP32? Does it work, considering SSL's memory/cpu needs for, say, 2-10 simultaneous clients with reasonable connect/read timeouts set?

I'm noticing some 200ms delays in transferring very small http files.

Wireshark shows that the delay appears to happen when the server waits for an ACK from the client before sending the next frame.

The delay seems dependent on the size of the file (for example, a 1kB file takes ~220ms, but a 1.5kB file takes ~8ms)

I think it may have to do with the TCP, specifically Nagle's algorithm. Is it possible to eliminate this delay?
Can we set TCP_NODELAY?

FYI:
There's tons of references to this online for example:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/1.2/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Application_Tuning_and_Deployment-TCP_NODELAY_and_Small_Buffer_Writes.html

(I do see one reference to Nagle's algorithm here: https://dl.espressif.com/doc/esp-idf/latest/api-reference/kconfig.html#tcp-oversize)

I was writing a CGI function which generates quite big responses (>2k). So it utilizes consecutive httpdSend() calls and HTTPD_CGI_MORE return code.
At some point I've stumbled upon corrupted responses. curl says Malformed encoding found in chunked-encoding.

Further digging revealed that I managed to fill conn->priv.sendBuff to the brim (exactly 2048 bytes) before returning from the function with HTTPD_CGI_MORE. That, in turn, caused a problem in httpdFlushSendBuffer(), because there is no room for closing a chunk with \r\n, and chunk length is calculated incorrectly.

I'm going to try to fix this issue myself. If I succeed I'll open a pull request.

Hi, it looks like library will not work on nonos-sdk because methods in httpd-nonos.c is not compatibe with their prototypes. For example, in httpd.c there is a method CallbackStatus ICACHE_FLASH_ATTR httpdDisconCb(HttpdInstance *pInstance, HttpdConnData *pConn) but in httpd-nonos.c it is invoked as httpdDisconCb(conn, (char*)conn->proto.tcp->remote_ip, conn->proto.tcp->remote_port);
Do you plan to support nonos-layer?

Hi,

I am currently playing around with WebSockets in the newest build of the libesphttpd server.

I seems to run fine except from sending continuous frames frames. I have tried out a lot of different ways to get it working and have given up for now, but i will be happy if someone can tell me if there is something wrong with my code.

The following code is sending the frames ok but chrome i keeps saying:
Error: WebSocket connection failed: Received start of new message but previous message is unfinished.

......
void ws_send_frames( WebSocket *m, 	P_NODE p) {
	P_BME_LOG_REC b;
	char message[100];
	
	sprintf( message, "{\"bme280_log\":");
	
	cgiWebsocketSend( &httpdFreertosInstance.httpdInstance, ws, message, strlen( message), WEBSOCK_FLAG_MORE);
	
	P_NODE t = p;
	while(t != NULL) {
		b = (P_BME_LOG_REC)t->data;
		
		sprintf( buffer, "{\"bme280\":{\"ts\":\"%llu\",\"t\":\"%0.1f\",\"p\":\"%0.3f\",\"h\":\"%0.2f\"}}",
				b->timestamp, b->humidity, b->pressure, b->temperature);
				
		cgiWebsocketSend( &httpdFreertosInstance.httpdInstance, ws, message, strlen( message), WEBSOCK_FLAG_NONE);
		
		t = t->next;
	}
	
	cgiWebsocketSend( &httpdFreertosInstance.httpdInstance, ws, message, strlen( message), WEBSOCK_FLAG_NONE);
	
}
......

Cheers - HPSchultz

It would be useful to be able to set both the task priority and core affinity (esp32) when initialising httpd.

cgiflash.c calls
if (esp_image_verify(ESP_IMAGE_VERIFY_SILENT, &part_pos, &data) != ESP_OK) {
line 614.
However, this function doesn't exist for this SDK. It exists as of 3.2 in ESP-IDF.

So modern web-browsers will try to open 6+ simultaneous connections to the webserver. If I have more than 2~3 clients connected to my web App, the server starts dropping connections and the browser hangs for a while.

It seems that if (MAX_CONNECTIONS in libesphttpd) is >= (LWIP_MAX_SOCKETS in menuconfig), then httpd doesn't even know that LWIP is dropping the connections. And LWIP_MAX_SOCKETS has a max value of 16 (why?).
I propose putting a compiler warning if MAX_CONNECTIONS >= LWIP_MAX_SOCKETS.

When MAX_CONNECTIONS = 32, but LWIP_MAX_SOCKETS = 16, I see lots of these errors when connect 4 clients:

E (21552) httpd-freertos: accept failed
accept: Too many open files in system
E (21562) httpd-freertos: accept failed
accept: Too many open files in system

So I decreased MAX_CONNECTIONS to 14. Now when exceeded I see error in console:

I (2260322) httpd-freertos: all 14 connections in use on '0.0.0.0'
E (2260332) httpd: send buf tried to write 1032 bytes, wrote -1
I (2260332) httpd-freertos: listening for new connections on '0.0.0.0'
I (2260342) httpd-freertos: all 14 connections in use on '0.0.0.0'
I (2263762) httpd-freertos: listening for new connections on '0.0.0.0'

But the browser still hangs.

I feel there should be a more graceful way to refuse the connections. Then the browser could use one of it's existing connections instead of hanging waiting for a response on a broken one.

Also, I would like a way to limit the number of connections per client (by IP?) to 2 or 3. That way I can support more active clients. (Actually I need to always allow the websocket connection, but limit the normal http to 1~2).

I found some guidance on what the server should do when it reaches it's connection limit:
https://stackoverflow.com/questions/35186725/how-to-limit-the-number-of-connections-to-a-socket-and-trigger-timeout-on-client

as long as the listening socket is open, connections will be accepted by the OS, even if not being handled by the application. If you want to reject connections after a set number, you have basically two options:

- simply accept and close directly after accepting.
- close the listening socket upon reaching the limit and reopen when done.

Looking a the httpd code,
I see the message coming from https://github.com/chmorgan/libesphttpd/blob/master/core/httpd-freertos.c#L344
But I never see the close happen at https://github.com/chmorgan/libesphttpd/blob/master/core/httpd-freertos.c#L375

Hi Chris,
when I open a second web session to the https server the browser hangs, and the ESP32 disconnects from the Wifi. But it doesn't crash. Since this didn't happened in an older version of libesphttpd, I have the suspicion that this could be a problem in the last version. Stackoverflows were not reported.
Do you have an idea in which corner I should investigate further?
Viele Gruesse
Axel

https://github.com/ataweg/libesphttpd/tree/awe

In libesphttpd/core/httpd.c there is an incorrect mime type for "htm":
there is no such mime type as text/htm, this should be text/html, the same as for "html"

//The mappings from file extensions to mime types. If you need an extra mime type,
//add it here.
static const ICACHE_RODATA_ATTR MimeMap mimeTypes[]={
	{"htm", "text/htm"},
	{"html", "text/html"},

I've just tried to run esphttpd-freertos example of libesphttpd on ESP32. First I go to menuconfig to select esphttpd and uart port etc. Compilation works fine, and server starts without any problems, I can connect to AP, and load default webpage with cats ;)

But, first page loading slow about 20-30seconds, and if I tried to run tests, all tests filed. When I tried to load led.tpl it's page is not responding.

Maybe I miss something in menuconfig ?

Hi,

do you already plan to support he new CMake based toolchain? The current CMakeLists.txt is unfortunately not compatible. Depending on your timeline it is possible I support but I can't promise anything at the moment.

Best regards,

Paul

Folks,

I need to only temporarily start esphttpd, and once it is done - I need to properly shut it down, freeing all allocated resources.

What will be the proper way to do it with the existing code?

Regards

Is there any kind of how-to or step-by-step anywhere on setting this up and installing it on an ESP? I haven't had much luck with the Arduino based captive portals when trying to access them from newer Android and Ios devices and I'm looking for other captive portal servers to try. But I downloaded this, the freeRTOS example and the freeRTOS SDK and I'm not sure how to plug all the pieces together.