Enable camera has big impact on networking about esp32-camera HOT 22 OPEN

@me-no-dev

Do you have any idea to debug such issue?
It does not show error, the device just becomes slow response after init camera.
Sometimes, I even cannot join the device's AP.
e.g.
I (197303) wifi:removing station <20:1e:88:5e:2b:3f> after unsuccessful auth/assoc, AID = 0
W (197304) wifi:rm mis
I (197304) wifi:new:<6,0>, old:<6,0>, ap:<6,0>, sta:<0,0>, prof:6
I (255407) wifi:removing station <20:1e:88:5e:2b:3f> after unsuccessful auth/assoc, AID = 0
W (255408) wifi:rm mis
I (255408) wifi:new:<6,0>, old:<6,0>, ap:<6,0>, sta:<0,0>, prof:6

I tried move cam_task to core1, but the behavior is not consistent.
It looks better sometimes, but sometimes still very slow.

Or is there any configuration in menuconfig I should try?

from esp32-camera.

You can try to lower the priority, but then you might miss the window to handle some DMA buffers from the camera and images could break. What is the camera config when testing this?

from esp32-camera.

You can try to lower the priority, but then you might miss the window to handle some DMA buffers from the camera and images could break. What is the camera config when testing this?

Which task's priority?
cam_task priority is hard-coded.
In additional, I don't know why the cam_task uses such high priority.

Below is my config:

static camera_config_t camera_config = {
    .pin_pwdn = CAM_PIN_PWDN,
    .pin_reset = CAM_PIN_RESET,
    .pin_xclk = CAM_PIN_XCLK,
    .pin_sccb_sda = CAM_PIN_SIOD,
    .pin_sccb_scl = CAM_PIN_SIOC,

    .pin_d7 = CAM_PIN_D7,
    .pin_d6 = CAM_PIN_D6,
    .pin_d5 = CAM_PIN_D5,
    .pin_d4 = CAM_PIN_D4,
    .pin_d3 = CAM_PIN_D3,
    .pin_d2 = CAM_PIN_D2,
    .pin_d1 = CAM_PIN_D1,
    .pin_d0 = CAM_PIN_D0,
    .pin_vsync = CAM_PIN_VSYNC,
    .pin_href = CAM_PIN_HREF,
    .pin_pclk = CAM_PIN_PCLK,

    .xclk_freq_hz = 20000000,
    .ledc_timer = LEDC_TIMER_0,
    .ledc_channel = LEDC_CHANNEL_0,

    .pixel_format = PIXFORMAT_JPEG,
    .frame_size = FRAMESIZE_SVGA,
    .jpeg_quality = 6,
    .fb_count = 1,
    .grab_mode = CAMERA_GRAB_WHEN_EMPTY,
};

from esp32-camera.

You do have access to the source, so you can change that hardcoded priority and see if it helps the network tasks. Priority is high, because the camera is constantly pushing bytes to the ESP, it's not the ESP asking for data, therefore if data is not handled on time, the image will be broken.

CAMERA_GRAB_WHEN_EMPTY should cause the task to sleep when images are not consumed

from esp32-camera.

You do have access to the source, so you can change that hardcoded priority and see if it helps the network tasks. Priority is

Lower the cam_task priority does not help too much.
There must be other issues causes the networking problem. (ping unreachable after join the esp32s3 soft-ap).

from esp32-camera.

the driver itself is just that task and hardware+DMA. It's not expected to cause such issues

from esp32-camera.

1. When join esp32s3 (AP) failure:
   // My laptop dmesg shows below log:
   [10167.095196] wlp0s20f3: authenticate with dc:da:0c:7c:d8:c1
   [10167.095223] wlp0s20f3: 80 MHz not supported, disabling VHT
   [10167.101847] wlp0s20f3: send auth to dc:da:0c:7c:d8:c1 (try 1/3)
   [10167.231054] wlp0s20f3: send auth to dc:da:0c:7c:d8:c1 (try 2/3)
   [10167.332200] wlp0s20f3: send auth to dc:da:0c:7c:d8:c1 (try 3/3)
   [10167.436376] wlp0s20f3: authentication with dc:da:0c:7c:d8:c1 timed out

// esp32s3 shows below messages
I (47537) wifi:removing station <20:1e:88:5e:2b:3f> after unsuccessful auth/assoc, AID = 0
W (47538) wifi:rm mis
I (47538) wifi:new:<10,0>, old:<10,0>, ap:<10,0>, sta:<0,0>, prof:10
I (95100) wifi:removing station <20:1e:88:5e:2b:3f> after unsuccessful auth/assoc, AID = 0
W (95100) wifi:rm mis
I (95101) wifi:new:<10,0>, old:<10,0>, ap:<10,0>, sta:<0,0>, prof:10

2. When join success:

// My laptop dmesg shows below log:
[10446.188896] wlp0s20f3: authenticate with dc:da:0c:7c:d8:c1
[10446.188930] wlp0s20f3: 80 MHz not supported, disabling VHT
[10446.193459] wlp0s20f3: send auth to dc:da:0c:7c:d8:c1 (try 1/3)
[10446.266845] wlp0s20f3: authenticated
[10446.271430] wlp0s20f3: associate with dc:da:0c:7c:d8:c1 (try 1/3)
[10446.375290] wlp0s20f3: associate with dc:da:0c:7c:d8:c1 (try 2/3)
[10446.479294] wlp0s20f3: associate with dc:da:0c:7c:d8:c1 (try 3/3)
[10446.480157] wlp0s20f3: RX AssocResp from dc:da:0c:7c:d8:c1 (capab=0x431 status=0 aid=1)
[10446.485983] wlp0s20f3: associated

// esp32s3 shows below messages

D (25972) wifi:ap recv assoc/reassoc request
I (25973) wifi:new:<10,0>, old:<10,0>, ap:<10,0>, sta:<0,0>, prof:10
D (25973) wifi:bss=0x3c1484c4, resp=16 join
I (25975) wifi:station: 20:1e:88:5e:2b:3f join, AID=1, bgn, 20
D (25981) wpa:   New STA
D (25983) wpa: add sm, index=0 bitmap=1
D (25987) wpa: init wpa sm=0x3fcd3ee4
D (25992) wifi:join success bss=0x3c1484c4, ap send assoc response
D (25997) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state INITIALIZE

D (26004) wpa: wpa_auth_set_key : key idx  0 alg 0 vlan_id 0 key_len 0 key
D (26011) wpa: ELOOP: wpa_remove_ptk:1252 called to remove timer handler=0x42048998, removed count=0
0x42048998: wpa_rekey_ptk at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:262

D (26020) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK_GROUP entering state IDLE

D (26027) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state AUTHENTICATION

D (26034) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state AUTHENTICATION2

D (26041) wpa: WPA: Re-initialize GMK/Counter on first station
D (26047) wpa: GMK - hexdump(len=32):
D (26051) wpa: a6 28 8f 60 a9 2e ca bf 5c a8 9f 99 6a a7 92 03 
D (26057) wpa: 1b da 78 ba 35 c3 71 16 56 af 5a 68 aa ef d1 62 
D (26064) wpa: Key Counter - hexdump(len=32):
D (26068) wpa: 2c 75 91 51 84 8a 15 6f ef 33 81 6b e5 e0 bf 76 
D (26073) wpa: 8d a2 2e 5c 1f 19 c0 91 94 ad 3c 2d f6 d7 7f 93 
D (26080) wpa: GTK - hexdump(len=16):
D (26083) wpa: 29 78 29 81 f7 61 d1 25 c7 68 3d 16 a6 92 86 66 
D (26090) wpa: IGTK - hexdump(len=16):
D (26093) wpa: 74 a6 8d b6 5b fc b7 f6 33 48 6c eb 54 a9 5a 5d 
D (26099) wpa: wpa_auth_set_key : key idx  1 alg 3 vlan_id 0 key_len 16 key
D (26106) wpa: wpa_auth_set_key : igtk idx 4
D (26110) wpa: WPA: Assign ANonce - hexdump(len=32):
D (26115) wpa: 26 15 26 19 22 74 39 c0 b3 76 11 2f 8d bc 15 0e 
D (26121) wpa: e6 bb 94 36 60 b7 c1 11 0a 04 0e 1b 7a 26 e7 44 
D (26127) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state INITPSK

D (26134) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state PTKSTART

D (26140) wpa: wpa_auth=0x3fccfddc sm=0x3fcd3ee4  kdersc=0x0 kde=0x0 nounce=0x3fcd3f34 kde_len=0 keyidx=0 encr=0 force=0
D (26152) wpa: WPA: Send EAPOL(version=2 secure=0 mic=0 ack=1 install=0 pairwise=1 kde_len=0 keyidx=0 encr=0)
D (26162) wpa: ELOOP: wpa_send_eapol:1212 called to remove timer handler=0x42046fe0, removed count=0
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26171) wpa: ELOOP: Added one timer from wpa_send_eapol:1213 to call 0x42046fe0, current order=0
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26180) wifi:ap recv assoc/reassoc request
D (26184) wifi:Ignore immediate assoc req as processing other assoc req for 20:1e:88:5e:2b:3f
D (26192) wifi:ap recv assoc/reassoc request
D (26196) wifi:Ignore immediate assoc req as processing other assoc req for 20:1e:88:5e:2b:3f
D (26232) wpa: WPA: Received EAPOL-Key from 20:1e:88:5e:2b:3f key_info=0x10a type=2 key_data_length=28

D (26233) wpa: WPA: Received Key Nonce - hexdump(len=32):
D (26236) wpa: df 23 89 a0 aa 8b 58 9e 2c 2f 5a 10 61 cc 4c 29 
D (26242) wpa: 0e 63 0f f4 b6 be 79 4d 48 a2 21 ce ba 5f d6 3e 
D (26248) wpa: WPA: Received Replay Counter - hexdump(len=8):
D (26254) wpa: 00 00 00 00 00 00 00 01 
D (26258) wpa: wpa_rx: free eapol=0x0
D (26262) wpa: wpa_rx: new eapol=0x3fcd40d8
D (26266) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state PTKCALCNEGOTIATING

D (26273) wpa: wpa psk
D (26277) wpa: WPA: PTK derivation - A1=dc:da:0c:7c:d8:c1 A2=20:1e:88:5e:2b:3f

D (26284) wpa: WPA: PTK - hexdump(len=48):
D (26288) wpa: 1b ed 27 bf 9f 80 7c 1a 51 8c 43 ad b6 49 d5 b0 
D (26294) wpa: ca 69 9d 48 70 5a c2 00 df 5b 42 7e 60 3a dc 17 
D (26300) wpa: bb be 83 8f c8 f4 14 d0 ab bd b7 a2 2f c6 b0 aa 
D (26306) wpa: WPA: KCK - hexdump(len=16):
D (26310) wpa: 1b ed 27 bf 9f 80 7c 1a 51 8c 43 ad b6 49 d5 b0 
D (26316) wpa: WPA: KEK - hexdump(len=16):
D (26320) wpa: ca 69 9d 48 70 5a c2 00 df 5b 42 7e 60 3a dc 17 
D (26326) wpa: WPA: TK - hexdump(len=16):
D (26330) wpa: bb be 83 8f c8 f4 14 d0 ab bd b7 a2 2f c6 b0 aa 
D (26336) wpa: ELOOP: sm_WPA_PTK_PTKCALCNEGOTIATING_Enter:1724 called to remove timer handler=0x42046fe0, removed count=1
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26347) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state PTKCALCNEGOTIATING2

D (26355) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state PTKINITNEGOTIATING

D (26362) wpa: wpa_auth=0x3fccfddc sm=0x3fcd3ee4  kdersc=0x3fcbc2c0 kde=0x3fcd415c nounce=0x3fcd3f34 kde_len=82 keyidx=1 encr=1 force=0
D (26374) wpa: WPA: Send EAPOL(version=2 secure=1 mic=1 ack=1 install=1 pairwise=1 kde_len=82 keyidx=1 encr=1)
D (26385) wpa: Plaintext EAPOL-Key Key Data - hexdump(len=96):
D (26391) wpa: 30 1a 01 00 00 0f ac 04 01 00 00 0f ac 04 01 00 
D (26396) wpa: 00 0f ac 02 80 00 00 00 00 0f ac 06 dd 16 00 0f 
D (26402) wpa: ac 01 01 00 29 78 29 81 f7 61 d1 25 c7 68 3d 16 
D (26408) wpa: a6 92 86 66 dd 1c 00 0f ac 09 04 00 00 00 00 00 
D (26414) wpa: 00 00 74 a6 8d b6 5b fc b7 f6 33 48 6c eb 54 a9 
D (26420) wpa: 5a 5d dd 00 00 00 00 00 00 00 00 00 00 00 00 00 
D (26427) wpa: ELOOP: wpa_send_eapol:1212 called to remove timer handler=0x42046fe0, removed count=0
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26436) wpa: ELOOP: Added one timer from wpa_send_eapol:1213 to call 0x42046fe0, current order=0
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26525) wpa: WPA: Received EAPOL-Key from 20:1e:88:5e:2b:3f key_info=0x30a type=2 key_data_length=0

D (26525) wpa: WPA: Received Key Nonce - hexdump(len=32):
D (26529) wpa: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
D (26535) wpa: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
D (26541) wpa: WPA: Received Replay Counter - hexdump(len=8):
D (26547) wpa: 00 00 00 00 00 00 00 02 
D (26551) wpa: ELOOP: wpa_receive:874 called to remove timer handler=0x42046fe0, removed count=1
0x42046fe0: resend_eapol_handle at /home/axel/esp/esp-idf/components/wpa_supplicant/src/ap/wpa_auth.c:1184

D (26560) wpa: wpa_rx: free eapol=0x3fcd40d8
D (26564) wpa: wpa_rx: new eapol=0x3fcd40d8
D (26568) wpa: WPA: 20:1e:88:5e:2b:3f WPA_PTK entering state PTKINITDONE

D (26575) wpa: wpa_auth_set_key : key idx  0 alg 3 vlan_id 0 key_len 16 key
D (26583) event: running post WIFI_EVENT:14 with handler 0x42011420 and context 0x3fcb9f04 on loop 0x3fcb35e4

I (26592) net_utils: station 20:1e:88:5e:2b:3f join, AID=1

D (26613) esp_netif_lwip: esp_netif_dhcps_cb esp_netif:0x3fcb9b3c
D (26612) wifi:I (26619) esp_netif_lwip: DHCP server assigned IP to a client, IP is: 192.168.10.2
eb is dhcp or dns sport = 68, dport = 67D (26628) esp_netif_lwip: Client's MAC: 20:1e:88:5e:2b:3f

D (26638) wifi:eb is dhcp or dns sport = 67, dport = 68
D (26643) event: running post IP_EVENT:2 with handler 0x42011c6c and context 0x3fcb9f38 on loop 0x3fcb35e4

I (26652) net_utils: STA IP assigned:192.168.10.2

I (27685) wifi:<ba-add>idx:2 (ifx:1, 20:1e:88:5e:2b:3f), tid:0, ssn:22, winSize:64

3. When camera is not init, I can always join the esp32s3's AP

from esp32-camera.

the driver itself is just that task and hardware+DMA. It's not expected to cause such issues

(my esp32s3 device does: join AP, start camera to take a photo, upload to http server)

I test by checking the device's ping response time.
I have confirmed that after calling ll_cam_config() the ping got much longer response time.
deinit camera does not make the ping response time recovery.

If you test when the AP is not strong (e.g. -6X), you will see big latency or no response after ll_cam_config() is called.
(And I can observe sometimes upload picture failure)
I'm not sure if this is S3 h/w limitation (ESP32 does not has such issue), anyway, just for your reference.

BTW, I can observe the long ping latency even when camera_probe() fail.
So you can test it without camera attached.

from esp32-camera.

@me-no-dev

While investigating this issue (enable camera has impact on networking even after deinit camera),
I notice the PERIPH_LCD_CAM_MODULE is still enabled after esp_camera_deinit().

So I think the esp_camera_deinit() should disable PERIPH_LCD_CAM_MODULE.

diff --git a/target/esp32s3/ll_cam.c b/target/esp32s3/ll_cam.c
index c618469..ebd88f0 100644
--- a/target/esp32s3/ll_cam.c
+++ b/target/esp32s3/ll_cam.c
@@ -193,6 +193,11 @@ esp_err_t ll_cam_deinit(cam_obj_t *cam)
     LCD_CAM.cam_ctrl1.cam_start = 0;
     LCD_CAM.cam_ctrl1.cam_reset = 1;
     LCD_CAM.cam_ctrl1.cam_reset = 0;
+
+    if (periph_ll_periph_enabled(PERIPH_LCD_CAM_MODULE)) {
+        periph_ll_disable_clk_set_rst(PERIPH_LCD_CAM_MODULE);
+    }
+
     return ESP_OK;
 }

Could you take a look? Maybe this needs fix. Thanks.

from esp32-camera.

This is left on, because you might be using the LCD part of the peripheral. Similar to the IO Interrupts, we do not want to just turn off the whole peripheral, so the clock stays on

from esp32-camera.

This is left on, because you might be using the LCD part of the peripheral. Similar to the IO Interrupts, we do not want to just turn off the whole peripheral, so the clock stays on

ok, then I will disable it in my application code since I don't use LCD.
And the question now is why simply keep PERIPH_LCD_CAM_MODULE enabled can impact on networking.
(After esp_camera_deinit() I don't use the camera any more, but I can still observe the networking is slow if PERIPH_LCD_CAM_MODULE is still enabled.)

from esp32-camera.

Let me know if network becomes better once you disable the module

from esp32-camera.

Let me know if network becomes better once you disable the module

Yes, it's actually much better no matter in STA or AP mode.
In STA mode, once the module is disabled, the networking behavior is similar to esp32 (which is ok).
When device in AP mode with camera enabled, sometimes it's very difficult to join the AP.

PS. Our hardware team measure the camera input clock.
on esp32, the input clock is stopped after deinit, the esp32s3's input clock is still on.

from esp32-camera.

BTW,
The comment in target/esp32s3/ll_cam.c ll_cam_config() is wrong.
LCD_CAM.cam_ctrl.cam_clk_sel = 3;//Select Camera module source clock. 0: no clock. 1: APLL. 2: CLK160. 3: no clock.

from esp32-camera.

On ESP32 and S2 the clock is provided by LEDC, on S3 the peripheral generates the clock. Maybe can be turned off in deinit

from esp32-camera.

@me-no-dev

For the "Enable camera has big impact on networking" issue,
I have confirmed this issue is realted to .xclk_freq_hz setting. ( interference issue?)
This is true for both esp32 and esp32s3.

Below symptom is exactly I observed:
espressif/arduino-esp32#5834
espressif/arduino-esp32#5834 (comment)

Is there any guideline to avoid such issue?
a. from s/w, any guideline to set a good .xclk_freq_hz setting.
currently, I did try-and-error to see the impact of different settings.
b. from h/w, is there any guideline to avoid such issue?
e.g. https://www.reddit.com/r/esp32/comments/r9g5jc/fixing_ymmv_the_poor_frame_rate_on_the_esp32cam/

from esp32-camera.

I found a similar issue on esp32-cam and I can repro with the 4 units I have, I believe this is caused by crosstalk because:

• lowering the camera clock freq reduces the problem
• if you put your finger in certain places (causing a ground plane that shields the antenna/camera from crosstalk) the problem goes away.

The solution was to use an external antenna, now all modules work consistently fine.

I am surprised maintainers haven't noticed this issue given we have records of so many cases, moreover they keep giving random advice making people waste time.

This is overall a flaky product and that is why espressif chips cost $5, what you save in one hand you lose it with a buggy platform(1600 open bugs) and unreliable support, add to that the issues some boards have (like the esp32-cam) .

Pay a bit more and get raspberry pi stuff, no more headaches.

from esp32-camera.

I am surprised maintainers haven't noticed this issue given we have records of so many cases

We don't have a record of so many cases. AMF when I test camera, I always test it with WiFi and WebServer running and have not noticed any issues ever. S3 works the best from them all, but all 3 work fine on my end. It is very much possible that such things get influenced by other changes in IDF through time and I just did not use the particular version... If the hardware misbehaves because of crosstalk, that would need to be really proven in order to even go through the process of changing the hardware (chip internals).

the issues some boards have (like the esp32-cam)

This is by far the works product that have ever been released with ESP32 on-board. Custom module (not made by us), bad design and routing, causing countless issues.

I can also add the fact that OmniVision have all of the supported sensors EOL and do no longer produce sensors that can work with the ESP32's parallel bus. All new sensors need MIPI. That all causes lots of bad sensors to be sold out in the wild with zero chance of support from the manufacturer. We can't even get the full registers definitions anymore

from esp32-camera.

It is very much possible that such things get influenced by other changes in IDF through time and I just did not use the particular version... If the hardware misbehaves because of crosstalk, that would need to be really proven in order to even go through the process of changing the hardware (chip internals).

@me-no-dev @igrr
I'm not sure if the espressif IDF guys will check the issues here, can you CC this issue to related people?

from esp32-camera.

We don't have a record of so many cases. AMF when I test camera, I always test it with WiFi and WebServer running and have not noticed any issues ever. S3 works the best from them all, but all 3 work fine on my end.

The issue of "slow frame rate on esp32-cam devices" is pretty much ubiquitous.

It is very much possible that such things get influenced by other changes in IDF through time and I
just did not use the particular version...

I think this issue has lasted for looong time so this discards a possible IDF issue.

If the hardware misbehaves because of crosstalk, that would need to be really proven
in order to even go through the process of changing the hardware (chip internals).

I meant the boards, not the chips.

This is by far the works product that have ever been released with ESP32 on-board.
Custom module (not made by us), bad design and routing, causing countless issues.

You should have spoken up and call-out bad buggy boards that don't follow your best practices. Otherwise you end up with hundreds of fustrated customers, who knows how many sales you have lost.

Another way to deal with this is by setting up a certification system where Espressif gives a seal of approval to HW boards that meet all the good practices and work as promised. (this is a common practice done by many HW companies)

from esp32-camera.

The issue of "slow frame rate on esp32-cam devices" is pretty much ubiquitous.

I've been pretty much able to get the max frame rates that the sensors provide over HTTP (up to other limits, like 8-10Mbps throughput). That is on Espressif made camera development boards.

Another way to deal with this is by setting up a certification system where Espressif gives a seal of approval to HW boards that meet all the good practices and work as promised.

Since the ESP32-CAM is a third-party, maker oriented board, I do not see how this could be done. Don't forget that sensors are EOL, so even of the hardware is good, the sensor might be from a bad batch and total trash.

from esp32-camera.

I've been pretty much able to get the max frame rates [...]. That is on Espressif made camera development boards

The ultimate goal should be to make your customers life easy.

The repo is called ep32-cam, the same name as the famous board. If you are using a different board to repro customers issues then that is not a real repro and, it's helping no one plus it's making everyone's life harder.

Another way to deal with this is by setting up a certification system
Since the ESP32-CAM is a third-party, maker oriented board, I do not see how this could be done.

Your marketing team should know how to do it, this is a very well known practice.

from esp32-camera.