josephhewitt / wardriver_rev3 Goto Github PK

With some degree of regularity I am getting "ESP-B NO DATA" messages on the display and no BLE or GSM data from side B. Restarting the Wispy pretty much always clears up the issue.

I have not been able to find a common cause, but it seems more likely to happen when I do not have a network connection for the WiSpy to connect to initially.
It also seems like not getting a quick GPS lock will cause the issue. In these cases I'll see the time displayed as 1970. If I eventually get a GPS lock, the time will update but side B will remain quiet.

I just fired up the unit in my basement and it shows "No GPS: No GSM pos / ESP-B NO DATA". It has a valid time from the network connection. I rebooted, and now have "No GPOS: GSM pos OK / BLE: 31 GSM: 1"

Not sure if there are debugs that could be enabled to see more -- I'm on version 1.1.0, flashed via the Arduino IDE.

When compiling for the B module I get this error using the Arduino IDE.

Sketch uses 1480121 bytes (112%) of program storage space. Maximum is 1310720 bytes.
Global variables use 57020 bytes (17%) of dynamic memory, leaving 270660 bytes for local variables. Maximum is 327680 bytes.
Sketch too big; see https://support.arduino.cc/hc/en-us/articles/360013825179 for tips on reducing it.
text section exceeds available space in board

Compilation error: text section exceeds available space in board

Benchmarks show that each logged WiFi network on side A takes ~21ms for the printf to the file object. The flush afterwards is ~8ms. The times have quite some jitter, but this seems to be about the average.

Worst case I have seen so far is ~40ms to write a single new network then flush. Since the WiFi scan part of the loop is ~1430ms (110ms per channel x 13 channels), we're wasting around 3% of the scan core on flushing out the data -- and that's if only a single new network is found. This slowdown will compound with more new networks found in a single channel scan.

Moving the writing to the other core will definitely see an increase in scan performance, but it may not be able to keep up. This will need very close monitoring after implementing, especially in extremely high WiFi dense areas.

It might be worth creating another xTaskCreatePinnedToCore for write operations which has a higher priority that simply yields when there is nothing to write.

Hi,

This doesn't seem to be covered in the questions, but how is the removal of power and cleanly closing the file handled?

There is no capacitor on board to sustain power to close the file cleanly.
To me it just looks like whatever state we are in at power removal is what we get. In my case, a corrupted SD card after a long war driving session.

Not complaining, I came here to help make it more robust 👍

Enhance the delete option to allow a delete all of w3d-* CSV files, or checkbox to delete selected files.

It is possible to flash A with the B firmware (or vice versa), a warning should be generated when this happens.

• Add LCD code to B and make it display an error on boot. Since B does not normally have an LCD connected, this error will only be seen if ESP A was flashed with the B code.
• Send "A" or "B" over serial to the other ESP; detect if another ESP responds with the same letter. If so, then display a warning indicating that both ESPs are running the same firmware.
• Show a warning if an expected "A" or "B" isn't sent over serial at all, this may indicate the other ESP32 isn't flashed at all or the serial connection isn't working.

This should cover all common wrong-firmware possibilities and will make troubleshooting easier.

When downloading files from the web interface (1.1.0b1) they are saved with the double-extension ".csv.html".
Could that be just ".csv" instead? Minor issue but I end up renaming them all the time.

It would also be nice to have the filename be a timestamp, another element I end up adding to all files. YYYY-MM-DD would be plenty to add, as the file number keeps things unique. YYYY-MM-DD-HH-MM would be another option.

Implement a config file on the SD card which contains all of the information the first-time setup requires. This being in place will allow the first-time setup to be skipped and would make it easier to configure multiple wardrivers identically.

This could also offer more configuration flexibility which would be hard to add nicely to the web interface.

SSIDs containing commas are saved literally and will cause Wigle to read the network information incorrectly.

Currently there is no indication of an SD card failure once everything is running correctly (eg, if the SD card is removed during use). We should check for this and also reset the SD card if needed.

I added a new shell/awk script that looks at *.csv files and identies MAC addresses seen at multiple locations.
It can identify the MAC address of people you come into contact frequently.

It's in my repository clone
https://github.com/grymoire/wardriver_rev3/tree/main/scripts

I was looking at why the time between the WiSpy showing it had 'got one' for an HTTP request to the server and a response on the client is so long. It's minor, but the pages aren't closing the < HTML> tag with a corresponding tag. This might(?) be causing Safari to do a big-think expecting more input. Closing the pages would be be good form and format in any case.

Also for the HTML purist, there could be a < BODY> after < /HEAD> and a < /BODY> before < /HTML>.

It would be useful to display the local time on the LCD instead of UTC.

The quick/easy option is to retrieve the timezone with Javascript whenever the user accesses the interface, but this means it will not update automatically when the user moves.

The better solution is to use something like ZoneDetect which converts coordinates to a TZ, but this adds quite some overhead which is probably overkill for such a small feature.

Detect whether the hardware is rev3 or rev4 automatically.

This might be as simple as reading the Side B standard messages (eg, BLE,) which will not be implemented on rev4 - might be problematic with #23

The data from the additional radios in Rev4 may be delayed, but we know the exact amount of delay. Since this delay can be multiple seconds, we need a way to determine our location x seconds ago. This probably won't ever be more than about 30 seconds.

The LCD information could be improved when the web interface is in use. It would be good to include a countdown indicating how much time is remaining before the device switches to normal wardriving mode.

This isn't a big issue since print_mac is unused, however, any octets starting with 0 aren't printed correctly (the zero isn't printed) meaning the printed MAC address is wrong.

This can be immediatly closed out its more as a note for others.

Sometimes on starting an error is seen "Unexpected Reset Error Code 9" and the Wardriver may hang or be unresponsive.

If you see this error check the Arduino serial monitor on the A ESP. In my case it was caused by the USB power supply not being able to supply enough power and intermittant brownouts causing issues.

Easily solved by using the 2.1A output of my power bank, and it's very obvious if you look at the Serial Output to Debug, but the OLED error itself isnt too helpful.

When configuring a WiFi network to connect to or as a fallback network, special characters can cause issues with the parsing.

Specifically, the HTML form uses GET to transfer the values to the wardriver and the URL encoding isn't handled causing the credentials to be misread.

Switching to POST is probably the best solution which is more suited for the task anyway.

I've used the device several times and noticed some minor inconsistencies in the csv files.
Most of the lines end with \r\n
However - some of the lines just end with \n.

Also some of the MAC addresses use lower case letters while others use upper case letters.
This might cause issue with those that analyze or filter the data.

The web interface calls get_latest_datetime once per CSV file on the SD card to populate the Finish Date HTML column. This function is slow because it parses each file to read the most recent timestamp. The date is set within the file metadata which should be used instead for a significant speed boost to the web interface.

Additional notes

• The date is added to the file metadata by the time library and appears to only happen when NTP is used.
• The date appears to be the creation datetime but the HTML is meant to show when the wardriving session ended (last modification).

Running 1.2.0b1, an invalid checksum is generated for the firmware (#85) and then the device crashes right after online verification during the first phase of install_firmware

When attempting to update side B, the following is printed:

21:38:42.953 -> Firmware install requested: /B.bin82e50cc91daf02044bc6f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2Will install firmware
21:38:43.045 -> Validating firmware
21:39:27.675 -> Contacting OTA server -> /hashes/82e50cc91daf02044bc6f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2.txt
21:39:27.675 -> primary cert
21:39:28.365 -> Guru Meditation Error: Core 1 panic'ed (Unhandled debug exception).
21:39:28.411 -> Debug exception reason: Stack canary watchpoint triggered (loopTask)
21:39:28.411 -> Core 1 register dump:
21:39:28.411 -> PC : 0x400917ab PS : 0x00060336 A0 : 0x800939e0 A1 : 0x3ffb0320
21:39:28.411 -> A2 : 0x3ffb0300 A3 : 0xb33fffff A4 : 0x0000cdcd A5 : 0x00060323
21:39:28.411 -> A6 : 0x00060320 A7 : 0x0000abab A8 : 0xb33fffff A9 : 0xffffffff
21:39:28.411 -> A10 : 0x3ffb0540 A11 : 0x00000011 A12 : 0x00000000 A13 : 0x3ffe0684
21:39:28.411 -> A14 : 0x007b0300 A15 : 0x003fffff SAR : 0x00000019 EXCCAUSE: 0x00000001
21:39:28.411 -> EXCVADDR: 0x00000000 LBEG : 0x40089ed8 LEND : 0x40089ee3 LCOUNT : 0x00000000
21:39:28.456 ->
21:39:28.456 ->
21:39:28.456 -> Backtrace: 0x400917a8:0x3ffb0320 0x400939dd:0x3ffb0360 0x40093c75:0x3ffb0380 0x40083fb6:0x3ffb03a0 0x40084251:0x3ffb03d0 0x40085c9c:0x3ffb03f0 0x401312a5:0x3ffb0410 0x40126f6c:0x3ffb0430 0x401270a8:0x3ffb0450 0x4012782a:0x3ffb0470 0x40127add:0x3ffb0500 0x40127f92:0x3ffb0520 0x4012aa43:0x3ffb05c0 0x4012b351:0x3ffb0630 0x4012bd3b:0x3ffb0820 0x40129d41:0x3ffb0980 0x40129f40:0x3ffb09b0 0x40124a84:0x3ffb09d0 0x401224fd:0x3ffb0a70 0x40122537:0x3ffb0a90 0x400e10b9:0x3ffb0ab0 0x400e0991:0x3ffb0d30 0x400e0a07:0x3ffb0d70 0x400e0a96:0x3ffb0db0 0x400d6169:0x3ffb0dd0 0x400d694a:0x3ffb0eb0 0x400d6cea:0x3ffb0f40 0x400d9e65:0x3ffb1fc0 0x400db3bf:0x3ffb21c0 0x400e5492:0x3ffb2290
21:39:28.502 ->
21:39:28.502 ->
21:39:28.502 ->
21:39:28.502 ->
21:39:28.502 -> ELF file SHA256: 969ebbe44366f8fc
21:39:28.502 ->
21:39:28.688 -> Rebooting...
21:39:28.688 -> ets Jun 8 2016 00:22:57

My dashcam (Road Angel Halo ) generates wifi hotspot to allow you to download data, the hotspot is active all the time the camera is running, so in effect any time the car is running.

Would you consider adding an exclude list where you can enter a few (maybe even just one or two) SSIDs and they are ignored so are not written to the SD card or show in the detected nubmers, My Wardriver never shows zero as even in emptt areas its picking up the dashcam.

I can also see it being an advantage for peeple who have cars that generate a passenger use hotspot (Audi, Tesla, and BMW offer it as a subscription I think), or have a phone in tether mode to provide data in car (say to an Alexa or Chromecast) to play music in car.

The default configuration SSID is set to "wardriver.uk" and is static. This will cause problems if multiple people in close proximity are setting up their Wardrivers (eg, during workshops) at the same time. A few bytes of the MAC address should be appended to the SSID to reduce the possibility of multiple Wardrivers having the same SSID at the same time.

The old working title Portable Wardriver still appears at various places throughout the project. This should be changed to wardriver.uk in the documentation, source code, and UI.

I have a local network with WPA & WPA2 Personal security.
I've setup my wardriver system to join this network.
When it collects information, it always lists the SSID as [OPEN]

Data sourced from OpenCelliD will soon be used by the project, so it would be great to encourage contributions to OpenCelliD in return. Ideally data captured by the wardriver should be compatible with OpenCelliD in some way, most likely by converting the standard CSV to something they can handle.

Users should have the ability to use the SIM800L module for connectivity if desired. Being able to use cellular for automatic data uploads would be very useful for remote unattended Wardriver installations. It should also be used for time/date sync.

I'm happy to implement this feature on my own and will assign the issue to myself, but would like to hear from people regarding how this feature should actually work/look from a user perspective, hence the "help wanted" label.

Things to consider:

• The SIM800L is connected to ESP B but only A has direct access to the data. This means that uploading files will require proxying the data through A which will probably prevent scanning during the transfer.
• The user configuration is stored on ESP A meaning it would need sending to B on boot (and also when/if B restarts). Perhaps it can be stored on B instead somehow?
• Uploading directly to Wigle is a bad idea as the SIM800L generally does not support good HTTPS encryption and Wigle API keys will need to be stored on the Wardriver. Maybe create a basic script which people can run on a server to accept the data via HTTP and forward to Wigle?
• Having a customisable upload interval is very important, especially if the upload will pause the scanning.

The repo doesn't include information about how a normal boot (after first time setup) looks. The wardriver.uk usage instructions does but has some inconsistencies. Specifically, it is not clear that the wardriver will connect to a configured network and then drop without warning after 10 seconds to continue booting; LCD feedback (#33) will help but the documentation needs updating too.

There are references to rev3 throughout the project. Either change these dynamically after hardware detection (#41) or remove the references entirely.

Update for clarity: This issue only refers to the UI/software; not the documentation.

Rev4 will only have a single BLE compatible radio, and that is Side A. Without this, rev4 will not be able to scan for BLE.

This should probably be a checkbox so the user can decide if they want it.

First off, thank you for the great project!! I love this, I have it assembled and working, well sorta. As of right now, the display shows wifi XX, ble XX etc... but there is not test.txt file and on start up all I get is connecting.... / failed to connect. Time does not sync etc.

I have tried multiple AP, with differing names (ie. no special char, no spaces etc) and still no success. I am not sure where to start with the trouble shooting. It is strange, as the first time setup does show the wifi AP etc.... I enter the psk and seems ok. BUT on boot, no connection?? Is there a log I can check? OR ... some suggestion?

Thanks for the time / project, it will be fun once up and running... ;)

Now that binaries are generated automatically, over the air updates are much easier to implement.

It would be best to deliver the binary over HTTPS and use custom certificates (eg, self signed) so that rotation of certificates can be in our control. This would be implemented by having a primary and a standby certificate in the firmware. The server would use the primary until rotation is needed (such as the cert expiring) then the standby would take the place of the primary, a new cert would be generated, and then the new cert would be pushed in a firmware update to replace the expired primary.

This requires us to host the binaries outside of GitHub (in addition to GitHub) so we can have a server with the custom HTTPS certs. This is easy, but it would be nice to have some kind of proof the same binaries are being hosted so people can see that we're not trying to sneak hidden stuff onto the wardrivers.

It would also be nice to have an upload form on the ESP web interface so people can generate (or download) the binaries themselves, do whatever checks they deem necessary, and then upload once they are satisfied.

We also need to find a nice method to update both ESPs. The obvious solution is to allow ESP B to connect to WiFi and run the same OTA code as A. This does mean we need to transfer WiFi credentials from A->B which could introduce a low complexity attack against the device. The firmware binary could be sent over serial from A, but we need to verify integrity of the transfer fully.

C:\Users\Barry\Desktop\wardriver_rev3-1.0.1rc1\wardriver_rev3-1.0.1rc1\A\A.ino:1:10: fatal error: parseUtils.h: No such file or directory
#include <parseUtils.h>
^~~~~~~~~~~~~~
compilation terminated.

exit status 1

Compilation error: parseUtils.h: No such file or directory

The temperature sensor is not working on many/all builds since 1.0.1

I have a 32GB newly formatted Samsung SD card (using the Microsoft formatter) and a Micro SD Storage Expansion Board Mini Micro-SD TF Card Memory Shield Module With Pins for Arduino ARM AVR. But no SD card is found. - SD.begin() fails.I'm trying to get further into the debugging but I wondered if you can give me a few tips if you have a;

Please install version 2.0.0 (at the time of writing, later versions have not been tested but might work).

Just confirming that you definitely want to be on the 2.0.0 version of the esp32 board version. I was on the 2.0.3 version for another project and the B board code was failing to compile due to some onewire issues. Downgrading to the 2.0.0 version and trying again worked fine.

When rev4 is detected (#41), create a loop which can address each radio and change their channel if necessary. We need to find a nice way to reduce hopping but maximise coverage across the 10 radios.

I recently completed the hackerbox 089 project and it is now working fine. I can remove the sd card and read captured data with no problems. I can also attach to the wardriver.uk SSID and setup various networks I have in the house and after powering up the display shows the IP it received from DHCP but will drop the connection after about 8 seconds. I checked power, different USB cables, different networks for main and backup with the same results. During the short time it connects I cannot bring up a web browser with the displayed IP either.

Brian.

Could the auto update mechanism be used to flag when the Cells data has an update? Track the current version on the OTA site and post a message to the screen during the boot cycle or on the web page when a new version is available.

I understand it is likely not practical to update the cells data OTA, but leveraging the version check to alert users to a new version would be valuable to keeping the GSM data current.

Add some mechanism (e.g. button, switch) for switching between captive portal and scanning mode without requiring to reboot the device. This also allows for the device to automatically enter in scanning mode on boot, and the user can change to captive when needed (e.g. with a two-state switch).

To ensure messages from B to A are not corrupt, implement error detection/correction.

Either auto detect the data structure, or use the hardware ID from #41 to correctly parse the lines received.

It might be worthwhile to log the radio ID received by each radio in the cluster to determine their individual performance, but there is currently no nice way to present that information.

Hi,

I'd like to find out if you've measured the power draw on this setup? I'd like to buy a powerbank that would power it for 12 hours along with the other components.

Now that mine is boxed up, I really don't want to have to take it apart to clean up the SD card. Having the option to Delete selected records through the web UI would be a great quality of life upgrade.

When running 1.2.0b1, the current latest B.bin with checksum 82e50cc91daf02044bc60f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2 is being generated as 82e50cc91daf02044bc6f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2 on device.

For side-by-side comparison:

82e50cc91daf02044bc60f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2
82e50cc91daf02044bc6f1d3da8d5c7fef12d6cbf0dffc697cb33e29921a8f2

A zero is missing in the ESP32-generated hash.

This causes verifications to fail. Since the current default is to reject firmware updates with mismatched checksums, it won't be possible to update the device OTA if the new update triggers this issue. The workaround (untested) for this is to set the advanced config enforce_checksums to false.

Not sure if it would be more efficient than the Open Cell ID data, but have you looked at Wigle.net's Magic 8 ball positioning data? Would be interesting to use the data we contribute to back into the system.

https://github.com/wiglenet/m8b

"Side B" will stop sending the "BLE," (Bluetooth device count) messages to "side A" in Bluetooth dense areas. I observed this happening when the device count was in the region of 145 - 165 or higher. Other data (GSM) was still sent normally from side B.

I haven't yet checked the logs or collected data to determine if this issue causes data loss.

This issue manifests as a stuck BLE counter on the LCD. Rebooting the wardriver will cause the "ESP-B NO DATA" warning as this warning is triggered by a lack of "BLE," messages on boot. Moving away from the Bluetooth-dense area fixes this issue.

I'm pretty new to these boards (and microcontrollers in general)... I'm using a hardware version from Hacker Boxes and the issue started with Side A only (v1.1.0rc). I noticed the screen was black and tried power cycling, no luck. I also checked the SD Card and no data was written to it. I've been using the project for several weeks now (over 171k new additions to Wigle since getting this working).

Originally using Firmware v1.1.0rc. When connecting to a PC, Side B works fine and I can see the serial monitor feeding out scanned data. If I take the exact same cable and plug it into side A... The COM port just cycles connected/disconnected. Putting it into Download mode... I was able to upload a basic sketch to confirm the board isn't dead.

void setup() {
Serial.begin(115200);
Serial.println("Starting");
}
void loop() {
Serial.println("Running...");
}

But, if I upload Side A (v1.1.0) or Side B to the same board, it goes back to cycling the COM port. I now have the board sitting on the bench with nothing attached to it. So I'm not sure if its software crashing something, or the board died.

test.txt, the last few entries were since updating to 1.1.0rc:
_BOOT_1.1.0rc1, ut=2266413, rr=1, id=4003788, bc=86, ep=1685629906
_BOOT_1.1.0rc1, ut=2443699, rr=1, id=4003788, bc=87, ep=0
_BOOT_1.1.0rc1, ut=2377789, rr=1, id=4003788, bc=88, ep=1685630756
_BOOT_1.1.0rc1, ut=2269792, rr=1, id=4003788, bc=89, ep=1685630974
_BOOT_1.1.0rc1, ut=2394085, rr=1, id=4003788, bc=90, ep=1685637787
_BOOT_1.1.0rc1, ut=2427782, rr=1, id=4003788, bc=91, ep=1685661791
_BOOT_1.1.0rc1, ut=2364740, rr=1, id=4003788, bc=92, ep=1685740600
_BOOT_1.1.0rc1, ut=2356746, rr=1, id=4003788, bc=93, ep=0
_BOOT_1.1.0rc1, ut=2393182, rr=1, id=4003788, bc=94, ep=0
_BOOT_1.1.0rc1, ut=2437381, rr=1, id=4003788, bc=95, ep=0
_BOOT_1.1.0rc1, ut=2434632, rr=1, id=4003788, bc=96, ep=0
_BOOT_1.1.0rc1, ut=2421691, rr=1, id=4003788, bc=97, ep=0
_BOOT_1.1.0rc1, ut=2389497, rr=1, id=4003788, bc=98, ep=0
_BOOT_1.1.0rc1, ut=2372779, rr=1, id=4003788, bc=99, ep=0
_BOOT_1.1.0rc1, ut=2432408, rr=1, id=4003788, bc=100, ep=1686098241
_BOOT_1.1.0rc1, ut=2352215, rr=1, id=4003788, bc=101, ep=1686185254
_BOOT_1.1.0rc1, ut=2436653, rr=1, id=4003788, bc=102, ep=1686270357
_BOOT_1.1.0rc1, ut=2419991, rr=1, id=4003788, bc=103, ep=1686337353
_BOOT_1.1.0rc1, ut=2438708, rr=1, id=4003788, bc=104, ep=0
_BOOT_1.1.0rc1, ut=2414774, rr=1, id=4003788, bc=105, ep=0
_BOOT_1.1.0rc1, ut=2430203, rr=1, id=4003788, bc=106, ep=0
_BOOT_1.1.0rc1, ut=2393580, rr=1, id=4003788
_BOOT_1.1.0rc1, ut=2405253, rr=1, id=4003788, bc=108, ep=0
_BOOT_1.1.0rc1, ut=2383849, rr=1, id=4003788, bc=109, ep=1686704961
_BOOT_1.1.0rc1, ut=6397698, rr=4, id=4003788, bc=110, ep=1686705037
_BOOT_1.1.0rc1, ut=2396251, rr=1, id=4003788, bc=111, ep=1686791300
_BOOT_1.1.0rc1, ut=2385380, rr=1, id=4003788, bc=112, ep=1686876561
_BOOT_1.1.0rc1, ut=2432352, rr=1, id=4003788, bc=113, ep=0
_BOOT_1.1.0rc1, ut=2403651, rr=1, id=4003788, bc=114, ep=0
_BOOT_1.1.0rc1, ut=2460080, rr=1, id=4003788
_BOOT_1.1.0rc1, ut=6447069, rr=7, id=4003788

When updated to 1.1.0, no log is ever created. I also don't see the screen still or the GPS powering up (blinking light) and cycling the COM port.

Having a USB power meter connected, seems to never pull more that 0.04A with just the board. The entire project (both boards, gos, gsm, sd card, and with an added small fan) pulls 0.35A

The GSM core loop2 on B.ino has a logic flaw which could cause an infinite loop, blocking SIM800L communication.

This does not affect the other core which does Bluetooth and WiFi scanning.

The following code block is affected:

int i = 0;
while (i < 120){
        if (Serial2.available()){
          char c = Serial2.read();
          if (c == '\n'){
            Serial.print("Read from SIM: ");
            Serial.print(i);
            Serial.println(" bytes");
            linebuf[i] = '\0';
            break;
          }
          linebuf[i] = c;
          i++;
        }
      }

This loop is meant to run 120 times, to read up to 120 bytes into a buffer from the SIM800L. The loop is broken when the SIM800L sends a \n or when 120 bytes are received.

The issue arises due to the fact that i only increments when the SIM800L sends a byte. If the SIM800L stops sending data and does not send a \n, this code gets stuck in an infinite loop. Nothing will prompt the SIM800L to send more data because this loop blocks future messages from being sent.

This bug affects all releases and manifests as a stuck GSM counter, no GSM data being logged, and GSM tower lookups failing but no warnings are generated.