Giter Club home page Giter Club logo

find-lf's Introduction

find-lf

This is a extension of FIND, the Framework for Internal Navigation and Discovery, which is based on the idea of Lucius Fox's sonar system in The Dark Knight that is used to track cellphones.

The system uses a network of Raspberry Pis which sniff the WiFi probe requests from WiFi-enabled devices and sends these parcels to a central server which compiles and forwards the fingerprint to the FIND server which then uses machine learning to classify the location based on the unique WiFi fingerprints.

This system does not require being logged into a particular WiFi - it will track any phone/device with WiFi enabled! (Caveat: for iOS devices it will only track if Wi-Fi is associated with a network - any network, though - because of MAC spoofing it uses for security). This system also does not require installing any apps on a phone. Tracking occurs anytime a WiFi chip makes a probe request (which is every minute or so). For this to work, it requires a one-time setup to populate the system with known fingerprints of known locations before it can pinpoint locations (see #3 below).

Note: It may be illegal to monitor networks for MAC addresses, especially on networks that you do not own. Please check your country's laws (for US Section 18 U.S. Code § 2511) - discussion.

How does it work?

You can setup Raspberry Pis around a house in a way that they will see different signal strengths from various WiFi devices. Each WiFi-enabled device see requests from the various Raspberry Pis with different strengths (see colored bars in Figure below). These graphs are a fingerprint which can be used to uniquely classify the location.

The Raspberry Pi cluster gets probe requests to various WiFi devices, and compiles these and forwards them to a FIND-LF server. This FIND-LF server then sends a compiled fingerprint to a main server. This system uses WiFi probe requests, which occur on any WiFi enabled device, as long as WiFi is enabled. These probe requests generally occur when a scan takes place, which is every few minutes when the phone is being used.

Once sent to the FIND server, the location can be directly determined. For more information about this, see the FIND README.

Requirements

You will need a Raspberry Pi. Ones with built-in Wifi work best:

You will need a monitor-mode enabled wifi USB adapter. There are a number of possible USB WiFi adapters that support monitor mode. Here's a list that are popular:

Namely you want to find a USB adapter with one of the following chipsets: Atheros AR9271, Ralink RT3070, Ralink RT3572, or Ralink RT5572.

Setup

1. Initialize Pis

Install Raspbian lite onto a Pi. If you don't use the pi user, make sure to give the user sudo acess. Then initialize the Raspberry Pi with the following script

sudo sh -c "$(curl -fsSL https://raw.githubusercontent.com/schollz/find-lf/master/node/initialize.sh)"

Alternatively, you can do this using my script for PiBakery.

Also edit /etc/network/interfaces and remove the wpa-conf line for wlan1.

Do this for several Pis and then plug in the WiFi adapter that has "monitor" mode.

2. Start Pi cluster

First make sure you have SSH installed and nmap.

sudo apt-get install openssh-client openssh-server nmap

Its useful to add your SSH key to each Pi, which you can do just using:

ssh-keygen                                               # do this once
cat ~/.ssh/id_rsa.pub | ssh pi@1YOURADDRESS \
  "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"     # do for every Pi

Then download cluster.py for accessing the cluster

wget https://raw.githubusercontent.com/schollz/find-lf/master/tools/cluster.py

Then, to initialize, just run

python3 cluster.py initialize

to which you'll be asked for the information about your cluster. Choose any group that you want, but remember it, as you will need it to login to the FIND server. For the lf address, you can use the default (a public server) or set it to your own. See find-lf/server/README.md for more information.

To easily find which hostnames/IPs are on your network, use

python3 cluster.py list

Startup the Pi cluster using python3.py cluster start. You can check the status with python3 cluster.py status

3. Classify locations using Pi cluster

After the cluster is up in running, you need to do learning. Take a smart phone and identify its mac address, something like AA:BB:CC:DD:EE:FF. Take your phone to a location. Then activate the find-lf server to do learning either by switching on the find-lf website or running python3 cluster.py -u AA:BB:CC:DD:EE:FF -l location learn.

This is important! Before moving to a new location, make sure to turn off learning by switching to tracking. Activate this on the find-lf server using the the find-lf website or use python3 cluster.py track.

Repeat these steps for as many locations as you want.

4. Track all the cellphones!

Now just go to https://ml.internalpositioning.com and login using your group name to see tracking of all the phones!

You can also track a single phone using https://ml.internalpositioning.com/GROUP/dashboard?user=AA:BB:CC:DD:EE:FF.

License

Copyright 2015-2017 Zack Scholl ([email protected], @zack_118). All rights reserved. Use of this source code is governed by a AGPL license that can be found in the LICENSE file.

Todo

  • Determine the wlan0/1 of the Wi-Fi adapter automatically (currently it defaults to wlan1, which is right most of the time)

find-lf's People

Contributors

mzealey avatar schollz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

find-lf's Issues

Not sure if it is running properly

Hi:
I run the cluster.py script, but it says it was unable to connect to localhost, but then I check the status and it says "scanning". After that I sent the stop signal, but the status continues showing "scanning".
I set up my own server, but I can't login with the group name.

ESP8266 as node

Hello, Is it possbilbe to use a ESP8266 as Wifi Sniffer and report the data to your Server?
I will automate my house via Home Assistant and found your interresting project.
But i have the problem that i can not install any apps on my buisness phone but half a day i am searching it.
So find if would be the perfect solution.
As i automate everything at home i use NODEMCU dev boards everywhere. In light switch door opener etc pp.
This boards are more powerfull than aruinos and really cheap.
I send and receive mqtt packages for the given tasks.

With your server and a sniffing code like this
https://github.com/rw950431/ESP8266mini-sniff

Also asked rw950431 if it could be done
rw950431/ESP8266mini-sniff#1

I would have allready everthing installt and can locate my buisness phone.
No expensiv RPi3 and Wifi Dongles are needed.

ESP8266 can be used as wifi repeater, not for video streaming, but txt pages are good.
So running in monitormode with connection via Wifi to your server should also be possible.

Any opinion how to send the data to server?
Do not know a lot about mqtt but maybe this can be used to send.

Pulling rssi values from ap

Looking at PR #28 I could easily get that data out of my unifi setup, and even pull from more than just one ap. But of course each client is only going to be seen by one ap, so would that data have any actual value?

If I'm understanding the code correctly it looks like i just need to submit some json like the following

{  
   "node":"ap identifier",
   "signals":[  
      {  
         "mac":"client mac",
         "rssi":"##"
      }
   ],
   "timestamp":"Date.now()"
}

I've already banged out a quick piece of code to extract the data but I don't have a working find cluster yet so I didn't bother writing anything to submit the data yet and wanted to know about the viability of this before proceeding.

Bluetooth communication

My idea here is to continuously sniff WiFi with tshark and then have all nodes communicate with a single node that is devoted to sending the information to the central server. I'd like to also support a single WiFi card (instead of having to have two). Communication would have to be done with Bluetooth.

sudo hciconfig hci0 piscan   # setup bluetooth
sudo apt-get install libbluetooth-dev python3-pip bluez
sudo python3 -m pip install pybluez

Inaccurate results when using find-lf

Hi, I've managed to set up find and find-lf using 5 Raspberry Pis. However I'm not getting accurate results.

My set-up:

  • 1 Raspberry Pi running Find server
  • 1 Raspberry Pi running Find-lf server
  • 1 Raspberry Pi from where I'm initializing the cluser
  • 3 Raspberry Pis (the one running the Find-lf server and another 2) within the cluser (set up in different corners of the room)

Initial plan was to have the one from where I'm initializing the cluser also part of the cluster, however I'm not managing to get it to work with that set-up (it never transmits even though the status says that it's running)

Currently my main issue though is that even after giving it quite some time to learn, it is still highly inaccurate and does not correctly guess which area I'm in - even if I'm exactly next to a PI which is in the cluster and if that's a place from where I initially set it to learn.

Some logs which might be helpful:

When set to learning mode:

Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"my_desk","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-59}]}
Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"my_desk","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-62}]}

Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"window","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-67}]}
Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"window","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-51}]}

Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"door","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-71}]}
Sending to http://192.168.11.21:8003/learn: {"group":"mytest","username":"bcf5acdfa3d9","location":"door","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-61}]}

When set to tracking mode:

Sending to http://192.168.11.21:8003/track: {"group":"mytest","username":"bcf5acdfa3d9","location":"unknown","timestamp":0,"wifi-fingerprint":[{"mac":"raspberrypi","rssi":-49}]}

fingerprint.go:241: Tracking fingerprint containing 1 APs for bcf5acdfa3d9 (mytest) at window (guess)

(In reality I was back to location 'my_desk' over here)

Any help is greatly appreciated. Thanks

Could not connect to RPi

When I try running python3 cluster.py initialize, I get "Could not connect to {'address': 'pi@..., 'notes': 'Test', 'wlan': 'wlan0'}"

Any way to debug this? Thank you.

Historical Information? Generate trending graphs over time?

I couldn't find this in the README - apologies if I somehow missed it.

Does find-lf maintain any kind of historical information?

For example - is it possible to see how many devices were in an area at a given time? Or is there some server/backend we can poll to generate graphs like say, the following?

location-analytics

Is it possible to run find-lf on a single RPi?

schollz,

  • Presence Detector
  • Internal Positioning (seems 2-3s delay)

I am hoping to use find-lf for two purposes, in order of importance above. I am running HomeAssistant and as I'm sure you're already aware, reliable presence detection has always been a challenging problem to solve. To my understanding, FIND works by allowing an Android user to connect to the platform via an app. find-lf works much the same, but without the app. Please correct me if I am wrong!

If this is the case, find-lf seems more attractive, as it would work better with both Android and iPhone users.. however I know the intention is to emulate the LF sonar system which implies multiple polling devices. So, in saying all of this..

Is it possible to run find-lf on a single RPi?

README refers to "wpa-conf" in /etc/network/interfaces - does this still exist?

The README mentions the following:

Also edit /etc/network/interfaces and remove the wpa-conf line for wlan1.

However, on a default Raspbian Lite install (September 2017), my /etc/network/interfaces contains:

# interfaces(5) file used by ifup(8) and ifdown(8)

# Please note that this file is written to be used with dhcpcd
# For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

Is the wpa-conf line still there by default?

Add Bluetooth sniffing

Sniffing

Using a Raspberry Pi 3, compile BlueZ.

Then make sure Bluetooth adapter is up: sudo hciconfig hci0 up.

Then run btmon in the background: sudo btmon &

Then make a scan using: sudo hcitool lescan or sudo hcitool scan

(Source)

Problem

The RSSI and Phone MAC address is only seen when the phone is on the Bluetooth screen. I.e., when any other process is running, it doesn't seem to allow the phone to be discoverable.

When I scan from the Pi3, sudo hcitool scan it gives me the following if I'm not on the Bluetooth screen:

$ sudo hcitool scan
Scanning ...
< HCI Command: Inquiry (0x01|0x0001) plen 5                                            [hci0] 97.534966
        Access code: 0x9e8b33 (General Inquiry)
        Length: 10.24s (0x08)
        Num responses: 0
> HCI Event: Command Status (0x0f) plen 4                                              [hci0] 97.535425
      Inquiry (0x01|0x0001) ncmd 1
        Status: Success (0x00)
> HCI Event: Inquiry Complete (0x01) plen 1                                           [hci0] 107.777731
        Status: Success (0x00)

However, if I goto the Bluetooth screen on my phone, then the scan sees the following:

$ sudo hcitool scan
Scanning ...
< HCI Command: Inquiry (0x01|0x0001) plen 5                                           [hci0] 120.701658
        Access code: 0x9e8b33 (General Inquiry)
        Length: 10.24s (0x08)
        Num responses: 0
> HCI Event: Command Status (0x0f) plen 4                                             [hci0] 120.702118
      Inquiry (0x01|0x0001) ncmd 1
        Status: Success (0x00)
> HCI Event: Extended Inquiry Result (0x2f) plen 255                                  [hci0] 127.292361
        Num responses: 1
        Address: 34:FC:EF:41:E6:F7 (OUI 34-FC-EF)
        Page scan repetition mode: R1 (0x01)
        Page period mode: P0 (0x00)
        Class: 0x5a020c
          Major class: Phone (cellular, cordless, payphone, modem)
          Minor class: Smart phone
          Networking (LAN, Ad hoc)
          Capturing (Scanner, Microphone)
          Object Transfer (v-Inbox, v-Folder)
          Telephony (Cordless telephony, Modem, Headset)
        Clock offset: 0x32cb
        RSSI: -73 dBm (0xb7)
        Name (complete): VS985 4G LTE
        16-bit Service UUIDs (complete): 11 entries
          OBEX Object Push (0x1105)
          OBEX File Transfer (0x1106)
          Audio Source (0x110a)
          A/V Remote Control Target (0x110c)
          Headset AG (0x1112)
          PANU (0x1115)
          NAP (0x1116)
          Handsfree Audio Gateway (0x111f)
          Phonebook Access Server (0x112f)
          PnP Information (0x1200)
          Message Access Server (0x1132)
> HCI Event: Inquiry Complete (0x01) plen 1                                           [hci0] 130.945272
        Status: Success (0x00)
< HCI Command: Remote Name Request (0x01|0x0019) plen 10                              [hci0] 130.945509
        Address: 34:FC:EF:41:E6:F7 (OUI 34-FC-EF)
        Page scan repetition mode: R1 (0x01)
        Page scan mode: Mandatory (0x00)
        Clock offset: 0xb2cb
> HCI Event: Command Status (0x0f) plen 4                                             [hci0] 130.946235
      Remote Name Request (0x01|0x0019) ncmd 1
        Status: Success (0x00)
> HCI Event: Remote Host Supported Features (0x3d) plen 14                            [hci0] 133.437920
        Address: 34:FC:EF:41:E6:F7 (OUI 34-FC-EF)
        Features: 0x0f 0x00 0x00 0x00 0x00 0x00 0x00 0x00
          Secure Simple Pairing (Host Support)
          LE Supported (Host)
          Simultaneous LE and BR/EDR (Host)
          Secure Connections (Host Support)
> HCI Event: Remote Name Req Complete (0x07) plen 255                                 [hci0] 133.440673
        34:FC:EF:41:E6:F7       VS985 4G LTE
        Status: Success (0x00)
        Address: 34:FC:EF:41:E6:F7 (OUI 34-FC-EF)
        Name: VS985 4G LTE

Another way for discovering, but it doesn't get around the above problem:

sudo apt-get install python-gi python-dbus
git clone https://github.com/pauloborges/bluez.git
cd bluez/test
sudo ./test-discovery

cluster.py list seems to hardcode 192.168.1.0/24 subnet?

The cluster.py list command seems to assume you're using the 192.168.1.0/24 subnet?

c = 'nmap -sP 192.168.1.0/24'

    elif command == "list":
        print("scanning all ips...please wait")
        c = 'nmap -sP 192.168.1.0/24'
        r, code = run_command(c)
        logger.debug(r)
        logger.debug(code)
        lines = []
        for line in r.splitlines():
            if "scan report" in line:
                lines.append(line.split("for ")[1])
        r, code = run_command(c)
        for line in r.splitlines():
            if "scan report" in line:
                lines.append(line.split("for ")[1])
        print("\n".join(sorted(list(set(lines)))))
        return

Is this intentional?

Role of WiFi adapter

In Readme.md

at the line

Do this for several Pis and then plug in the WiFi adapter that has "monitor" mode.

Could you please explain a bit on the role of this adapter in the system?

Thank you!

Do I have to use a WiFi adapter?

I was wondering if I need to go out and buy a WiFi adapter when I could just modify the WiFi on the Pi itself? Point me in the right direction.

Cheers!

building error for server

go build in the server directory leads to the following result:

server.go:20:2: cannot find package "github.com/gin-gonic/gin" in any of:
/usr/lib/go-1.6/src/github.com/gin-gonic/gin (from $GOROOT)

Continual tshark scanning

There is really no reason (except for single-wifi computers) for the tshark scanning to stop. I would like to implement a thread for the tshark scanning and then have the output parsed directly from the log file.

Command for scanning, something like:

sudo /usr/bin/tshark -I -i wlan1 -b files:2 -b filesize:10000 -w /tmp/temp

Then, to read:

sudo tshark -r /tmp/tempXXXX -T fields -e frame.time_epoch -e wlan.sa -e wlan.bssid -e radiotap.dbm_antsignal

The two files are to ensure that the data will rollover and not get overwritten. Both files can be read by the python script and determined accordingly.

In Python3, this can be implemented:

# Run tshark in the background
ls_output=subprocess.Popen("/usr/bin/tshark -I -i wlan1 -b files:2 -b filesize:10000 -w /tmp/temp".split(), stdout=subprocess.PIPE,stderr=subprocess.PIPE)

# Check if its running
ls_output1=subprocess.Popen("ps aux".split(), stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output = ls_output1.stdout.read()
isRunning = "tshark" in output and "[tshark] <defunct>" not in output

Question: WiFi adapter

Hi,
First of all really love the idea, see a lot of potential in this solution.

Quick question: would this wlan adapter be compatible?
https://www.aliexpress.com/item/NEW-Ralink-RT5572-300Mbps-802-11AC-2-4G-5G-dual-band-wireless-card-300M-wireless-n/32768906733.html

optionally would use https://www.aliexpress.com/item/RaLink-RT3572-2-4GHz-5-0GHz-300Mbps-WiFi-USB-Adapter-PCB-Module-Wireless-WiFi-Adapter-with/32815492744.html

I intend to buy RasPiZW + this adapted for every room in the house.
Let me know what you think!

How to rewrite information?

I tried to run the following command:
python3 cluster.py initialize
in the part of "2. Start Pi cluster," but I realized that I mistyped some information.

Is it possible to rewrite/reset information such as the IP address for Raspberry Pi?
If possible, could you tell me how?

Thank you.

invalid memory address crashing find-lf?

I'm getting the following error and it kills the find-lf server and breaks everything else that relies on the server, any ideas on how I can prevent this from happening?

[GIN] 2017/09/27 - 11:01:54 | 200 | 735.98µs | 10.0.1.71 | POST /reversefingerprint
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4036ba]

goroutine 21 [running]:
panic(0xa50720, 0xc8200100c0)
/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.sendFingerprints(0xc820194bd0)
/home/justin/find-lf/server/server.go:278 +0x13da
created by main.parseFingerprints
/home/justin/find-lf/server/server.go:204 +0x62

How to install find-lf server?

I don't understand how to install the find-lf server. Could someone explain it more detailed?

I've created a new VM with debian 8.11, installed "go" and built gin-gonic. The readme says, that i have to run the server with "./server". But how?

Not able to initiate learn command

I have setup a server using docker. mosquitto, rftcp, findserver are running as per the docker output. But when I'm trying to learn the finger print from using cluster, it is returning html content of the login page. I have enter x.x.x.x:18003 as LF server, where x.x.x.x is my public IP.

Also, i am able to create a group and learn the finger print using android device.

Please help

No tracking for other WiFi devices than learning device.

I am using public find-lf server.
When start my cluster and turn on "Track" mode,
I can only see single device tracked (the one used during "Learn" mode).
Dashboard and GET /location is showing just single smartphone device
and shows only once I turn on Find-it app on that smartphone.

How to start tracking of all WiFi enabled devices?
Should I establish my private find-lf server to scan more devices?

lf.internalpositioning.com does not seem to create group correctly on ml.internalpositioning.com?

I've setup a find-lf cluster using some Raspberry PIs.

I've set the group name to "" in config.json, and I'm using the default https://lf.internalpositioning.com/ server for now.

I set https://lf.internalpositioning.com/ to learn, and it seemed to learn OK - however, if I then go to https://ml.internalpositioning.com, and try with that group name, I keep getting "incorrect login".

However, I am able to access a group that I created previously via the Android app.

Is there something else you need to do when using https://lf.internalpositioning.com/ to setup a group on https://ml.internalpositioning.com?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.