Giter Club home page Giter Club logo

gamutrf's Introduction

gamutRF

An SDR orchestrated scanner and collector.

gamutRF is a system enabling a compact network of one or more modest machines (such as Pi4s), each with their own USB SDR (such as an Ettus B200mini or a BladeRF XA9), to operate collectively as a configurable wideband scanner and I/Q sample recorder.

A gamutRF system comprises an "orchestrator" machine which typically runs at least one scanner service (typically scanning 0.1GHz to 6GHz in 30s) and the sigfinder service, which then can command potentially many gamutRF "workers" to make I/Q sample recordings for later analysis. sigfinder can command multiple scanners which can be on the same machine or be connected over a network.

gamutRF provides tools to work with I/Q sample recordings, and to also record GPS location/compass metadata for the system itself. gamutRF typically runs on networks of Raspberry Pi4s, but can also run on x86 machines, and is based on gnuradio.

See also instructions on how to build a gamutRF system.

Scanner theory of operation

gamutRF's scanner container connects to a local SDR and sweeps over a configured frequency range or ranges collecting samples. The samples are sent an FFT block, and then to a streaming retuning gnuradio block, which aggregates and then serves the frequency-annotated FFT points as JSON objects over ZMQ to the "sigfinder" container (see below) and also commands the source SDR block to retune to a new frequency in the range when enough FFT points have been accumulated.

The sigfinder container consumes these FFT points over ZMQ (potentially from many scanners), does some noise processing (correcting FFT points to be in frequency order, computing mean power over 10kHz, and then a rolling mean over 1MHz) and then submits them to scipy.signals.find_peaks.

If workers have been provisioned, the sigfinder will then command the workers to make an approximately 10 second I/Q recording at approximately 20Msps of each signal. Each signal peak is assigned a 20MHz bin, which means that if a signal is repeatedly detected with some frequency variation, the assigned recording bin will be constant, and if multiple signals are detected within 20MHz they can be collected simultaneously. A worker by default records at a higher sample rate than the bin size, so that 20MHz signal margins can be recorded.

As there will almost certainly be more signals than workers available, sigfinder will prioritize signals that it least often observed over a configurable number of scanner cycles. It is possible to configure this to off so that the recording choice will be random. It is also possible to configure the workers to tell the sigfinder to exclude that worker from certain frequency ranges (if for example the worker SDR cannot handle some part of the frequency spectrum scanned).

Build and Operating gamutRF

See the build doc for instruction on setting up a GamutRF system. Next see the operating instructions for instructions on operating GamutRF.

Documentation

1 - Introduction
2 - System Overview
3 - Build
4 - Operation
5 - ML Tooling
6 - Testing
7 - Troubleshooting
AIRT Support

Contributing

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b dev)
  3. Commit your Changes (git commit -m 'adding some feature')
  4. Run (and make sure they pass):
black --diff --check gamutrf
  1. Push to the Branch (git push origin dev)
  2. Open a Pull Request

See CONTRIBUTING.md for more information.

License

Distributed under the Apache 2.0. See LICENSE for more information.

Contact IQTLabs

See our other projects: https://github.com/IQTLabs/

gamutrf's People

Contributors

anarkiwi avatar cglewis avatar dependabot[bot] avatar emmair avatar lk-iqt avatar ltindall avatar rashley-iqt avatar renovate-bot avatar renovate[bot] avatar robotastic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gamutrf's Issues

Streamline docker builds

Currently docker builds are taking on the order of 6 hours to complete and frequently timeout. Please:

  1. profile the builds to determine where the bottleneck is occurring
  2. Further modularize build workflow
  • potentially seperate the build of amd64 and arm64 images
  1. refactor builds or images to improve speed

sigfinder won't open new connections to a collector if it tries too many times and fails

2021-11-19 22:26:20,350 HTTPConnectionPool(host='192.168.111.12', port=8000): Max retries exceeded with url: /v1/info (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff7c0cee90>: Failed to establish a new connection: [Errno 111] Connection refused'))

It should try to establish a new connection each time, not hold on the initial failure and not try again.

SDR Test Harness

Create a test harness compatible with supported SDRs to test signal detection and performance tune scanning and accuracy.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

dockerfile
Dockerfile
  • ubuntu 22.04
  • ubuntu 22.04
  • iqtlabs/gnuradio 3.10.9.2
docker/Dockerfile.airspyfm
  • ubuntu 22.04
  • ubuntu 22.04
docker/Dockerfile.base
  • ubuntu 22.04
  • iqtlabs/gnuradio 3.10.9.2
  • ubuntu 22.04
  • iqtlabs/gnuradio 3.10.9.2
docker/Dockerfile.driver
  • ubuntu 22.04
  • iqtlabs/gnuradio 3.10.9.2
  • ubuntu 22.04
docker/Dockerfile.sigmf
  • ubuntu 22.04
  • ubuntu 22.04
docker/Dockerfile.test
docker/Dockerfile.torchsig
  • ubuntu 22.04
docker/Dockerfile.uhd-sr
  • ubuntu 22.04
  • ubuntu 22.04
docker/Dockerfile.vkfft
  • ubuntu 22.04
  • ubuntu 22.04
docker/Dockerfile.waterfall
  • ubuntu 22.04
  • ubuntu 22.04
utils/mavlink-api/Dockerfile.mavlink-api
  • python 3.11-slim
utils/mavlink-api/Dockerfile.mqtt-publisher
  • python 3.11-slim
github-actions
.github/workflows/ci-test.yml
  • actions/checkout v4
  • actions/setup-python v5
  • snok/install-poetry v1
  • codecov/codecov-action v4
  • ubuntu 22.04
.github/workflows/codeql-analysis.yml
  • actions/checkout v4
  • github/codeql-action v3
  • github/codeql-action v3
  • github/codeql-action v3
.github/workflows/docker-extras.yml
  • actions/checkout v4
  • docker/setup-qemu-action v3
  • docker/setup-buildx-action v3
  • docker/build-push-action v6
  • actions/checkout v4
  • docker/setup-qemu-action v3
  • docker/setup-buildx-action v3
  • docker/build-push-action v6
.github/workflows/docker-test.yml
  • actions/checkout v4
  • actions/checkout v4
  • actions/checkout v4
  • actions/checkout v4
.github/workflows/docker.yml
  • actions/checkout v4
  • docker/setup-qemu-action v3
  • docker/setup-buildx-action v3
  • docker/build-push-action v6
  • docker/build-push-action v6
  • docker/build-push-action v6
  • docker/build-push-action v6
  • docker/build-push-action v6
  • docker/build-push-action v6
  • actions/checkout v4
  • docker/setup-qemu-action v3
  • docker/setup-buildx-action v3
  • docker/build-push-action v6
  • docker/build-push-action v6
.github/workflows/pypi.yml
  • actions/checkout v4
  • snok/install-poetry v1
.github/workflows/secrets-scan.yml
  • actions/checkout v4
  • actions/github-script v7
  • actions/upload-artifact v4.3.3
.github/workflows/semgrep.yml
  • actions/checkout v4
  • actions/setup-python v5
  • actions/github-script v7
  • actions/upload-artifact v4.3.3
  • actions/checkout v4
  • actions/setup-python v5
  • actions/github-script v7
  • actions/upload-artifact v4.3.3
.github/workflows/stale.yml
  • actions/stale v9
pep621
gamutrflib/pyproject.toml
  • poetry-core >=1.0.0
gamutrfwaterfall/pyproject.toml
  • poetry-core >=1.0.0
pyproject.toml
  • poetry-core >=1.0.0
utils/mavlink-api/pyproject.toml
poetry
gamutrflib/pyproject.toml
  • numpy 2.0.0
  • scipy 1.14.0
  • pandas 2.2.2
  • python >=3.10,<3.13
  • pyzmq 26.0.3
  • zstandard 0.22.0
gamutrfwaterfall/pyproject.toml
  • Flask 3.0.3
  • cairocffi 1.7.1
  • matplotlib 3.9.0
  • numpy 2.0.0
  • pandas 2.2.2
  • pycairo 1.26.1
  • python >=3.10,<3.13
  • pyzmq 26.0.3
  • scipy 1.14.0
  • zstandard 0.22.0
  • requests 2.32.3
pyproject.toml
  • Flask 3.0.3
  • Jinja2 3.1.4
  • bjoern 3.2.2
  • falcon 3.1.3
  • falcon-cors 1.1.7
  • gpsd-py3 0.3.0
  • numpy 1.26.4
  • paho-mqtt 2.1.0
  • pandas 2.2.2
  • pbr 6.0.0
  • prometheus_client 0.20.0
  • python >=3.9,<3.13
  • pyzmq 26.0.3
  • requests 2.32.3
  • sigmf 1.2.2
  • sysrsync 1.1.1
  • webcolors 24.6.0
  • zstandard 0.22.0
  • attr 0.3.2
  • attrs 23.2.0
  • black 24.4.2
  • docker 7.1.0
  • pdbpp ^0.10.3
  • pylint 3.2.5
  • pytest 8.2.2
  • pytest-cov 5.0.0
  • pytype 2024.4.11
utils/mavlink-api/pyproject.toml
  • python >=3.10,<3.13
  • Flask ^3.0.0
  • pymavlink ^2.4.40
  • pyserial ^3.5
  • httpx 0.27.0
  • paho-mqtt ^2.0.0

  • Check this box to trigger a request for Renovate to run again on this repository

The gamutrF fails to connect to the BladerF

Hello, how does GamuTRf connect to Bladerf? Bladerf is running correctly on the VM. Run the "docker run -v/TMP :/ TMP -ti iqtlabs/gamutrf gamutrf-scan -- SDR bladerf "command to return "[ERROR] bladerf_open_with_devinfo() Returned -7 - No device(s) available" Please see the screenshot for details
bladerf_info
gamutrf-scan
gamutrf-scan_freq
system_version

Add GUI for configuration

Please add a graphical user interface allowing the modification of the gamutRF scanner configuration during operation.

Standalone install for NUC

Setup a method for creating an instance of GamutRF that doesn't need to be split across multiple Raspberry Pis

BladeRF can't find it's lib for A4

[WARNING @ host/libraries/libbladeRF/src/board/bladerf2/bladerf2.c:456] FPGA bitstream file not found.
[WARNING @ host/libraries/libbladeRF/src/board/bladerf2/bladerf2.c:457] Skipping further initialization...

  Error: Command requires FPGA to be loaded

Waterfall bug (bad db min/max + bad data)

I've observed unexpected behavior where the waterfall dB limits do not seem to update properly. In image 1 the minimum dB on the y-axis seems to be about -95 dB despite the true minimum from visual inspection being closer to -84 dB. This behavior appears after restarting the scanner with a different --peak_fft_range value (in this case 0 -> 1) which results in the noise floor being raised and thus the expectation of the minimum dB increasing.

Additionally, I've noticed what appears to be incorrect FFT values in the first column as shown in images 2 and 3. I've included these in the issue since I speculate they could be the root cause of the issue above.

db_min/db_max are dependent on db_data

state.db_min = np.nanmin(state.db_data)
state.db_max = np.nanmax(state.db_data)

db_data should roll in new data as the scanner restarts and eventually all the db_data from the first run of the scanner container will be rolled out. This behavior remains even when all db_data has been replaced.

state.db_data = np.roll(state.db_data, -1, axis=0)
state.db_data[-1][idx] = scan_df.db.values.flatten()

Image 3 appears to show a single non-nan value for the entire first column. This by itself may be a separate issue but could also be the root cause of this issue. My initial guess is that there could be a mistake in the following lines which calculate the indexing used to populate db_data.

idx = (
((scan_df.freq - config.min_freq) / config.freq_resolution)
.round()
.clip(lower=0, upper=(state.db_data.shape[1] - 1))
.values.flatten()
.astype(int)
)

Image 1
issue_1

Image 2
issue_2

Image 3
issue_3

To replicate:

  1. Run waterfall and scanner containers. Set --peak_fft_range=0 (value is arbitrary, this value is used for clarity of the resulting difference)
  2. Set --peak_fft_range=1 and restart scanner container.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.