I prepared and tested a script to setup the latest WSL (WSL2) and usbipd services but I want to know if its needed for easy-of-test of the gcc-port on Windows.

IMHO, I think it would be helpful and better than a traditional virtual machine as WSL2 comes with kernels and one can easily swap between its version and that of the kernels.

I currently don't know how the firmware is deployed (not yet familiar with ansible) but I added usbipd just incase if we need usb sharing services between Windows and WSL2, all in a single click.

Lastly, if I am to add the script and its README, what location is preferred? ./software/gcc-port and ./docs or both in ./software/gcc-port or add none?

• uv allows easy install & use of newer py versions.
• even py3.12.5 with full LTO
• install sheep as tool into virtual env for easier updates

Currently, the programmers for SBW and SWD live in the PRU and expose a simple firmware upload interface through the kernel module. This has the advantage that the low level SWD and SBW transactions execute with well defined timing on the PRU. However, the interface is rather simple and allows only uploading firmware while the underlying implementation would generally allow debugging attached devices with, for example, gdb. Also, the information how to control the specific device, e.g, an nRF52840 must be implemented by us on the PRU.

A better approach would be to offer a standard interface for programming/debugging that is accessed and controlled by existing userspace applications like pyOCD for SWD and mspdebug for SBW. Our software would just execute the lowest-level operations and leave the device-specific operations to these well-maintained tools. There exist well-defined interfaces that a debug probe needs to implement in order to work with the tools. CMSIS-DAP is the reference implementation of such protocol for ARM Cortex M microcontrollers. These protocols exchange commands and responses between the host and the debug probe through a serial interface, mostly USB.

We could implement two virtual debug probes that expose standard interfaces for debugging attached MSP430 and ARM Cortex-M microcontrollers as character devices. A tool like pyOCD opens the corresponding character device and writes commands into it. A kernel module receives the commands and either executes them itself by bitbanging the corresponding sequence on the programmer pins or passes them to the PRU for processing. The response is fed back to the character device and received by the user space tool.

python-package -> datalog.py has already configs for "complevel" and "compalg"

Latest Version. Sporadic.
GPIO-Traces repeat in extracted csv after ominous 6.4s, which coincides with the ring-buffer-size.

packet_indicator.csv

block after 1724410961.0002775;0 exists twice

@orgua I have been thinking about regenerating the toolchain we've been using to see if some of the errors (here and there) that seem internal to the compiler will reduce.
I've little experience with crosstool-ng.
Do I try it?
I'm asking before hand because it takes a while to generate the toolchain.
What do you think?

Open Tasks 2022, see posts below for newer tasks

• Programmer Mod #20
• Target PCB (msp430 + nRF) design + test
• Target FWs DeepSleep
• postponed: Cape Mods & make operational, Web Frontend, backend & Api
• Misc
  • FW-node ID Patching
  • PTP-Sync fix
  • POE Problem with TP-Link Splitter

The SPI - Chip select (CS) delay between two nodes is oscillating with ~ 0.2 Hz -> see this picture

PTP shows stable sync so the problem must be after that.

Current progress is documented here

shepherd-herd -v resync

Expected: give warning or error if timestamps are too far off

• i.e. > 5 s
• usually they only differ <= 1 s (13 nodes in same network) due to ssh delays

also give a note to wait some minutes to stabilize clocks after PTP restart. Herd should also check time-difference before starting an experiment-run (maybe it already does -> check).

Modus Operandi

• users can report
  • missing features that improve usage
  • bugs
  • overly complicated workflow-details
  • etc.
• list is ordered roughly by importance (order of execution)

List

• fix programming of unaligned hex, #33
• log systemd services
• fastapi-server / client
  • SW, scheduling
  • website -> link to doc at first
  • webservice -> open & secure fastapi-server (in progress)
  • make API available to the internet (in progress)
• determine cause of bad rf-performance for targets
• allow .elf-embedding with windows (datalib)
• simplify extraction of gpio-traces (datalib) -> is csv ok? shepherd-data extract-gpio
• add api-documentation (good example: pydantic)
• lint all subprojects (ruff, pyright)
• update unittests
• replace Fifo-system between userspace & pru
• allow static voltages for emulator without supplying trace-recordings
• gpio actuation in PRU (precondition: FiFo-replacement)
• gpio direction still not configurable!! while actuation does not work, shp sets variable pins to input atm
• #11
• verification of vsrc (c in PRU vs py-port)
• #19
  • latest results show, that BBone is only capable of +-300 ns sync performance (time it needs to update kernel-time)
• (web-) dataviewer for recorded traces
• sheep - error handling + saving message log & files -> report-model?
• sheep - cleanup between Testbed-Runs (program default FWs and wipe FRAM)
• energy environments - work up existing recordings - include base-set
• explore speedups with mypyc (example black-formatter)
• allow reading back memory of target MCUs (i.e. FRAM of MSP)

Note: most of these topics are pretty self contained and could be the base for student projects.

check if 2.4 us is still a valid threshold
possible culprit -> timing code in kernel-driver that provides ctrl_rep->clock_corr

there seems to be a bug where the update_kernel.sh does not activate the downgrade to kernel 4.19 (from time to time).
-> current dev-branch has guards implemented in the ansible playbooks before deleting the active kernel-image.

Quick Fix: switch out entry for active kernel in /boot/uEnv.txt during install

Better Fix: double-check in ansible (determine installed kernel, adapt uEnv.txt), TODO

Workaround: there is a ready-to-use image (last update 2022-12-14)

codon might bring an improvement.
But packages like numpy and probably h5py are not yet supported -> documented here

but using numba could be an option

• Flush with blocking character (for more precise timestamps)
• Poll more frequent (configurable)

Roadmap for May-August 2023

• targets
  • SW, prepare set of firmwares
  • SW, firmware patching (verification, conversion, modification)
  • SW, programmer (swd tested, sbw tbd)
  • SW, firmwares for selftesting
  • HW, assembly
  • HW, test (10/18 fully working)
• capes
  • HW, THT assembly (0/10) (in progress)
  • SW, test & calibration
• BBone
  • HW, POE Converter assembly & test
  • SW, Update OS (ubuntu 22.04, py3.10)
  • SW, sheep & herd -> task, core-integration
  • HW, deployment (update, test) (in progress)
• misc
  • SW, datalayout (config, metadata, meas) -> core-lib
  • SW, time sync investigation
  • SW, include Trafficbench to get linkmatrix
    • find schedule and prepare FW
    • prepare python-code for inclusion
    • include software into shepherd
    • digital waveform to uart converter (& disable flow control)
    • test & bugfixes
  • switch to pydantic v2
  • update general documentation
  • describe testbed (usage, structure)
  • generate &| integrate energy-envs
  • integrate default-firmwares
  • get allowances for deployment in offices
  • case - laser-cut (in production)
• testbed-frontend
  • add DNS to server: shepherd.cfaed.tu-dresden.de
  • testbedClient & Mockup (fixture-loader)

shepherd-core currently ships with zstandard. v0.22 has no arm7l-package prebuilt, so compilation starts and takes forever.

https://github.com/indygreg/python-zstandard

Updates: indygreg/python-zstandard#205

Workaround on BBone:

cd
git clone https://github.com/orgua/shepherd-datalib
cd shepherd-datalib/shepherd-core
nano pyproject.toml
# edit out zstandard line
sudo pip install .[elf] -U

zstandard must be installed anyway - it compiles, but needs ~20min

sudo pip install zstandard

pru should continue to sample ADC with 100 kHz -> python can filter in userspace. this safes space for long measurements and improves noise levels

some could be used for bachelor or master thesis, other only qualify for HiWi-Jobs

• IO Sampling
  • High-Speed, high temporal resolution (>= 1 MHz, <= 100 ns)
  • FPGA or MCU as Interface in front of SBC
  • optional: bidirectional, also actuating
• Network Topology of Testbed
  • find and characterize interesting topologies (multi-hop, ring, mesh, ...)
  • layout-optimization to include topos as subnets (while keeping node-count small)
  • bootstrapping - how to start (with smaller number of nodes)
  • optional: different PHY (protocol or target dependent)
  • link-measurements or channel characterization could be a baseline for research
• Target-Design (more practical work)
  • motivation: one free port on shepherd
  • what system to choose, featureset
  • survey: what is popular, missing or coming, i.e. https://www.iot-lab.info/
  • lora, wifi, 15.4, uwb,
  • design, validation
• optional sync via DCF77
  • https://blog.blinkenlight.net/experiments/dcf77/local-clock/
  • phase-detection
  • what min error would be possible
  • interesting for far away nodes (uwb) -> distant subnets
• shepherd-cape redesign - howto - feasibility-study
  • characterize constraints
  • survey of SBC with included featureset
• concept for measuring / recording energy environments
  • statistically valid data
  • how to scale (50 .. 80 .. 100 nodes)
• (web-) dataviewer for recorded traces
• new programmer design (OpenOCD + virtual debug probe, mixed with a new controller like RP2040), more details in #23
• evaluate current target (or new revision)

the firmware doesn't know yet how and when to change the IO-Pin direction

Programming-Pins

# SWD1 CLK - jtag TCK   - always TX
# SWD1 IO  - jtag TDI   - pDir1-pin / rxtx
# SWD2 CLK - jtag TDO   - always TX
# SWD2 IO  - jtag TMS   - pDir2-pin / rxtx

swd1_clk = 5
swd1_io = 4
swd2_clk = 8
swd2_io = 9

Direction-Pins

swd1_io = 10  # P8_31, GPIO0[10]
swd2_io = 11  # P8_32, GPIO0[11]

Link to Source

TLDR

Programmer is miss-guided by padding / alignment of .hex-file.

https://developer.arm.com/documentation/ka003694/latest

helps to realign the .hex. Store config into file 'srec_cfg'

# BL51 hex files are not sorted for ascending addresses. Suppress this warning
-disable-sequence-warning
# load input HEX file
build.hex -Intel
# fill all incomplete 16-bit words with 0xFF. The range is limited to the application
-fill 0xFF -within build.hex -Intel -range-padding 4
# generate hex records with 16 byte data length (default 32 byte)
-Output_Block_Size=16
#generate 16-bit address records. Do no use for address ranges > 64K
-address-length=2
# generate a Intel hex file
-o buildPadded.hex -Intel

install and run srec_cat

sudo apt install srecord
srec_cat @srec_cfg

TODO: integrate into shepherd

Long Version

investigation so far:

• firmwares from https://github.com/orgua/shepherd-targets/releases work, except rf-survey / trafficbench
• trafficbench is built with segger tools (automatically in github workflow)
• flashing the hex onto the nRF triggers a verification error (read-back of written data)
  • ignoring that verification-error leads to a write-error right after
• programmer now bubbles up address and value, see below
• tools used
  • latest shepherd dev-branch
  • targets fw from release v0.3.1

Debug output

Will set pru0-firmware to 'am335x-pru0-programmer-SWD-fw'
Shared memory address:  0x9CA00000, size: 15607808 byte
Samples per buffer:     10000
Number of buffers:      64
Buffer period:          0.100 s
Size of 1 Buffer:       243872 byte
wrote Firmware-Data to SharedMEM-Buffer (size = 120989 bytes)
set programmerCTRL
        target = 'nrf52'
        datarate = '500000'
        pin_tck = '5'
        pin_tdio = '4'
        pin_dir_tdio = '10'
        pin_tdo = '0'
        pin_tms = '0'
        pin_dir_tms = '0'
Programmer initialized, will start now
Programming in progress,        pgm_state = init, shp_state = reset
Programming in progress,        pgm_state = running - 4640 B written, shp_state = reset
Programming in progress,        pgm_state = running - 10256 B written, shp_state = reset
Programming in progress,        pgm_state = running - 15864 B written, shp_state = reset
Programming in progress,        pgm_state = running - 21468 B written, shp_state = reset
SystemError - Failed during Programming, p_state = error (-3)
Programming - Procedure failed - will exit now!
        shepherdState   = idle
        programmerState = error (-3)
        programmerCtrl  = ['nrf52', '500000', '5', '4', '10', '0', '0', '0']
PROGRAMMER-WRITE-ERROR: ihex to target @0x5FB7, data=7238995 [0x6E7553]
Now exiting ShepherdIO

Panicking Line in Programmer-Code

Section in Hex (Line 1570, +-3 Lines)

:105F90004108300842083608252D32352E32357337
:105FA00020252D32352E32357320666C75736820AE
:075FB0006275666665720070
:105FB70053756E004D6F6E0054756500576564002C
:0D5FC7005468750046726900536174000053
:105FD4004108300842083608252D32352E323573F3
:105FE40020252D32352E3235732066696E616C6939

Decoded iHex

Byte Count: x10
Address:    x5FB7
RecordType: x00
Data:       x53756E004D6F6E005475650057656400
Checksum:   x2C

Error happens right at the beginning of that data-segment.
NOTE: the offset is !=mod4 here, as there are 7 bytes written in der prev. hex-line. According to nRF-Doc Section 4.3.1 that is a problem as only whole 32bit-values can be written.

Removing both write & verify error-panicks the programmer runs till the end - but the firmware is not working. last data-segment in hex-file is at 0xA58A which would be 42378 in base10 and corresponds to the blocks written (as reported by the interface).

Programmer initialized, will start now
Programming in progress,        pgm_state = init, shp_state = reset
Programming in progress,        pgm_state = running - 3088 B written, shp_state = reset
Programming in progress,        pgm_state = running - 6824 B written, shp_state = reset
Programming in progress,        pgm_state = running - 10544 B written, shp_state = reset
Programming in progress,        pgm_state = running - 14280 B written, shp_state = reset
Programming in progress,        pgm_state = running - 18012 B written, shp_state = reset
Programming in progress,        pgm_state = running - 21744 B written, shp_state = reset
Programming in progress,        pgm_state = running - 28012 B written, shp_state = reset
Programming in progress,        pgm_state = running - 41176 B written, shp_state = reset
Finished Programming!
        shepherdState   = idle
        programmerState = idle
        programmerCtrl  = ['nrf52', '500000', '5', '4', '10', '0', '0', '0']

Reducing data-rate from 500k to 300k, 200k, 100k works but triggers the same verification-error.

Try to fix Alignment

The Segger-tools seem to produce incompatible hex-files. Unfortunately a generic objcopy makes the same error, only later in the hex. Even with special commands.

objcopy -O ihex build.elf build.hex --file-alignment 4 --section-alignment 4
objcopy -O ihex build.elf build.hex --gap-fill 0x00 --srec-len 0x08 --file-alignment 0x04 --section-alignment 0x04 

• this affects pipenv and pre-commit
• must be fixed in ubuntu / python
• workaround

Roadmap for May-August 2023

• targets
  • SW, prepare set of firmwares
  • SW, firmware patching (verification, conversion, modification)
  • SW, programmer (swd tested, sbw tbd)
  • SW, firmwares for selftesting
  • HW, assembly
  • HW, test (10/18 fully working)
• capes
  • HW, THT assembly (0/10) (in progress)
  • SW, test & calibration
• BBone
  • HW, POE Converter assembly & test
  • SW, Update OS (ubuntu 22.04, py3.10)
  • SW, sheep & herd -> task, core-integration
  • HW, deployment (update, test) (in progress)
• misc
  • SW, datalayout (config, metadata, meas) -> core-lib
  • SW, time sync investigation
  • SW, include Trafficbench to get linkmatrix
    • find schedule and prepare FW
    • prepare python-code for inclusion
    • include software into shepherd
    • digital waveform to uart converter (& disable flow control)
    • test & bugfixes
  • switch to pydantic v2
  • update general documentation
  • describe testbed (usage, structure)
  • generate &| integrate energy-envs
  • integrate default-firmwares
  • get allowances for deployment in offices
  • case - laser-cut (in production)
• testbed-frontend
  • add DNS to server: shepherd.cfaed.tu-dresden.de
  • testbedClient & Mockup (fixture-loader)

current state:

• current ubuntu images for bbone are currently here
• the custom path is something like ./ubuntu-armhf-focal-minimal/2022-09-26/am335x-ubuntu-20.04.4-console-armhf-2022-09-26-4gb.img.xz
• these get build by image-builder
• current focal-v5.10 config for the builder is here

new approach:

• build your own, add latest ubuntu 22.04 (jammy) with python 3.10, if possible include shepherd
• the rcn-ee apt repo already has jammy-support (newest, last checked on 2023-03-10)
• a manually updated ubuntu on bbone works fine

We are currently locked in with kernel 4.19. For normal testbed-usage there is no problem, but unittests remove and reload the module for every test (~ 100 tests), which crashes the system at a random point in time eventually. I already removed (some) possible leaks and start to suspect the dusty kernel itself. There are kernel version 5.4 and 5.10 available for the beaglebone, but some internal interfaces changed with 5.4 and more with 5.10, so the transition is not done with a simple switch.

At least one kernel-change also affects PRU-Code regarding the intc-ressource:

• prepared in shepherd-code
• kernel 5.10 needs switch to pru-software-support-package (PSSP) v6

Another problem are changes in the API and include system. part of the problem seems to be documented here

UPDATE: crashes were caused by the shepherd kernel-module and are fixed now. Update to 5.10 or 5.15 needs to happen anyway, as the next ubuntu-release might not allow downgrading the kernel to 4.19 anymore.

Issues:

• having no "recordings" folder directly writes to /var/shepherd/recordings (as a file)
• having rec.h5 and rec.0.h5, emulating from rec.0 overwrites rec.h5 with result-data

better approach:

• check if folder is present, create if needed
• recording should get a timestamp and source-name (sheep0).
• make sure no file is overwritten