chilloutcharles / brainflowsintovrchat Goto Github PK

Currently, the parameters provided by the app are all sent over as top-level snake_case variables. While this works, it poses some issues in terms of clarity and usability at scale. When configuring animation transitions in unity, it will leave all current variables in the root scope of the conditions list. This will additionally become an issue with retreiving scoped parameter endpoints if OSCQuery is ever implemented by any apps which want to remain compatible with the standard parameter list.

For these reasons I believe it would be beneficial to establish a proper naming schema for brainflow OSC parameters to ensure clarity and ease-of-use, as well as provide a standard format that can be implemented and extended by developers of other brain tracking apps.

Here is a baseline schema layout that I put together:

Brainflow:
  Sensors:
    Focus:
      - Left [float]
      - Right [float]
      - Average [float]
    Relax:
      - Left [float]
      - Right [float]
      - Average [float]
    Heart:
      - BeatsPerMinute [int]
      - BeatsPerSecond [float]
    Oxygen:
      - OxygenPercent [float]
      - RespBreathsPerMinute [int]
  Bands:
    Power:
      Left:
        - Alpha [float]
        - Beta [float]
        - Theta [float]
        - Delta [float]
        - Gamma [float]
        ...
      Right:
        - Alpha [float]
        - Beta [float]
        - Theta [float]
        - Delta [float]
        - Gamma [float]
        ...
      Overall:
        - Alpha [float]
        - Beta [float]
        - Theta [float]
        - Delta [float]
        - Gamma [float]
        ...
  Device:
    - ActiveBands [int]
    - BatteryPercent [float]
    - Connected [bool]
    - HeadsetId [int]
    - TimeSinceLastSample [float]
  AddOn:
    [AppName]:
    - HueShift [float]

This would map out to the following new parameters:

NEW

• Brainflow/Device/HeadsetId
  • Maps the hardware ID of the connected device, according to a mapping table. Supports up to 255 unique device types.
• Brainflow/Device/ActiveeBands
  • The number of named bands the connected device supports. Example: The Muse 2 having Alpha, Beta, Theta, Delta, and Gamma, this would be 5. For other interfaces this may be greater or fewer.
• Brainflow/Addon//...
  • Added scope to allow for app-specific features or custom schemas. HueShift could be moved to this for example.

The benefits of this approach would allow us to provide scoping to both the Unity conditions list, as Unity supports using / to consolidate parameters into collections. OSCQuery also uses this to retrieve groups of parameters whenever you query for a specific group on an OSC app's TCP server.

Migration Steps

To move the existing parameters over to the new ones, a deprecation phase will likely be needed to ensure compatibility. This can be done in several steps:

1. Move parameter transport logic into a new class OSCParameter which has the primary focus of sending data over to VRChat or any other app that is listening.
2. Create an enum class OSCEndpoint which provides mappings for each supported endpoint and its backup values (these being the deprecated endpoints).
3. Pass items from this to build instances of OSCParameter which will send a given value to all endpoints in the OSCEndpoint class.
4. Decide on an EOL support date for the backup endpoints, and announce it to users so they have time to migrate their prefabs over.
5. After deprecation cycle, remove the backup endpoints from the OSCEndpoint class, formally drop support at this point.

python recognized as py in other systems, account for this in readme

Add launch argument for amount of connection attempts, so that the program can be lauched earlier without giving up so soon.
Something along the lines of:

--connection-attempts
--retry-count

Or whatever seems most appropriate.

Due to complaints about the usability of the new parameter schema, internal discussions were had about how the schema itself should be laid out. The discussions were productive and yielded some solutions for ways we can rework the schema to be a bit more user-friendly. Summarizing the requested changes, they are as follows:

• Shorten Brainflow to BFI
• Flatten neurofeedback scopes together and simplify naming
  • FocusLeftSigned
  • FocusAvg
  • RelaxAvgSigned
  • etc...
• Shorten Neurofeedback to NeuroFB
• Shorten PowerBands to PwrBand
• Move all heart/respiration parameters to Biometrics scope
• Move related device and meta parameters into new Info scope
  • Move related "supported" variables into Info scope with added prefix
• Rename Device/Connected to Info/DeviceConnected

This set of changes will yield us the following layout:

New™️ new schema

BFI:
  Info:
    - VersionMajor [int]
    - VersionMinor [int]
    - SecondsSinceLastUpdate [float]
    - DeviceConnected [bool]
    - BatterySupported [bool]
    - BatteryLevel [float]
  NeuroFB:
    - FocusLeft
    - FocusLeftPos
    - FocusRight
    - FocusRightPos
    - FocusAvg
    - FocusAvgPos
    - RelaxLeft
    - RelaxLeftPos
    - RelaxRight
    - RelaxRightPos
    - RelaxAvg
    - RelaxAvgPos
  PwrBand:
    Left:
      - Alpha [float]
      - Beta [float]
      - Theta [float]
      - Delta [float]
      - Gamma [float]
    Right:
      - Alpha [float]
      - Beta [float]
      - Theta [float]
      - Delta [float]
      - Gamma [float]
    Avg:
      - Alpha [float]
      - Beta [float]
      - Theta [float]
      - Delta [float]
      - Gamma [float]
  Addons:
    - Hueshift [float 0-1]
  Biometrics:
    - HeartrateSupported [bool]
    - HeartBeatsPerSecond [float]
    - HeartBeatsPerMinute [int]
    - OxygenSupported [bool]
    - OxygenPercent [float]
    - BreathsPerSecond [float]
    - BreathsPerMinute [int]

Migration sheet

This will then result in the following parameter migrations:

Upon testing things within VRChat's debug menus and AV3Emulator, it was discovered that parameter names both in the old system and the current one were having their names truncated in the UI. This is partly a problem on VRChat's side, as well as a UX issue in the emulator, however we do not know currently if this will eventually be addressed by either group.

In the interest of attempting to work around this limitation, we should research ways we could send over a set of debug parameters which do their best to remain within the current limits for parameter names. These parameters will not be useful for anything outside of debugging errors with the application, should any arise.

Would be very useful to have a condensed list of all the parameters in one long string trough a webhook output to localhost,
the list could look like this:

,"parameter 1":"parameter 1 value";"parameter 1 type" ,"parameter 2":"parameter 2 value";"parameter 2 type" ,[...]

Important to have the space before the comma for easy pacing and indexing
(can always use a character that is not used in any of the parameters names/values/types to replace the ,)

In the interest in saving users memory, we should add the ability for the app to set custom thresholds which can be mapped to single-bit boolean values. This can be done on a per-direction basis, and would be beneficial in providing a simple way to trigger a set action when data matches up with a specific state.

Just a suggestion! Since it's not the first time I have dealt with running out of parameters memory, but I would like to suggest alternative osc parameters that use booleans instead of floats, which could be 4 per value

ex: the float osc_focus_left can be broken into 4 booleans to reduce the memory usage from 8 to just 4 bits. Booleans can go true for a certain threshold of a brain wave (10% > 25% > 50% > 85%). This in terms of usage on animators should make little to no difference since animations can smoothly transition with booleans going true.

Currently using all float values on this project takes 64 bits of the available custom parameters space, reducing it to 4 bools per metric, that would reduce its usage to 32 bits! Leaving enough space to add more parameters.

This is just a suggestion for those here who run multiple osc hardware! I also face track my avatars and gotta have all that sweet memory available to use all the hardware I have!

Currently the schema supports explicitly 5 classes of bands for general use in prefabs: Alpha, Beta, Theta, Delta, and Gamma. With the way Brainflow reports these parameters back, all values for each corresponding group will always add up to 1.

The Problem

While this is generally fine, it inherently limits prefabs from using consistent ranges if the app is running a greater or fewer number of bands. When looking at the default count of 5, to reach equilibrium each value can be at most 0.2. For a band count of 4 and 6, the equilibrium is 0.25 and 0.1666667 respectively. This also imposes a limit on parameter resolution the greater the number of bands as this will decrease each value by a set amount to accommodate the added band-space.

The Solution

A set of normalized parameters can be added, which scales the value to fit within the full 0-1 range relative to the provided band count. This will allow prefab authors to use these parameters consistently regardless of band count. In addition to the normalized parameters, we should also provide a parameter such as Brainflow/Device/BandCount [int] to report back the total band count available to users.

Neuroscientists looove their greek symbols. Change the order of them to correspond to their frequency range to make sense of the list. Highest to lowest: [Gamma, Beta, Alpha, Theta, Delta]

So, after a quick glance through the code, it looks like the OSC data is being transmitted through the OSCReporter. Resonite uses websockets instead of OSC. Is it possible to support websockets as an alternative method to specify as an argument when starting the program? Most of the code seems easy to change to support it, so it shouldn't be too hard to accomplish. Just a suggestion.

using board ids isn't user friendly, names would be better.