This project is a framework for xR creators to extend inputs and outputs easily beyond the capabilities of conventional devices.

The framework consists of firmware for M5Stack and an SDK for Unity with the STYLY plugin. The M5Stack toolkit allows you to easily create IoT projects by connecting various sensors and actuators without soldering. The controller functions as a Bluetooth keyboard and a Wi-Fi server, sending keyboard events to a device such as a smartphone in response to input from sensors. Moreover, you can control actuators by sending HTTP requests to the controller.

With this framework, you may be able to expand your imagination of the concept of xR and realize your ideas as works.

本プロジェクトは、xRクリエーターが、従来のデバイスの能力を超える入出力を簡単に拡張するためのフレームワークです。

フレームワークは、M5Stack用のファームウェアと、STYLYプラグインを組み込んだUnity用のSDKで構成されています。M5Stackは、各種センサーやアクチュエーターをハンダづけなしで接続し、IoTプロジェクトを簡単に作成できるツールキットです。コントローラは、BluetoothキーボードおよびWi-Fiサーバーとして機能し、センサーからの入力に応じてキーボードイベントをスマートフォンなどのデバイスに送信します。さらに、コントローラにHTTPリクエストを送ることで、アクチュエータを制御できます。

このフレームワークを使うことにより、xRの概念に対する想像力を膨らませ、アイデアを作品として実現できるかもしれません。

🚧 Heavily work in progress; therefore, only for experimental purposes 🚧

• Mini Dual Button Unit
• Light Sensor Unit
• Mini Angle Unit
• I2C Joystick Unit V1.1
• Distance Ranging Sensor Unit
• TVOC/eCO2 Gas Sensor Unit
• RFID 2 Unit
• Gesture recognition sensor
• 12 Key Capacitive I2C Touch Sensor V3 by Seeed Studio

• Servo Kit 180° Brick-compatible
• Vibration Motor Unit

⚠️ Not thoroughly tested yet.

• iOS/iPadOS - STYLY app
• Android - STYLY app
• Oculus Quest 2 - STYLY app (from App Lab)
• Windows - STYLY VR app
• Windows - Chrome
• macOS - Chrome

The connected Unit(s) input is transmitted on three different channels: analog, joystick, and buttons. The analog and joystick channels can handle up to one sensor Unit input for each at once, and the buttons channel can handle up to six button inputs simultaneously.

• If you want to connect multiple Units to port A, please connect via a 1 to 3 HUB Expansion Unit.
• When a Ranging Sensor Unit is connected, you can't use a Joystick Unit nor Gas Sensor Unit.
• To change the connection, make sure to turn off the power (i.e., M5Stack Fire: double-clicking the power button, M5Stack Core 2: pressing the power button for 6 seconds). You cannot turn it off while your controller is connected to a USB port.

1. Download M5Burner v3.0 for your platform from the official website
2. Extract and launch the M5Burner (move to your Applications folder before launching on macOS)
3. Choose "I/O Framework for xR" from the projects
4. Click on the Download button of the project
5. Once finished downloading, the Download button becomes the Burn button
6. Click on the Burn button
7. Choose the serial port in the COM field and click on the Start button to burn the firmware

1. Install EspTouch for iOS or Android (choose the esptouch-v2.0.0.apk) to your smartphone
2. Connect your smartphone to the Wi-Fi router
3. Open your EspTouch app and tap on the EspTouch item (not EspTouch V2)
4. Input the router’s password on the EspTouch app
5. Power on (or reboot) your M5Stack and press the A button within three seconds¹
6. Tap the Confirm button on the EspTouch app and wait for a while
7. You will see an IP address on the screen of the M5Stack

1. Choose a pattern from the table above and connect Unit(s) to your M5Stack controller (e.g., M5Stack FIRE)
2. Power on your M5Stack controller
3. If you want to use a Unit to be connected to Port B, please refer to the "How to setup" section and setup (Units to be connected to Port A will be recognized automatically)
4. Connect the controller as a Bluetooth device to your device (please follow standard instruction for the device)
5. Open the IOFrameworkWidget (BLE and Wi-Fi) scene in a browser and bring the browser frontmost
6. Press the Send (C) button to start sending
7. Control the joystick, sensor, etc.
8. Once confirmed, press the Stop (C) button again to stop sending

1. Press the Setup (A) button to enter the preferences screen
2. Press the Next (C) button (if needed) to choose the PORT B: NONE line
3. Press the Go (B) button
4. Press the - (A) or + (C) button to be matched to the device connected to the Port B (i.e., DUAL BUTTON, ANALOG IN², SERVO or VIBRATOR)
5. Press the Done (B) button
6. Press the Exit (A) button to back to the main screen

1. Press the Setup (A) button to enter the preferences screen
2. Press the Next (C) button (if needed) to choose the RFID 1: **:**:**:** line
3. Press the Go (B) button
4. Press the Reset (C) button
5. Put an RFID Tag on the RFID Unit, then remove the Tag
6. Press the Next (C) button to choose the next line
7. Repeat from step 3 for 3 times to register the remaining 3 RFID Tags
8. Press the Exit (A) button to back to the main screen

• "IO Framework M5" is shown as connected but no input to the STYLY scene. → Please try to unpair the controller in the Bluetooth preference and pair it again.
• Bluetooth connection status on my controller keeps switching between Connected and Disconnected when not connected. → The controller might have been paired with an old host (i.e., a PC or smartphone). If you no longer use the controller with the host, please remove the device from the host.

The SDK consists of components as follows.

• IOFrameworkManager is a manager for the I/O Framework
• IOFrameworkTestUI is a UI to test the framework
• IOFrameworkAnalogHandler is a handler for events in the analog channel
• IOFrameworkJoystickHandler is a handler for events in the joystick channel
• IOFrameworkButtonsHandler is a handler for events in the buttons channel
• IOFrameworkGestureHandler is a handler for events from a Gesture sensor (in both joystick and buttons channel)
• IOFrameworkOutputHandler is a handler for events in the output channel via Wi-Fi

IOFrameworkTestUI

IOFrameworkAnalogHandler

IOFrameworkJoystickHandler

IOFrameworkButtonsHandler

IOFrameworkGestureHandler

IOFrameworkOutputHandler

• PlayMaker
• STYLY Plugin for Unity

1. Import the IOFrameworkSDK.unitypackage file to your Unity project
2. Instantiate an IOFrameworkManager
3. Instantiate an IOFrameworkTestUI as a Child of the IOFrameworkManager object if necessary
4. Instantiate a handler from handlers in the SDK (e.g., IOFrameworkAnalogHandler) as a Child of the IOFrameworkManager object
5. Unpack the handler
6. Navigate PlayMaker editor to the FSM of the handler and edit for your scene

How to layout the SDK components in Unity

IOFrameworkManager broadcasts events to all Children of the IOFrameworkManager.

• ANALOG VALUE CHANGED Int: analog value
• JOYSTICK VALUE CHANGED String: joystick value
• BUTTON 1 DOWN
• BUTTON 1 UP
• BUTTON 2 DOWN
• BUTTON 2 UP
• BUTTON 3 DOWN
• BUTTON 3 UP
• BUTTON 4 DOWN
• BUTTON 4 UP
• BUTTON 5 DOWN
• BUTTON 5 UP
• BUTTON 6 DOWN
• BUTTON 6 UP

Send events to IOFrameworkManager: Event Handlers FSM.

• SET OUTPUT VALUE REQUEST String: output value

IOFrameworkManager broadcasts events to all FSMs.

• HTTP RESPONSE String: HTTP response

Send events to IOFrameworkManager: Event Handlers FSM.

• SET IP ADDRESS String: IP address

Send events to IOFrameworkManager: HTTP Request Loop FSM.

• START LISTENING
• STOP LISTENING
• LISTEN ONCE

sequenceDiagram
    participant S as STYLY app
    participant M as M5Stack
    M-->>S: keyboard events over BLE

The keys used for the protocol

• Keys in green: used in STYLY Web Player
• Keys in yellow: used for the Analog channel
• Keys in orange: used for the Joystick channel
• Keys in red: used for the Buttons channel

sequenceDiagram
    participant S as STYLY app
    participant M as M5Stack
    S-->>M: GET HTTP Requests over Wi-Fi
    M-->>S: HTTP Responses over Wi-Fi

GET http://{ip_address}/input

{analog value},{joystick value},{button 1 value},{button 2 value},{button 3 value},{button 4 value},{button 5 value},{button 6 value},

Example:

9,Left,0,1,0,0,0,0

GET http://{ip_address}/output?val={value}

• SERVO: {value} is servo angle in degree, between 0 and 180
• VIBRATOR: {value} is on duration in ms, between 0 and 100

Example:

GET http://192.168.0.10/output?val=123

An example of controlling output without SDK in PlayMaker: build an URL (this part can be parameterized) and issue an HTTP request (you need to add states to handle errors properly)

• English: Arduino IDE environment - M5Core
• Chinese (simplified): Arduino IDE 环境搭建 - M5Core

• M5Unified v0.0.7 by M5Stack
• M5GFX v0.0.20 by M5Stack
• ESP32 BLE Keyboard library v0.3.0 by T-vK
• VL53L0X library for Arduino v1.3.1 by Pololu³
• Adafruit SGP30 Gas / Air Quality I2C sensor v2.0.0 by Adafruit
• Adafruit MPR121 Library v1.1.1 by Adafruit
• ESP32Servo v0.11.0 by Kevin Harrington and John K. Bennett
• ServoEasing v3.0.0 by Armin Joachimsmeyer
• DFRobot_PAJ7620 v1.0.1 by DFRobot

1. Download the ZIP file at the ESP32-BLE-Keyboard v0.3.0 page (you don’t have to extract it after downloading)
2. In the Arduino IDE, navigate to Sketch → Include Library → Add .ZIP Library...
3. Select the file you just downloaded

1. In the Arduino IDE, navigate to Tools → Manage Libraries...
2. Type in a part of each library (e.g., VL53L0X, Adafruit SGP30 etc.) in the text field in the top right corner, choose the right library and press the install button
3. Repeat the second step for all required libraries

• The included MFRC522 I2C Library is from the RFID_RC522 example in the public domain, originally developed by arozcan based on the findings of the pioneers and modified by M5Stack.
• The technique for detecting that an RFID tag has been removed was implemented by referring to the example proposed by uluzox and modified for cooperative multitasking