Practical case: BLE gesture gamepad on Arduino Nano 33 BLE

Objective and use case

What you’ll build: Create a BLE gesture gamepad using Arduino Nano 33 BLE, APDS9960, and MPU6050 to stream gamepad states via hand gestures and tilt.

Why it matters / Use cases

• Enable intuitive gaming controls through hand gestures, enhancing user experience in mobile and PC games.
• Facilitate accessibility for users with limited mobility by providing alternative input methods.
• Demonstrate the integration of multiple sensors (APDS9960 for gesture detection and MPU6050 for tilt) in a compact device.
• Showcase the capabilities of Arduino Nano 33 BLE in developing low-power, wireless applications.

Expected outcome

• Achieve a latency of less than 50ms between gesture input and gamepad state transmission.
• Maintain a stable BLE connection with a packet loss rate below 2% during gameplay.
• Stream gamepad states at a rate of 10 packets per second, ensuring smooth gameplay.
• Demonstrate accurate gesture recognition with a success rate of over 90% in various lighting conditions.

Audience: Hobbyists and developers interested in IoT and gaming; Level: Intermediate

Architecture/flow: Arduino Nano 33 BLE processes inputs from APDS9960 and MPU6050, transmitting gamepad states via BLE to connected devices.

Advanced Hands‑On: BLE Gesture Gamepad with Arduino Nano 33 BLE + APDS9960 + MPU6050

Objective: Build a BLE “gesture gamepad” that streams a compact gamepad state over Bluetooth Low Energy (BLE), using hand gestures (APDS9960) for D‑pad/button inputs and tilt (MPU6050) for analog axes.

We will develop, build, and flash the firmware using PlatformIO (CLI). The target device is EXACTLY: Arduino Nano 33 BLE + APDS9960 + MPU6050.

Note on tooling policy: The family default is Arduino UNO with Arduino CLI. Because we are using a different board (Arduino Nano 33 BLE, nRF52840), we will use PlatformIO (CLI) as required.

---

Prerequisites

• OS:
• Windows 10/11 (x64) or
• macOS 12+ or
• Ubuntu 22.04+ (or equivalent Linux)
• Software:
• Python 3.10+ (recommended 3.11+)
• PlatformIO Core 6.1.13 or newer (CLI)
• USB cable:
• High‑quality data cable (USB Micro‑B to USB)
• BLE Central for validation:
• Smartphone with Nordic “nRF Connect” app OR
• Laptop BLE adapter and Python (bleak) for optional host script
• Drivers:
• Arduino Nano 33 BLE uses native USB CDC (ACM). Typically no additional drivers are needed on macOS/Linux. Windows 10/11 installs automatically. If Windows driver issues arise, install “Arduino Mbed OS Boards” drivers via Arduino IDE package (only driver component) or allow Windows Update to complete.

---

Materials (exact model)

• 1 × Arduino Nano 33 BLE (Model: ABX00030; MCU: nRF52840; 3.3 V I/O only)
• 1 × APDS9960 gesture/proximity/color breakout (e.g., SparkFun APDS-9960, Part: SEN‑12787; default I2C address 0x39; 3.3 V logic)
• 1 × MPU6050 6‑axis accelerometer/gyro breakout (common module: GY‑521; default I2C address 0x68; ensure it can run at 3.3 V)
• 4–8 × Female‑female jumper wires (Dupont)
• Optional:
• 1 × Breadboard
• 2 × additional jumpers if you want to use INT lines for low‑latency gesture interrupts (we’ll use polling by default)

---

Setup / Connection

The Arduino Nano 33 BLE uses 3.3 V logic. Do not connect 5 V logic to its I/O. Both APDS9960 and MPU6050 operate over I2C; you can connect both sensors in parallel to SDA/SCL.

• I2C pins on Nano 33 BLE:
• SDA = A4
• SCL = A5
• Power rails:
• 3V3 pin provides regulated 3.3 V
• GND for ground reference

We will poll the APDS9960 for gestures (so INT is optional). MPU6050 INT is also optional.

Wire Connections

• Power:
• Nano 33 BLE 3V3 → APDS9960 VCC; MPU6050 VCC
• Nano 33 BLE GND → APDS9960 GND; MPU6050 GND
• I2C:
• Nano 33 BLE A4 (SDA) → APDS9960 SDA; MPU6050 SDA
• Nano 33 BLE A5 (SCL) → APDS9960 SCL; MPU6050 SCL
• Optional interrupts:
• APDS9960 INT → D2
• MPU6050 INT → D3

Ensure your APDS9960 breakout is 3.3 V compatible (SparkFun SEN‑12787 is). Many GY‑521 MPU6050 boards include a regulator; when in doubt, power with 3.3 V and confirm it works reliably at that voltage.

Expected I2C Addresses

• APDS9960: 0x39
• MPU6050: 0x68 (AD0 low). If AD0 is tied high, address is 0x69.

---

Signal/Pin Mapping Table

---

Design Overview

• BLE GATT custom “Gamepad” service with two characteristics:
• Buttons (1 byte): bitfield for Up/Down/Left/Right, A, B (from APDS9960 gestures)
• Axes (2 bytes): X, Y in signed int8 range −127..127 based on tilt from MPU6050
• Gesture mapping:
• Up/Down/Left/Right gestures map to D‑pad bits.
• Near/Far gestures map to A/B buttons.
• Tilt mapping:
• Roll → X axis; Pitch → Y axis
• Simple low‑pass filtered accelerometer‑only tilt to avoid gyro drift.
• Report rate:
• 50 Hz default (20 ms), with change‑detection to reduce BLE traffic.
• Debug:
• Serial log at 115200 baud for quick inspection.

---

Full Code

Create the PlatformIO project with the following files.

File: platformio.ini

; Project: ble-gesture-gamepad
; Board: Arduino Nano 33 BLE (ABX00030)
; PlatformIO Core >= 6.1.13

[env:nano33ble]
platform = nordicnrf52
board = nano33ble
framework = arduino
upload_protocol = cmsis-dap

; Lock known-good library versions for reproducibility
lib_deps =
  arduino-libraries/ArduinoBLE @ ^1.3.6
  sparkfun/SparkFun APDS9960 RGB and Gesture Sensor @ ^1.4.3
  adafruit/Adafruit MPU6050 @ ^2.2.6
  adafruit/Adafruit Unified Sensor @ ^1.1.14

monitor_speed = 115200

Notes:
– upload_protocol cmsis‑dap works with the Nano 33 BLE’s on‑board debugger. If your upload fails, PlatformIO will fall back to the serial bootloader. You can omit this line if needed.

File: src/main.cpp

#include <Arduino.h>
#include <Wire.h>
#include <ArduinoBLE.h>
#include <SparkFun_APDS9960.h>
#include <Adafruit_MPU6050.h>
#include <Adafruit_Sensor.h>
#include <math.h>

// ====== Sensor instances ======
SparkFun_APDS9960 apds;
Adafruit_MPU6050 mpu;

// ====== BLE definitions (custom service) ======
// 128-bit UUIDs generated for this project
#define GP_SERVICE_UUID   "12345678-1234-5678-1234-56789abcdef0"
#define GP_BUTTONS_UUID   "12345678-1234-5678-1234-56789abcdef1"
#define GP_AXES_UUID      "12345678-1234-5678-1234-56789abcdef2"

BLEService gpService(GP_SERVICE_UUID);
// Buttons bitfield (1 byte): [bit0:Up][1:Down][2:Left][3:Right][4:A][5:B][6:reserved][7:reserved]
BLECharacteristic btnChar(GP_BUTTONS_UUID, BLERead | BLENotify, 1);
// Axes (2 bytes): int8 X, int8 Y, range -127..127
BLECharacteristic axesChar(GP_AXES_UUID, BLERead | BLENotify, 2);

// ====== Gamepad state ======
volatile uint8_t buttons = 0x00;
int8_t axisX = 0;
int8_t axisY = 0;

// Gesture mapping timings
const uint16_t GESTURE_HOLD_MS = 150; // keep button asserted briefly per gesture
uint32_t gestureHoldUntilMs = 0;
uint8_t gestureBitsLatched = 0;

// Tilt filter
float filtX = 0.0f;
float filtY = 0.0f;
const float alpha = 0.25f; // low-pass filter coeff (0..1)

// Rate limiting
const uint32_t REPORT_INTERVAL_MS = 20; // 50 Hz
uint32_t lastReportMs = 0;

// Helpers
static inline int8_t clampToI8(float v) {
  if (v < -127) return -127;
  if (v > 127) return 127;
  return (int8_t)lroundf(v);
}

void updateAxesFromMPU() {
  sensors_event_t a, g, temp;
  mpu.getEvent(&a, &g, &temp);

  // Compute tilt angles from accelerometer only (degrees)
  // Roll: rotation around X axis, Pitch: around Y axis
  float ax = a.acceleration.x;
  float ay = a.acceleration.y;
  float az = a.acceleration.z;

  // Protect against divide-by-zero issues
  if (isnan(ax) || isnan(ay) || isnan(az)) return;

  float roll  = atan2f(ay, az) * 57.29578f; // deg
  float pitch = atan2f(-ax, sqrtf(ay * ay + az * az)) * 57.29578f; // deg

  // Map angles to -127..127. Choose ~35 deg full-scale for responsiveness.
  const float FS_DEG = 35.0f;
  float xRaw = (roll / FS_DEG) * 127.0f;
  float yRaw = (pitch / FS_DEG) * 127.0f;

  // Dead-zone to avoid jitter
  const float DZ = 4.0f;
  if (fabsf(xRaw) < DZ) xRaw = 0.0f;
  if (fabsf(yRaw) < DZ) yRaw = 0.0f;

  // Low-pass filter
  filtX = (alpha * xRaw) + ((1.0f - alpha) * filtX);
  filtY = (alpha * yRaw) + ((1.0f - alpha) * filtY);

  axisX = clampToI8(filtX);
  axisY = clampToI8(filtY);
}

void processGesture() {
  // Non-blocking polling for gesture
  if (apds.isGestureAvailable()) {
    uint8_t g = apds.readGesture();
    uint8_t newBits = 0;

    switch (g) {
      case DIR_UP:    newBits |= (1 << 0); break; // Up
      case DIR_DOWN:  newBits |= (1 << 1); break; // Down
      case DIR_LEFT:  newBits |= (1 << 2); break; // Left
      case DIR_RIGHT: newBits |= (1 << 3); break; // Right
      case DIR_NEAR:  newBits |= (1 << 4); break; // A
      case DIR_FAR:   newBits |= (1 << 5); break; // B
      default: break;
    }

    if (newBits != 0) {
      gestureBitsLatched = newBits;
      gestureHoldUntilMs = millis() + GESTURE_HOLD_MS;
    }
  }

  // Apply latched gesture bits for a short time window
  uint32_t now = millis();
  if (gestureBitsLatched != 0) {
    if (now <= gestureHoldUntilMs) {
      // Assert gesture bits
      buttons |= gestureBitsLatched;
    } else {
      // Release after hold time
      buttons &= ~gestureBitsLatched;
      gestureBitsLatched = 0;
    }
  }
}

bool publishIfChanged() {
  static uint8_t lastButtons = 0xFF;
  static int8_t lastX = 127, lastY = 127;

  bool changed = false;
  if (buttons != lastButtons) {
    btnChar.writeValue(&buttons, 1);
    lastButtons = buttons;
    changed = true;
  }

  if (axisX != lastX || axisY != lastY) {
    int8_t axes[2] = { axisX, axisY };
    axesChar.writeValue((uint8_t*)axes, 2);
    lastX = axisX; lastY = axisY;
    changed = true;
  }
  return changed;
}

void setupAPDS() {
  if (!apds.init()) {
    Serial.println("[APDS9960] init failed");
  } else {
    // Optional tuning
    apds.setGestureGain(GGAIN_4X);
    apds.setGestureLEDDrive(LED_DRIVE_100MA);
    apds.setGestureProximityThreshold(30);
    apds.enableGestureSensor(true);
    Serial.println("[APDS9960] gesture sensor enabled");
  }
}

void setupMPU() {
  if (!mpu.begin(0x68, &Wire)) {
    Serial.println("[MPU6050] begin failed (check wiring/address)");
    return;
  }
  mpu.setAccelerometerRange(MPU6050_RANGE_4_G);
  mpu.setGyroRange(MPU6050_RANGE_500_DEG);
  mpu.setFilterBandwidth(MPU6050_BAND_21_HZ);
  Serial.println("[MPU6050] online");
}

void setupBLE() {
  if (!BLE.begin()) {
    Serial.println("[BLE] init failed");
    while (1) delay(1000);
  }
  BLE.setDeviceName("Nano33BLE");
  BLE.setLocalName("GestureGamepad");
  BLE.setAdvertisedService(gpService);

  gpService.addCharacteristic(btnChar);
  gpService.addCharacteristic(axesChar);
  BLE.addService(gpService);

  // Initialize characteristic values so a central can read immediately
  uint8_t b = 0;
  int8_t axes[2] = {0, 0};
  btnChar.writeValue(&b, 1);
  axesChar.writeValue((uint8_t*)axes, 2);

  BLE.advertise();
  Serial.println("[BLE] advertising as 'GestureGamepad'");
}

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  Serial.begin(115200);
  while (!Serial && millis() < 2500) { /* wait briefly for monitor */ }

  Wire.begin();
  Wire.setClock(400000); // 400 kHz I2C

  setupAPDS();
  setupMPU();
  setupBLE();
}

void loop() {
  // Handle BLE events
  BLEDevice central = BLE.central();

  if (central) {
    Serial.print("[BLE] Connected: "); Serial.println(central.address());

    // Connected loop
    lastReportMs = 0; // force immediate report
    while (central.connected()) {
      updateAxesFromMPU();
      processGesture();

      uint32_t now = millis();
      if (now - lastReportMs >= REPORT_INTERVAL_MS) {
        bool changed = publishIfChanged();
        if (changed) {
          digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); // blink on update
        }
        lastReportMs = now;

        // Debug print (comment out if too chatty)
        Serial.print("btn=0b"); Serial.print(buttons, BIN);
        Serial.print(" X="); Serial.print(axisX);
        Serial.print(" Y="); Serial.println(axisY);
      }
      // Give time to BLE stack
      BLE.poll();
    }

    Serial.println("[BLE] Disconnected");
  }

  // Not connected: still poll BLE stack
  BLE.poll();
}

Key points:
– The APDS9960 is used in gesture mode only; polling avoids wiring INT.
– The MPU6050 uses accelerometer data to derive tilt angles and map them to gamepad axes.
– BLE exposes a custom Gamepad service with two characteristics (buttons, axes). A host app can subscribe to notifications and interpret the gamepad state.

---

Build / Flash / Run commands

We will use PlatformIO CLI end‑to‑end.

1) Install/verify PlatformIO:

python3 -m pip install --upgrade platformio
pio --version

2) Create project folder and files:

mkdir -p ~/projects/ble-gesture-gamepad/src
cd ~/projects/ble-gesture-gamepad

3) Fetch dependencies and build:

pio pkg install
pio run -e nano33ble

4) Put the board into normal mode (power via USB), then upload:

# Identify the serial port if needed:
pio device list

# Upload firmware
pio run -e nano33ble -t upload

5) Open the serial monitor for debugging output:

pio device monitor -b 115200

6) BLE run procedure:
– Keep the board powered via USB.
– It will advertise as “GestureGamepad”.

Driver notes:
– Windows: The Nano 33 BLE enumerates as a COM port (CDC ACM). No CP210x/CH34x drivers are required.
– macOS/Linux: Appears as /dev/cu.usbmodem (macOS) or /dev/ttyACM (Linux).

---

Step‑by‑Step Validation

1) I2C sanity check (power and address)

• Power the board and open the serial monitor:
• Expect messages like “[APDS9960] gesture sensor enabled” and “[MPU6050] online”.
• If either init fails, revisit wiring. Ensure both sensors share SDA/SCL/GND/3V3.

Optional: Run an I2C scanner sketch (not provided here) if you suspect bus issues. Expected addresses: 0x39 (APDS9960), 0x68 (MPU6050).

2) BLE advertisement

• On a smartphone, open “nRF Connect” (iOS or Android).
• Scan: You should see “GestureGamepad” advertising.
• Tap it and Connect. In the GATT browser you should see:
• Service UUID 12345678‑1234‑5678‑1234‑56789abcdef0
• Characteristics:
  • Buttons (UUID …ef1), length 1
  • Axes (UUID …ef2), length 2

3) Subscribe and observe values

• In nRF Connect, enable notifications (bell icon) on both characteristics.
• With the board flat and stationary:
• Buttons should be 0x00
• Axes near 0,0 (allow slight noise)
• Tilt the board:
• Rolling right should increase X toward +127; left toward −127.
• Pitching forward/back should move Y accordingly.
• Perform gestures over the APDS9960 sensor window:
• Swipe UP: Buttons bit0 set briefly (expect reported byte 0x01 during hold).
• Swipe DOWN: byte 0x02
• Swipe LEFT: byte 0x04
• Swipe RIGHT: byte 0x08
• NEAR: byte 0x10
• FAR: byte 0x20

Because the gesture is latched for GESTURE_HOLD_MS (150 ms), you’ll see the corresponding bit asserted briefly after each gesture, then return to zero.

4) Desktop validation with Python (optional)

If you prefer a desktop BLE central, install bleak and run a quick monitor:

python3 -m pip install bleak

Example script (replace MAC/UUIDs as needed by your OS):

# file: host_monitor.py
import asyncio, struct
from bleak import BleakScanner, BleakClient

SERVICE = "12345678-1234-5678-1234-56789abcdef0"
BTN_UUID = "12345678-1234-5678-1234-56789abcdef1"
AX_UUID  = "12345678-1234-5678-1234-56789abcdef2"

async def main():
    print("Scanning for GestureGamepad...")
    dev = None
    devices = await BleakScanner.discover(timeout=5.0)
    for d in devices:
        if "GestureGamepad" in (d.name or ""):
            dev = d
            break
    if not dev:
        print("Device not found.")
        return

    async with BleakClient(dev) as client:
        print("Connected:", dev)
        async def btn_cb(_, data: bytearray):
            btn = data[0]
            print(f"Buttons=0b{btn:08b}")

        async def ax_cb(_, data: bytearray):
            x, y = struct.unpack("bb", data)
            print(f"Axes: X={x:4d}, Y={y:4d}")

        await client.start_notify(BTN_UUID, btn_cb)
        await client.start_notify(AX_UUID, ax_cb)
        print("Listening (Ctrl+C to quit)...")
        while True:
            await asyncio.sleep(1)

if __name__ == "__main__":
    asyncio.run(main())

Run:

python3 host_monitor.py

Perform gestures and tilts. You should see button bitfields and axis values printed in real time.

5) End‑to‑end checks

• Latency: You should observe <100 ms end‑to‑end from gesture to notification with the default 50 Hz reporting.
• Stability: Axes should be stable near zero when the board is stationary (thanks to low‑pass filtering and dead‑zone).
• BLE reconnection: Disconnect from nRF Connect; the device should resume advertising automatically.

---

Troubleshooting

• No BLE advertisement:
• Ensure BLE.begin() succeeded in the serial log. If not, power‑cycle the board and close any BLE central app that might be caching the connection.

• Avoid multiple centrals connecting at once.

• Upload fails:

• Try: pio run -e nano33ble -t upload –upload-port
• On Windows, check Device Manager for the COM port. On macOS/Linux, check /dev/cu.usbmodem or /dev/ttyACM.

• Press the reset button twice quickly to enter the bootloader (LED pulsing), then retry upload.

• APDS9960 not detected:

• Recheck SDA/SCL orientation. APDS9960 address should be 0x39.

• Some boards need a clean sensor window; ensure there’s no tape/dust blocking the IR.

• MPU6050 not detected:

• Default address is 0x68. If your board ties AD0 high, change code to mpu.begin(0x69).

• Ensure power at 3.3 V; some GY‑521 boards are flaky at 3.3 V if their regulator drops too much—verify with a multimeter. If the breakout expects 5 V only, replace it with a 3.3 V‑friendly version.

• Choppy axes or jitter:

• Increase filter bandwidth smoothing (lower alpha, e.g., 0.15).
• Increase dead‑zone DZ to 6–8.

• Reduce REPORT_INTERVAL_MS to 30–40 ms to lower traffic.

• Gesture misses:

• Adjust APDS9960 gain/LED drive or proximity threshold; ensure good ambient lighting and keep 3–10 cm above the sensor for swipes.

• If polling is insufficient, wire INT to a pin and switch to interrupt‑driven reads (SparkFun library supports this pattern).

• Duplicate I2C pull‑ups:

• Many breakouts include their own pull‑ups; if you have instability on long wires, prefer a single set of ~4.7 kΩ pull‑ups to 3.3 V or keep wiring short.

• BLE central sees raw data but you want native OS “gamepad”:

• This tutorial exposes a custom GATT service. For OS‑recognized gamepad (HID over GATT, HOGP), you’d implement a HID descriptor and HID service. See “Improvements” below.

---

Improvements

• BLE HID Gamepad (HOGP):
• Replace the custom service with a standard HID service (UUID 0x1812) and a Gamepad HID report descriptor (buttons + X/Y axes).
• The ArduinoBLE library includes HID support on Nano 33 BLE in recent versions; you’ll define a HID report map and input report characteristic, then the device will enumerate as a “Gamepad” on hosts that support HOGP.

• This yields native compatibility with games and OS input mapping, removing the need for a host‑side script.

• Interrupt‑driven gesture:

• Connect APDS9960 INT to a digital pin and attach an ISR or event flag to promptly read gestures, reducing latency and power.

• Sensor fusion:

• Use complementary or Kalman filters to blend accelerometer and gyro for smoother axes, especially during dynamic motion.

• The MPU6050 DMP (Digital Motion Processor) can offload some fusion tasks if you adopt a suitable library.

• Calibration routine:

• Record zero‑tilt baseline on startup (press a “calibrate” button), compute offsets, and store in NVM.

• Battery operation:

• Power the Nano 33 BLE with a LiPo + charger backpack and manage advertising intervals for power savings.

• Debounce and gesture customization:

• Add a gesture queue to handle repeated swipes and differentiate short/long gestures mapped to different buttons.

• Expand buttons:

• Use APDS9960 proximity levels to map analog threshold to additional buttons (e.g., “Select/Start”).

---

Final Checklist

• Materials
• Arduino Nano 33 BLE (ABX00030)
• APDS9960 breakout (SparkFun SEN‑12787 or equivalent, 3.3 V)
• MPU6050 breakout (GY‑521 or equivalent, 3.3 V safe)

• Jumpers, USB cable

• Wiring

• 3V3 and GND shared to both sensors
• A4 → SDA on both sensors
• A5 → SCL on both sensors

• Optional: APDS INT → D2, MPU INT → D3

• Software

• PlatformIO Core installed
• platformio.ini configured for nano33ble and libraries
• src/main.cpp created with BLE + APDS9960 + MPU6050 logic
• Build: pio run -e nano33ble
• Upload: pio run -e nano33ble -t upload

• Monitor: pio device monitor -b 115200

• BLE validation

• “GestureGamepad” is advertising
• Connect with nRF Connect
• Subscribe to buttons and axes characteristics
• Swipe: buttons bits change briefly

• Tilt: axes vary in −127..127 range

• Optional host

• bleak installed

• host_monitor.py receives notifications and prints states

• Troubleshooting

• Addressed I2C address mismatches, driver notes, and sensor noise
• Adjusted filter parameters if needed

With this build, you have a working BLE gesture gamepad: APDS9960 handles discrete inputs (D‑pad + buttons), and MPU6050 tilt drives analog axes—streamed over BLE at a fixed rate to any central that subscribes to your custom gamepad service.