ESP32 StreamDeck-Like Touchscreen Interface

Short Description

This project transforms an ESP32 microcontroller into a StreamDeck-like touchscreen interface featuring three customizable tabs—Apps, Commands, and Custom—each arranged in a 3x3 grid. Utilizing a modern UI with consistent spacing, vibrant colors, and rounded buttons, it allows users to launch applications, execute system commands, and perform user-defined actions seamlessly via touch. The system leverages BLE keyboard functionality for effortless integration with other devices and supports dynamic customization through SD card-loaded commands.

Detailed Overview

Project Overview

The ESP32 StreamDeck-like touchscreen interface is designed to provide users with a compact and versatile control panel for managing applications and executing commands effortlessly. By mimicking the functionality of professional StreamDeck devices, this project offers a cost-effective and customizable solution ideal for streamers, developers, and power users seeking streamlined workflows.

Key Features

  • Three Tabs:
    • Apps: Launch frequently used applications with a single tap.
    • Commands: Execute predefined system commands such as copy, paste, save, etc.
    • Custom: Perform user-defined actions loaded from an SD card, allowing for extensive customization.
  • 3x3 Grid Layout: Each tab contains a 3x3 grid of buttons, providing nine actionable items per tab for efficient organization and access.
  • Modern UI Design: The interface boasts consistent spacing, modern color schemes, and rounded-corner buttons, enhancing both aesthetics and usability.
  • BLE Keyboard Integration: Utilizes Bluetooth Low Energy (BLE) to emulate a keyboard, enabling the ESP32 to send keyboard shortcuts and text inputs to connected devices seamlessly.
  • Dynamic Customization: Supports loading custom commands from an SD card, allowing users to tailor the interface to their specific needs without modifying the code.

Hardware Components

  • ESP32 Microcontroller: Serves as the core processing unit, handling touch inputs, UI rendering, and BLE communication.
  • TFT_eSPI Display: A 320x240 pixel touchscreen that renders the user interface.
  • XPT2046 Touch Controller: Manages touch inputs from the TFT display.
  • SD Card Module: Stores custom commands in a text file (custom_cmds.txt) for dynamic loading.
  • BLE Keyboard Module: Facilitates BLE-based keyboard emulation, enabling the ESP32 to send keystrokes to paired devices.

Software Structure

The software is organized into several key sections, each responsible for different aspects of the system's functionality:

  • Initialization (setup Function):
    • Initializes serial communication for debugging.
    • Mounts the SD card and loads custom commands from custom_cmds.txt.
    • Initializes the touchscreen and TFT display.
    • Begins BLE keyboard communication.
    • Draws the initial user interface.
  • Main Loop (loop Function):
    • Continuously checks for touch events.
    • Debounces touch inputs to prevent accidental multiple triggers.
    • Handles touch interactions by determining whether the touch occurred on the tab bar or within the grid area.
  • User Interface Rendering:
    • drawUI Function: Clears the screen and redraws the tab bar and the current tab's grid.
    • drawTabBar Function: Renders the top tab bar with three tabs, highlighting the currently selected tab.
    • drawTabGrid Function: Draws a 3x3 grid of buttons based on the active tab, displaying appropriate labels.
    • drawRoundedButton Function: Renders individual buttons with rounded corners and centered labels.
  • Touch Handling:
    • handleTouch Function: Determines the location of the touch and delegates actions based on whether a tab or grid button was pressed.
    • handleGridButtonPress Function: Executes actions corresponding to the pressed button, varying by the active tab.
  • Action Execution:
    • Apps Tab: Launches applications by simulating "Cmd + Space," typing the application name, and pressing Enter.
    • Commands Tab: Executes predefined system commands by sending specific key combinations (e.g., Cmd-S for Save).
    • Custom Tab: Performs user-defined actions, which can include typing text or sending complex hotkey sequences loaded from the SD card.
  • Custom Command Handling:
    • loadCustomCommands Function: Reads custom commands from custom_cmds.txt, parsing each line into label-command pairs.
    • populateCustomLabels Function: Converts loaded custom commands into C-style strings for display on the Custom tab buttons.
    • handleCustom Function: Processes custom commands, distinguishing between text typing actions and hotkey sequences.
    • parseAndSendHotkey Function: Parses hotkey strings (e.g., "SHIFT+CMD+4") and sends the corresponding key presses via BLE.
  • Helper Functions:
    • sendKeyCombo Function: Presses and releases a combination of modifier and main keys.
    • pressKey Function: Maps string representations of keys (e.g., "SHIFT") to actual key codes and presses them.
    • typeText Function: Types out a string character by character with slight delays to emulate natural typing.
Download ZIP

Function Descriptions

Initialization Functions

  • setup()
    • Initializes serial communication for debugging.
    • Mounts the SD card and loads custom commands from custom_cmds.txt using loadCustomCommands().
    • Converts loaded custom commands into displayable labels with populateCustomLabels().
    • Initializes the touchscreen and TFT display with appropriate rotations and fills the background.
    • Starts BLE keyboard communication.
    • Draws the initial user interface with drawUI().

Main Loop

  • loop()
    • Continuously monitors for touch events.
    • When a touch is detected, it maps the raw touch coordinates to screen coordinates.
    • Calls handleTouch(xTouch, yTouch) to process the touch event.
    • Implements a basic debounce with a delay to prevent multiple triggers from a single touch.

User Interface Rendering

  • drawUI()
    • Clears the screen with the background color.
    • Draws the tab bar using drawTabBar().
    • Renders the current tab's grid of buttons with drawTabGrid().
  • drawTabBar()
    • Draws the background for the entire tab bar.
    • Iterates through each tab (Apps, Commands, Custom) and highlights the selected tab with a different color.
    • Centers and displays the tab labels within their respective sections.
  • drawTabGrid()
    • Determines which set of labels to use based on the active tab.
    • Iterates through each cell in the 3x3 grid, calculating the position and drawing each button with drawRoundedButton().
  • drawRoundedButton(int x, int y, int w, int h, uint16_t bgColor, const char* label)
    • Draws a rounded rectangle for the button background.
    • Adds a border around the button.
    • Centers and displays the button label within the button area.

Touch Handling

  • handleTouch(int x, int y)
    • Checks if the touch occurred on the tab bar. If so, it switches the current tab and redraws the UI.
    • If the touch is within the grid area, it determines which button was pressed and calls handleGridButtonPress(index).
  • handleGridButtonPress(int index)
    • Identifies the label of the pressed button based on the active tab.
    • Depending on the tab, it calls the appropriate function to execute the action:
      • Apps: Calls pressCmdSpaceAndType(label) to launch the application.
      • Commands: Calls handleCommand(label) to execute the system command.
      • Custom: Calls handleCustom(label, index) to perform the custom action.

Action Execution

  • Apps Tab:
    • pressCmdSpaceAndType(const char* text)
      • Simulates pressing "Cmd + Space" to open the application launcher.
      • Types the application name.
      • Presses "Enter" to launch the application.
  • Commands Tab:
    • handleCommand(const char* label)
      • Matches the label to predefined commands (e.g., "Cmd-S" for Save).
      • Calls sendKeyCombo(modifierKey, mainChar) to execute the command.
    • sendKeyCombo(uint8_t modifierKey, char mainChar)
      • Presses the modifier key (e.g., Cmd) and the main character key (e.g., 'S').
      • Releases all keys after a short delay.
  • Custom Tab:
    • handleCustom(const char* label, int index)
      • Retrieves the command associated with the custom button from the loaded commands.
      • If the command starts with "Type:", it extracts and types the subsequent text using typeText(text).
      • Otherwise, it interprets the command as a hotkey sequence and calls parseAndSendHotkey(commandStr).
    • parseAndSendHotkey(const String& hotkey)
      • Splits the hotkey string by '+' to identify individual keys (e.g., "SHIFT", "CMD", "4").
      • Presses each identified key using pressKey(token).
      • Releases all keys after a short delay.
    • pressKey(const String& keyStr)
      • Maps string representations of keys to their corresponding key codes (e.g., "SHIFT" to KEY_LEFT_SHIFT).
      • Presses the identified key using the BLE keyboard.

Helper Functions

  • loadCustomCommands()
    • Initializes the SD card.
    • Opens and reads the custom_cmds.txt file line by line.
    • Parses each line into a label and command pair, storing them in the customCommands array.
  • populateCustomLabels()
    • Copies the labels from the customCommands array into the customCmdLabels C-string array for display purposes.
  • typeText(const String& text)
    • Iterates through each character in the provided text string.
    • Sends each character as a keystroke via the BLE keyboard with a short delay between characters to emulate natural typing.

Usage Instructions

Hardware Setup

  • Connect the TFT touchscreen and XPT2046 touch controller to the ESP32 using the defined pin configurations.
  • Insert an SD card into the SD card module. Ensure the SD card contains a custom_cmds.txt file with custom commands formatted as Label|Command per line.
  • Power the ESP32 and pair it with your target device as a BLE keyboard.

Software Setup:

  • Ensure all required libraries (TFT_eSPI, XPT2046_Touchscreen, BleKeyboard, SD, FS) are installed in your Arduino IDE.
  • Download the ESP32_StreamDeck_Interface.zip file.
  • Extract the ZIP file and replace the User_Setup.h file in the TFT_eSPI library with the one provided in the ZIP.
  • Important: If you are using an ESP32 board with different pinouts, ensure that the pin configurations in the User_Setup.h file match your hardware setup.
  • The system is configured to work only with the US keyboard layout. Ensure your target device is set to the US keyboard layout for proper functionality.
  • This interface is compatible with macOS, iOS (iPhone and iPad).
  • Upload the provided code to the ESP32 using the Arduino IDE.

Defining Custom Commands

  • Create a custom_cmds.txt file on the SD card with each line defining a custom command in the format Label|Command.
  • Example:
    • Screen Shot|SHIFT+CMD+4
    • Open Terminal|Type:Terminal
  • Insert the SD card into the ESP32 and restart the device to load the custom commands.

Operating the Interface

  • Navigate between the Apps, Commands, and Custom tabs by tapping the respective tabs on the top bar.
  • Tap any button within the 3x3 grid to execute the associated action:
    • Apps: Launches the selected application.
    • Commands: Executes the predefined system command.
    • Custom: Performs the user-defined action, which could be typing text or executing a hotkey sequence.

Troubleshooting

  • Monitor the serial output for debugging information such as touch coordinates, loaded commands, and action executions.
  • Ensure the SD card is properly formatted and the custom_cmds.txt file follows the correct syntax.
  • Verify that the ESP32 is successfully paired as a BLE keyboard with the target device.

Customization and Expansion

  • Adding More Buttons:
    • To expand beyond the 3x3 grid, adjust the GRID_COLS and GRID_ROWS constants and update the UI rendering logic accordingly.
  • Additional Tabs:
    • Introduce more tabs by extending the Tab enum and updating the tab bar rendering and touch handling functions to accommodate additional sections.
  • Enhanced Command Parsing:
    • Improve the parseAndSendHotkey function to support more complex key sequences and special keys as needed.
  • UI Enhancements:
    • Modify the color scheme, button styles, and font sizes to match personal preferences or specific application themes.
  • Performance Optimization:
    • Adjust delays and optimize touch handling to achieve faster response times and a smoother user experience.

Conclusion

This ESP32-based StreamDeck-like touchscreen interface offers a powerful and flexible tool for managing applications and executing commands with ease. By combining robust hardware components with a well-structured and customizable software framework, it provides users with a seamless and efficient control panel tailored to their specific needs. Whether for streaming, development, or general productivity, this project exemplifies the potential of embedded systems in creating intuitive and user-friendly interfaces.

Back to Projects