Skip to main content

Voice Translation

Voice Translation allows you to understand in real time what others say during games or voice calls in VR, just like reading subtitles, and it can also translate your own voice in real time for others. It runs entirely locally—audio never has to be uploaded, keeping your privacy secure.

Feature Overview​

Voice Translation captures and recognizes two audio streams simultaneously, translating each independently:

  • Speaker Channel (loopback capture): Captures the voices of others during games, voice calls, or meetings, then recognizes and translates them into your native language, displayed as an overlay in front of you, achieving a "real-time subtitle" effect.
  • Microphone Channel: Captures your own speech, then recognizes it and translates it into the target language (such as English), making it easy to communicate with foreign players in multilingual social settings (such as VRChat).

The recognition language, translation language, and display style of each channel are configured independently and do not affect each other.

How It Works​

Voice Translation is accomplished cooperatively by the main program and an independent ASR (Automatic Speech Recognition) worker process:

  1. The main program communicates with the ASR subprocess through a named pipe, capturing microphone and speaker (system loopback) audio.
  2. The ASR subprocess uses Silero VAD for real-time voice activity detection, automatically segmenting the start and end of each utterance.
  3. The detected speech segments are transcribed locally into text by the Qwen3-ASR model (pure CPU inference based on llama.cpp).
  4. The transcribed results are then handed to the translation engine for translation (supporting local translation models or your configured custom translation API), and finally displayed in real time in the main program's overlay.

The entire process happens on your machine; audio never leaves your computer.

Audio Sources​

  • Microphone: Captures your own voice. The device list shows available input devices.
  • Speaker (loopback capture): Captures the sound currently being played by the system, i.e., the voices of others in the game or voice call.

In the Voice Translation Settings page, in the "Audio Device Settings" card, you can select a specific device for the microphone and speaker separately (click "Refresh Device List" to re-enumerate available devices).

Automatic Language Recognition​

The speech recognition language is set to "Auto" by default, in which case the ASR will determine the language being spoken on its own, with no manual switching needed. You can also choose to specify a fixed recognition language for the speaker and microphone separately in settings, to improve recognition accuracy for specific languages.

Streaming Output​

After enabling "Simulated Streaming Output", recognition results are displayed in real time in increments of roughly once per second (intermediate results appear first, and are replaced once the final result stabilizes), making the display feel more like real-time subtitles; when disabled, the full result is displayed at once after an utterance is completely recognized. You can switch this at any time in the Voice Translation Settings.

Pause and Resume​

In voice translation mode, a single short press of the trigger will pause / resume speech recognition. When paused, the model remains in memory, so resuming requires no reload and is very fast. Flicking the left or right joystick clears the currently displayed voice translation result.

For detailed operational pacing in the mode, please get familiar with it through the software's built-in in-VR step-by-step tutorial.

Translation Engine Selection​

The translation engine for voice translation has two modes to choose from:

  • Local: Uses the bundled Hy-MT2 local translation model for offline translation, with no internet connection or API keys required, running fully locally. The local model automatically unloads after 30 seconds of inactivity to free memory, and reloads on demand when needed.
  • Custom: Uses your own API keys to access third-party translation services (general machine translation or LLM large models), offering a richer choice of languages and quality.

Note: Voice Translation currently does not support the "Built-in Translation Service" mode; only local or custom modes are supported. Please configure it in the "Voice Translation Engine Settings" card on the Translation Engine page.

VRChat Chatbox Sending​

When enabled, voice translation results are automatically sent to VRChat's Chatbox via the OSC protocol, so your translations appear in the VRChat interface in real time. You can control separately:

  • Whether to send the translation of the speaker (what others say)
  • Whether to send the translation of the microphone (what you say)
  • The OSC port (VRChat's default receiving port is 9000)
  • Message mode: Comparison mode (translation on top, original below) or translation-only mode
  • Number of retained messages (1 to 10)

Subject to VRChat Chatbox limitations, a single message supports at most about 135 characters and up to 9 lines; overly long content wraps automatically. There is also a minimum interval between two sends to ensure the chat box displays stably.

Memory Requirements​

  • Speech recognition model: Selecting "Fast (0.6B)" requires about 2.5 GB of available memory; selecting "High-Precision (1.7B)" requires about 3 GB of available memory. If the system's available memory is insufficient, a prompt will be shown and loading will be skipped.
  • Local translation model (Hy-MT2): When local translation is enabled, the model is additionally loaded as needed, so please reserve enough available memory.

The number of CPU inference cores can be adjusted in settings (range 1 to the device core count, default 4). A higher core count improves recognition and translation speed, but consumes more computing resources.

Usage Steps​

  1. Make sure the software is installed and launched, and the VR device is running.
  2. Enable "Automatically enable voice translation at startup" in Basic Settings (or manually enable it after launching while in voice translation mode).
  3. In the Translation Engine page, in the "Voice Translation Engine Settings", select a local or custom translation engine and complete the configuration.
  4. In the Voice Translation settings page, confirm that the microphone and speaker devices, recognition language, and translation language are all set correctly.
  5. In VR (in voice translation mode), long press the trigger for about 1 second to start voice translation, and you can see the translation of others or yourself in real time; short press the trigger to pause / resume recognition.

For more detailed setting descriptions, please refer to Voice Translation Settings.