-
Notifications
You must be signed in to change notification settings - Fork 13k
feat(voice): add animated waveform visualizer for voice mode state feedback #21115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
ayush31010
wants to merge
31
commits into
google-gemini:main
from
ayush31010:feat/voice-waveform-visualizer
Closed
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
3374238
feat(voice): add animated waveform visualizer for voice mode state fe…
ayush31010 d307109
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 6c0ddb1
Update packages/cli/src/ui/components/AudioWaveform.tsx
ayush31010 4cd0faa
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 4733147
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 83c0900
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 f50befd
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 ac18e8c
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 141e9d9
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 3087e5f
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 46f9079
Merge branch 'main' into feat/voice-waveform-visualizer
spencer426 19e55cd
Merge branch 'main' into feat/voice-waveform-visualizer
spencer426 39eddf5
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 da8a3fd
Merge branch 'main' into feat/voice-waveform-visualizer
spencer426 ea06d7d
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 e5b0a0b
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 8016ec6
Merge branch 'main' into feat/voice-waveform-visualizer
spencer426 4e85185
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 d81615a
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 1c72287
Merge branch 'main' into feat/voice-waveform-visualizer
spencer426 af772a5
fix(voice): update copyright year to 2026 in AudioWaveform files
ayush31010 31752cf
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 8c46df3
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 758e4c6
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 65b3079
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 d51ba9f
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 f0f09bf
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 3c2671c
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 acafaad
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 0b5f73a
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 523330d
Merge branch 'main' into feat/voice-waveform-visualizer
ayush31010 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; | ||
| import { act } from 'react'; | ||
| import { render } from '../../test-utils/render.js'; | ||
| import stripAnsi from 'strip-ansi'; | ||
| import { AudioWaveform } from './AudioWaveform.js'; | ||
| import type { VoiceState } from './AudioWaveform.js'; | ||
|
|
||
| describe('<AudioWaveform />', () => { | ||
| beforeEach(() => { | ||
| vi.useFakeTimers(); | ||
| }); | ||
|
|
||
| afterEach(() => { | ||
| vi.useRealTimers(); | ||
| }); | ||
|
|
||
| // ── idle ───────────────────────────────────────────────────────────────── | ||
|
|
||
| it('renders nothing in idle state', async () => { | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="idle" />, | ||
| ); | ||
| await waitUntilReady(); | ||
| expect(lastFrame({ allowEmpty: true })).toBe(''); | ||
| unmount(); | ||
| }); | ||
|
|
||
| // ── active states render content ───────────────────────────────────────── | ||
|
|
||
| it.each(['listening', 'processing', 'speaking', 'error'] as VoiceState[])( | ||
| 'renders a non-empty waveform in %s state', | ||
| async (state) => { | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state={state} width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| expect(stripAnsi(lastFrame()).trim().length).toBeGreaterThan(0); | ||
| unmount(); | ||
| }, | ||
| ); | ||
|
|
||
| // ── state labels ───────────────────────────────────────────────────────── | ||
|
|
||
| it.each([ | ||
| ['listening', 'listening'], | ||
| ['processing', 'processing'], | ||
| ['speaking', 'speaking'], | ||
| ['error', 'error'], | ||
| ] as Array<[VoiceState, string]>)( | ||
| 'shows "%s" label in %s state', | ||
| async (state, label) => { | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state={state} width={30} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| expect(stripAnsi(lastFrame()).trim()).toContain(label); | ||
| unmount(); | ||
| }, | ||
| ); | ||
|
|
||
| // ── block characters ────────────────────────────────────────────────────── | ||
|
|
||
| it('renders unicode block characters for non-zero amplitudes', async () => { | ||
| const amplitudes = [0.25, 0.5, 0.75, 1.0, 0.75, 0.5, 0.25]; | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="speaking" amplitudes={amplitudes} width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| expect(/[▁▂▃▄▅▆▇█]/.test(lastFrame())).toBe(true); | ||
| unmount(); | ||
| }); | ||
|
|
||
| it('uses all-low bars for all-zero amplitudes', async () => { | ||
| const amplitudes = new Array(10).fill(0); | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="listening" amplitudes={amplitudes} width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| // amplitude=0 maps to index 0 → ' ' (space); higher chars should be absent | ||
| const text = lastFrame(); | ||
| expect(/[▃▄▅▆▇█]/.test(text)).toBe(false); | ||
| unmount(); | ||
| }); | ||
|
|
||
| it('uses all-full bars for all-one amplitudes', async () => { | ||
| const amplitudes = new Array(10).fill(1.0); | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="speaking" amplitudes={amplitudes} width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| // amplitude=1 maps to '█' | ||
| expect(lastFrame()).toContain('█'); | ||
| unmount(); | ||
| }); | ||
|
|
||
| // ── animation ──────────────────────────────────────────────────────────── | ||
|
|
||
| it('generates synthetic animation in listening state (frames differ over time)', async () => { | ||
| const { lastFrameRaw, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="listening" width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| const frame1 = stripAnsi(lastFrameRaw()).trim(); | ||
|
|
||
| // Advance by enough ticks to guarantee a visually different frame. | ||
| // The sine wave shifts by tick*0.3 rad/tick; ~21 ticks complete one full | ||
| // cycle (2π / 0.3 ≈ 21). 500ms at 80ms/tick = ~6 ticks — well within | ||
| // the first half-period where every tick changes at least one bar. | ||
| // Use lastFrameRaw() (Ink's raw output) rather than lastFrame() (xterm | ||
| // buffer) to avoid the async write queue blocking the read. | ||
| await act(async () => { | ||
| await vi.advanceTimersByTimeAsync(500); | ||
| }); | ||
| const frame2 = stripAnsi(lastFrameRaw()).trim(); | ||
|
|
||
| // Both frames should be non-empty and the waveform should have changed. | ||
| expect(frame1.length).toBeGreaterThan(0); | ||
| expect(frame2.length).toBeGreaterThan(0); | ||
| expect(frame1).not.toBe(frame2); | ||
| unmount(); | ||
| }); | ||
|
|
||
| it('error state is static (no animation)', async () => { | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform state="error" width={20} />, | ||
| ); | ||
| await waitUntilReady(); | ||
| const frame1 = stripAnsi(lastFrame()).trim(); | ||
|
|
||
| await act(async () => { | ||
| await vi.advanceTimersByTimeAsync(800); | ||
| }); | ||
| const frame2 = stripAnsi(lastFrame()).trim(); | ||
|
|
||
| expect(frame1).toBe(frame2); | ||
| unmount(); | ||
| }); | ||
|
|
||
| // ── width ───────────────────────────────────────────────────────────────── | ||
|
|
||
| it('respects a narrow width', async () => { | ||
| const { lastFrame, waitUntilReady, unmount } = render( | ||
| <AudioWaveform | ||
| state="listening" | ||
| amplitudes={new Array(5).fill(1.0)} | ||
| width={10} | ||
| />, | ||
| ); | ||
| await waitUntilReady(); | ||
| // Strip ANSI and label; remaining bar section should be short | ||
| const text = stripAnsi(lastFrame()).replace('listening', '').trim(); | ||
| // bar section width ≤ requested width | ||
| expect(text.length).toBeLessThanOrEqual(10); | ||
| unmount(); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2026 Google LLC | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import type React from 'react'; | ||
| import { useEffect, useState } from 'react'; | ||
| import { Text, Box } from 'ink'; | ||
|
|
||
| export type VoiceState = | ||
| | 'idle' | ||
| | 'listening' | ||
| | 'processing' | ||
| | 'speaking' | ||
| | 'error'; | ||
|
|
||
| export interface AudioWaveformProps { | ||
| state: VoiceState; | ||
| /** | ||
| * Amplitude samples in [0, 1]. Array length determines the number of bars | ||
| * rendered. When omitted a synthetic animation is generated. | ||
| * @default 20 bars (synthetic) | ||
| */ | ||
| amplitudes?: number[]; | ||
| /** | ||
| * Total width available in terminal columns (bars + label). | ||
| * @default 40 | ||
| */ | ||
| width?: number; | ||
| } | ||
|
|
||
| // Unicode block characters ordered from empty → full (9 levels, indices 0-8). | ||
| const BARS = ' ▁▂▃▄▅▆▇█'; | ||
|
|
||
| const STATE_COLOR: Record<VoiceState, string> = { | ||
| idle: 'gray', | ||
| listening: 'green', | ||
| processing: 'yellow', | ||
| speaking: 'cyan', | ||
| error: 'red', | ||
| }; | ||
|
|
||
| const STATE_LABEL: Record<VoiceState, string> = { | ||
| idle: '', | ||
| listening: ' listening', | ||
| processing: ' processing', | ||
| speaking: ' speaking', | ||
| error: ' error', | ||
| }; | ||
|
|
||
| const ANIMATION_INTERVAL_MS = 80; | ||
|
|
||
| /** | ||
| * Returns a synthetic amplitude array for animated states. | ||
| * | ||
| * - listening / speaking: rippling sine wave across bar positions | ||
| * - processing: all bars pulse together (breathing effect) | ||
| */ | ||
| function syntheticAmplitudes( | ||
| tick: number, | ||
| barCount: number, | ||
| state: VoiceState, | ||
| ): number[] { | ||
| if (state === 'processing') { | ||
| const pulse = 0.2 + 0.6 * Math.abs(Math.sin((tick * Math.PI) / 15)); | ||
| return Array.from({ length: barCount }, () => pulse); | ||
| } | ||
| return Array.from({ length: barCount }, (_, i) => { | ||
| const phase = (i / barCount) * Math.PI * 2; | ||
| return 0.3 + 0.6 * Math.abs(Math.sin(phase + tick * 0.3)); | ||
| }); | ||
| } | ||
|
|
||
| function amplitudeToChar(amp: number): string { | ||
| const index = Math.round(Math.max(0, Math.min(1, amp)) * (BARS.length - 1)); | ||
| return BARS[index] ?? '█'; | ||
| } | ||
|
|
||
| /** | ||
| * Animated terminal waveform that visualises the current voice session state. | ||
| * | ||
| * Renders nothing in `idle` state. In all other states a bar chart built from | ||
| * Unicode block characters is shown alongside a text label: | ||
| * | ||
| * ``` | ||
| * ▃▄▆█▇▅▃▂▁▂▃▅▆▇█▇▆▅▃▂ listening | ||
| * ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ processing | ||
| * ``` | ||
| */ | ||
| export function AudioWaveform({ | ||
| state, | ||
| amplitudes, | ||
| width = 40, | ||
| }: AudioWaveformProps): React.ReactElement | null { | ||
| const isAnimated = | ||
| state === 'listening' || state === 'processing' || state === 'speaking'; | ||
|
|
||
| const [tick, setTick] = useState(0); | ||
|
|
||
| useEffect(() => { | ||
| if (!isAnimated) return; | ||
| const id = setInterval(() => setTick((t) => t + 1), ANIMATION_INTERVAL_MS); | ||
| return () => clearInterval(id); | ||
| }, [isAnimated]); | ||
|
|
||
| if (state === 'idle') return null; | ||
|
|
||
| const label = STATE_LABEL[state]; | ||
| // Reserve columns for the label so the total stays within `width`. | ||
| const barCount = Math.max(0, width - label.length); | ||
ayush31010 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| let bars: number[]; | ||
| if (amplitudes && amplitudes.length > 0) { | ||
| // Resample caller-supplied amplitudes to fit exactly `barCount` bars. | ||
| bars = Array.from({ length: barCount }, (_, i) => { | ||
| const srcIdx = Math.round( | ||
| (i / (barCount - 1 || 1)) * (amplitudes.length - 1), | ||
| ); | ||
| return amplitudes[Math.min(srcIdx, amplitudes.length - 1)] ?? 0; | ||
| }); | ||
| } else if (isAnimated) { | ||
| bars = syntheticAmplitudes(tick, barCount, state); | ||
| } else { | ||
| // error state without supplied amplitudes: low flat line | ||
| bars = Array.from({ length: barCount }, () => 0.15); | ||
| } | ||
|
|
||
| const color = STATE_COLOR[state]; | ||
| const waveform = bars.map(amplitudeToChar).join(''); | ||
|
|
||
| return ( | ||
| <Box> | ||
| <Text color={color}>{waveform}</Text> | ||
| <Text color={color} bold> | ||
| {label} | ||
| </Text> | ||
| </Box> | ||
| ); | ||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.