WIP: Ffmpeg decoder, encoders and filters.#10990
Draft
lgirdwood wants to merge 14 commits into
Draft
Conversation
Add a loadable (LLEXT) audio module that wraps FFmpeg. Two build modes: - decoder (default): a compressed elementary stream (FLAC/AAC/Opus, selected via Kconfig) is parsed and decoded to PCM (process_raw_data); - filter (CONFIG_FFMPEG_DEC_FILTER_MODE): a PCM source/sink effect that runs an FFmpeg audio filter graph (afftdn noise reduction). FFmpeg is a west-pinned source (see west.yml) cross-built by ffmpeg.cmake as a CMake ExternalProject, enabling only the Kconfig-selected decoders and filters. The module supplies the libc surface FFmpeg needs that the SOF core does not export to LLEXT (ffmpeg_dec-shims.c), a SOF-heap-backed malloc (ffmpeg_dec-alloc.c), fast single-precision float math (fastmathf.c), and routes av_log() into the Zephyr log. The avfilter graph backend and the PCM effect ops are in ffmpeg_dec-filter.c. Also adds the rimage module manifest (ffmpeg_dec.toml + per-platform includes), the topology widget (ffmpeg_dec.conf), host test tooling (ffmpeg_dec_prepare.sh) and docs (README/TESTING). Verified building and load-ready (all symbols resolved, signed) for ace30 (ptl) with the Zephyr SDK: FLAC and FLAC+AAC decoders, and the afftdn filter effect. Runtime decode/denoise verification is pending hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the ffmpeg_dec component to the host->component->host benchmark harness so a topology instantiating the module can be built and driven (testbench or target). Adds the per-format bench configs (generated with bench_comp_generate.sh), registers the widget include and the ffmpeg_dec32 BENCH_CONFIG in cavs-benchmark-hda.conf, and adds ffmpeg_dec to the s32 component list in tplg-targets-bench.cmake (the filter effect path processes S32). Also drop the codec-setup bytes control from the ffmpeg_dec widget class so it is a plain 1-in/1-out effect; the decoder's STREAMINFO control is added at the instance level in a decoder topology instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_MP3: enables the libavcodec mp3 decoder and mpegaudio parser in the cross-build, the FFMPEG_DEC_CODEC_MP3 codec id mapping, and the float math layer (the mp3 decoder uses it). Verified building for ace30 (ptl) with decoders [flac,mp3]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FFmpeg has no native MP3 encoder, so add MP3 encode through libshine, a small fixed-point encoder (good fit for a DSP - no float dependency). libshine is a west-pinned project; ffmpeg.cmake cross-builds it (its lib needs no config.h and its autotools CLI/shared link cannot work bare-metal, so the objects are compiled and archived directly), generates a shine.pc, and enables --enable-libshine --enable-encoder=libshine in FFmpeg. FFmpeg's require_pkg_config link test for -lshine pulls newlib malloc and hence Zephyr-runtime symbols (z_errno_wrap, ...) that only exist at module load; a configure-test-only stub is passed via --extra-ldflags so the test links (--extra-ldflags does not affect the static-archive build). Gated by CONFIG_FFMPEG_ENC_MP3. Verified building for ace30 (ptl): libshine.a cross-builds and libavcodec.a contains the libshine encoder. A module encode path (PCM->MP3) is a separate build mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_ENCODE_MODE: build the module as an encoder using the modern .process_raw_data path in reverse of the decoder - avcodec_send_frame/avcodec_receive_packet. ffmpeg_dec-encode.c opens the libshine MP3 encoder, converts SOF interleaved S32 to the encoder sample format per frame, and emits the compressed elementary stream. The interface ladder in ffmpeg_dec.c selects encoder / filter / decoder ops by Kconfig. libshine is linked into the module (after libavcodec, which references it) from its own cross-build install dir. ffmpeg.cmake now allows an encoder-only build (no decoder) and makes the decoder configure flags conditional. The encoder path pulls a few more libc symbols, added to the local surface: calloc/posix_memalign (alloc), modf/localtime_r/iconv*/__xpg_strerror_r (shims). fastmathf is now always built for the real backend (its sofm_log2f backs the shims' log10()). Verified building and load-ready (no unresolved externals, signed) for ace30 (ptl) as an encoder-only module. Runtime encode + real-time framing need hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add HIFI.md: analysis of the hot audio-processing paths in the module (decode/ filter/encode) and the linked FFmpeg DSP, and where Xtensa HiFi intrinsics would help. FFmpeg is built --disable-asm (scalar C on Xtensa) and has no _xtensa DSP init, so the generic C kernels run. Documents: the module's own PCM conversion loops (easy HiFi wins we own), FFmpeg's DSP dispatch contexts (float_dsp, flacdsp, mpegaudiodsp, tx) as fork-patch targets mirroring the other arch inits, fastmathf vectorisation, and a cost/benefit priority order. References SOF's existing HiFi kernels (src/math/*_hifi*) as the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the webrtc_vad SOF module wrapping libfvad — a standalone C port of the WebRTC Gaussian mixture model VAD (identical to the algorithm shipping in Chrome/WebRTC since 2011). Module characteristics: - Single-source, single-sink PCM pass-through - Accumulates 10/20/30 ms frames (configurable via Kconfig) - Broadcasts NOTIFIER_ID_VAD event on every frame classification - Supports S16_LE and S32_LE at 8/16/32/48 kHz, mono or stereo - Fixed-point; no FPU required - Two backends: real (libfvad) and pass-through stub for CI/LLEXT Build: - libfvad is cross-compiled by src/audio/webrtc_vad/webrtc_vad.cmake - Source is fetched via west (modules/audio/libfvad, pinned SHA 532ab666c20d) - LLEXT packaging supported via llext/ subdirectory Add NOTIFIER_ID_VAD to the notifier_id enum so downstream components (e.g. webrtc_ns2 / RNNoise) can share the same VAD event channel. UUID: c790b11d-5d14-e54e-be36ba4ad732cc14 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns SOF module wrapping the classic WebRTC Noise Suppression algorithm (spectral Wiener filter, fixed-point) from the webrtc-audio-processing 0.3.1 library. Module characteristics: - Single-source, single-sink PCM effect - Fixed-point implementation; no FPU required - Operates at 8 or 16 kHz on 10 ms frames (160/80 samples) - Supports S16_LE and S32_LE, mono or stereo (per-channel instances) - Four suppression levels: Mild/Medium/Aggressive/VeryAggressive (configurable at build time or via IPC4 set_configuration) - Two backends: real (WebRTC NS) and pass-through stub for CI/LLEXT - Designed as a build-time alternative to webrtc_ns2 (RNNoise): lower CPU, no 48 kHz constraint, fully fixed-point Build: - WebRTC NS subset is cross-compiled by src/audio/webrtc_ns/webrtc_ns.cmake (6 C files from modules/audio/webrtc-apm) - Source fetched via west (modules/audio/webrtc-apm, tag v0.3.1) - LLEXT packaging supported UUID: 0fc8faef-945f-004b-8d5f315047f1136a Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_aec SOF module wrapping WebRTC AECm (Acoustic Echo Canceller Mobile) — the fixed-point Q15 echo canceller designed for mobile and embedded devices. Module characteristics: - Dual-source (2 input pins): pin 0 = microphone capture, pin 1 = playback echo reference - Single-sink: echo-cancelled microphone output - Fixed-point Q15; no FPU required - Operates at 8 or 16 kHz on 10 ms frames - Supports S16_LE and S32_LE, up to WEBRTC_AEC_CHANNELS_MAX channels (one AecmCore handle per channel) - Pipeline-ID heuristic (matching google_rtc_audio_processing) used to distinguish mic vs. reference at prepare() time - Two backends: real (AECm) and pass-through stub for CI/LLEXT Pin binding follows the same pattern as google-rtc-aec: both DAI capture copiers are connected as sources; the firmware resolves mic-vs-ref from pipeline membership. Reference topology: tools/topology/topology2/development/ cavs-nocodec-webrtc-aec.conf (dual SSP: SSP0=mic, SSP2=ref) UUID: e5da7b5b-133a-ba46-b517651d6300bb83 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns2 SOF module wrapping xiph/rnnoise — a recurrent neural network (GRU-based) noise suppressor originally developed by Jean-Marc Valin at Mozilla. Module characteristics: - Single-source, single-sink PCM effect - Floating-point; internal float math, hardware FPU beneficial - Hard requirement: pipeline sample rate MUST be 48000 Hz - Processes 480-sample (10 ms) frames; partial periods accumulated internally - Supports S16_LE and S32_LE, mono or stereo (one DenoiseState per channel; up to WEBRTC_NS2_CHANNELS_MAX channels) - Float scale bridging: pipeline samples normalised to ±1.0 are scaled to RNNoise's native ±32768 range and back - VAD dual-use: rnnoise_process_frame() returns per-frame speech probability [0.0, 1.0]; this fires NOTIFIER_ID_VAD events at a configurable threshold (WEBRTC_NS2_VAD_THRESHOLD_PCT) - Two backends: real (RNNoise) and pass-through stub for CI/LLEXT Design notes: - rnnoise_init() used at prepare/reset instead of rnnoise_create() to avoid per-init heap allocation (embedded-friendly path) - Model weights (rnn_data.c) are const float[] placed in .rodata (Flash/ROM); ~85-340 KB depending on model variant - Peak stack ~12-16 KB per rnnoise_process_frame() call - No expf/tanhf in hot path: replaced by 201-entry tansig LUT Build: - RNNoise cross-compiled by src/audio/webrtc_ns2/webrtc_ns2.cmake (6 C files: denoise.c rnn.c rnn_data.c pitch.c celt_lpc.c kiss_fft.c) - Source fetched via west (modules/audio/rnnoise, SHA 70f1d256) - LLEXT packaging supported UUID: eacfacdc-2a87-c942-97e894c917a740db Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Register all four WebRTC audio processing modules in the SOF build
system:
Kconfig (src/audio/Kconfig):
- rsource the four module Kconfig files (alphabetical order)
CMakeLists (src/audio/CMakeLists.txt):
- add_subdirectory() guards for COMP_WEBRTC_{VAD,NS,AEC,NS2}
UUID registry (uuid-registry.txt):
- c790b11d-5d14-e54e-be36ba4ad732cc14 webrtc_vad
- 0fc8faef-945f-004b-8d5f315047f1136a webrtc_ns
- e5da7b5b-133a-ba46-b517651d6300bb83 webrtc_aec
- eacfacdc-2a87-c942-97e894c917a740db webrtc_ns2
west.yml:
- Add remotes: libfvad (dpirch), webrtc-apm (freedesktop.org),
rnnoise (xiph)
- Add project entries with pinned revisions:
libfvad @ 532ab666c20d → modules/audio/libfvad
webrtc-apm v0.3.1 → modules/audio/webrtc-apm
rnnoise @ 70f1d256 → modules/audio/rnnoise
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add Topology 2.0 component widgets, capture pipeline templates, and
standalone development/test topologies for all four WebRTC modules.
Component widgets (include/components/):
webrtc-vad.conf — pass-through VAD; S16/S32; any rate; 1→1
webrtc-ns.conf — fixed-point spectral NS; 8/16 kHz; S16/S32; 1→1
webrtc-aec.conf — AECm echo canceller; 10ms IBS/OBS; 2 input pins
(pin 0 = mic, pin 1 = echo ref); S16/S32; 2→1
webrtc-ns2.conf — RNNoise GRU NS; 48 kHz only; S16/S32; 1→1
Capture pipeline templates (include/pipelines/cavs/):
webrtc-ns-capture.conf — DAI-copier → VAD → NS → module-copier
webrtc-aec-capture.conf — copier → AEC(2-pin) → copier
webrtc-ns2-capture.conf — RNNoise → module-copier (48 kHz locked)
Development/test topologies (development/):
cavs-nocodec-webrtc-ns.conf — single SSP loopback; VAD+NS
cavs-nocodec-webrtc-aec.conf — dual SSP (SSP0=mic, SSP2=ref); AEC
mirrors cavs-nocodec-rtcaec.conf
cavs-nocodec-webrtc-ns2.conf — single 48 kHz SSP; RNNoise
All three development topologies registered in tplg-targets.cmake
targeting TGL nocodec with NHLT preprocessing.
Compile example:
alsatplg -c development/cavs-nocodec-webrtc-aec.conf -D PLATFORM=tgl -o sof-tgl-nocodec-webrtc-aec.tplg
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the root Kconfig that exposes the ffmpeg_dec module configuration tree (codec selection, filter mode, cold-split experimental option) and the standalone ffmpeg.cmake cross-build helper for out-of-tree builds. These files were omitted from the earlier ffmpeg_dec commits and are grouped here to keep all ffmpeg_dec build system pieces together. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…and webrtc modules Create and update README.md documentation for the FFmpeg decoder/encoder/filter module and all four WebRTC audio modules: - ffmpeg_dec/README.md: Added Mermaid architecture diagram, detailed decoder/encoder/filter configurations, build choices, and topology guides. - webrtc_vad/README.md: Included Mermaid data flow, GMM classification details, ring-buffer accumulation behavior, and notification topology mappings. - webrtc_ns/README.md: Documented fixed-point spectral Wiener filter complexity, channel separation, and Kconfig rules. - webrtc_aec/README.md: Described dual-input AECm echo cancellation routing, pipeline-ID heuristics, and layout designs. - webrtc_ns2/README.md: Documented deep-learning/RNNoise GRU topology, 48 kHz lock rules, VAD probability thresholds, and DSP scaling metrics. Each document contains a tailored Mermaid diagram mapping data/control flows. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP build support for llext based and runtime linked ffmpeg audio encoding, decoding and filters as a west module. Builds and links today. Next steps are integration with topology and kernel then optimization.