Opus hardware decoding release features
Historically, XMA was the only offering on the XBOX One family of consoles that offered hardware offload decoding. Now, on the XBOX Series X|S generation of consoles, we’re introducing Opus hardware offload decoding through the XAPU (XBOX Audio Processing Unit) API. Opus is a royalty-free audio compression codec that’s designed to efficiently code speech and general audio into a single format. Opus is low latency. In internal evaluations, Opus surpasses XMA in quality and file-size compression ratio. Additionally, a High-quality Sample Rate Convertor (HSRC) has been added to work in conjunction with the hardware decoding functionality (Opus). When combining Opus hardware offload decoding with the HSRC feature to change pitch, developers will notice improved output quality over the previous-generation SHAPE offering. (It’s still supported in XBOX Series X|S consoles for backward compatibility.)New in this release
- New with June FAL QFE3: For Quick Resume scenarios, it is no longer required for titles to re-establish its connection to the hardware after a resume.
- New with June FAL QFE4: 2.5 and 5 ms length Opus packets are now supported for CELT.
- August 2020 Preview Recovery version 10.0.19041.4124 (rs_xbox_release_2008- 19041.4124.200814-0000) / 10.0.19041.3562 (rs_xbox_release_sirius- 19041.3562.200814-2300) or higher: Breaking Change to XApuDecodeConvertCommand implementation: now residual data when using the firstFrameIndex is not included when it used to be. Please see the Best Practice section or the API reference page XApuDecodeConvertCommand for additional details.
- If the HSRC output processing buffer has data in it and the flush command is used, previously there would be a loss of data (you might lose some converted audio data and the output would not be as expected)
- If unsupported Opus packets are sent for processing, an error is now thrown instead of a hang. (ex. Sending a 2.5 ms Opus packet where only 10 ms or 20 ms is supported)
- FIFO is on by default
- Overall performance improvements and stabilization fixes
Opus hardware decoding release specifications
The following table outlines what commands and features are available:Important note on HSRC usage
The High-quality Sample Rate Convertor (HSRC) has a frequency-dependent group delay that is approximately 5 frames for frequencies below 10 kHz. The expected delay in output is 5 frames when the sample rate converter is engaged and the caller can adjust accordingly. If using a middleware solution for audio, this delay may already be handled for your title.Best Practices
- Regarding XApuDecodeConvertCommand implementation: firstFrameIndex passed into XApuDecodeConvertCommand specifies the first frame that will be copied to the PCM output buffer. firstFrameIndex coupled with the frameCount specify what gets copied to the output buffer. Our implementation does not include the residual data when using the firstFrameIndex. If there are 100 residual frames and firstFrameIndex is set to 50 and the frameCount is set to 200, 200 frames will be added to the entire residual starting from the 50th frame in the decoded data. This means the buffer into the HSRC will have the original 100 frames from residual followed by 200 frames starting from the 50th frame in the current decoded packet.
- We recommend that you create XAPU clients with care. It is best to create XAPU clients up front (during activation of audio engine resources) and keep using them throughout the lifetime of the game. Calls to XApuConnect and XAapuDisconnect are computationally expensive. You run the risk of running out of resources if clients with varying memory requirements are created and deleted frequently.
- For looping purposes, it is up to the caller to use XApuCommandType::Reset or not, as it might cause transient effect in the output. However when XApuCommandType::Reset is used, pre-roll packets must be submitted to avoid transient effect.
- For getting the best performance, use Opus streams encoded with 20ms packet size
- Performance wise it might be better using fewer clients with more streams rather than fewer streams with more clients. For example, having 5 clients with 20 streams per client giving a total of 100 streams gives better performance when compared to 50 clients with 2 streams each client. Both can be used to decode 100 streams but the first one will perform better
- OPUS_SET_PREDICTION_DISABLED disables the inter-packet prediction and is supported. Encoding with this flag reduces compression performance and thereby degrading the quality of the decoded output.
- XAPU returns XAPU_E_DEVICE_FATAL error when the hardware device is in a bad state. All the XAPU clients should be disconnected at this point before creating any new clients to recover from this error.
- Signaling mechanism for command completions works on a 3ms timer and signals once for all the commands completed so far for each XAPU client irrespective of the how may commands used the XApuCommandOptions::SignalOnCompletion flag.
- If the title cannot tolerate the 3ms signaling mechanism, it can look for responses in a loop or use Sleep(0) and check for responses.
- It is important to wait for all responses before disconnecting the client. XApuDisconnect now returns XAPU_E_PENDING_RESULTS error if there are outstanding requests that are not picked up. This error is not actionable and should be used only during development time. XApuDisconnect releases all resources allocated for the client and if it is called while the hardware is still processing commands for this client, memory corruption can happen as hardware writes to memory that has been freed.
- All memory pointers passed to the hardware (ex: inputData, outPutData and the processingBuffer) should be 16byte aligned.
- When using ConvertOnly or DecodeConvert modes, larger output frame count sizes will result in better performance. The highest output frame size supported when using decode convert and convert is 1024.
- Only Variable Bit Rate (VBR) encoding is supported. Constant Bit Rate (CBR) encoding is not supported because it adds transition packets, which are less efficient and worse quality than the redundancy packets used in VBR encoding. Therefore, this packet type, and by extension, CBR, is unsupported.
- For HYBRID and SILK: Only mono and stereo Opus streams are supported with 10 or 20 ms packets and at 48000 Hz sample rate.
- For CELT: Only mono and stereo Opus streams are supported with 2.5, 5, 10 or 20 ms packets and at 48000 Hz sample rate.
Pausing and resuming a title
A title must be able to pause and resume, for example, when the user puts it into Constrained mode. When the suspend handler is called, simply stop submitting XAPU commands to the hardware. The commands that were already submitted will complete normally and placed in their corresponding response queues.Samples (source code available in an external .zip file, request by emailing AnaAud@microsoft.com)
- Decode One Opus Stream. (Overview of this sample available here.)
- Decode Multi-Opus Streams.
- Play One Opus Stream. (Threading is different than “Decode One Opus Stream.”)
- Play Multi-Opus Streams.
- Decode Convert One Opus Stream with an option to loop and sample accurate seek.
- Decode Convert Multi-Opus Streams.
- Play Decode Convert One Opus Stream.
- SimpleXAPU (Example of an Opus Stream decode within a GDK title with a UI.)
Contact
If you have questions or concerns about this feature, please email AnaAud@microsoft.com or use the online forums.Reference API documentation
- Xapu (API contents)
- Functions
- Structures