UniVoice 4.6.0: Per peer output filters

UniVoice 4.6.0 is out! The main change in this release is per-peer output filters (instead of global filters) which fixes a bug. 

But before I get into it, there are a couple smaller changes that I'd like to go over:

UniMic 3.3.0

UniVoice now uses UniMic 3.3.0. This was a major improvement that I posted about in this post 

ClientSession<T> constructor enhancements
You don't need to implement an IAudioOutputFactory the client session object can now be constructed using a factory method instead.

// This required you to create a StreamedAudioSourceOutput.Factory class that implements IAudioOutputFactory
// even though it doesn't do much, it just implements Create() that returns a new object
ClientSession = new ClientSession<int>(client, input, new StreamedAudioSourceOutput.Factory());

// But you you can do this. No need to create the Factory class
ClientSession = new ClientSession<int>(client, input, () => new StreamedAudioSourceOutput());

Probably only useful if you're making your own IAudioOutput implementations. Regardless of what you use, ClientSession understands if you're using a factory object or a method and handles it internally.

--------------------------

This brings me to the main change in this release. 

Per peer output filter

Investigation
I started the post about UniMic 3.3.0 with the description of an issue. When two people are in voice chat, the audio is clear. But as soon as a third person joins, the audio degrades substantially for all users and the voice is crackly. 

I have been looking into that issue for over a month. In fact, the improvements that UniMic 3.3.0 introduced were initially me trying to fix this problem. 

Around the same time that post went up, Github user @FilipBarva opened an issue Robotic voices with more than two clients which sounded like the root cause was the same.

Turns out the issue is that the Concentus objects, such as decoder, encoder and resampler cannot be shared for the audio of different peers. This is because they probably have some mutable data inside that's based on what they previously processed. 

A while ago I posted this sample concentus test script. It basically does these things (in sequence)
- starts the mic and captures the audio frames
- resamples the audio to the desired frequency using Concentus, if required
- encodes the audio 
- decodes the audio
- plays it back

You can attach that script to a gameobject, assemble some components in the inspector that it needs and it'll work. You can do this with multiple gameobjects and they will work in parallel.

I had a suspicion that using the same Concentus objects might lead to problems. So I did something pretty simple, I marked the Concentus objects static so that all instances of this script use the same objects.

static IOpusEncoder encoder;
static IOpusDecoder decoder;
static IResampler resampler;

And suddenly, the multiple gameobjects that were previously working perfectly in parallel were not sounding so good. The audio had crackles and pops.

Bug
Before this release, the ClientSession class had a single List<IAudioFilter> OutputFilters. You can see this in the 4.5.1 release here and the usage here

Basically to add an output filter, you create the filter object and add it to the list.

But this is exactly what was wrong. Internally ClientSession iterated over the same output filters over the audio of every peer. This meant the filters were being shared for different audio streams. 

Fix

See the 4.6.0 commit

To fix this, instead of creating and add the filter objects to the ClientSession, you register a lambda that creates and returns the filter object. This lambda can then be executed for every peer, allowing the session to internally maintain a Dictionary<T, List<IAudioFIlter>> which is a map of every filter for every peer.

The adding and removal of output filters is using using: .AddOutputFilter and .RemoveOutputFilter. The usage looks like this:

ClientSession.AddOutputFilter<ConcentusDecodeFilter>(() => new ConcentusDecodeFilter());

ClientSession.RemoveOutputFilter<ConcentusDecodeFilter>();

You can also check if an output filter type has already been added using 

ClientSession.HasOutputFilter<ConcentusDecodeFilter>();

The previous usage has been deprecated. Using ClientSession.OutputFilters.Add(new ConcentusDecodeFilter()); will now lead to an error that says you should use the new methods. So this release does have some breaking changes.

Comments

Post a Comment

Popular posts from this blog

UniVoice 4+: Now a mono repo

UniVoice has been around since Jan 2019 and has been evolving towards becoming a robust VoIP solution for Unity. Recently I released version 4.x, which is a big upgrade from v3. One of the things that's changed is that UniVoice is now a mono repo. Why mono repo Previously the main repository only housed the interfaces and implementations for them were separate repositories . This caused a lot of pain for both me and any users of UniVoice. For me because if I wanted to change something in, let's say, the univoice-unimic-input  repository. I would need to open UniVoice, change the manifest to point to a local copy of the univoice-unimic-input repo so that I can test the changes locally.  For users because when they'd install the univoice package, it wouldn't really have anything usable. They'd need to install implementation packages separately. In v3, to get UniVoice to work you needed at implementations of at least three interfaces: IAudioInput, IAudioOutput and ICha...
Read more

UniVoice 4.4.0: Easy drag and drop integration with Mirror (also, YouTube tutorial)

Image
A tutorial for this blogpost is available on YouTube I've recently helped a lot of people setup a basic UniVoice integration in their projects. And it became clear that the sample scene is not minimal enough. The sample scene is basically an example app that also does some UI stuff. Going through the code shows how it can be used. But what most people are looking for is to get basic voice chat working as soon as possible. I've seen some people adding the example app script to their scenes, which will not work as expected. For this reason, I have now created a basic setup sample that provides a component that you can just add to your Mirror Network manager and get voice chat working immediately. This UniVoiceMirrorSetupSample.cs  easily allows you to check if voice chat is working by making sure you have setup UniVoice properly, pretty much just adding the UNIVOICE_NETWORK_MIRROR  compilation symbol to your Mirror project. The script also provides access to the IAudioServer<...
Read more

UniMic 3.3.0: Many StreamedAudioSource improvements, thanks to MetaVoiceChat

For a couple of months, I have been getting on a weekly call with a team that's integrating UniVoice into a project but they've been facing an issue. When it's just 2 people in the room, they can hear each other perfectly. But as soon as a third person joins, everyone's audio starts popping and gets a little jittery. You can still understand the person but the audio becomes noisy and irritating after a while. The popping effect is often the cause of writing audio data to an AudioClip on a buffer strip that's still being read. Or it can happen when you write data that's too close to your reading head and before the writing operation completes the read head is already on it. I've been looking into StreamedAudioSource.cs to fix a fix for this. But it was only last week that we finally figured out that the issue is in the Concentus Decode Filter instead and I've been looking in the wrong place.  But the great thing about this detour is that during this time ...
Read more