UCMA 3.0 Advanced media woes

When it comes to conference management, im and presence, even basic media projects like ‘recording audio in wma format’ or even detecting tones then UCMA is definitely your friend – and I really love it. When it comes to doing advanced stuff related to audio and video, you have your work cut out.

Some recent requests I have had that are hard –

Detecting who is talking in a conference (like the way the icons show up against attendees in the Lync client – not to be confused with the active speaker) – that event only works when your using an audio recorder (because it comes from data in the RTP stream directly, so there needs to be media flowing). So the solution is to make a recording to a temporary file and throw away the file.

Oh and don’t even try and implement your own flow consumer class using the AudioVideoFlow provided because whilst many classes are public and it seems like it should work – under the hood it is specifically implemented for its own known types (Player, Recorder,SpeechRecognitionConnector,SpeechSynthesisConnector,ToneController).

Things like immersive collaboration are tricky when you can’t do stereo sound (let alone 3d sound) or any kind of video, in or out (you can pass it through using the back to back user agent, but that’s food for another post).   Resorting to the lower level bits  of the API just leaves SIP signaling – which is handy of course.

The only way to access the audio stream itself is to use the Speech connectors and they are hard wired to PCM @ 8k (tip SpeechRecognitionConnector.Start() returns a SpeechRecognitionStream which supports pretty much only the Read method  but still that’s often enough).

I like things this, a common feature in the AudioVideo namspace:

If you want to get fancy, and don’t need to stay in a supported configuration and don’t care if your code breaks after the next hotfix… you can use reflection to grab a reference to an active Microsoft.Rtc.Internal.Media.RtpMediaReceiver – notice that whilst its public and has some tasty methods and events – be careful it is in the .Internal. namespace for a reason, your not supposed to use it and its hidden behind internal and private members so you cant without reflection. – And just in case you didn’t read this right, DON’T DO IT THIS WAY (in production).

So I will get back to being busy writing my own managed media stack - contact me if you’re interested in using it in commercial projects.