Media Mixer Control for XCON

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119.

This works tries to solve the problem of allowing a conference participant to manipulate the media flow in a mixer. It defines a protocol between the end users software from manipulating the conference and the centralized conference. This needs to be rich enough for a mixer to express what information it wants from a mixer yet simple enough to allow the client to render a useful user interface to the user. This work takes into account that real mixers have constraints on what media flows are possible and that UIs have buttons, knobs, etc that users manipulate. The goal is for a conferencing end point made by one vendor to work with mixers or conference systems made by another vendor.

There are several topics that are completely internal to the conference systems and are out of scope for this this work. These include: How the focus manipulates the mixer. How one describes what a mixer is capable of doing.

When a conference is created, it is instantiated from a template. The template describes what controls are available for the client to manipulate the media. The conference also describes roles that the client can take on, such as Moderator. The template can have parameters that are set when it is instantiated to help control the type of the conference. This document describes the templates and ways for the client to understand and manipulate the media in the conference. It allows for the following things: A conference consists of several participants and multiple streams of media. Sidebars are mini conferences that are just like conferences except that they can not contain sidebars. Clients can discover the virtual streams in a conference. Clients can request new virtual streams be created. Clients can receive media and contribute to a virtual stream. Clients can discover the Participants in a conference and their role (this is more conference policy than media policy). Clients can join a conference as a participant and assume a particular role. Conferences, Streams, and Participants can have controls that manipulate the media received. The role of the participant will control what view of the conference they have one which media streams they can manipulate.

The conference has a list of Participants. Each Participant has some Controls such as Name and Hidden. Each conference has a list of sidebars. Each conference has a list of Virtual Streams. Each Virtual Stream has some attributes like name, type, priority and list of contributing participants.

The protocol between the client and the conference server allows the client to get the semantic information in the conference, find out when it changes, and make changes to it. It's probably something like XCAP. [TODO add ref]

Templates define what information is valid semantic information for the conference. A template provides enough information that the client can intelligently render a useful GUI to the end user. There is a registry of well known templates, but a conference server can define new ones. A client can find all the templates a conference server supports and select one to use when creating the conference. The template is in some ways a schema that describes what information can be sent in the protocol to manipulate the conference and also provides hints on how to build a GUI that can control the conference. A template for a basic audio conference may indicate that there is one audio stream named "primary" and each participant in the stream has a single binary control called "Mute". There is only one Role that can be used, called "participant". Parameters are variables in the template that are set when the conference is created. For example, in the audio conference, the maximum number of users is 10 when the conference is instantiated but the template can indicate that the valid range is from 2 to 128.

Controls are what describe the parts of the conference that can be modified by the clients. Conferences can have controls, participants in a conference can have controls, and streams in a conference can have Controls.

Participants in a conference can take on different Roles that change what they can do. The template defines what Roles are available for the client. The moderator can change the role of a particular participant.

The client selects the basic audio template that looks like:

<template name="basic-audio"> <parameter type="integer" name="max-participants" min="2" max="128"/> <role name="Participant"> <parameter type="integer" name="max-participants" min="2" max="128"/> </role> <stream type="audio" name="primary" priority="1.0"> <participant name="*"> <control type="boolean" name="mute"/> </participant> </stream> </template> The client retrieves this template and uses it to create a conference where it sets the max-participants to 10. Alice and Bob join this conference and the conference server tells Bob about the state of the conference media. There is a single control, called mute, for each participant contributing to the audio stream. After Alice and Bob have joined, the conference server informs the clients that the current state of the conference is as shown in the xml below.

<conference type="basic-audio" name="Fluffy Conference"> <parameter type="integer" name="max-participants"> 10 </parameter> <role name="Participant"/> <participant name="Alice" role="Participant"/> <participant name="Bob" role="Participant"/> <stream type="audio" name="primary" url="sip:fluffy22-audio-primary@cs.example.com" priority="1.0"> <contributor name="Alice"> <csrc> 60123 </csrc> <control type="boolean" name="mute" perm="readonly"> 0 </control> </contributor> <contributor name="Bob"> <csrc> 12345 </csrc> <control type="boolean" name="mute" perm="readwrite"> 0 </control> </contributor> </stream> </conference> There are two participants, Alice and Bob, who both contribute to the audio stream and neither is muted. Bob's client decides to change the Mute state for its audio stream and sends the following to the conference server to change the state of the conference.

<conference type="basic-audio" name="Fluffy Conference"> <stream type="audio" name="primary"> <contributor name="Bob"> <control type="Boolean" name="mute"> 1 </control> </contributor> </stream> </conference> A key part of this is that Bob's client may have known about this basic audio template and what the semantics of the "mute" control implied. The client may have connected this up with a button of the client's that was labeled mute. On the other hand, Bob's client may not have known anything about this template and simply rendered a button on the screen and labeled it "mute" with no idea what this would do. A third client may not have been table to deal with the control at all and may have just ignored it. Clearly the user interface can be better if the client understands the semantics of what the template means, but the user interface is still functional when the client does not.

Templates contain a list of stream types, roles for participants, parameters that need to be set, and controls for the conference.

Participants are the logical user entities participating in a conference. Just to keep it confusing, there is a role called Participant too.

The virtual stream is a named logical stream of media. An example is a simple audio conference with 6 participants and a mixer that mixes the loudest three. There are several participants that are all contributing a real media stream and also receiving a real mixed audio stream. Generally speaking, all the participants are hearing the same thing, and this one thing is referred to as a single virtual audio stream so that it can be named. In a simple audio conference it may be named "primary" for example. In truth, not all the participants are actually receiving the same real stream for this virtual stream because the real stream to a particular contributor does not include the audio contributed by that participant. Likewise, all 6 people are considered Contributors to the virtual stream even though only 3 of them are truly contributing at any given time. Streams all have a type, a name with one or more URLs, and a priority. The name is the name of the virtual stream so that it can be selected by clients. The URL is a URL that can be used to join or connect to this virtual stream. The priority indicates how important this particular stream is to the conference and the type indicates the type of media carried in this steam. Streams have types. These correspond to the major MIME types of the media they send.

Some number of the Participants sending media to this stream will be selected by some speaker selection algorithm, mixed together, and sent to the Participants listening to this stream. Note that different participants might hear different things because the mixed audio does not generally include the audio contributed from the Participant receiving the stream.

Some number of Participants are contributing media to this stream. The mixer chooses how to select and combine these images for each Participant receiving the result of this stream.

This is "chat" session where the results of multiple IM conversations are mixed. All the messages sent to the conference are played out to all the Participants receiving this stream.

At a minimal level, this consist of a URL that defines the application. Many systems will simply update an http URL that fetches an HTML page that shows the current presentation.

Streams have names that users can select them by and URLs that can be used to join the stream. These would typically be a SIP, H323 or XMPP URL. A stream may have multiple types of URLS if the mixer is capable of transcoding from one type to another. There are some well known stream names for each stream type. They are "primary" and "secondary". Often the names of all the participants will also be stream names. For text chat, the convention is Primary refers to everyone in the conference and Secondary refers to the moderators.

This generally has the main media for the conference. For audio, this will generally be the mix of the participants (including Presenters and Moderators) that are not muted. For video it will often follow the active speaker or be a tiled video representation of all the cameras. For text it will be a general chat session. For applications it will be the main presentation.

This is less common. For video it may be a second video view of the main scene. For example, in a lecture situation the secondary video may display the view from one wide angle camera that gets the whole presenter and presentation area, while the primary has a narrow angle camera that tracks the head and shoulders of the presenter.

Streams have a priority from 0 to 1. Zero indicates that a client, by default, should not display this stream unless the user specifically requests it. A priority of 1 indicates that, by default, the client should render this stream and should warn the user if it cannot. Other values only define an ordering, and clients should attempt to use their resources to display the higher priority streams before the lower.

Roles are more part of Conference Policy than Media Policy but are mentioned here so that the Media Policy will make sense. Arbitrary new roles can be defined by a template, but the basic set of roles are: Observer: Can only receive media from the conference. May be a hidden participant. Has fewer privileges than a general participant. Participant: Can receive media from the conference. May be able to contribute. Often will not be able to Mute other participants. May be a hidden participant. Is a superset of Observer. Presenter: Is generally not muted and can contribute media. Generally has more privileges than a Participant. Can often control whether a Participant is Muted or not. Is a superset of Participant. Moderator: Can control Roles of other participants and may have other privileges, like the ability to Mute and join special streams. Generally has more privileges than a Presenter. Is a superset of Presenter. Monitor: Receives all the media that some other particular Participant receives. Coach: Can send media on behalf of some other particular participant.

These are input to the conference server that can be manipulated in real time while the conference is ongoing. All controls have a name, a type, and permissions that indicate whether or not the current client can modify them.

These are parameters to the template that are fixed at the point when the template is instantiated and the conference is created.

The parameters in the templates customize a generic template for a specific conference. One control parameter is "max-participants". When the CS generates the template for the client, it can customize the min and max value of this parameter to match what it is capable of. When the client instantiates the template and creates the conference, it can specify the value that has been requested. This max-participants value does not mean the conference can never go over that value - it just means the conference is highly likely to be able to get at least that big, but there is no guarantee for the client that it will be able to grow bigger than that. Parameter Types: Integer Real Enumeration String Templates need to define all the Roles that a participant can take and the max number of participants of each role. They also need to define all the types of streams available and their names. If the name of the stream is "*" in the template, then the actual conference will have a stream of that type for each participant that corresponds to that participant's input of that type. Inside a stream, the Participants that can contribute can be listed. A name of "*" means all the participants in the conference. A control can be inside the template, participant, stream, or participant in a stream. The control will apply to the appropriate context.

The conference state contains all the participants, all the streams, and the values of all the controls. The conference contains the current list of participants and current streams. The conference can also contain sidebars. Each stream contains the contributors that could contribute media to that stream. A very common control for each contributor to a stream is "mute".

The client can attempt to change the state of various controls in the CS by sending a document that contains just the things it wants to change.

How the client finds out about a change in Conference state.

TODO: Need to define how the information is sent between the client and the conference server. XCAP?

Note the XML syntax in this section is not consistent with the rest of the document. Probably want to use something more like what is shown here.

The controls need to collect information. This can be classified into several types. It should be possible to provide default values, a name for the control and text it displays, help text, control if a value is required, and control of whether or not the value is editable. It should be possible to express constraints on the form an input can take by specifying a minimum or maximum for types where that makes sense, or specifying a regular expression that must be satisfied. For numeric values in a constrained range, it should be possible to provide an increment value used by the control. For strings it should be possible to indicate that they should not be displayed when they are entered for things like passwords. Need the ability to internationalize any text that is displayed to the user. There are controls for: Strings Multi-line Strings Integer Real Boolean Date Time Date Time URI File Selection Select SingleSelect Multiple Controls all have a type that constrains what can be entered in them and provides a hint for rendering the control. If an unknown control is encountered, it should be treated as a string type. Controls also have a reference attribute which is just a name for the program to refer to them by. The <label> element controls what is displayed to the user and the <value> element contains the current setting of the control. An optional <description> element provides some text that can be used as help text for the control.

This is typically rendered as a text input field.

<control type="string" ref="month" private="true" > <label> The Month </label> <value>Jan</value> <description>Enter a month with the letter r</description> <regex>.*[rR].*</regex> </control> The "private" attribute indicates that the string should not be displayed as it is entered.

This can be rendered as a slider or volume knob if it has a constrained range; otherwise it is a text field. The text field may have increment or decrement buttons.

<control type="integer" ref="gain"> <label> Volume </label> <value>0</value> <range min="-18" max="6" increment="3"/> </control>

This is typically rendered as a toggle button.

This is typically rendered as a pull down menu or as a radio button box.

<control type="select1" ref="foo"> <label> the thing </label> <value>2</value> <item> <label>one</label> <value>1</value> </item> <item> <label>two</label> <value>2</value> </item> </control> The list of items that can be selected is contained in <item> elements. Each item has a label that is displayed and a value that is returned when it is selected.

This is typically rendered as a combo box or list. This is the same as a selection, except that the type is selected and the initial value is a space-separated list of values.

Range restrictions can be done with

<range min="-18" max="6" increment="3"/> Regular expression restrictions can be done with

Probably not needed. Just provides a grouping of things so that the User Interface can provide some hints to display these together. Like a frame or tab display.

Probably not needed. Was getting at the idea that you might want to make an array of Boolean values indexed by something like the roster. Seems the roster is the only useful index and can do this a different way. Might need this to select what video gets composited into a "Hollywood Squares" style stream.

The following is a more complex template with bits of text explaining it:

<template name="audio-video-presentation"> Note how the max-participants parameter is defined for each type of role and for the conference as a whole.

<parameter type="integer" name="max-participants" min="2" max="2000"/> <role name="Participant"> <parameter type="integer" name="max-participants" min="0" max="2000"/> </role> <role name="Presenter"> <parameter type="integer" name="max-participants" min="0" max="2000"/> </role> <role name="Moderator"> <parameter type="integer" name="max-participants" min="0" max="1"/> </role> <stream type="audio" name="primary"> This stream has a control for the master gain and for the input gain of each participant.

<control type="real" name="master-gain" min="-18" max="+6" note="Gain in DB"/> <participant name="*"> <control type="binary" name="mute"/> <control type="real" name="gain" min="-18" max="+6" note="Gain in DB"/> <control type="binary" name="force-in-speaker-selection"/> </participant> </stream> Note that the parameters in this stream define how the MCU does Hollywood Squares. The parameters set the maximum number of tiles to display, and then the priority provides the hint for which tile to display. How the tiles are positioned is left up to the video composition unit. The client could change the layout by changing the priorities. There is no way to, say, put the stream "Alice" in the bottom left hand corner. If you want better control of layout than this, use SMIL.

<stream type="video" name="primary"> <parameter type="integer" name="vertical-tiles" min="1" max="4"/> <parameter type="integer" name="horizontal-tiles" min="1" max="8"/> <parameter type="integer" name="number-tiles" min="1" max="32"/> <participant name="*"> <control type="binary" name="mute"/>  <control type="real" name="priority" min="0.0" max="1.0"/> </participant> </stream> <stream type="video" name="secondary"> <participant name="*"> <control type="binary" name="mute"/> </participant> </stream> The next set of video streams forms an individual video stream named for each participant. This allows one user to specifically request the video from a particular client.

<stream type="video" name="*"/> The next two streams are from two slide presentations such as PowerPoint slides.

<stream type="application" name="primary"> <participant name="*"> <control type="binary" name="mute"/> </participant> </stream> <stream type="application" name="secondary"> <participant name="*"> <control type="binary" name="mute"/> </participant> </stream> Finally an IM chat session is set up for the conference.

The templates often include the name Basic and Standard. Basic is minimal stuff. Standard is stuff that occurs in lots of systems that are available today but does not include the kitchen sink. Audio is audio, Video is video, Presentation is web or application sharing. Could have style hints for things like Meeting, Presentation, Q & A session, Lecture. [Still need TODO this]. Two templates will be defined here. Basic and Standard. The templates have parameters that allow them to support, or not support, Audio, Video, Text, and Presentation. They also have parameters to indicate the for roles and styles that are allowed.

[TODO]

What is the boundary between conference control, media control, and policy control for both of them.

Many thanks to Brian Rosen, Nermeen Ishmail and Rohan Mahy