Recording proposal...still having corruption issues in recorded content

1

Hi Jeremy,

First of all I want to thank you for your effort and contributions to Licode. We really appreciate your work and your input. Get ready for a long answer :slight_smile:

I understand your concerns regarding the actual state of recording, as it is, Licode does not implements the mechanisms you mention to manage packet loss. This is due to lack of development effort but also there are some design decisions involved.
It doesn’t handle packet loss correctly; it should be sending a NACK on a lost packet on the RTCP back-channel.
It doesn’t handle FEC correctly (there’s an entire packet type of 117 that it licode is droppoing on the floor).
Licode allows recording, yes. But it is primarily designed to be used in a videoconferencing scenario. We give a low priority to implementing FEC handling and RTCP reporting because Chrome clients (subscribers), DO implement this, and Licode makes sure this RTCP messages (NACK, REMB, etc.) ARE forwarded (in a quite brute way at this point, we plan on improving this).

Publisher ————(A)———> Licode ———(B) ———> subscriber.

For the communication, it is important to control packet loss both in A and B. With our limited resources, we decided it was better to pass RTCP packets from the subscriber instead of generating a new RTCP stream from Licode.

This, of course, means that if there are no subscribers in the session, no packet loss will be reported. That is the case of a record-only session. Also, as you point out, we discard red data and fec packets WHEN recording, we know that is quite far from optimal. Do you have other subscribers in your recording session?

Another difference with recording is that you need periodic keyframes to be able to seek within a recorded video, we manually ask for them via FIR (Frame-intra-request) messages.

This approach has provided very good communication experiences in our projects and decent-to-good recording quality.

On how to proceed: As I mentioned, we want to add more intelligence to Licode regarding RTCP. I would definitely improve ExternalOutput and the (now almost totally yours) RtpQueue so they can report packets loss, use redundancy packets and so on.

We don’t want to use Google’s library in Licode’s core for a variety of engineering, scientific, socio-political and philosophical reasons. We prefer to increasingly improve what we have according to the standards while being compatible with Google.

However, there is another option. You can use Google’s webRTC library for recording without altering Licode, you can create a stand-alone webRTC client, implement the negotiation as in erizoClient and implement recording there (possibly by reusing our ExternalOutput code if you manage to get the encoded frames). That would require, IMO less work that replacing Erizo’s WebRTC stack and you could still use Licode as the MCU for room management, etc. In fact, I believe that there are similar projects out there.

Finally, I would like to ask you to send me an example of a corrupted video so I can see exactly the kind of corruption you are referring to.

Cheers! –
Pedro Rodriguez

On 15 Sep 2014 at 22:11:46, Jeremy Noring (jnoring@hirevue.com) wrote:

I’ve been working on recording a lot in licode, and I’ve come to two conclusions:
It doesn’t handle packet loss correctly; it should be sending a NACK on a lost packet on the RTCP back-channel.
It doesn’t handle FEC correctly (there’s an entire packet type of 117 that it licode is droppoing on the floor).
WebRTC itself seems to use a hybrid approach to deal with packet loss where they’ll use retransmits if the latency is low enough and FEC to provide stream-level redundancy. I know this approach must work well because I rarely (never?) see visible corruption in a WebRTC video stream, even with significant packet loss and latency in the local network connection. Unfortunately in my recorded content, corruption is fairly common and often pretty bad.

To solve this…I’m thinking the best approach may be to actually use libjingle-peerconnection in licode to handle RTP packet reordering on the video stream. This would involve sucking down the entire WebRTC project, building it, linking it against licode, stripping out the RTP packet queue I recently re-wrote, etc. It’s a ton of work. But I think it’d still be dramatically less work than re-writing the RTP/RTCP/NACK/FEC code in WebRTC, which is part of a half-dozen RFCs that definitely not trivial to implement. So I wanted to get the opinion of others here on this approach and see if there’s any alternate proposals?

I really, really need video recording that is largely free of corruption. So I have to do something. Any advice is hugely appreciated.

-Jeremy

You received this message because you are subscribed to the Google Groups “lynckia” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lynckia+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I’ve started pulling code in to see if I can make this work. My first guy
impression is it’s going to be very hard to granularly remove
fec_receiver_impl.cc/fec_receiver_impl.h, because the sheer number of
#includes they pull in is sort of mind-boggling. Still trying to see if
there’s a granular approach I can adopt here. Advice welcome.On Tuesday, September 23, 2014 1:43:32 AM UTC-6, Pedro Rodriguez wrote:

I’d swear I sent the response to the list but I cannot find it here now…

Short answer:
Yes, we need to implement FEC/redundancy support to improve recording
quality.
Generating RTCP from the MCU is a lower priority at this point, I find
more interesting improving the way we forward the RTCP RR/Feedback packets
from clients by adding more intelligence to OneToManyProcessor. That said,
generating RTCP RR from the MCU is quite simple at this point.

We are also working on an statistics monitor that will hopefully help
diagnose network problems.

We really don’t want to use full libjingle in Licode’s core for a variety
of reasons. Using isolated parts of the code, like we are doing with VP8
fragmenting and parsing is ok, though. Another route is to create a
stand-alone erizo client for recording using libjingle.

3

To have libjingle included in Licode would definitly make sense. I also
need to handle packet loss correctly for similar reasons ( I am forwarding
the webRTC stream to an http streaming server ).

I don’t realize how much work this represents but I’d also love to hear the
vision from Licode’s team and their plans on the recording functionnality.

4

I’d swear I sent the response to the list but I cannot find it here now…

Short answer:
Yes, we need to implement FEC/redundancy support to improve recording
quality.
Generating RTCP from the MCU is a lower priority at this point, I find more
interesting improving the way we forward the RTCP RR/Feedback packets from
clients by adding more intelligence to OneToManyProcessor. That said,
generating RTCP RR from the MCU is quite simple at this point.

We are also working on an statistics monitor that will hopefully help
diagnose network problems.

We really don’t want to use full libjingle in Licode’s core for a variety
of reasons. Using isolated parts of the code, like we are doing with VP8
fragmenting and parsing is ok, though. Another route is to create a
stand-alone erizo client for recording using libjingle.

Cheers!El lunes, 22 de septiembre de 2014 14:49:43 UTC+2, Yannick Modah Gouez escribiĂł:

To have libjingle included in Licode would definitly make sense. I also
need to handle packet loss correctly for similar reasons ( I am forwarding
the webRTC stream to an http streaming server ).

I don’t realize how much work this represents but I’d also love to hear
the vision from Licode’s team and their plans on the recording
functionnality.

5

I’ve been working on recording a lot in licode, and I’ve come to two
conclusions:

  1. It doesn’t handle packet loss correctly; it should be sending a NACK
    on a lost packet on the RTCP back-channel.
  2. It doesn’t handle FEC correctly (there’s an entire packet type of 117
    that it licode is droppoing on the floor).

WebRTC itself seems to use a hybrid approach to deal with packet loss where
they’ll use retransmits if the latency is low enough and FEC to provide
stream-level redundancy. I know this approach must work well because I
rarely (never?) see visible corruption in a WebRTC video stream, even with
significant packet loss and latency in the local network connection.
Unfortunately in my recorded content, corruption is fairly common and
often pretty bad.

To solve this…I’m thinking the best approach may be to actually use
libjingle-peerconnection in licode to handle RTP packet reordering on the
video stream. This would involve sucking down the entire WebRTC project,
building it, linking it against licode, stripping out the RTP packet queue
I recently re-wrote, etc. It’s a ton of work. But I think it’d still be
dramatically less work than re-writing the RTP/RTCP/NACK/FEC code in
WebRTC, which is part of a half-dozen RFCs that definitely not trivial to
implement. So I wanted to get the opinion of others here on this approach
and see if there’s any alternate proposals?

I really, really need video recording that is largely free of corruption.
So I have to do something. Any advice is hugely appreciated.

-Jeremy

6

Hi Pedro,

Thanks so much for the response!

I understand your concerns regarding the actual state of recording, as it
is, Licode does not implements the mechanisms you mention to manage packet
loss. This is due to lack of development effort but also there are some
design decisions involved.

  1. It doesn’t handle packet loss correctly; it should be sending a
    NACK on a lost packet on the RTCP back-channel.
  2. It doesn’t handle FEC correctly (there’s an entire packet type of
    117 that it licode is droppoing on the floor).

Licode allows recording, yes. But it is primarily designed to be used in a
videoconferencing scenario. We give a low priority to implementing FEC
handling and RTCP reporting because Chrome clients (subscribers), DO
implement this, and Licode makes sure this RTCP messages (NACK, REMB, etc.)
ARE forwarded (in a quite brute way at this point, we plan on improving
this).

Publisher ————(A)———> Licode ———(B) ———> subscriber.

For the communication, it is important to control packet loss both in A
and B. With our limited resources, we decided it was better to pass RTCP
packets from the subscriber instead of generating a new RTCP stream from
Licode.

This, of course, means that if there are no subscribers in the session, no
packet loss will be reported. That is the case of a record-only session.
Also, as you point out, we discard red data and fec packets WHEN recording,
we know that is quite far from optimal. Do you have other subscribers in
your recording session?

Yes, our typical scenario involves two to four parties in a room. All of
the incoming streams are being recorded. Are you saying I should be seeing
packets get retransmitted? In general, I’d describe the video recording
quality as “adequate?” Definitely during any sort of packet loss and/or
latency situation, I see pretty frequent breakup in recorded streams. It’s
particularly bad if a keyframe is corrupted (which makes sense, since
subsequent packets rely on it–also more likely to lose a packet for a
keyframe)

On how to proceed: As I mentioned, we want to add more intelligence to

Licode regarding RTCP. I would definitely improve ExternalOutput and the
(now almost totally yours) RtpQueue so they can report packets loss, use
redundancy packets and so on.

I think that’d be a good approach. Also, technically to do recording
right, I need the RTCP channel there so I can do proper timestamping.
Right now, I’m sort of winging it. The RTCP channel is needed to relate
timestamps in audio/video streams.

I know it’s an open source project, but do you have any idea when someone
would work on this? I’d be happy to help in any way, since this is an
issue I have to find some way to fix.

However, there is another option. You can use Google’s webRTC library for
recording without altering Licode, you can create a stand-alone webRTC
client, implement the negotiation as in erizoClient and implement recording
there (possibly by reusing our ExternalOutput code if you manage to get the
encoded frames). That would require, IMO less work that replacing Erizo’s
WebRTC stack and you could still use Licode as the MCU for room management,
etc. In fact, I believe that there are similar projects out there.

I thought about this, but there’s a number of problems with the approach;
the biggest is it’s no longer tightly integrated with the MCU, and that
creates a headache when parties connect/disconnect/on room events/etc.

Finally, I would like to ask you to send me an example of a corrupted video

so I can see exactly the kind of corruption you are referring to.

I’ll send you an email with the details of this.On Wednesday, September 17, 2014 8:14:16 AM UTC-6, Pedro Rodriguez wrote:

7

@Pedro_Rodriguez it seems this issue still persists.

Can you please recommend a project that readily does it? I will greatly appreciate any suggestion that you can offer.

Thank you,
Rohit

8

@Pedro_Rodriguez if Chrome takes care of packet loss, perhaps I can create a subscriberOnServer, who subscribes to the speaker on Licode server. We can then record client-side using subscriberOnServer, but since this client is on the server, we will have the recording on the server.

How does it sound to you? I can try something like this to run client/subscriber on server: https://github.com/relekang/docker-webrtc-test

If it sounds ok to you, then I need to figure out how to record video from the client/subscriber in Licode room. I will greatly appreciate your suggestions on this.

9

This is a very outdated thread. Licode now has is able to report packet loss and ExternalOutput should reorder packets accordingly. We’ve come a long way since we discussed this here.
You can certainly use a browser somewhere else to record the whole screen. I haven’t personally tried to record individual streams in Licode that way but, since you have access to the underlying peerConnection and streams, you should be able to do it.

10

@pedro So would you have any idea why recording still gets corrupted? And typically this corruption happens only in the starting 30 to 110 secs and then the rest of the recording is fine. I must note that subscriber sees speaker’s video good all the time, even when recording gets corrupted at the start.

  1. Could this be that subscriber’s bandwidth is affecting publisher’s recording? We use this setting to publish speaker’s room.
    room.publish(localStream, { minVideoBW: 1000, scheme: "notify-break-recover" });

  2. Would providing only audio to a subscriber and no video to subscriber prevent any recording degradation of speaker’s video?

Any suggestion on how to prevent recording corruption will be highly appreciated.