FLV (H.264 + AAC) support

[jeoliva]

FLV is not one of the most used containers right now, and of course not a priority for any player, but I though it could be useful adding it to Exoplayer for 2 reasons:

1. FLV was pretty used not too much time ago and still there are a lot of legacy video content libraries that are using this container. I have even seen a couple of reported issues asking for FLV support so there are some interest on it.
2. And second, and most important, a share knowledge reason. In other words, I think this is really useful to allow other developers to better understand how to build their own extractors for Exoplayer. FLV is one of the simplest and best documented video/audio container formats so it is easy going through the code, looking at the parser, flv extractor, FLV specification, and see what is the architecture behind Exoplayer extractors. MP4 type of containers are much more complex, and are a so hard way of starting coding or trying to understand extractors :).

Regarding the implementation, as title stated, this pull request is adding support for FLV container, and more specifically, when content within FLV is encoded using H.264 and AAC.

[ojw28]

In general I'm not a fan of adding support for legacy container formats. If we take it, it's something we have to commit to maintaining. However, as you say, this does look pretty simple, and so it seems like a reasonable request!

I'll post a few initial comments now, and try and have a more in-depth look later this week. Thanks!

[ojw28]

Please update the list in the javadoc at the top of this class to say that FLV is supported. Also, if seeking is not supported (see comment below) then update the sentence that says seeking in AAC and MPEG-TS is not supported, to include FLV too.

[jeoliva]

Taking a look to all these things.

[jeoliva]

Just incorporated suggested changes. Thanks @ojw28 for all your comments!

[ojw28]

Sorry for the delay looking at this. Will try and get to it tomorrow!

[ojw28]

This list should contain FLV.

[jeoliva]

Changes applied

[winlinvip]

FLV is useful in live streaming with low-latency, similar to RTMP for flash.
For example, most live streaming server supports RTMP/HLS, but HLS is large latency 10s+ while RTMP 1s+. Actually the RTMP is similar to HTTP-FLV and some server(for instance, srs) already supported the HTTP-FLV/RTMP.

[winlinvip]

@jeoliva Thanks for your work, I merged this pull request to my fork, 👍

[winlinvip]

Compile failed when merge to dev-1.5.1-rc：

git/ExoPlayer/library/src/main/java/com/google/android/exoplayer/extractor/flv/AudioTagPayloadReader.java
Error:(111, 44) error: method createAudioFormat in class MediaFormat cannot be applied to given types;
required: int,String,int,int,long,int,int,List<byte[]>,String
found: String,int,int,long,Integer,Integer,List<byte[]>,
reason: actual and formal argument lists differ in length

[winlinvip]

Seems change to the bellow is ok:

MediaFormat.createAudioFormat(MediaFormat.NO_VALUE, MimeTypes.AUDIO_AAC, 

[winlinvip]

I fix it at: winlinvip@115aee8

[ojw28]

Yes. I've merged this and fixed the break. Thanks @jeoliva / @winlinvip !

[ojw28]

I've also submitted some stylistic tweaks just to bring the code in line with the rest of the source: 950cc70

[ojw28]

And some further changes. I was able to simplify ScriptTagPayloadReader somewhat. Please take a look, and let me know if I accidentally broke anything!

[ojw28]

One final question: Where did the test link you've added come from? I want to know if it's fine to copy it across to our own hosting. Obviously the underlying content (big buck bunny) is fine for us to copy. Depending on who packaged it, we might need to re-package it ourselves to copy it over. Please advise. Thanks!

[winlinvip]

I test the flv, seems ok for vod stream, for example, it's ok for http://www.ossrs.net/mp4/source.200kbps.768x320.flv。
But failed to play a flv live stream, which is send by chunked encoding by SRS server, the demo stream is http://ossrs.net:8081/live/livestream.flv or link。
I will find out why, maybe ExoPlayer depends on the file Content-Length? So not support chunked http stream?

[ojw28]

Thanks. for the info. I made some further changes to the extractor in 4422e8a based on an internal review. I've put a couple of comments in the old version of the code in that change that explain why the new state "skipping to next tag header" state is necessary :).

[winlinvip]

@ojw28 Thanks~ I will try to add hasVideo and hasAudio in header in SRS server.

[winlinvip]

Fixed in ossrs/srs@50a7b9c, set hasVideo and hasAudio tag in flv header.
It works! And the ExoPlayer can play the HTTP-FLV and HTTP-TS stream, both live and vod:

• HTTP-FLV: http://ossrs.net:8081/live/livestream.flv
• HTTP-TS: http://ossrs.net:8081/live/livestream.ts

[chodison]

But live time is not moving

[winlinvip]

@chodison Not understand you, what's time moving means? The timestamp in FLV stream is increasing, maybe the player not display it?

[winlinvip]

I see, you means the timeline of ExoPlayer, that's caused by the flv onMetaData packet:

String onMetaData Object (20 items) Property 'duration' Number 210.7 

Because the flv stream is generated by ffmpeg to publish a flv file, which contains the duration, I will remove this field for live stream.

[chodison]

The player‘s position is always zero. And Cycle two times, it don't work.

[winlinvip]

Fixed in ossrs/srs@ef00005, remove the duration of flv for live streaming.

[chodison]

@ojw28 For flv vod video, can not ExoPlayer to support seeking?

[ojw28]

See @jeoliva's note on this thread above seeking: In fact, FLV container doesn't include any kind of information to help on implementing seeking.

If you want your media to be seekable you should use a more appropriate container format, like MP4.

[winlinvip]

@ojw28@jeoliva@chodison There is some workaround for flv to support seek, the service which produce the flv file must inject all keyframe offset to the onMetaData script tag, then player can got the offset to seek to specified time. However, it's not the standard of flv, it's possible to implements by this workaround.

[winlinvip]

Please refer to https://github.com/simple-rtmp-server/srs/wiki/v2_EN_FlvVodStream.

[chodison]

@ojw28 why the ffplay player can seek flv vod video?

[winlinvip]

@chodison The ffplay generally play a local flv vod file, it can skip one tag by tag, it's ok for disk but not good for http flv vod stream. The network file always need a header which store the keyframe and offset map, similar to mp4 container.

[chodison]

@winlinvip My tested url is http flv vod stream.

[winlinvip]

@chodison Http flv vod stream is ok to seek, to skip one tag by tag, it's not efficient way however. The acceptable way for a big flv vod file to seek, is inject the keyframe offset to the onMetaData.

[ojw28]

There's an alternative "accepted" way where the server will allow you to specify the seek position as, for example, a query parameter. In that case the server does the brute force tag-by-tag search (it's probably implemented to have built an index already) and serve you the media from the specified seek position.

The takeaway here, however is that (a) FLV does not support seek properly, (b) we don't intend to support any hacks to make seeking in FLV possible in ExoPlayer, unless they're external contributions and do not extend anywhere outside of the FLV extractor package, and (c) if you're knowingly packaging your own media as FLV, you should stop and choose a more appropriate container like MP4.

[winlinvip]

@ojw28 Aggree to use MP4 instead for http vod stream. 😄

[chodison]

@winlinvip@ojw28 flv vod stream server supports http start or http range to seek.

[winlinvip]

@chodison Yep, but need the player to finger it out where to seek, for example:

player.seek(10s) 

The player must conver the 10s to file offset, for instance, 3421bytes, then player must request from the offset:

http://server/f.flv?start=3421 

Or request by range whatever.

The question is: how to know the offset and seconds relationship? The flv container never describe it.

There is a workaround, to save the offset-seconds map to the onMetadata(the scirpt tag) of flv, which need the player to parse this tag -- this is exactly what @ojw28 said about the standard flv not support seek.

I aggree, should never use none-standard, especially the player.

[chodison]

@winlinviphttp://server/f.flv?start=time or request by range, file offset is from he proportion of the position to the duration.
seekTime/flvDuration == position/flvFileLength.

[winlinvip]

@chodison seekTime/flvDuration != position/flvFileLength.

[mondain]

The offset and time relationships are often sent in the metadata response; depending upon your server's implementation of course.

For information on this, look here: http://h264.code-shop.com/trac/wiki/FlashPlayer

[ojw28]

We have no plans to support seeking in FLV. All of the approaches mentioned would add complexity to the upstream package, and would not work in a consistent way across different DataSource implementations. If you want seeking, you should use a container format the properly supports it (or DASH, SmoothStreaming or HLS).

[chodison]

@winlinvip Estimation calculation. The flv vod stream server will return the key frame for closest position

[winlinvip]

@chodison It is possible to implement the flv stream seek, but not official standard for it, so I think @ojw28 is right to not support this feature and use HLS or DASH.

For the server to seek the closest position, not all server support it, that is why not official standard.