[FFmpeg-devel,FFmpeg-cvslog] lavc: add new API for iterating codecs and codec parsers

Submitted by Muhammad Faiz on Feb. 10, 2018, 10:13 a.m.

Details

Message ID CAJsO9v7iY+wGuMJZdz_owNmu1P19Vepkb7EQkJxtydRFoKYHqw@mail.gmail.com
State New
Headers show

Commit Message

Muhammad Faiz Feb. 10, 2018, 10:13 a.m.
On Sat, Feb 10, 2018 at 7:51 AM, Michael Niedermayer
<michael@niedermayer.cc> wrote:
> On Fri, Feb 09, 2018 at 10:21:04PM +0700, Muhammad Faiz wrote:
>> On Fri, Feb 9, 2018 at 6:53 PM, James Almer <jamrial@gmail.com> wrote:
>> > On 2/9/2018 7:56 AM, Muhammad Faiz wrote:
>> >> On Thu, Feb 8, 2018 at 7:04 AM, Michael Niedermayer
>> >> <michael@niedermayer.cc> wrote:
>> >>> On Wed, Feb 07, 2018 at 01:52:33PM +0100, Nicolas George wrote:
>> >>>> Josh de Kock (2018-02-06):
>> >>>>> ffmpeg | branch: master | Josh de Kock <josh@itanimul.li> | Fri Dec 22 22:17:00 2017 +0000| [7e8eba2d8755962d9dca5eade57bf8f591a73c0c] | committer: Josh de Kock
>> >>>>>
>> >>>>> lavc: add new API for iterating codecs and codec parsers
>> >>>>>
>> >>>>> Based on an unfinished patch by atomnuker.
>> >>>
>> >>> This commit also breaks
>> >>>
>> >>> ./configure --enable-libsoxr && make -j12 fate-checkasm
>> >>>  make -j12 fate-checkasm
>> >>>  ...
>> >>> LD      tests/checkasm/checkasm
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_out_samples':
>> >>> ffmpeg/libswresample/soxr_resample.c:118: undefined reference to `soxr_delay'
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_delay':
>> >>> ffmpeg/libswresample/soxr_resample.c:100: undefined reference to `soxr_delay'
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `flush':
>> >>> ffmpeg/libswresample/soxr_resample.c:70: undefined reference to `soxr_delay'
>> >>> ffmpeg/libswresample/soxr_resample.c:72: undefined reference to `soxr_process'
>> >>> ffmpeg/libswresample/soxr_resample.c:77: undefined reference to `soxr_process'
>> >>> ffmpeg/libswresample/soxr_resample.c:78: undefined reference to `soxr_delay'
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `process':
>> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_num_channels'
>> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_error'
>> >>> ffmpeg/libswresample/soxr_resample.c:90: undefined reference to `soxr_process'
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `destroy':
>> >>> ffmpeg/libswresample/soxr_resample.c:65: undefined reference to `soxr_delete'
>> >>> libswresample/libswresample.a(soxr_resample.o): In function `create':
>> >>> ffmpeg/libswresample/soxr_resample.c:46: undefined reference to `soxr_io_spec'
>> >>> ffmpeg/libswresample/soxr_resample.c:48: undefined reference to `soxr_quality_spec'
>> >>> ffmpeg/libswresample/soxr_resample.c:56: undefined reference to `soxr_delete'
>> >>> ffmpeg/libswresample/soxr_resample.c:57: undefined reference to `soxr_create'
>> >>> collect2: error: ld returned 1 exit status
>> >>> make: *** [tests/checkasm/checkasm] Error 1
>> >>
>> >> Fixed in 81d6501be77b273053a66eeced94d78e2021f1d1
>> >>
>> >> Thank's.
>> >
>> > This is not a proper solution. swr is pulled by avcodec only if Opus
>> > decoder is enabled. There's no reason to hardcode it for checkasm otherwise.
>> > The problem here is that the Makefile should pull all the dependencies
>> > of its hardcoded dependencies. This is what FFLIBS and FFEXTRALIBS do in
>> > common.mak to link the actual libraries.
>>
>> Probably, this is not a proper solution, but it is trivial enough.
>> (I'm sorry that I pushed it without posting).
>> So, please fix it with the proper solution. Probably, by adding
>> swresample-extralibs to avcodec-extralibs when avcodec has dependency
>> to swresample. I don't know how to do it.
>>
>
>> >
>> > That said, was this really a regression generated by this commit? It
>> > looks unrelated.
>>
>> Actually, it was a bug even before this commit. Just, previously it
>> was hidden because linker was smart enough to discard unneeded
>> dependency. But now when the list is changed to array, the linker is
>> unable to do it.
>
> iam not sure i understand correctly. But does this mean that
> tools/target_dec_"codec"_fuzzer will now include everything and not just
> the "codec" ?
> If so this will possibly prevent FFmpeg from being tested in googles ossfuzz
> framework. As their diskspace was already rather tight.
>
> i do see on my disk that the more recently build fuzzers have gottem MUCH
> larger:
> -rwxr-x--- 1 michael michael  17588987 Feb  3 18:53 tools/target_dec_scpr_fuzzer*
> -rwxr-x--- 1 michael michael  17476326 Feb  4 02:16 tools/target_dec_paf_video_fuzzer*
> -rwxr-x--- 1 michael michael 143210465 Feb  9 13:53 tools/target_dec_h264_fuzzer*
> -rwxr-x--- 1 michael michael 143210465 Feb  9 13:56 tools/target_dec_vp3_fuzzer*
>

Does the attached patch fix the problem?

Thank's.

Comments

Michael Niedermayer Feb. 10, 2018, 11:37 p.m.
On Sat, Feb 10, 2018 at 05:13:10PM +0700, Muhammad Faiz wrote:
> On Sat, Feb 10, 2018 at 7:51 AM, Michael Niedermayer
> <michael@niedermayer.cc> wrote:
> > On Fri, Feb 09, 2018 at 10:21:04PM +0700, Muhammad Faiz wrote:
> >> On Fri, Feb 9, 2018 at 6:53 PM, James Almer <jamrial@gmail.com> wrote:
> >> > On 2/9/2018 7:56 AM, Muhammad Faiz wrote:
> >> >> On Thu, Feb 8, 2018 at 7:04 AM, Michael Niedermayer
> >> >> <michael@niedermayer.cc> wrote:
> >> >>> On Wed, Feb 07, 2018 at 01:52:33PM +0100, Nicolas George wrote:
> >> >>>> Josh de Kock (2018-02-06):
> >> >>>>> ffmpeg | branch: master | Josh de Kock <josh@itanimul.li> | Fri Dec 22 22:17:00 2017 +0000| [7e8eba2d8755962d9dca5eade57bf8f591a73c0c] | committer: Josh de Kock
> >> >>>>>
> >> >>>>> lavc: add new API for iterating codecs and codec parsers
> >> >>>>>
> >> >>>>> Based on an unfinished patch by atomnuker.
> >> >>>
> >> >>> This commit also breaks
> >> >>>
> >> >>> ./configure --enable-libsoxr && make -j12 fate-checkasm
> >> >>>  make -j12 fate-checkasm
> >> >>>  ...
> >> >>> LD      tests/checkasm/checkasm
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_out_samples':
> >> >>> ffmpeg/libswresample/soxr_resample.c:118: undefined reference to `soxr_delay'
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_delay':
> >> >>> ffmpeg/libswresample/soxr_resample.c:100: undefined reference to `soxr_delay'
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `flush':
> >> >>> ffmpeg/libswresample/soxr_resample.c:70: undefined reference to `soxr_delay'
> >> >>> ffmpeg/libswresample/soxr_resample.c:72: undefined reference to `soxr_process'
> >> >>> ffmpeg/libswresample/soxr_resample.c:77: undefined reference to `soxr_process'
> >> >>> ffmpeg/libswresample/soxr_resample.c:78: undefined reference to `soxr_delay'
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `process':
> >> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_num_channels'
> >> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_error'
> >> >>> ffmpeg/libswresample/soxr_resample.c:90: undefined reference to `soxr_process'
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `destroy':
> >> >>> ffmpeg/libswresample/soxr_resample.c:65: undefined reference to `soxr_delete'
> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `create':
> >> >>> ffmpeg/libswresample/soxr_resample.c:46: undefined reference to `soxr_io_spec'
> >> >>> ffmpeg/libswresample/soxr_resample.c:48: undefined reference to `soxr_quality_spec'
> >> >>> ffmpeg/libswresample/soxr_resample.c:56: undefined reference to `soxr_delete'
> >> >>> ffmpeg/libswresample/soxr_resample.c:57: undefined reference to `soxr_create'
> >> >>> collect2: error: ld returned 1 exit status
> >> >>> make: *** [tests/checkasm/checkasm] Error 1
> >> >>
> >> >> Fixed in 81d6501be77b273053a66eeced94d78e2021f1d1
> >> >>
> >> >> Thank's.
> >> >
> >> > This is not a proper solution. swr is pulled by avcodec only if Opus
> >> > decoder is enabled. There's no reason to hardcode it for checkasm otherwise.
> >> > The problem here is that the Makefile should pull all the dependencies
> >> > of its hardcoded dependencies. This is what FFLIBS and FFEXTRALIBS do in
> >> > common.mak to link the actual libraries.
> >>
> >> Probably, this is not a proper solution, but it is trivial enough.
> >> (I'm sorry that I pushed it without posting).
> >> So, please fix it with the proper solution. Probably, by adding
> >> swresample-extralibs to avcodec-extralibs when avcodec has dependency
> >> to swresample. I don't know how to do it.
> >>
> >
> >> >
> >> > That said, was this really a regression generated by this commit? It
> >> > looks unrelated.
> >>
> >> Actually, it was a bug even before this commit. Just, previously it
> >> was hidden because linker was smart enough to discard unneeded
> >> dependency. But now when the list is changed to array, the linker is
> >> unable to do it.
> >
> > iam not sure i understand correctly. But does this mean that
> > tools/target_dec_"codec"_fuzzer will now include everything and not just
> > the "codec" ?
> > If so this will possibly prevent FFmpeg from being tested in googles ossfuzz
> > framework. As their diskspace was already rather tight.
> >
> > i do see on my disk that the more recently build fuzzers have gottem MUCH
> > larger:
> > -rwxr-x--- 1 michael michael  17588987 Feb  3 18:53 tools/target_dec_scpr_fuzzer*
> > -rwxr-x--- 1 michael michael  17476326 Feb  4 02:16 tools/target_dec_paf_video_fuzzer*
> > -rwxr-x--- 1 michael michael 143210465 Feb  9 13:53 tools/target_dec_h264_fuzzer*
> > -rwxr-x--- 1 michael michael 143210465 Feb  9 13:56 tools/target_dec_vp3_fuzzer*
> >
> 
> Does the attached patch fix the problem?

in my local build this makes no difference to the size

make distclean ; ./configure   --optflags='-O1 -Os' --enable-debug --disable-encoders --disable-parsers --disable-avformat --disable-avfilter --disable-ffplay --disable-ffmpeg --disable-avdevice  --disable-stripping --enable-ossfuzz --libfuzzer='~/libfuzzer/libFuzzer.a -lstdc++'    --cc='ccache clang' --cxx='ccache clang++' --disable-vaapi --disable-cuvid   --pkg-config-flags="--static" --enable-libx265  --enable-libx264 --enable-libvorbis --enable-libopus --enable-libvpx --enable-libmp3lame --enable-libfdk-aac  --enable-gpl --enable-nonfree --enable-libfreetype --enable-libtheora --enable-libass --assert_level=2 --enable-debug && time make -j12 tools/target_dec_h264_fuzzer


after the patch:
-rwxr-x--- 1 michael michael 143237303 Feb 11 00:18 tools/target_dec_h264_fuzzer

before the patch:
-rwxr-x--- 1 michael michael 143237007 Feb 11 00:27 tools/target_dec_h264_fuzzer*

caa4bd7a9fe9186a56fdcb8194263b066978cbbe
-rwxr-x--- 1 michael michael  35470813 Feb 11 00:31 tools/target_dec_h264_fuzzer*


[...]
Muhammad Faiz Feb. 11, 2018, 12:35 a.m.
On Sun, Feb 11, 2018 at 6:37 AM, Michael Niedermayer
<michael@niedermayer.cc> wrote:
> On Sat, Feb 10, 2018 at 05:13:10PM +0700, Muhammad Faiz wrote:
>> On Sat, Feb 10, 2018 at 7:51 AM, Michael Niedermayer
>> <michael@niedermayer.cc> wrote:
>> > On Fri, Feb 09, 2018 at 10:21:04PM +0700, Muhammad Faiz wrote:
>> >> On Fri, Feb 9, 2018 at 6:53 PM, James Almer <jamrial@gmail.com> wrote:
>> >> > On 2/9/2018 7:56 AM, Muhammad Faiz wrote:
>> >> >> On Thu, Feb 8, 2018 at 7:04 AM, Michael Niedermayer
>> >> >> <michael@niedermayer.cc> wrote:
>> >> >>> On Wed, Feb 07, 2018 at 01:52:33PM +0100, Nicolas George wrote:
>> >> >>>> Josh de Kock (2018-02-06):
>> >> >>>>> ffmpeg | branch: master | Josh de Kock <josh@itanimul.li> | Fri Dec 22 22:17:00 2017 +0000| [7e8eba2d8755962d9dca5eade57bf8f591a73c0c] | committer: Josh de Kock
>> >> >>>>>
>> >> >>>>> lavc: add new API for iterating codecs and codec parsers
>> >> >>>>>
>> >> >>>>> Based on an unfinished patch by atomnuker.
>> >> >>>
>> >> >>> This commit also breaks
>> >> >>>
>> >> >>> ./configure --enable-libsoxr && make -j12 fate-checkasm
>> >> >>>  make -j12 fate-checkasm
>> >> >>>  ...
>> >> >>> LD      tests/checkasm/checkasm
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_out_samples':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:118: undefined reference to `soxr_delay'
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `get_delay':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:100: undefined reference to `soxr_delay'
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `flush':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:70: undefined reference to `soxr_delay'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:72: undefined reference to `soxr_process'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:77: undefined reference to `soxr_process'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:78: undefined reference to `soxr_delay'
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `process':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_num_channels'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:88: undefined reference to `soxr_set_error'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:90: undefined reference to `soxr_process'
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `destroy':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:65: undefined reference to `soxr_delete'
>> >> >>> libswresample/libswresample.a(soxr_resample.o): In function `create':
>> >> >>> ffmpeg/libswresample/soxr_resample.c:46: undefined reference to `soxr_io_spec'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:48: undefined reference to `soxr_quality_spec'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:56: undefined reference to `soxr_delete'
>> >> >>> ffmpeg/libswresample/soxr_resample.c:57: undefined reference to `soxr_create'
>> >> >>> collect2: error: ld returned 1 exit status
>> >> >>> make: *** [tests/checkasm/checkasm] Error 1
>> >> >>
>> >> >> Fixed in 81d6501be77b273053a66eeced94d78e2021f1d1
>> >> >>
>> >> >> Thank's.
>> >> >
>> >> > This is not a proper solution. swr is pulled by avcodec only if Opus
>> >> > decoder is enabled. There's no reason to hardcode it for checkasm otherwise.
>> >> > The problem here is that the Makefile should pull all the dependencies
>> >> > of its hardcoded dependencies. This is what FFLIBS and FFEXTRALIBS do in
>> >> > common.mak to link the actual libraries.
>> >>
>> >> Probably, this is not a proper solution, but it is trivial enough.
>> >> (I'm sorry that I pushed it without posting).
>> >> So, please fix it with the proper solution. Probably, by adding
>> >> swresample-extralibs to avcodec-extralibs when avcodec has dependency
>> >> to swresample. I don't know how to do it.
>> >>
>> >
>> >> >
>> >> > That said, was this really a regression generated by this commit? It
>> >> > looks unrelated.
>> >>
>> >> Actually, it was a bug even before this commit. Just, previously it
>> >> was hidden because linker was smart enough to discard unneeded
>> >> dependency. But now when the list is changed to array, the linker is
>> >> unable to do it.
>> >
>> > iam not sure i understand correctly. But does this mean that
>> > tools/target_dec_"codec"_fuzzer will now include everything and not just
>> > the "codec" ?
>> > If so this will possibly prevent FFmpeg from being tested in googles ossfuzz
>> > framework. As their diskspace was already rather tight.
>> >
>> > i do see on my disk that the more recently build fuzzers have gottem MUCH
>> > larger:
>> > -rwxr-x--- 1 michael michael  17588987 Feb  3 18:53 tools/target_dec_scpr_fuzzer*
>> > -rwxr-x--- 1 michael michael  17476326 Feb  4 02:16 tools/target_dec_paf_video_fuzzer*
>> > -rwxr-x--- 1 michael michael 143210465 Feb  9 13:53 tools/target_dec_h264_fuzzer*
>> > -rwxr-x--- 1 michael michael 143210465 Feb  9 13:56 tools/target_dec_vp3_fuzzer*
>> >
>>
>> Does the attached patch fix the problem?
>
> in my local build this makes no difference to the size
>
> make distclean ; ./configure   --optflags='-O1 -Os' --enable-debug --disable-encoders --disable-parsers --disable-avformat --disable-avfilter --disable-ffplay --disable-ffmpeg --disable-avdevice  --disable-stripping --enable-ossfuzz --libfuzzer='~/libfuzzer/libFuzzer.a -lstdc++'    --cc='ccache clang' --cxx='ccache clang++' --disable-vaapi --disable-cuvid   --pkg-config-flags="--static" --enable-libx265  --enable-libx264 --enable-libvorbis --enable-libopus --enable-libvpx --enable-libmp3lame --enable-libfdk-aac  --enable-gpl --enable-nonfree --enable-libfreetype --enable-libtheora --enable-libass --assert_level=2 --enable-debug && time make -j12 tools/target_dec_h264_fuzzer
>
>
> after the patch:
> -rwxr-x--- 1 michael michael 143237303 Feb 11 00:18 tools/target_dec_h264_fuzzer
>
> before the patch:
> -rwxr-x--- 1 michael michael 143237007 Feb 11 00:27 tools/target_dec_h264_fuzzer*
>
> caa4bd7a9fe9186a56fdcb8194263b066978cbbe
> -rwxr-x--- 1 michael michael  35470813 Feb 11 00:31 tools/target_dec_h264_fuzzer*

Yeah, this isn't simple. I have no idea about this.

Thank's.

Patch hide | download patch | download mbox

From 46d5a15dba7578d312aae59048ad608c9afdacbf Mon Sep 17 00:00:00 2001
From: Muhammad Faiz <mfcc64@gmail.com>
Date: Sat, 10 Feb 2018 17:08:04 +0700
Subject: [PATCH] tools/target_dec_fuzzer: don't use avcodec_register()

Signed-off-by: Muhammad Faiz <mfcc64@gmail.com>
---
 tools/target_dec_fuzzer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/target_dec_fuzzer.c b/tools/target_dec_fuzzer.c
index 62d62a9de3..6c575f8845 100644
--- a/tools/target_dec_fuzzer.c
+++ b/tools/target_dec_fuzzer.c
@@ -144,7 +144,8 @@  int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
 #define DECODER_SYMBOL0(CODEC) ff_##CODEC##_decoder
 #define DECODER_SYMBOL(CODEC) DECODER_SYMBOL0(CODEC)
         extern AVCodec DECODER_SYMBOL(FFMPEG_DECODER);
-        avcodec_register(&DECODER_SYMBOL(FFMPEG_DECODER));
+        if (DECODER_SYMBOL(FFMPEG_DECODER).init_static_data)
+            DECODER_SYMBOL(FFMPEG_DECODER).init_static_data(&DECODER_SYMBOL(FFMPEG_DECODER));
 
         c = &DECODER_SYMBOL(FFMPEG_DECODER);
 #else
-- 
2.13.2