diff mbox

[FFmpeg-devel,GSOC] TensorFlow backend introduction for DNN module

Message ID CAAeE=qqPeZiJ8tSraDwGZQW_6yq1KZ+wjJ20Raz-4-tre_KsrQ@mail.gmail.com
State New
Headers show

Commit Message

Sergey Lavrushkin June 7, 2018, 3:29 p.m. UTC
2018-06-06 17:22 GMT+03:00 Pedro Arthur <bygrandao@gmail.com>:

> Hi,
>
> 2018-06-05 20:23 GMT-03:00 Sergey Lavrushkin <dualfal@gmail.com>:
> > Here is the patch, that fixes described issues.
> When I try to run (video input), when tf is not enabled in configure it
> crashes.
>
>
> $ffmpeg -i in.mp4 -vf srcnn=dnn_backend=tensorflow out.mp4
>
> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
> developers
>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>   configuration:
>   libavutil      56. 18.102 / 56. 18.102
>   libavcodec     58. 19.105 / 58. 19.105
>   libavformat    58. 17.100 / 58. 17.100
>   libavdevice    58.  4.100 / 58.  4.100
>   libavfilter     7. 25.100 /  7. 25.100
>   libswscale      5.  2.100 /  5.  2.100
>   libswresample   3.  2.100 /  3.  2.100
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mp4':
>   Metadata:
>     major_brand     : isom
>     minor_version   : 512
>     compatible_brands: isomiso2mp41
>     encoder         : Lavf58.17.100
>   Duration: 00:06:13.70, start: 0.000000, bitrate: 5912 kb/s
>     Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v /
> 0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 5777 kb/s, 29.97
> fps, 29.97 tbr, 30k tbn, 30k tbc (default)
>     Metadata:
>       handler_name    : VideoHandler
>     Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz,
> stereo, fltp, 128 kb/s (default)
>     Metadata:
>       handler_name    : SoundHandler
> Stream mapping:
>   Stream #0:0 -> #0:0 (mpeg4 (native) -> mpeg4 (native))
>   Stream #0:1 -> #0:1 (aac (native) -> aac (native))
> Press [q] to stop, [?] for help
> free(): invalid pointer
> Aborted (core dumped)
>
>
>
> When the output is an image, t does not crashes but neither fallback to
> native
>
>
> $ffmpeg -i in.jpg -vf srcnn=dnn_backend=tensorflow out.png
>
> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
> developers
>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>   configuration:
>   libavutil      56. 18.102 / 56. 18.102
>   libavcodec     58. 19.105 / 58. 19.105
>   libavformat    58. 17.100 / 58. 17.100
>   libavdevice    58.  4.100 / 58.  4.100
>   libavfilter     7. 25.100 /  7. 25.100
>   libswscale      5.  2.100 /  5.  2.100
>   libswresample   3.  2.100 /  3.  2.100
> Input #0, image2, from 'in.jpg':
>   Duration: 00:00:00.04, start: 0.000000, bitrate: 43469 kb/s
>     Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown),
> 1192x670 [SAR 1:1 DAR 596:335], 25 tbr, 25 tbn, 25 tbc
> Stream mapping:
>   Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
> Press [q] to stop, [?] for help
> [Parsed_srcnn_0 @ 0x557d3ea55980] could not create DNN module for
> requested backend
> [AVFilterGraph @ 0x557d3ea102c0] Error initializing filter 'srcnn'
> with args 'dnn_backend=tensorflow'
> Error reinitializing filters!
> Failed to inject frame into filter network: Cannot allocate memory
> Error while processing the decoded data for stream #0:0
> Conversion failed!
>
>
> I think you could disable the tensorflow option if it is not enable in
> configure or fallback to native, either solution is ok for me.


I disabled tensorflow option when it is not configured with it. Here is the
updated patch.
I think, crash occurred due to improper call to av_freep for dnn_module.
Here is also the patch, that fixes this bug.

Comments

Pedro Arthur June 8, 2018, 1:28 p.m. UTC | #1
Pushed.

Thanks.

2018-06-07 12:29 GMT-03:00 Sergey Lavrushkin <dualfal@gmail.com>:
> 2018-06-06 17:22 GMT+03:00 Pedro Arthur <bygrandao@gmail.com>:
>>
>> Hi,
>>
>> 2018-06-05 20:23 GMT-03:00 Sergey Lavrushkin <dualfal@gmail.com>:
>> > Here is the patch, that fixes described issues.
>> When I try to run (video input), when tf is not enabled in configure it
>> crashes.
>>
>>
>> $ffmpeg -i in.mp4 -vf srcnn=dnn_backend=tensorflow out.mp4
>>
>> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
>> developers
>>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>>   configuration:
>>   libavutil      56. 18.102 / 56. 18.102
>>   libavcodec     58. 19.105 / 58. 19.105
>>   libavformat    58. 17.100 / 58. 17.100
>>   libavdevice    58.  4.100 / 58.  4.100
>>   libavfilter     7. 25.100 /  7. 25.100
>>   libswscale      5.  2.100 /  5.  2.100
>>   libswresample   3.  2.100 /  3.  2.100
>> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'in.mp4':
>>   Metadata:
>>     major_brand     : isom
>>     minor_version   : 512
>>     compatible_brands: isomiso2mp41
>>     encoder         : Lavf58.17.100
>>   Duration: 00:06:13.70, start: 0.000000, bitrate: 5912 kb/s
>>     Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v /
>> 0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 5777 kb/s, 29.97
>> fps, 29.97 tbr, 30k tbn, 30k tbc (default)
>>     Metadata:
>>       handler_name    : VideoHandler
>>     Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz,
>> stereo, fltp, 128 kb/s (default)
>>     Metadata:
>>       handler_name    : SoundHandler
>> Stream mapping:
>>   Stream #0:0 -> #0:0 (mpeg4 (native) -> mpeg4 (native))
>>   Stream #0:1 -> #0:1 (aac (native) -> aac (native))
>> Press [q] to stop, [?] for help
>> free(): invalid pointer
>> Aborted (core dumped)
>>
>>
>>
>> When the output is an image, t does not crashes but neither fallback to
>> native
>>
>>
>> $ffmpeg -i in.jpg -vf srcnn=dnn_backend=tensorflow out.png
>>
>> ffmpeg version N-91232-g256386fd3e Copyright (c) 2000-2018 the FFmpeg
>> developers
>>   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
>>   configuration:
>>   libavutil      56. 18.102 / 56. 18.102
>>   libavcodec     58. 19.105 / 58. 19.105
>>   libavformat    58. 17.100 / 58. 17.100
>>   libavdevice    58.  4.100 / 58.  4.100
>>   libavfilter     7. 25.100 /  7. 25.100
>>   libswscale      5.  2.100 /  5.  2.100
>>   libswresample   3.  2.100 /  3.  2.100
>> Input #0, image2, from 'in.jpg':
>>   Duration: 00:00:00.04, start: 0.000000, bitrate: 43469 kb/s
>>     Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown),
>> 1192x670 [SAR 1:1 DAR 596:335], 25 tbr, 25 tbn, 25 tbc
>> Stream mapping:
>>   Stream #0:0 -> #0:0 (mjpeg (native) -> png (native))
>> Press [q] to stop, [?] for help
>> [Parsed_srcnn_0 @ 0x557d3ea55980] could not create DNN module for
>> requested backend
>> [AVFilterGraph @ 0x557d3ea102c0] Error initializing filter 'srcnn'
>> with args 'dnn_backend=tensorflow'
>> Error reinitializing filters!
>> Failed to inject frame into filter network: Cannot allocate memory
>> Error while processing the decoded data for stream #0:0
>> Conversion failed!
>>
>>
>> I think you could disable the tensorflow option if it is not enable in
>> configure or fallback to native, either solution is ok for me.
>
>
> I disabled tensorflow option when it is not configured with it. Here is the
> updated patch.
> I think, crash occurred due to improper call to av_freep for dnn_module.
> Here is also the patch, that fixes this bug.
>
diff mbox

Patch

From e223a00972ec8b29f557d7ae55d597238abe9004 Mon Sep 17 00:00:00 2001
From: Sergey Lavrushkin <dualfal@gmail.com>
Date: Thu, 7 Jun 2018 18:22:22 +0300
Subject: [PATCH 2/2] libavfilter/dnn_interface.c: fix av_freep dnn_module.

---
 libavfilter/dnn_interface.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavfilter/dnn_interface.c b/libavfilter/dnn_interface.c
index ce64fe58f3..87c90526be 100644
--- a/libavfilter/dnn_interface.c
+++ b/libavfilter/dnn_interface.c
@@ -51,7 +51,7 @@  DNNModule* ff_get_dnn_module(DNNBackendType backend_type)
         dnn_module->execute_model = &ff_dnn_execute_model_tf;
         dnn_module->free_model = &ff_dnn_free_model_tf;
     #else
-        av_freep(dnn_module);
+        av_freep(&dnn_module);
         return NULL;
     #endif
     }
-- 
2.14.1