[FFmpeg-devel] avfilter/vf_overlay_cuda: check ctx->hwctx->internal null pointer

From: Steven Liu <lq@chinaffmpeg.org>

because it will cannot get ctx->hwctx->internal value when
get ctx->hwctx->internal->cuda_dl value. so should check if internal is
null.

before patch:
liuqi@chinaffmpeg:~/multimedia/build$ ./ffmpeg_g -hwaccel_device 0 -hwaccel cuda -pix_fmt nv12 -s 1920x1080 -i output.yuv -hwaccel_device 0 -hwaccel cuda -pix_fmt nv12 -s 800x480 -i output_small.yuv -filter_complex "[0:v]hwupload_cuda[a];[1:v]hwupload_cuda[b];[a][b]overlay_cuda=x=0:y=0[o];[o]hwdownload[o1]" -map "[o1]" -pix_fmt nv12 -y -t 1 xxx.yuv
ffmpeg version N-101879-g13bf797ced Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --cc='ccache cc' --enable-libx264 --enable-libx265 --enable-libopus --enable-libwebp --enable-libvpx --enable-ffnvcodec --enable-vaapi --enable-opengl --enable-nonfree --enable-gpl --enable-libfdk-aac --enable-libfreetype --enable-cuda --enable-libnpp --enable-cuda-nvcc --enable-cuda-llvm
  libavutil      56. 72.100 / 56. 72.100
  libavcodec     58.135.100 / 58.135.100
  libavformat    58. 78.100 / 58. 78.100
  libavdevice    58. 14.100 / 58. 14.100
  libavfilter     7.111.100 /  7.111.100
  libswscale      5. 10.100 /  5. 10.100
  libswresample   3. 10.100 /  3. 10.100
  libpostproc    55. 10.100 / 55. 10.100
[rawvideo @ 0x5584b7a9e4c0] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from 'output.yuv':
  Duration: 00:01:54.64, start: 0.000000, bitrate: 622080 kb/s
  Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 1920x1080, 622080 kb/s, 25 tbr, 25 tbn, 25 tbc
[rawvideo @ 0x5584b7aa30c0] Estimating duration from bitrate, this may be inaccurate
Input #1, rawvideo, from 'output_small.yuv':
  Duration: 00:04:07.16, start: 0.000000, bitrate: 115200 kb/s
  Stream #1:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 800x480, 115200 kb/s, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> hwupload_cuda
  Stream #1:0 (rawvideo) -> hwupload_cuda
  hwdownload -> Stream #0:0 (rawvideo)
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'xxx.yuv':
  Metadata:
    encoder         : Lavf58.78.100
  Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12(progressive), 1920x1080, q=2-31, 622080 kb/s, 25 fps, 25 tbn (default)
    Metadata:
      encoder         : Lavc58.135.100 rawvideo
frame=   25 fps=0.0 q=-0.0 Lsize=   75938kB time=00:00:01.00 bitrate=622080.0kbits/s speed=6.11x
video:75938kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Segmentation fault (core dumped)

This patch can fix the Segmentation fault problem.

Signed-off-by: Steven Liu <lq@chinaffmpeg.org>
---
 libavfilter/vf_overlay_cuda.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On 09.04.2021 12:34, Steven Liu wrote:
> -    if (ctx->hwctx && ctx->cu_module) {
> +    if (ctx->hwctx && ctx->cu_module && ctx->hwctx->internal) {
>           CUcontext dummy;
>           CudaFunctions *cu = ctx->hwctx->internal->cuda_dl;
>           CHECK_CU(cu->cuCtxPushCurrent(ctx->cu_ctx));
> 

How can this happen at this point?
Did something else prematurely free the hwctx?

This makes me think that the filter is missing a buffer ref somewhere to 
prevent things from getting freed too early.

Can you test if
https://github.com/BtbN/FFmpeg/commit/454d5c5d3eda7b5356f961a0e1b66f3a56d95363
Also fixes the issue?

I'm unable to reproduce the crash.

Timo Rothenpieler <timo@rothenpieler.org> 于2021年4月10日周六 下午8:34写道：
>
> Can you test if
> https://github.com/BtbN/FFmpeg/commit/454d5c5d3eda7b5356f961a0e1b66f3a56d95363
> Also fixes the issue?
Yes, it can fix the issue


Thanks Timo.

Thanks
Steven