[FFmpeg-devel] avcodec/videotoolboxenc: fix encoding frame crash on iOS 11

Submitted by sharpbai@gmail.com on Aug. 13, 2019, 6:18 a.m.

Details

Message ID 20190813061820.58120-1-sharpbai@gmail.com
State New
Headers show

Commit Message

sharpbai@gmail.com Aug. 13, 2019, 6:18 a.m.
On iOS 11, encoding a frame may return error with log
"Error encoding frame 0", which means vtenc_output_callback
is called with status=0 and sample_buffer=NULL. Then the
encoding session will be crashed on next callback wether or not
closing the codec context.

Let us look through the link below introducing VTCompressionOutputCallback,

https://developer.apple.com/documentation/videotoolbox/vtcompressionoutputcallback?language=objc

"status=0" (noErr) means compression was successful.
"sampleBuffer=NULL" means the frame was dropped when compression
was successful (status=0) or compression was not successful (status!=0).

So we should not set AVERROR_EXTERNAL on "status=0" and "sample_buffer=NULL"
as it is not a error.

The fix is that we only set AVERROR_EXTERNAL with status value non zero.
When sample_buffer is NULL and status value is zero, we simply return
with no other operation.

This crash often occurs on iOS 11 for example encoding 720p@25fps.

Signed-off-by: sharpbai <sharpbai@gmail.com>
---
 libavcodec/videotoolboxenc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Patch hide | download patch | download mbox

diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index d76bb7f646..8afdd125d2 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -569,12 +569,16 @@  static void vtenc_output_callback(
         return;
     }
 
-    if (status || !sample_buffer) {
+    if (status) {
         av_log(avctx, AV_LOG_ERROR, "Error encoding frame: %d\n", (int)status);
         set_async_error(vtctx, AVERROR_EXTERNAL);
         return;
     }
 
+    if (!sample_buffer) {
+        return;
+    }
+
     if (!avctx->extradata && (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER)) {
         int set_status = set_extradata(avctx, sample_buffer);
         if (set_status) {