From patchwork Fri Feb 16 03:52:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 7608 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.150.1 with SMTP id c1csp227409jai; Thu, 15 Feb 2018 19:58:22 -0800 (PST) X-Google-Smtp-Source: AH8x227cfg0vdFVNNaI8dgStJcWdAPNce603GVohXdNZRFaTH1attLA8W8s8jT59AmLnsePQoeSZ X-Received: by 10.223.197.12 with SMTP id q12mr4623572wrf.147.1518753502243; Thu, 15 Feb 2018 19:58:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518753502; cv=none; d=google.com; s=arc-20160816; b=QLUjGG0CcSHQLK0rMl9ib6BfnduNCxQbr2leqKClHlN4dOD9BbCNSIjwwaKkUZZtDd nFUcqNFnJ/sBD8xCPVwnOpbQtjSJywfjwa0WQHCcTmSabEldWUiJ0iLo2zk5SqhIWi63 SmbpFbs+1kCHiu2r1WpGMZRSDdtZyxcAfJZUXR/MSi4DP9bjuTxguoh8PD9jbpIUtGkj UmKFBth9CsawzbDPw7CpOxVOOH28mvF65R9XHRZn8E+y6u/zhDLtMtJjztKO4/rUzUSK 6pEReHWT2c3burKj+R7p4oA02YSg0l5F9TLIE5qSu6/SasGZpo4zWm7OZPItMu2CQhB2 F6PA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=sg5VFX+M06Ccs31kwEzhtQQs4X3QpYqvAHZ0Ruezjzo=; b=IlYTW8vOUQofN49WPWgNGG4fT1goFrD/S0zZIKknDyzJ9onTo3FY3BJXRcLfVT81c0 QziG+LwXzXb68p+wQcJVjAKb/ZiUxyyuGTZq11JhSvmEP+pgY5wap/U4qQG7nbhbfsjR ZexdJr+/pravHFlYyqEgbzyah6nWBNSZ5dsPOCngeSHUlvXak8jEF8F3i3sL0IehRqAP YWsxSxEQZ4ujSJ+GVzFOUxjIMJUPIF2zK8qcp+hvSVKn7cj8vTTsW/gl37kO8c5DyYs4 0aa7rFuu+A2K93ArkU5J2GAhPNN8cFR7qUyYoodgnGLFMJ9zjKe5ZiEtAtQ/Xv7htp7S Ff2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=qWdKL68M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 19si12814534wrt.156.2018.02.15.19.58.21; Thu, 15 Feb 2018 19:58:22 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=qWdKL68M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 78F3568A0F9; Fri, 16 Feb 2018 05:58:07 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f42.google.com (mail-pg0-f42.google.com [74.125.83.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BB47868A0C8 for ; Fri, 16 Feb 2018 05:58:00 +0200 (EET) Received: by mail-pg0-f42.google.com with SMTP id m19so1480286pgn.1 for ; Thu, 15 Feb 2018 19:58:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id; bh=MKrmO7s/7Q33vpI0PzP0v+N8Hxlr8KHDIkBSM6pe7Q0=; b=qWdKL68M7A3lyDMhMy9QzDbbbSE07GsIgb8TJJpX8SnOmRM3cRNrK5oXjAJPZnD0mR Kvr9mfw/lL2DXexNA0/cyIlKVkbkLpeHZx7blXABlLLmgMhRqU6pTBmAXylMA/mOPvPL 6NlbLiZ5LfhLV/Axh2bPn/X/XKnFiGFqwOKY1jf3TxAXiqisGxWZoOtJt13UUcP/3+dx 27XX7w8hPuQyTpDMGZyeOgo2MHCaGuq7ebMsv/vQTvBmHsI7dDLkmPcInoFSs9hP9DJ2 mlmxnqGL/GdQNZP75dBjv8isms9jBSkKcGLRE8oo8xsEEJ28lbJNeupctnOm/x1gv5V+ Dh3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id; bh=MKrmO7s/7Q33vpI0PzP0v+N8Hxlr8KHDIkBSM6pe7Q0=; b=J88NV/33pEeHFBep/QKrtia3mg0Y/97SenYGl9II25nWAdl2O0VPYmGqUkWVJXXwFH kBJc3dXphMEF+082CCk7WW/xZ+606qFF68RVbECMMwEBnuYSXeMyNWtba+mGG2RhpElg no7n7rkz/zsEnOZ3g6ADdSgA+mpJy12RzjSvF3+NmFmuzBc9XgWLka/ihjSPzRr9on/k 7cyJX/tbmfKoWat2dlITKyq4lATqh9f4iCntx5OW315ITcqV01ilcZxvjamrEXM/vH9E OX0kEcEbY+f4fvgid46uwusrJtctgJbdy7N8JP9DSD5HhBUpMVPXjDCoKJAqGO1bbQZD VgiA== X-Gm-Message-State: APf1xPAMjp99tZLucuUd6zhIt4wM+nwSPCilYfRgUQ+0iMTavhENJxMN VMGXUkyVcH2phQZjj0nOM/QyFo2F X-Received: by 10.99.166.10 with SMTP id t10mr4012330pge.198.1518753144608; Thu, 15 Feb 2018 19:52:24 -0800 (PST) Received: from localhost.localdomain (c-69-181-54-242.hsd1.ca.comcast.net. [69.181.54.242]) by smtp.gmail.com with ESMTPSA id j25sm53992770pfk.38.2018.02.15.19.52.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 15 Feb 2018 19:52:23 -0800 (PST) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Thu, 15 Feb 2018 19:52:14 -0800 Message-Id: <20180216035214.95097-1-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.14.2 Subject: [FFmpeg-devel] [PATCH] avcodec/mediacodecdec: refactor to take advantage of new decoding api X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@gmail.com, Aman Gupta MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta This refactor splits up the main mediacodec decode loop into two send/receive helpers, which are then used to rewrite the receive_frame callback and take full advantage of the new decoding api. Since we can now request packets on demand with ff_decode_get_packet(), the fifo buffer is no longer necessary and has been removed. This change was motivated by behavior observed on certain Android TV devices, featuring hardware mpeg2/h264 decoders which also deinterlace content (to produce multiple frames per field). Previously, this code caused buffering issues because queueInputBuffer() was always invoked before each dequeueOutputBuffer(), even though twice as many output buffers were being generated. With this patch, the decoder will always attempt to drain new frames first before sending more data into the underlying codec. --- libavcodec/mediacodecdec.c | 107 ++++++++++++++------------------------ libavcodec/mediacodecdec_common.c | 50 ++++++++++++------ libavcodec/mediacodecdec_common.h | 14 +++-- 3 files changed, 80 insertions(+), 91 deletions(-) diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c index cb1151a195..0c29a1113d 100644 --- a/libavcodec/mediacodecdec.c +++ b/libavcodec/mediacodecdec.c @@ -25,7 +25,6 @@ #include "libavutil/avassert.h" #include "libavutil/common.h" -#include "libavutil/fifo.h" #include "libavutil/opt.h" #include "libavutil/intreadwrite.h" #include "libavutil/pixfmt.h" @@ -43,8 +42,6 @@ typedef struct MediaCodecH264DecContext { MediaCodecDecContext *ctx; - AVFifoBuffer *fifo; - AVPacket buffered_pkt; } MediaCodecH264DecContext; @@ -56,8 +53,6 @@ static av_cold int mediacodec_decode_close(AVCodecContext *avctx) ff_mediacodec_dec_close(avctx, s->ctx); s->ctx = NULL; - av_fifo_free(s->fifo); - av_packet_unref(&s->buffered_pkt); return 0; @@ -400,12 +395,6 @@ static av_cold int mediacodec_decode_init(AVCodecContext *avctx) av_log(avctx, AV_LOG_INFO, "MediaCodec started successfully, ret = %d\n", ret); - s->fifo = av_fifo_alloc(sizeof(AVPacket)); - if (!s->fifo) { - ret = AVERROR(ENOMEM); - goto done; - } - done: if (format) { ff_AMediaFormat_delete(format); @@ -418,13 +407,33 @@ done: return ret; } +static int mediacodec_send_receive(AVCodecContext *avctx, + MediaCodecH264DecContext *s, + AVFrame *frame, bool wait) +{ + int ret; + + /* send any pending data from buffered packet */ + while (s->buffered_pkt.size) { + ret = ff_mediacodec_dec_send(avctx, s->ctx, &s->buffered_pkt); + if (ret == AVERROR(EAGAIN)) + break; + if (ret < 0) + return ret; + s->buffered_pkt.size -= ret; + s->buffered_pkt.data += ret; + if (s->buffered_pkt.size <= 0) + av_packet_unref(&s->buffered_pkt); + } + + /* check for new frame */ + return ff_mediacodec_dec_receive(avctx, s->ctx, frame, wait); +} + static int mediacodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) { MediaCodecH264DecContext *s = avctx->priv_data; int ret; - int got_frame = 0; - int is_eof = 0; - AVPacket pkt = { 0 }; /* * MediaCodec.flush() discards both input and output buffers, thus we @@ -452,74 +461,34 @@ static int mediacodec_receive_frame(AVCodecContext *avctx, AVFrame *frame) } } - ret = ff_decode_get_packet(avctx, &pkt); - if (ret == AVERROR_EOF) - is_eof = 1; - else if (ret == AVERROR(EAGAIN)) - ; /* no input packet, but fallthrough to check for pending frames */ - else if (ret < 0) + /* flush buffered packet and check for new frame */ + ret = mediacodec_send_receive(avctx, s, frame, false); + if (ret != AVERROR(EAGAIN)) return ret; - /* buffer the input packet */ - if (pkt.size) { - if (av_fifo_space(s->fifo) < sizeof(pkt)) { - ret = av_fifo_realloc2(s->fifo, - av_fifo_size(s->fifo) + sizeof(pkt)); - if (ret < 0) { - av_packet_unref(&pkt); - return ret; - } - } - av_fifo_generic_write(s->fifo, &pkt, sizeof(pkt), NULL); - } - - /* process buffered data */ - while (!got_frame) { - /* prepare the input data */ - if (s->buffered_pkt.size <= 0) { - av_packet_unref(&s->buffered_pkt); - - /* no more data */ - if (av_fifo_size(s->fifo) < sizeof(AVPacket)) { - AVPacket null_pkt = { 0 }; - if (is_eof) { - ret = ff_mediacodec_dec_decode(avctx, s->ctx, frame, - &got_frame, &null_pkt); - if (ret < 0) - return ret; - else if (got_frame) - return 0; - else - return AVERROR_EOF; - } - return AVERROR(EAGAIN); - } - - av_fifo_generic_read(s->fifo, &s->buffered_pkt, sizeof(s->buffered_pkt), NULL); - } + /* skip fetching new packet if we still have one buffered */ + if (s->buffered_pkt.size > 0) + return AVERROR(EAGAIN); - ret = ff_mediacodec_dec_decode(avctx, s->ctx, frame, &got_frame, &s->buffered_pkt); + /* fetch new packet or eof */ + ret = ff_decode_get_packet(avctx, &s->buffered_pkt); + if (ret == AVERROR_EOF) { + AVPacket null_pkt = { 0 }; + ret = ff_mediacodec_dec_send(avctx, s->ctx, &null_pkt); if (ret < 0) return ret; - - s->buffered_pkt.size -= ret; - s->buffered_pkt.data += ret; } + else if (ret < 0) + return ret; - return 0; + /* crank decoder with new packet */ + return mediacodec_send_receive(avctx, s, frame, true); } static void mediacodec_decode_flush(AVCodecContext *avctx) { MediaCodecH264DecContext *s = avctx->priv_data; - while (av_fifo_size(s->fifo)) { - AVPacket pkt; - av_fifo_generic_read(s->fifo, &pkt, sizeof(pkt), NULL); - av_packet_unref(&pkt); - } - av_fifo_reset(s->fifo); - av_packet_unref(&s->buffered_pkt); ff_mediacodec_dec_flush(avctx, s->ctx); diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c index a9147f3a08..b44abaef7f 100644 --- a/libavcodec/mediacodecdec_common.c +++ b/libavcodec/mediacodecdec_common.c @@ -555,23 +555,17 @@ fail: return ret; } -int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, - AVFrame *frame, int *got_frame, - AVPacket *pkt) +int ff_mediacodec_dec_send(AVCodecContext *avctx, MediaCodecDecContext *s, + AVPacket *pkt) { - int ret; int offset = 0; int need_draining = 0; uint8_t *data; ssize_t index; size_t size; FFAMediaCodec *codec = s->codec; - FFAMediaCodecBufferInfo info = { 0 }; - int status; - int64_t input_dequeue_timeout_us = INPUT_DEQUEUE_TIMEOUT_US; - int64_t output_dequeue_timeout_us = OUTPUT_DEQUEUE_TIMEOUT_US; if (s->flushing) { av_log(avctx, AV_LOG_ERROR, "Decoder is flushing and cannot accept new buffer " @@ -584,13 +578,14 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, } if (s->draining && s->eos) { - return 0; + return AVERROR_EOF; } while (offset < pkt->size || (need_draining && !s->draining)) { index = ff_AMediaCodec_dequeueInputBuffer(codec, input_dequeue_timeout_us); if (ff_AMediaCodec_infoTryAgainLater(codec, index)) { + av_log(avctx, AV_LOG_TRACE, "Failed to dequeue input buffer, try again later..\n"); break; } @@ -621,13 +616,15 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, return AVERROR_EXTERNAL; } + av_log(avctx, AV_LOG_TRACE, "Queued input buffer %zd" + " size=%zd ts=%" PRIi64 "\n", index, size, pts); + s->draining = 1; break; } else { int64_t pts = pkt->pts; size = FFMIN(pkt->size - offset, size); - memcpy(data, pkt->data + offset, size); offset += size; @@ -643,11 +640,32 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, } } - if (need_draining || s->draining) { + if (offset == 0) + return AVERROR(EAGAIN); + return offset; +} + +int ff_mediacodec_dec_receive(AVCodecContext *avctx, MediaCodecDecContext *s, + AVFrame *frame, bool wait) +{ + int ret; + uint8_t *data; + ssize_t index; + size_t size; + FFAMediaCodec *codec = s->codec; + FFAMediaCodecBufferInfo info = { 0 }; + int status; + int64_t output_dequeue_timeout_us = OUTPUT_DEQUEUE_TIMEOUT_US; + + if (s->draining && s->eos) { + return AVERROR_EOF; + } + + if (s->draining) { /* If the codec is flushing or need to be flushed, block for a fair * amount of time to ensure we got a frame */ output_dequeue_timeout_us = OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US; - } else if (s->output_buffer_count == 0) { + } else if (s->output_buffer_count == 0 || !wait) { /* If the codec hasn't produced any frames, do not block so we * can push data to it as fast as possible, and get the first * frame */ @@ -656,9 +674,7 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, index = ff_AMediaCodec_dequeueOutputBuffer(codec, &info, output_dequeue_timeout_us); if (index >= 0) { - int ret; - - av_log(avctx, AV_LOG_DEBUG, "Got output buffer %zd" + av_log(avctx, AV_LOG_TRACE, "Got output buffer %zd" " offset=%" PRIi32 " size=%" PRIi32 " ts=%" PRIi64 " flags=%" PRIu32 "\n", index, info.offset, info.size, info.presentationTimeUs, info.flags); @@ -686,8 +702,8 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, } } - *got_frame = 1; s->output_buffer_count++; + return 0; } else { status = ff_AMediaCodec_releaseOutputBuffer(codec, index, 0); if (status < 0) { @@ -737,7 +753,7 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, MediaCodecDecContext *s, return AVERROR_EXTERNAL; } - return offset; + return AVERROR(EAGAIN); } int ff_mediacodec_dec_flush(AVCodecContext *avctx, MediaCodecDecContext *s) diff --git a/libavcodec/mediacodecdec_common.h b/libavcodec/mediacodecdec_common.h index 10f38277b5..32d16d3e3a 100644 --- a/libavcodec/mediacodecdec_common.h +++ b/libavcodec/mediacodecdec_common.h @@ -25,6 +25,7 @@ #include #include +#include #include #include "libavutil/frame.h" @@ -69,11 +70,14 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, const char *mime, FFAMediaFormat *format); -int ff_mediacodec_dec_decode(AVCodecContext *avctx, - MediaCodecDecContext *s, - AVFrame *frame, - int *got_frame, - AVPacket *pkt); +int ff_mediacodec_dec_send(AVCodecContext *avctx, + MediaCodecDecContext *s, + AVPacket *pkt); + +int ff_mediacodec_dec_receive(AVCodecContext *avctx, + MediaCodecDecContext *s, + AVFrame *frame, + bool wait); int ff_mediacodec_dec_flush(AVCodecContext *avctx, MediaCodecDecContext *s);