From patchwork Thu Sep 23 18:37:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Shun Ho X-Patchwork-Id: 30517 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp736130iob; Thu, 23 Sep 2021 11:38:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwu8dGQrgwXzHnFN07XkYhRke3HuQZiXXSAeIeNwgAXJvUMTPLfM7w06/yyX52Za78glz5J X-Received: by 2002:a50:bf02:: with SMTP id f2mr267340edk.98.1632422287574; Thu, 23 Sep 2021 11:38:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1632422287; cv=none; d=google.com; s=arc-20160816; b=Oe/NgVMZgFFmynNJumWZSUUMcqogKDjNLIVkmvQV8LJjEa/axwQcEABPPe50wtAosP 2OvqgotKAxnGBS97x4u3F3WE5IoUQ7oEJZu0K5IGXj60RPyR0XHEtGc7E0a0/l/W/aUH 8gVai5weM6uk5hokSlfykv4uVQS2IIswErE7iYIpRmH6F5vui7UBMfVQvzQnQ18iuaaR 8350YfmQ1w/5BMwSrBI8I32a3QMKs5Eum8wy1lj96cnxWZZrRZQm2b6qmFV7Rqz/jR8q svlMmA2fl5y5anaPUpQwO53GjdjPWLS4qFbtJkuFtcwu5CESwL5SKkmFTeZdvZy54c4Y Li5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=NWWfnIcusWJkbD/XDA0itz7a+o/T8ryslLJZDeHF5zU=; b=c/BeXWanBbDzZicbBogWbE9UPQNqYfmAdl3YBu0Hcuzd1zA+0FQiEd2HAYGNFZfis3 GwylY0SWh8/jOvfKCPn1daLWNJmZzXH92WLqLWzd1weKORHp1Lx5RoQ+dTwd+a36lXhZ wGSb0jkoc2SPYuAX+epDBoi9ksW4VvvPNmLsKt48+jf4rEs1uEMIZ6XleqR/Ul45vJOU D4tZter4z643Tz+2S3B1IF7fvb+RBlpW3/iAh+SeS6CrKl7r79I4Z+IyHozEV7r1GeKy s2ouuhM90BgHmAV+zsnl2Yx5n+2jK+27f7EPhlpc5Fz87hZGREaJacprpK5n4FoM2Mqf DDxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=lY+sSbJG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 5si6696794ejj.698.2021.09.23.11.38.06; Thu, 23 Sep 2021 11:38:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=lY+sSbJG; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BC8046898F5; Thu, 23 Sep 2021 21:38:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5BE3F688113 for ; Thu, 23 Sep 2021 21:37:57 +0300 (EEST) Received: by mail-pl1-f171.google.com with SMTP id n2so4561062plk.12 for ; Thu, 23 Sep 2021 11:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ncs5v99kUvbSRhJelnI1jUI+FBJJ5HoSARev+j3Dx/g=; b=lY+sSbJG1Qoo/Nh+uZfSNj6CYSxm2oui5WAdD5bVHuNX8BGlpcS4r+QDLZG3JL4Nd9 Yh2Nch6Op33D+kXOtADugE2MdvFKmSElDSzl6Bix+ikQRp4ckmWy7ne/Db/8nsmL+Oyh 2J93OL4I5yxD2r3RNVNxvBeIqE2BK5eZfBw6h3OBXcbWBronj3ayuKHJ+EVeriAyBy7W 3Tqmukb9asKqn3/6dxFYT3UsPxO926ox1NVDY1oRwhqJE7qUpmJTzBTcE/L3B28WHx1T VtnHhVrKPA5YVHGpPf42SiGyL2csJ8TKmmFTqHAhX/rErDgz1rSViZ3yRm0tM67AUFUp p36Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ncs5v99kUvbSRhJelnI1jUI+FBJJ5HoSARev+j3Dx/g=; b=q1yoao0sTxcQ4k8vm3yYwVL2K9S5uWVEnTHSWpy3ixbzkfk/FFXUimFK+1+GWH9SR4 +e/DAffu5vTFsfvshQiRxCAjpdi5f5MnX58/O2DhRvgRWotDHOilFw0tpa/UBezqkiKi wmfE1ko4gLrS4AuCvPSmkaNlAeYe9LYX28YDTHvHJWk0szeE0uDaSYI2O1LE1IMLQpbv 6W7BxHemhxV6l6f9WFcpbY30GRWhhUOP8JOz2G67HMIRah8uCU91muBbBf6GWmMgRP6c sAMem5zzV1fGYX2tK8JF9VshSoV51L26mQXdqrOzoAug5uCwOanE6oKL/tjJNDd4TfLr 0RKQ== X-Gm-Message-State: AOAM531MfsGQkAbc7LhVsj8uYC0ANZVHH24pQw0iqcERQixw7ltSPlhb NeZ/wX/g28AOb6Y1XCvSHyWFqZpq+Z1crw== X-Received: by 2002:a17:903:4112:b0:13a:7afa:f9c4 with SMTP id r18-20020a170903411200b0013a7afaf9c4mr5167599pld.66.1632422275240; Thu, 23 Sep 2021 11:37:55 -0700 (PDT) Received: from localhost.localdomain ([103.252.202.177]) by smtp.gmail.com with ESMTPSA id h10sm9286502pjs.51.2021.09.23.11.37.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 11:37:54 -0700 (PDT) From: Ho Ming Shun To: ffmpeg-devel@ffmpeg.org Date: Fri, 24 Sep 2021 02:37:35 +0800 Message-Id: <20210923183735.27400-1-cyph1984@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: References: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/mmaldec: use decoupled dataflow X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Ho Ming Shun Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: b2A6/7Ec96Cz MMAL is an fundamentally an asynchronous decoder, which was a bad fit for the legacy dataflow API. Often multiple packets are enqueued before a flood of frames are returned from MMAL. The previous lockstep dataflow meant that any delay in returning packets from the VPU would cause ctx->queue_decoded_frames to grow with no way of draining the queue. Testing this with mpv streaming from an RTSP source reduced decode latency from 2s to about 0.2s. Signed-off-by: Ho Ming Shun --- libavcodec/mmaldec.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/libavcodec/mmaldec.c b/libavcodec/mmaldec.c index 96140bf53d..3d7cc90cd2 100644 --- a/libavcodec/mmaldec.c +++ b/libavcodec/mmaldec.c @@ -570,6 +570,7 @@ static int ffmmal_add_packet(AVCodecContext *avctx, AVPacket *avpkt, done: av_buffer_unref(&buf); + av_packet_unref(avpkt); return ret; } @@ -655,6 +656,12 @@ static int ffmal_copy_frame(AVCodecContext *avctx, AVFrame *frame, avctx->pix_fmt, avctx->width, avctx->height); } + frame->sample_aspect_ratio = avctx->sample_aspect_ratio; + frame->width = avctx->width; + frame->width = avctx->width; + frame->height = avctx->height; + frame->format = avctx->pix_fmt; + frame->pts = buffer->pts == MMAL_TIME_UNKNOWN ? AV_NOPTS_VALUE : buffer->pts; frame->pkt_dts = AV_NOPTS_VALUE; @@ -763,12 +770,12 @@ done: return ret; } -static int ffmmal_decode(AVCodecContext *avctx, void *data, int *got_frame, - AVPacket *avpkt) +static int ffmmal_receive_frame(AVCodecContext *avctx, AVFrame *frame) { MMALDecodeContext *ctx = avctx->priv_data; - AVFrame *frame = data; int ret = 0; + AVPacket avpkt; + int got_frame = 0; if (avctx->extradata_size && !ctx->extradata_sent) { AVPacket pkt = {0}; @@ -782,7 +789,11 @@ FF_ENABLE_DEPRECATION_WARNINGS return ret; } - if ((ret = ffmmal_add_packet(avctx, avpkt, 0)) < 0) + ret = ff_decode_get_packet(avctx, &avpkt); + if(ret == 0) { + if ((ret = ffmmal_add_packet(avctx, &avpkt, 0)) < 0) + return ret; + } else if(ret < 0 && !(ret == AVERROR(EAGAIN))) return ret; if ((ret = ffmmal_fill_input_port(avctx)) < 0) @@ -791,7 +802,7 @@ FF_ENABLE_DEPRECATION_WARNINGS if ((ret = ffmmal_fill_output_port(avctx)) < 0) return ret; - if ((ret = ffmmal_read_frame(avctx, frame, got_frame)) < 0) + if ((ret = ffmmal_read_frame(avctx, frame, &got_frame)) < 0) return ret; // ffmmal_read_frame() can block for a while. Since the decoder is @@ -803,7 +814,12 @@ FF_ENABLE_DEPRECATION_WARNINGS if ((ret = ffmmal_fill_input_port(avctx)) < 0) return ret; - return ret; + if(!got_frame && ret == 0) + return AVERROR(EAGAIN); + else + return ret; + + } static const AVCodecHWConfigInternal *const mmal_hw_configs[] = { @@ -835,7 +851,7 @@ static const AVOption options[]={ .priv_data_size = sizeof(MMALDecodeContext), \ .init = ffmmal_init_decoder, \ .close = ffmmal_close_decoder, \ - .decode = ffmmal_decode, \ + .receive_frame = ffmmal_receive_frame, \ .flush = ffmmal_flush, \ .priv_class = &ffmmal_##NAME##_dec_class, \ .capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE, \