From patchwork Thu Dec 9 13:08:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 32243 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp343101iog; Thu, 9 Dec 2021 05:08:28 -0800 (PST) X-Google-Smtp-Source: ABdhPJybsqshi9cK5C7GihA5iPVTnR9JMxlL8SjCmGDl1QLCsaegHm5MgfRsvrF6IOC1kr12bO7R X-Received: by 2002:a05:6402:27cd:: with SMTP id c13mr28590263ede.57.1639055308511; Thu, 09 Dec 2021 05:08:28 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id do13si11023804ejc.780.2021.12.09.05.08.28; Thu, 09 Dec 2021 05:08:28 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=sNs+jUXy; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC6D468AFCE; Thu, 9 Dec 2021 15:08:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05olkn2068.outbound.protection.outlook.com [40.92.89.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 39CF36897E2 for ; Thu, 9 Dec 2021 15:08:18 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cIPI0WHiRdgJY8T04CYkNRQpgz41TYO6ogr6GlgZrDUdkcZXBiqQ8DfXUq7P7XPh0Nao3kUekbuFwciKSusdQjl65JaS7gI/SHk0PtcipGZ3eykvUu18Qz3wcPuHP0mgyPaOLac5VXNL9YqdWhUbQs8rMv4bcuPhMcYB1JC+izwwWNx5vR6JCpq31ZxzsAeTcYvHYaId54B9N8R4LPlW+jooj39J4oSXqf/ULVugNlkfPeEKX4Nna1C3yMo9spgYjsqaXqSHmvrQDRoqZP+4QasCq3Y7Fz6i5i/5MLSnVIDL7UFgsb527HUS+f2QYWJr6hUlnHaozdZoQ1vBOYDR9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bfsh0aGT+S16A2RfO8Kmy3KFyI6BVmtcRkoBeAuUVEI=; b=oMDt3F/EEtl38hHB+Xy8ApJdGpdlCf+3c08WgA65PBb9koMl1evHJpHrBDEWs0CIY5vTsulEJnNx/AMwf9NWl8ADezWTL2PllljoSBAiwoaycE4MQDT76BNq1xEvkbdmINAf0xSqpM4YRFyCLp4Ce7mHPAdS7F8qNRm55H2h8QCJOvcU5QTVkkf+r11Evkg2DKznMgSeAxmgm8To9iHhLzALJsyiuRfy7OcSqH7JgogwzeHLntuJvicfUHSSXm0L4oYuTLZEcR6KKmvRhpx7/gkfwQFX/BCN9r+n0BiAOK/sYvE2vLV1Dc4Yfvk4XBCwcxRv1HCkyDJg1c2mevC20w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bfsh0aGT+S16A2RfO8Kmy3KFyI6BVmtcRkoBeAuUVEI=; b=sNs+jUXyGtkBkgFx5ny+HAC/kwrPD7BzmWFpgHm70VKJXZiCJKaN/aM+WT5xGnURWr7Oztgi6WNSQdqbBWfZRroSVIqwIStfHCnxPW010rHT1XbryD8NaKshtX9YOkPtq2U9LhSjZdNvzQl0sk7fSCvpeKcf0HxCB34x5VaD9/rHLz5+EW/DoXlIl4AybhitQr0FhsgrqLAiggDc7x37ivqmGL+MB2wbSqmLUiiAM6upLYo5TmyXzQBVYrDTCVtDMUulX1VnfksAnLCZ4mS3c+oN9jzxMrTBDCZWrswORRC7P5ifkT8rrwcL8B59wMsJwhdoCR2oI3hVOjbLJhntlQ== Received: from AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) by AM6PR0302MB3400.eurprd03.prod.outlook.com (2603:10a6:209:1d::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4755.21; Thu, 9 Dec 2021 13:08:16 +0000 Received: from AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::f0dc:92f5:6bc2:45ca]) by AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::f0dc:92f5:6bc2:45ca%4]) with mapi id 15.20.4755.025; Thu, 9 Dec 2021 13:08:16 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Thu, 9 Dec 2021 14:08:01 +0100 Message-ID: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: X-TMN: [NkMLvkph2Cbvy1jDaL11DVGXF8cVSuoL] X-ClientProxiedBy: AM5PR1001CA0039.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:206:15::16) To AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) X-Microsoft-Original-Message-ID: <20211209130805.678224-1-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sblaptop.fritz.box (188.192.248.74) by AM5PR1001CA0039.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:206:15::16) with Microsoft SMTP Server (version=TLS1_2, cipher=) via Frontend Transport; Thu, 9 Dec 2021 13:08:16 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4d94b3a1-2aa2-43b4-bc57-08d9bb14f6b1 X-MS-Exchange-SLBlob-MailProps: mBy7Mai7yE5TX1J4zfzcjryh+3QMv6knbaPSeRZOoQcu+A3r647Brd3d3AvMnoSybIekQxEoyUUHBF162/3AXmigV1mR5kjWcuGAaQ9ryVwrWrrD7h/d1okCiMK/afn/3as6K2CMdrQy9yGArUSsY3GQXEM7bJgYB/QbaPrZyl9Kt4zn3yyTCMWMlowkSEIZLDIECNr48e7BIuSCjp4JK181aUMMqByLLZHEKjWvYTctni4CzFtnwbzxFUuEoaVd0l8bRlxpw5SEakd0pQvW3CAY0NWRlBm2t52rgYRZOxv+BktCrZp25+lbgq2OQgbNeMgpz6sXcNS+2kqn0JUFnHl581EgO12+fTDIshoR/BYUz3DI1A3XjvMr0deXKBZpHq+I6+2W7Sgtf+FFdFDactPTsKjegbQ97tmSTDVm2UGqQdDUwFJj/9OfX7w/2I4hJxCOwyaR8lc2TPXetR9tdpr5LamJTx2gTXswmcbIe1h2OLrSP1+v+vkbrmLhMwohbtwYqyjT56GKpshzT3fKyi8q9X+AjpIYIWQcI3BRHfVc70XxftGTgyMKR9jWfBClXuRKz0GFynk+RCTT/vLwsJbNC+hkUGugI5SURr/eUju8T1vuVxpOODHRfaYLfr7mgUljqGOyaCXXuFhgUeoZJACJSeNi9OyudLBsv/9k//XwuyRIHaF2zwDRAacyy/z1udkVkcyjeAoG0LSrsM3tfA6gGr5Wg9sF X-MS-TrafficTypeDiagnostic: AM6PR0302MB3400:EE_ X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JTAQpMT0n8IR8GpftUzlUPUCtyfObT5arMs6AU06fNpzh8XJQOzYwquRtX4jSuKLV/9z1RUJO37WQReHlGDOHdr59GXevLRGJw0ZO6folrjckp8BdIK60dRt1gGk17iLehUO9mYlc6OMOxCtoSvebcQZmwbuQ5tmX//CZaU1sk+OCsnSzX5LM/+poD9yky8zODdmkpItZzgb6offMc1BK3OQzwyHsbGBvIIchNEnXNJPITgWIJmwmGgrSUPQvAIUrOzUMqAgdo1fdO/mjj4n2WWTymZwDozxHeeslDSNuxoDcb1MtnXsUB3D9kN6An32i1oDTfz8U+sKcRQFwP3kAiwtYXoOY4g5WepF3kk8oPz/v8wGWjy7QWLtuIyjOQTLQFUGIhzRyTXZ3XEC52D6IFVcXaB51CZ+ogZFdX52U/wxbVbRUOmOGl7yYTHtasd10FALqfGBzwGc4DSUvgyCRcuAnF23aSUf2qlzE7LQ9D6Emisahwx1Bq28dVxZTUzMY1a+gJU8BL7Mure8j3bQTKJIIKQ3XmE+bqd3ztR1uSbzAIUfCXdnoDTxPoftGeG0OhlwmFUzV0gEYJEf39qPFQ== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HR4s9/J0CbDqmTu8dhyhQ5U0eJIPDh9mn6BqSd0I635xjL21Yzl3hV5VSy1U8yeERBbR+8Hk1XJwzlUqtrr/g5lAB1TF+EpcEjbPFQiRs166Esb7+3G0GDsmXXDONSzSPb/d0aLYs6QBWGsu5mRTFJi3J4PaIkx4TLuIqnBsrwdBWda7wd609n84MGgPmx7F/zYYWlGD5GiD8qJ57/BSCMBtDhdiBXvikr6bnbqBJHjkQoZtSlO2Lou6Hol+AbsQA11KPD3JXCHbXDY8GE77NDaHRz8U0xJL7MDz6CCfkkAOTOtYKMjDxaNYUtjkME43sySNRw0/p8sAojsgnFXofuL6HVUVWP7GPYTzB8HrWfyVdXpS//Z8jPtvPtQHL3XE48JfcPeO81rHvqTf9R7ggK6hJEFlpvKxA5HTyc542gjozoe3HNH0fakGo8x0yOpi/Z9cPNwGbp0k6jTpV9zXAXhpAsH6uuCecinwcCjR35randFl96SteSzP5smZo0TfIwv+kYA71Qheib3zolSNaviWYzOmTNviJgwGFhnZBbjUpGvd9qRJYuFChZLEc6AYvP5l7gJXDDf56vnTeRRqnnyjlbLwEyGX+4/uCREeAXHCzUhKlXKBw0lZzRm+2ndLdIhfa1WLT6BikoaKsYRU4TbabWr2kZRWD68owWsrE8t9DqbXMBkop08uh/iFu3CvpbMllUZWEn7wjP/MVypXwQEPS5QiZcNIv01Kv5z+gHFP1UzYciCylkUldsWTAD566CawK10bezZaBeLmkVwEK6C8WQL6xqS+AxxB8hh+xsm66oLxUp67CbOZAyO2oVIZepKOw8xzBltoMh1Ngr1avi9Fl/YOLOgkFULsd4BhkWOklOWKY5MtHalBgEIx/+G/rME9aDv7QMNJdFFwtKcPuK15cVyUEKgD3XTdPTp3MuBhBcqHgafyh1maKgMtoEYjh8p8dqfxETjdwr9dRIJyJg== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d94b3a1-2aa2-43b4-bc57-08d9bb14f6b1 X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6660.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Dec 2021 13:08:16.4874 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0302MB3400 Subject: [FFmpeg-devel] [PATCH v3 1/5] avcodec/mmaldec: use decoupled dataflow X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt , Ho Ming Shun Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8kNazVSN/M/m From: Ho Ming Shun MMAL is an fundamentally an asynchronous decoder, which was a bad fit for the legacy dataflow API. Often multiple packets are enqueued before a flood of frames are returned from MMAL. The previous lockstep dataflow meant that any delay in returning packets from the VPU would cause ctx->queue_decoded_frames to grow with no way of draining the queue. Testing this with mpv streaming from a live RTSP source visibly reduced latency introduced by frames waiting in queue_decoded_frames from roughly 2s to 0. Signed-off-by: Ho Ming Shun Signed-off-by: Andreas Rheinhardt --- libavcodec/mmaldec.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/libavcodec/mmaldec.c b/libavcodec/mmaldec.c index 8c7d749742..d336f10350 100644 --- a/libavcodec/mmaldec.c +++ b/libavcodec/mmaldec.c @@ -83,6 +83,8 @@ typedef struct MMALDecodeContext { // libavcodec API can't return new frames, and we have a logical deadlock. // This is avoided by queuing such buffers here. FFBufferEntry *waiting_buffers, *waiting_buffers_tail; + /* Packet used to hold received packets temporarily; not owned by us. */ + AVPacket *pkt; int64_t packets_sent; atomic_int packets_buffered; @@ -355,6 +357,8 @@ static av_cold int ffmmal_init_decoder(AVCodecContext *avctx) MMAL_COMPONENT_T *decoder; int ret = 0; + ctx->pkt = avctx->internal->in_pkt; + bcm_host_init(); if (mmal_vc_init()) { @@ -570,6 +574,7 @@ static int ffmmal_add_packet(AVCodecContext *avctx, AVPacket *avpkt, done: av_buffer_unref(&buf); + av_packet_unref(avpkt); return ret; } @@ -655,6 +660,12 @@ static int ffmal_copy_frame(AVCodecContext *avctx, AVFrame *frame, avctx->pix_fmt, avctx->width, avctx->height); } + frame->sample_aspect_ratio = avctx->sample_aspect_ratio; + frame->width = avctx->width; + frame->width = avctx->width; + frame->height = avctx->height; + frame->format = avctx->pix_fmt; + frame->pts = buffer->pts == MMAL_TIME_UNKNOWN ? AV_NOPTS_VALUE : buffer->pts; frame->pkt_dts = AV_NOPTS_VALUE; @@ -763,12 +774,12 @@ done: return ret; } -static int ffmmal_decode(AVCodecContext *avctx, void *data, int *got_frame, - AVPacket *avpkt) +static int ffmmal_receive_frame(AVCodecContext *avctx, AVFrame *frame) { MMALDecodeContext *ctx = avctx->priv_data; - AVFrame *frame = data; + AVPacket *const avpkt = ctx->pkt; int ret = 0; + int got_frame = 0; if (avctx->extradata_size && !ctx->extradata_sent) { AVPacket pkt = {0}; @@ -780,7 +791,11 @@ static int ffmmal_decode(AVCodecContext *avctx, void *data, int *got_frame, return ret; } - if ((ret = ffmmal_add_packet(avctx, avpkt, 0)) < 0) + ret = ff_decode_get_packet(avctx, avpkt); + if(ret == 0) { + if ((ret = ffmmal_add_packet(avctx, avpkt, 0)) < 0) + return ret; + } else if(ret < 0 && !(ret == AVERROR(EAGAIN))) return ret; if ((ret = ffmmal_fill_input_port(avctx)) < 0) @@ -789,7 +804,7 @@ static int ffmmal_decode(AVCodecContext *avctx, void *data, int *got_frame, if ((ret = ffmmal_fill_output_port(avctx)) < 0) return ret; - if ((ret = ffmmal_read_frame(avctx, frame, got_frame)) < 0) + if ((ret = ffmmal_read_frame(avctx, frame, &got_frame)) < 0) return ret; // ffmmal_read_frame() can block for a while. Since the decoder is @@ -801,7 +816,10 @@ static int ffmmal_decode(AVCodecContext *avctx, void *data, int *got_frame, if ((ret = ffmmal_fill_input_port(avctx)) < 0) return ret; - return ret; + if(!got_frame && ret == 0) + return AVERROR(EAGAIN); + else + return ret; } static const AVCodecHWConfigInternal *const mmal_hw_configs[] = { @@ -833,7 +851,7 @@ static const AVOption options[]={ .priv_data_size = sizeof(MMALDecodeContext), \ .init = ffmmal_init_decoder, \ .close = ffmmal_close_decoder, \ - .decode = ffmmal_decode, \ + .receive_frame = ffmmal_receive_frame, \ .flush = ffmmal_flush, \ .priv_class = &ffmmal_##NAME##_dec_class, \ .capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE, \