From patchwork Thu Jun 30 15:22:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timo Rothenpieler X-Patchwork-Id: 36554 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp391906pzh; Thu, 30 Jun 2022 08:22:49 -0700 (PDT) X-Google-Smtp-Source: AGRyM1v4MMQtLmM72Cuszt+LsMbdbgeMJa4lT+ycZpLV6VVk8uL2/REn/fdOnvzHVCM6fHr6Gkl8 X-Received: by 2002:a17:907:1612:b0:722:e1b9:45d0 with SMTP id hb18-20020a170907161200b00722e1b945d0mr9524101ejc.439.1656602568939; Thu, 30 Jun 2022 08:22:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656602568; cv=none; d=google.com; s=arc-20160816; b=t2MORZ1BM6fYla2OiryAqLtJWJP05d+nvZRpX1xOXAC5LNCfYsRuD5BiAai8oDeX9F 2LxIIo91gsjAYAhMjifetkoxXtE/E5eVg4mzyXyeUvqYE9YMRUyBue9na93va4QhF0pi Eq+VP1O90odQat7o/QPjQy3wCOdFOBws4vgYEeBiGAe07epRkT8AaJUjhFT6mTnr0k9J NbO33ob9+5AJV6BknvT6o2iPVKx6MLObTdAa9MC0khY6how4jhDQPi6ttS5B8V8ensK0 Nc9bc7Zw7gKAzGl5JLL+ojDR5T6lCse6eaW61cvJKcySh1pzZlp0s3pB+iHbA7J8vaso 7w8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ww/WJjMk1puR40Q4Bx0D1Geo9zVgD61V5SkcO1KgLIM=; b=hp/w1hMs4TqkTIridSVyV4Hc3pe/8hNgmLgikVnbaD1F3SlMJTVWZc9c7LPp7odwd5 3ZsiMph2Qkc+qEbx8Ji1/OM6IMY3droI1llv55gw+Lxvd5I0EjwwdVyEDDl6WL4LXnL8 1uGRGvYUWO0swguGEmg+26AhWQvAKl+N0l/LYUaOtFH5owFR2UC4ozQ+ICJ5NsslYyXl 5QCU+227592CBVtoL5GFczINvl9tXCsyoc+k3IIlGxCNTuuvF6Q/g9hqwKe2NT0Clu6k A7XmhTD0SHH4/KZMFD2T/m97hp8n3tbU/6wnS7Ubova4QnFKRt1bE/Olq06pPT1SQG/Z mSuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=Tab0ysHS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id n25-20020aa7c459000000b00435802a2e93si3788126edr.504.2022.06.30.08.22.38; Thu, 30 Jun 2022 08:22:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rothenpieler.org header.s=mail header.b=Tab0ysHS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0D9B468B657; Thu, 30 Jun 2022 18:22:34 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from btbn.de (btbn.de [136.243.74.85]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3EFFA68B4F5 for ; Thu, 30 Jun 2022 18:22:27 +0300 (EEST) Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id D0E192FA492; Thu, 30 Jun 2022 17:22:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1656602546; bh=OAQi5VLTzRWb01TZslwZNJNlW8Tt9OL10B0TKX3nJSY=; h=From:To:Cc:Subject:Date; b=Tab0ysHS0Yo/XbvNscygX4NNEeKi8UDy2bzNcUYwNNlSIa9J/ya6Fi0xfpQkqdKCn QK616fy+OF+KHdrS8g01w/lc4fOS4tQmWyTtH2oKXQg5j6/U4q0uBrPl5zzvEtauIs qMEbx19Yf20/8tCa4VchvJrry0IJFl+jYcUMPxz/4474EvRi5fCtkE7PJIFcbMPgrB OJ5GtstrWF14zXYxRoVFJYp+fvkjxYNa1bhUKMjhCCYcAq6EwXSLHeRAUU5Bc8O0ok TblTgbxTnalapg2dp2tokUMrCYDgjdnzAIKadu2Ma9tS6x4GN9if02tcNV6E98lMRM q/lfG0n6PCwrw== From: Timo Rothenpieler To: ffmpeg-devel@ffmpeg.org Date: Thu, 30 Jun 2022 17:22:06 +0200 Message-Id: <20220630152206.15366-1-timo@rothenpieler.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avdevice/lavfi: output wrapped AVFrames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Timo Rothenpieler Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: sUgBS7HrBnqr This avoids an extra copy of potentially quite big video frames. Instead of copying the entire frames data into a rawvideo packet it packs the frame into a wrapped avframe packet and passes it through as-is. Unfortunately, wrapped avframes are set up to be video frames, so the audio frames unfortunately continue to be copied. Additionally, this enabled passing through video frames that previously were impossible to process, like hardware frames or other special formats that couldn't be packed into a rawvideo packet. --- libavdevice/lavfi.c | 87 +++++++++-------------- tests/ref/fate/filter-metadata-cropdetect | 3 +- 2 files changed, 36 insertions(+), 54 deletions(-) diff --git a/libavdevice/lavfi.c b/libavdevice/lavfi.c index db5d0b94de..61076f9950 100644 --- a/libavdevice/lavfi.c +++ b/libavdevice/lavfi.c @@ -54,32 +54,10 @@ typedef struct { int *sink_eof; int *stream_sink_map; int *sink_stream_subcc_map; - AVFrame *decoded_frame; int nb_sinks; AVPacket subcc_packet; } LavfiContext; -static int *create_all_formats(int n) -{ - int i, j, *fmts, count = 0; - - for (i = 0; i < n; i++) { - const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(i); - if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL)) - count++; - } - - if (!(fmts = av_malloc_array(count + 1, sizeof(*fmts)))) - return NULL; - for (j = 0, i = 0; i < n; i++) { - const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(i); - if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL)) - fmts[j++] = i; - } - fmts[j] = AV_PIX_FMT_NONE; - return fmts; -} - av_cold static int lavfi_read_close(AVFormatContext *avctx) { LavfiContext *lavfi = avctx->priv_data; @@ -90,7 +68,6 @@ av_cold static int lavfi_read_close(AVFormatContext *avctx) av_freep(&lavfi->sink_stream_subcc_map); av_freep(&lavfi->sinks); avfilter_graph_free(&lavfi->graph); - av_frame_free(&lavfi->decoded_frame); return 0; } @@ -125,15 +102,11 @@ av_cold static int lavfi_read_header(AVFormatContext *avctx) LavfiContext *lavfi = avctx->priv_data; AVFilterInOut *input_links = NULL, *output_links = NULL, *inout; const AVFilter *buffersink, *abuffersink; - int *pix_fmts = create_all_formats(AV_PIX_FMT_NB); enum AVMediaType type; int ret = 0, i, n; #define FAIL(ERR) { ret = ERR; goto end; } - if (!pix_fmts) - FAIL(AVERROR(ENOMEM)); - buffersink = avfilter_get_by_name("buffersink"); abuffersink = avfilter_get_by_name("abuffersink"); @@ -264,8 +237,6 @@ av_cold static int lavfi_read_header(AVFormatContext *avctx) ret = avfilter_graph_create_filter(&sink, buffersink, inout->name, NULL, NULL, lavfi->graph); - if (ret >= 0) - ret = av_opt_set_int_list(sink, "pix_fmts", pix_fmts, AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN); if (ret < 0) goto end; } else if (type == AVMEDIA_TYPE_AUDIO) { @@ -321,15 +292,11 @@ av_cold static int lavfi_read_header(AVFormatContext *avctx) avpriv_set_pts_info(st, 64, time_base.num, time_base.den); par->codec_type = av_buffersink_get_type(sink); if (par->codec_type == AVMEDIA_TYPE_VIDEO) { - int64_t probesize; - par->codec_id = AV_CODEC_ID_RAWVIDEO; + par->codec_id = AV_CODEC_ID_WRAPPED_AVFRAME; par->format = av_buffersink_get_format(sink); par->width = av_buffersink_get_w(sink); par->height = av_buffersink_get_h(sink); - probesize = par->width * par->height * 30 * - av_get_padded_bits_per_pixel(av_pix_fmt_desc_get(par->format)); - avctx->probesize = FFMAX(avctx->probesize, probesize); - st ->sample_aspect_ratio = + st ->sample_aspect_ratio = par->sample_aspect_ratio = av_buffersink_get_sample_aspect_ratio(sink); } else if (par->codec_type == AVMEDIA_TYPE_AUDIO) { par->sample_rate = av_buffersink_get_sample_rate(sink); @@ -348,11 +315,7 @@ av_cold static int lavfi_read_header(AVFormatContext *avctx) if ((ret = create_subcc_streams(avctx)) < 0) goto end; - if (!(lavfi->decoded_frame = av_frame_alloc())) - FAIL(AVERROR(ENOMEM)); - end: - av_free(pix_fmts); avfilter_inout_free(&input_links); avfilter_inout_free(&output_links); return ret; @@ -378,15 +341,20 @@ static int create_subcc_packet(AVFormatContext *avctx, AVFrame *frame, return 0; } +static void lavfi_free_frame(void *opaque, uint8_t *data) +{ + AVFrame *frame = (AVFrame*)data; + av_frame_free(&frame); +} + static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) { LavfiContext *lavfi = avctx->priv_data; double min_pts = DBL_MAX; int stream_idx, min_pts_sink_idx = 0; - AVFrame *frame = lavfi->decoded_frame; + AVFrame *frame; AVDictionary *frame_metadata; int ret, i; - int size = 0; AVStream *st; if (lavfi->subcc_packet.size) { @@ -394,6 +362,10 @@ static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) return pkt->size; } + frame = av_frame_alloc(); + if (!frame) + return AVERROR(ENOMEM); + /* iterate through all the graph sinks. Select the sink with the * minimum PTS */ for (i = 0; i < lavfi->nb_sinks; i++) { @@ -411,7 +383,7 @@ static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) lavfi->sink_eof[i] = 1; continue; } else if (ret < 0) - return ret; + goto fail; d = av_rescale_q_rnd(frame->pts, tb, AV_TIME_BASE_Q, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX); ff_dlog(avctx, "sink_idx:%d time:%f\n", i, d); av_frame_unref(frame); @@ -421,8 +393,10 @@ static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) min_pts_sink_idx = i; } } - if (min_pts == DBL_MAX) - return AVERROR_EOF; + if (min_pts == DBL_MAX) { + ret = AVERROR_EOF; + goto fail; + } ff_dlog(avctx, "min_pts_sink_idx:%i\n", min_pts_sink_idx); @@ -431,15 +405,19 @@ static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) st = avctx->streams[stream_idx]; if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) { - size = av_image_get_buffer_size(frame->format, frame->width, frame->height, 1); - if ((ret = av_new_packet(pkt, size)) < 0) + pkt->buf = av_buffer_create((uint8_t*)frame, sizeof(*frame), + &lavfi_free_frame, NULL, 0); + if (!pkt->buf) { + ret = AVERROR(ENOMEM); goto fail; + } - av_image_copy_to_buffer(pkt->data, size, (const uint8_t **)frame->data, frame->linesize, - frame->format, frame->width, frame->height, 1); + pkt->data = pkt->buf->data; + pkt->size = pkt->buf->size; + pkt->flags |= AV_PKT_FLAG_TRUSTED; } else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) { - size = frame->nb_samples * av_get_bytes_per_sample(frame->format) * - frame->ch_layout.nb_channels; + int size = frame->nb_samples * av_get_bytes_per_sample(frame->format) * + frame->ch_layout.nb_channels; if ((ret = av_new_packet(pkt, size)) < 0) goto fail; memcpy(pkt->data, frame->data[0], size); @@ -468,10 +446,13 @@ static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt) pkt->stream_index = stream_idx; pkt->pts = frame->pts; pkt->pos = frame->pkt_pos; - av_frame_unref(frame); - return size; + + if (st->codecpar->codec_type != AVMEDIA_TYPE_VIDEO) + av_frame_free(&frame); + + return pkt->size; fail: - av_frame_unref(frame); + av_frame_free(&frame); return ret; } diff --git a/tests/ref/fate/filter-metadata-cropdetect b/tests/ref/fate/filter-metadata-cropdetect index e7c03ead4e..104acdd77f 100644 --- a/tests/ref/fate/filter-metadata-cropdetect +++ b/tests/ref/fate/filter-metadata-cropdetect @@ -1,4 +1,5 @@ -pts=0 +pts=0| + pts=400 pts=800|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=719|tag:lavfi.cropdetect.y1=61|tag:lavfi.cropdetect.y2=424|tag:lavfi.cropdetect.w=720|tag:lavfi.cropdetect.h=352|tag:lavfi.cropdetect.x=0|tag:lavfi.cropdetect.y=68 pts=1200|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=719|tag:lavfi.cropdetect.y1=61|tag:lavfi.cropdetect.y2=424|tag:lavfi.cropdetect.w=720|tag:lavfi.cropdetect.h=352|tag:lavfi.cropdetect.x=0|tag:lavfi.cropdetect.y=68