From patchwork Tue Mar 6 18:49:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 7824 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.181.170 with SMTP id m39csp4093292jaj; Tue, 6 Mar 2018 10:49:39 -0800 (PST) X-Google-Smtp-Source: AG47ELuFk6yODO52z/QZsOEdTs9mZCJaTbM2Ea2wSms/TPu4xL8+WNlArEOVy6zXkbTOModyAGKq X-Received: by 10.28.210.135 with SMTP id j129mr13113880wmg.41.1520362179512; Tue, 06 Mar 2018 10:49:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520362179; cv=none; d=google.com; s=arc-20160816; b=Ll7jeVt/l+94zoITGu2avEXR4lWy2sdDMt9pu/HQiKp0iX6yXk4NgDCL08HaF9Le+1 9A8W4PHZJTmsCI+mEoPR4hmXKkVwqHi5IJ/gfbMsKtZ5wTX+KpUOLb20yuK/EQ124Kkq 48ehDeO/QgYVCR4uvcXdRNxgW6m5k80W7hEHiXog2qO5DlkGFk/noPE30NMnKWqLcpl2 cE6RWnDbNozfiQmAo5uIXneq2qmsT1xNXdMpwoKCNvFKdbFA1ILTIlCsIAcdDtVbWtmM 5S1w1qOK51Cz+rah4SeSfU2HbQKXcjf+T3VV/jT6sIRoq8sXSQqld0nFLsv5dPUzQ+y3 Dnjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:dkim-signature :delivered-to:arc-authentication-results; bh=LTiTvMjp4Jqew6OJUZb9wxqcAyvHUhjx/gNFG9qhSoY=; b=h9qDcs2QfhadWBquiFglEShPxNZ/qnd+n0RCYDFthtr2Kdm1AmhawFwdO4aDrCS2Ip tE3XWef/0frP4MWLMgJlzKuL+VJFXK4S9RQFUsSu/J3WYD/WD9kIoyPdxZYwY1tFyPTQ w8S+gYdZkTsN1IHhdPg2+eh+cio26zyDM75VGKIVvuiVBTdMd7Uoxv85Gmp1jpgOXoPT +JCmaw4pu5y3BkqNhIB+vr6yva9OCefS8LZLQU3B8oOUDZLYFaprgpz1lUaimEVQmHoV t5TcrLCeIoHRn/fAbUVK7cluwSw/aoDVKBg4JQr6LOBQNJQXeOD/3twsQcE4WNhpNRM4 siyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=WmtGa0iv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m71si3237774wmd.156.2018.03.06.10.49.38; Tue, 06 Mar 2018 10:49:39 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=WmtGa0iv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 57F8268A293; Tue, 6 Mar 2018 20:49:29 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr0-f172.google.com (mail-wr0-f172.google.com [209.85.128.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8456468A213 for ; Tue, 6 Mar 2018 20:49:22 +0200 (EET) Received: by mail-wr0-f172.google.com with SMTP id v18so21366102wrv.0 for ; Tue, 06 Mar 2018 10:49:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=w/idSYW2Ognm3CPIsmX4e6FTUAfUwbk6rk2VnM7YBJ0=; b=WmtGa0ivnWUv4ZhhPwiESiUT4XR4J9QyCnh1PVdLYf9l43PnaKXCZmJ6zq6VkbfP4Z YxbB+3NDFBEg1Ely9nj5ZjB+wdSHdRuAH+HXwdAiEZFlfsr3Nf88JFIqYOvp0gF4ldbo 1mEO66gPlYfTGb9lonsj3RDWC25bdl+n7tnvKLgajvSH/Fvd6KbMeU21P+n2eRUSj9HG q9iSnofxRBn3WawqHKgd7rNOKM7YuqjX+o9LpaiFPn6cUWKbEPOR0wanzgBJQwSudeHd xbUzCMxl40hnPcxyn8R1oTIAQtNsuTSXRFWSRPt8iNL34qbcxbF6qifY61hkBkgTvohq i1Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=w/idSYW2Ognm3CPIsmX4e6FTUAfUwbk6rk2VnM7YBJ0=; b=VCoVNjdtSYiv5p6qGWsFxmVY6Xc4MuF7IOMhl1Wh6IgPZwvAdfTLyMtHykrEF7dkW9 qI68YV86bpsx7yQcB2WoQY1oa/i9VcEiJCfS9VgGlQsKhxcv4yUWwAGg7MGEPmZKOrPR VpmlOdsLN9yXMgcrSMla9xsXZvn+6l5WumqxVDa897jT4tER2b/syZtk1h2hOxIvMk2P oNknjkpX2mXKQ5cP94dqtdgasaZEyd+dHrrTW4MxhueEUqyOr4iBcqzIi7CSyxsDI7bo qwXvGIOyZtMdsRLB/K0uorNmF15rcie77VMxBfRc9j0XWKavCFvF7jH9ne/qylXZReWn BdrQ== X-Gm-Message-State: APf1xPBXbVOdVqctZvc6Rnl/gzYMRWzFlTr0Mc0nn/9pEwpEruLO8mds 2NyKcZzc+cSZGg/yG3/L3K+0/f4V X-Received: by 10.223.150.194 with SMTP id u60mr16328129wrb.47.1520362169225; Tue, 06 Mar 2018 10:49:29 -0800 (PST) Received: from [10.0.0.193] ([213.210.30.6]) by smtp.gmail.com with ESMTPSA id p68sm2968888wmg.7.2018.03.06.10.49.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Mar 2018 10:49:28 -0800 (PST) To: ffmpeg-devel@ffmpeg.org References: From: Mark Thompson Message-ID: <15ed2c24-8b3d-02c8-4294-79ff741f7a32@jkqxz.net> Date: Tue, 6 Mar 2018 18:49:27 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Subject: [FFmpeg-devel] [PATCH] lavc: Add filter_units bitstream filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This can remove units with types in or not in a given set from a stream. For example, it can be used to remove all non-VCL NAL units from an H.264 or H.265 stream. --- On 06/03/18 17:27, Hendrik Leppkes wrote: > On Tue, Mar 6, 2018 at 3:51 PM, Eran Kornblau wrote: >> Hi all, >> >> The attached patch adds a parameter that enables the user to choose which AVC/HEVC NAL units to include in the output. >> The parameter is supplied as a bitmask in order to keep things simple. >> >> A short background on why we need it - in our transcoding process, we partition the video in chunks, the chunks are >> transcoded in parallel and packaged in MPEG-TS container. The transcoded TS chunks are then concatenated and >> packaged in MP4. These MP4 files are later repackaged on-the-fly to various protocols (HLS/DASH etc.) using our >> JIT packager. >> For performance reasons (can get into more detail if anyone's interested...), when packaging the MP4 to DASH/CENC, >> we configure the packager to assume that each AVC frame contains exactly one NAL unit. >> The problem is that the transition through MPEG-TS adds additional NAL units (NAL AUD before each frame + SPS/PPS >> before each key frame), and this assumption fails. >> Using the attached patch we can pass '-nal_types_mask 0x3e' which will make ffmpeg output only VCL NALs in the stream. >> > > Having such logic in one single muxer is not something we really like > around here. Next time someone needs something similar for another > codec, you're stuck re-implementing it. > > To achieve the same effect, Mark Thompson quickly wipped up a > Bitstream Filter using his CBS framework which achieves the same > result. He'll be sending that patch to the mailing list in a while. The suggested use-case would be '-bsf:v filter_units=pass_types=1-5' for H.264 or '-bsf:v filter_units=pass_types=0-31' for H.265. (Also note that filters already exist for some individual parts of this: h264_metadata can remove AUDs, extract_extradata can remove parameter sets.) - Mark libavcodec/Makefile | 1 + libavcodec/bitstream_filters.c | 1 + libavcodec/filter_units_bsf.c | 250 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 252 insertions(+) create mode 100644 libavcodec/filter_units_bsf.c diff --git a/libavcodec/Makefile b/libavcodec/Makefile index b496f0d..b99bdce 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1037,6 +1037,7 @@ OBJS-$(CONFIG_DUMP_EXTRADATA_BSF) += dump_extradata_bsf.o OBJS-$(CONFIG_DCA_CORE_BSF) += dca_core_bsf.o OBJS-$(CONFIG_EXTRACT_EXTRADATA_BSF) += extract_extradata_bsf.o \ h2645_parse.o +OBJS-$(CONFIG_FILTER_UNITS_BSF) += filter_units_bsf.o OBJS-$(CONFIG_H264_METADATA_BSF) += h264_metadata_bsf.o OBJS-$(CONFIG_H264_MP4TOANNEXB_BSF) += h264_mp4toannexb_bsf.o OBJS-$(CONFIG_H264_REDUNDANT_PPS_BSF) += h264_redundant_pps_bsf.o diff --git a/libavcodec/bitstream_filters.c b/libavcodec/bitstream_filters.c index 338ef82..e1dc198 100644 --- a/libavcodec/bitstream_filters.c +++ b/libavcodec/bitstream_filters.c @@ -29,6 +29,7 @@ extern const AVBitStreamFilter ff_chomp_bsf; extern const AVBitStreamFilter ff_dump_extradata_bsf; extern const AVBitStreamFilter ff_dca_core_bsf; extern const AVBitStreamFilter ff_extract_extradata_bsf; +extern const AVBitStreamFilter ff_filter_units_bsf; extern const AVBitStreamFilter ff_h264_metadata_bsf; extern const AVBitStreamFilter ff_h264_mp4toannexb_bsf; extern const AVBitStreamFilter ff_h264_redundant_pps_bsf; diff --git a/libavcodec/filter_units_bsf.c b/libavcodec/filter_units_bsf.c new file mode 100644 index 0000000..3126f17 --- /dev/null +++ b/libavcodec/filter_units_bsf.c @@ -0,0 +1,250 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include + +#include "libavutil/common.h" +#include "libavutil/opt.h" + +#include "bsf.h" +#include "cbs.h" + + +typedef struct FilterUnitsContext { + const AVClass *class; + + CodedBitstreamContext *cbc; + + const char *pass_types; + const char *remove_types; + + int remove; + CodedBitstreamUnitType *type_list; + int nb_types; +} FilterUnitsContext; + + +static int filter_units_make_type_list(const char *list_string, + CodedBitstreamUnitType **type_list, + int *nb_types) +{ + CodedBitstreamUnitType *list = NULL; + int pass, count; + + for (pass = 1; pass <= 2; pass++) { + long value, range_start, range_end; + const char *str; + char *value_end; + + count = 0; + for (str = list_string; *str;) { + value = strtol(str, &value_end, 0); + if (str == value_end) + goto invalid; + str = (const char *)value_end; + if (*str == '-') { + ++str; + range_start = value; + range_end = strtol(str, &value_end, 0); + if (str == value_end) + goto invalid; + + for (value = range_start; value < range_end; value++) { + if (pass == 2) + list[count] = value; + ++count; + } + } else { + if (pass == 2) + list[count] = value; + ++count; + } + if (*str == '|') + ++str; + } + if (pass == 1) { + list = av_malloc_array(count, sizeof(*list)); + if (!list) + return AVERROR(ENOMEM); + } + } + + *type_list = list; + *nb_types = count; + return 0; + +invalid: + av_freep(&list); + return AVERROR(EINVAL); +} + +static int filter_units_filter(AVBSFContext *bsf, AVPacket *out) +{ + FilterUnitsContext *ctx = bsf->priv_data; + AVPacket *in = NULL; + CodedBitstreamFragment au; + int err, i, j; + + while (1) { + err = ff_bsf_get_packet(bsf, &in); + if (err < 0) + return err; + + err = ff_cbs_read_packet(ctx->cbc, &au, in); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to read packet.\n"); + goto fail; + } + + for (i = 0; i < au.nb_units; i++) { + for (j = 0; j < ctx->nb_types; j++) { + if (au.units[i].type == ctx->type_list[j]) + break; + } + if (ctx->remove ? j < ctx->nb_types + : j >= ctx->nb_types) { + ff_cbs_delete_unit(ctx->cbc, &au, i); + --i; + } + } + + if (au.nb_units > 0) + break; + + // Don't return packets with nothing in them. + av_packet_free(&in); + ff_cbs_fragment_uninit(ctx->cbc, &au); + } + + err = ff_cbs_write_packet(ctx->cbc, out, &au); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write packet.\n"); + goto fail; + } + + err = av_packet_copy_props(out, in); + if (err < 0) + goto fail; + + err = 0; +fail: + ff_cbs_fragment_uninit(ctx->cbc, &au); + + av_packet_free(&in); + + return err; +} + +static int filter_units_init(AVBSFContext *bsf) +{ + FilterUnitsContext *ctx = bsf->priv_data; + int err; + + if (!(!ctx->pass_types ^ !ctx->remove_types)) { + av_log(bsf, AV_LOG_ERROR, "Exactly one of pass_types or " + "remove_types is required.\n"); + return AVERROR(EINVAL); + } + + if (ctx->pass_types) { + ctx->remove = 0; + err = filter_units_make_type_list(ctx->pass_types, + &ctx->type_list, &ctx->nb_types); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to parse pass_types.\n"); + return AVERROR(EINVAL); + } + } else { + ctx->remove = 1; + err = filter_units_make_type_list(ctx->remove_types, + &ctx->type_list, &ctx->nb_types); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to parse remove_types.\n"); + return AVERROR(EINVAL); + } + } + + err = ff_cbs_init(&ctx->cbc, bsf->par_in->codec_id, bsf); + if (err < 0) + return err; + + // Don't actually decompose anything, we only want the unit data. + ctx->cbc->decompose_unit_types = ctx->type_list; + ctx->cbc->nb_decompose_unit_types = 0; + + if (bsf->par_in->extradata) { + CodedBitstreamFragment ps; + + err = ff_cbs_read_extradata(ctx->cbc, &ps, bsf->par_in); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to read extradata.\n"); + } else { + err = ff_cbs_write_extradata(ctx->cbc, bsf->par_out, &ps); + if (err < 0) + av_log(bsf, AV_LOG_ERROR, "Failed to write extradata.\n"); + } + + ff_cbs_fragment_uninit(ctx->cbc, &ps); + } else { + err = 0; + } + + return err; +} + +static void filter_units_close(AVBSFContext *bsf) +{ + FilterUnitsContext *ctx = bsf->priv_data; + + av_freep(&ctx->type_list); + + ff_cbs_close(&ctx->cbc); +} + +#define OFFSET(x) offsetof(FilterUnitsContext, x) +static const AVOption filter_units_options[] = { + { "pass_types", "List of unit types to pass through the filter.", + OFFSET(pass_types), AV_OPT_TYPE_STRING, { .str = NULL } }, + { "remove_types", "List of unit types to remove in the filter.", + OFFSET(remove_types), AV_OPT_TYPE_STRING, { .str = NULL } }, + + { NULL } +}; + +static const AVClass filter_units_class = { + .class_name = "filter_units_bsf", + .item_name = av_default_item_name, + .option = filter_units_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +static const enum AVCodecID filter_units_codec_ids[] = { + AV_CODEC_ID_H264, + AV_CODEC_ID_HEVC, + AV_CODEC_ID_NONE, +}; + +const AVBitStreamFilter ff_filter_units_bsf = { + .name = "filter_units", + .priv_data_size = sizeof(FilterUnitsContext), + .priv_class = &filter_units_class, + .init = &filter_units_init, + .close = &filter_units_close, + .filter = &filter_units_filter, + .codec_ids = filter_units_codec_ids, +};