From patchwork Sun Mar 11 18:30:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Thompson X-Patchwork-Id: 7911 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.1.70 with SMTP id c67csp826336jad; Sun, 11 Mar 2018 11:35:58 -0700 (PDT) X-Google-Smtp-Source: AG47ELtaOXabCjxHHlzetv1oKx6rZd/LHsfllUFpg1U7+JnPux+BgjIw5zSDsZ2paoVSahlulG+M X-Received: by 10.28.177.86 with SMTP id a83mr3552239wmf.143.1520793358717; Sun, 11 Mar 2018 11:35:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1520793358; cv=none; d=google.com; s=arc-20160816; b=hx8EGhUFIRxAxOvYRR8f1TLchghYcBoP0HFjRwRWXkcPLANfPqW28XVMFnz7zx6uwH sgZLLNOYZJPmS3tTFuSalDMeAF5zMdLmtUN6D8gSRB/KLsU5Jdi1+sKMSHLaHnC/FLge vO4e/Mdh0U8IQG/7PJ9UQ6jthaXaOCHjR+dy0wGj5Nce9glTe5t1Kr+b6xyWr+VqOrma vuJPQko+89AOreC8DjqfzwlqGtIEC39Pf7abPsqE9sM5ARQ3t9Bo9k55E3JEsmsd6CQV MCPUhFb7cAeraJs2vSboVsD5LqrjX86XmQWVwvh0iX4hWzLLiGSunzaAUBuy+tro6Rom +Agw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=Xv5Zmc2dwFyZE2dcypuEgL4oi1lZmbCTdPMeqon56+s=; b=xQ0DzO/HOrqdx2IEHBLL+mOQ8b91uVQxXPhJD+TozJcKvEUYrSPM3FFFpehY1qXeCR iF+NC3MWMyPlyLOsQtnzzqvSCoVfcKcfCRySHnZn6jqSfaJwvR5cUHSAw1J1sOm9EPbC U+/OU1qoXLYjPE55iR8l9QMda5tdwOhSNF9bLoPHUdDA7hjwihyoRR5Srg9u0dV+OlOv rS6UYS5eOp17w5f4eDKGVJyYjDvdI0Q+05QC/ndxQrPIuu6UD/DAsl34jz6aroPLLe/r nQrw5IlXat09jKPV5nwcvGYCkCB77IPt8/Tlshk2BUuzsTxaPfqsXOhblBfpl3iWXkkl JAzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=WEPguTZK; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r200si2190581wmb.149.2018.03.11.11.35.58; Sun, 11 Mar 2018 11:35:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@jkqxz-net.20150623.gappssmtp.com header.s=20150623 header.b=WEPguTZK; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E7D1D68A23F; Sun, 11 Mar 2018 20:35:46 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 75150689F51 for ; Sun, 11 Mar 2018 20:35:40 +0200 (EET) Received: by mail-wm0-f68.google.com with SMTP id t3so12354710wmc.2 for ; Sun, 11 Mar 2018 11:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jkqxz-net.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=gJMnODy42+XBvQNxNHl1nGOmofdZq0+7zKVY+CwlEuE=; b=WEPguTZKYr66dY7Fq8cb4/FpU2zRwji+eGtBssVh4XCzshjfxCUvvMvju/h7pm/NV2 YbMBdtG7POAZoq4IT8yICvNYZfW6t0uCwpKY5XMbJuxVEzDgGcpDIiHh4ZoGJQJWGE59 FTDB05oa2TEWV5WLBGc8T8vF0s2of95qh0hqi2kjA1NmNWhBc9gN9bqEfH+unDxd825E u5SMlLmYvH250PQjuYm/l0dPql5Y8JH5ybdzDJ89Z15YXZK9ijIYFwwFL05TCEfT7dC1 zEJnzGSGOr0L3E/nz+3ZdUZ2V2HgO+TByfBXFYCmWp5j71NiRKx7j0Msp38a+UyyDner 63XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=gJMnODy42+XBvQNxNHl1nGOmofdZq0+7zKVY+CwlEuE=; b=uePL/33UHmZH13d72TY5hVVMF1z/o9IEkcL67uaE+fDsrPUCfGLzX8DBz+ohfUVHKX yCV9pE9Y0T501TalPDn41JJW3P5a+/nj/8FapzOMtYIepSH6ztzt1nTHgFLnbUN6DW4m /p8/goAGfOTrUvIIX/Nzzl2TmKuKFVGzPkjI9zW3sav8lTzeBZfr1Fk9ZZ1G+Mi0Pd0M QPNYgJiqPQ4wVNPNYy3/s8ZlSC06bNSHZvNwL96tNH8hiJTprEc+4nsjHiu7WIah+Lt1 2as4Odgv/hl/2he7Zr8QUvHWEb3ar92EOMHR9Xot/8mdyAgqeSON9ieMuQ0S13IhHkRC Aw4w== X-Gm-Message-State: AElRT7FzGksCkPP9yvJsMSkMB/afhnZ5hKjb61ED7i5JkdZoqW0D91R4 Wy1p6JU/kcAnMZQPEKlE4bx/eoFw X-Received: by 10.28.125.150 with SMTP id y144mr3420729wmc.75.1520793027348; Sun, 11 Mar 2018 11:30:27 -0700 (PDT) Received: from rywe.jkqxz.net (cpc91242-cmbg18-2-0-cust650.5-4.cable.virginm.net. [82.8.130.139]) by smtp.gmail.com with ESMTPSA id 31sm5064514wrr.59.2018.03.11.11.30.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Mar 2018 11:30:26 -0700 (PDT) From: Mark Thompson To: ffmpeg-devel@ffmpeg.org Date: Sun, 11 Mar 2018 18:30:16 +0000 Message-Id: <20180311183021.25556-3-sw@jkqxz.net> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180311183021.25556-1-sw@jkqxz.net> References: <20180311183021.25556-1-sw@jkqxz.net> Subject: [FFmpeg-devel] [PATCH 3/8] h264_metadata: Add support for display orientation SEI messages X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" --- libavcodec/h264_metadata_bsf.c | 169 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 165 insertions(+), 4 deletions(-) diff --git a/libavcodec/h264_metadata_bsf.c b/libavcodec/h264_metadata_bsf.c index 466823cda6..36047887ca 100644 --- a/libavcodec/h264_metadata_bsf.c +++ b/libavcodec/h264_metadata_bsf.c @@ -17,6 +17,7 @@ */ #include "libavutil/avstring.h" +#include "libavutil/display.h" #include "libavutil/common.h" #include "libavutil/opt.h" @@ -30,6 +31,12 @@ enum { PASS, INSERT, REMOVE, + EXTRACT, +}; + +enum { + FLIP_HORIZONTAL = 1, + FLIP_VERTICAL = 2, }; typedef struct H264MetadataContext { @@ -38,6 +45,8 @@ typedef struct H264MetadataContext { CodedBitstreamContext *cbc; CodedBitstreamFragment access_unit; + int done_first_au; + H264RawAUD aud_nal; H264RawSEI sei_nal; @@ -62,9 +71,12 @@ typedef struct H264MetadataContext { int crop_bottom; const char *sei_user_data; - int sei_first_au; int delete_filler; + + int display_orientation; + double rotate; + int flip; } H264MetadataContext; @@ -211,6 +223,8 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *out) AVPacket *in = NULL; CodedBitstreamFragment *au = &ctx->access_unit; int err, i, j, has_sps; + uint8_t *displaymatrix_side_data = NULL; + size_t displaymatrix_side_data_size = 0; err = ff_bsf_get_packet(bsf, &in); if (err < 0) @@ -292,15 +306,13 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *out) // Only insert the SEI in access units containing SPSs, and also // unconditionally in the first access unit we ever see. - if (ctx->sei_user_data && (has_sps || !ctx->sei_first_au)) { + if (ctx->sei_user_data && (has_sps || !ctx->done_first_au)) { H264RawSEIPayload payload = { .payload_type = H264_SEI_TYPE_USER_DATA_UNREGISTERED, }; H264RawSEIUserDataUnregistered *udu = &payload.payload.user_data_unregistered; - ctx->sei_first_au = 1; - for (i = j = 0; j < 32 && ctx->sei_user_data[i]; i++) { int c, v; c = ctx->sei_user_data[i]; @@ -387,6 +399,121 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *out) } } + if (ctx->display_orientation != PASS) { + for (i = 0; i < au->nb_units; i++) { + H264RawSEI *sei; + if (au->units[i].type != H264_NAL_SEI) + continue; + sei = au->units[i].content; + + for (j = 0; j < sei->payload_count; j++) { + H264RawSEIDisplayOrientation *disp; + int32_t *matrix; + + if (sei->payload[j].payload_type != + H264_SEI_TYPE_DISPLAY_ORIENTATION) + continue; + disp = &sei->payload[j].payload.display_orientation; + + if (ctx->display_orientation == REMOVE || + ctx->display_orientation == INSERT) { + err = ff_cbs_h264_delete_sei_message(ctx->cbc, au, + &au->units[i], j); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to delete " + "display orientation SEI message.\n"); + goto fail; + } + --i; + break; + } + + matrix = av_mallocz(9 * sizeof(int32_t)); + if (!matrix) { + err = AVERROR(ENOMEM); + goto fail; + } + + av_display_rotation_set(matrix, + disp->anticlockwise_rotation * + 180.0 / 65536.0); + av_display_matrix_flip(matrix, disp->hor_flip, disp->ver_flip); + + displaymatrix_side_data = (uint8_t*)matrix; + displaymatrix_side_data_size = 9 * sizeof(int32_t); + } + } + } + if (ctx->display_orientation == INSERT) { + H264RawSEIPayload payload = { + .payload_type = H264_SEI_TYPE_DISPLAY_ORIENTATION, + }; + H264RawSEIDisplayOrientation *disp = + &payload.payload.display_orientation; + uint8_t *data; + int size; + int write = 0; + + data = av_packet_get_side_data(in, AV_PKT_DATA_DISPLAYMATRIX, &size); + if (data && size >= 9 * sizeof(int32_t)) { + int32_t matrix[9]; + int hflip, vflip; + double angle; + + memcpy(matrix, data, sizeof(matrix)); + + hflip = vflip = 0; + if (matrix[0] < 0 && matrix[4] > 0) + hflip = 1; + else if (matrix[0] > 0 && matrix[4] < 0) + vflip = 1; + av_display_matrix_flip(matrix, hflip, vflip); + + angle = av_display_rotation_get(matrix); + + if (!(angle >= -180.0 && angle <= 180.0 /* also excludes NaN */) || + matrix[2] != 0 || matrix[5] != 0 || + matrix[6] != 0 || matrix[7] != 0) { + av_log(bsf, AV_LOG_WARNING, "Input display matrix is not " + "representable in H.264 parameters.\n"); + } else { + disp->hor_flip = hflip; + disp->ver_flip = vflip; + disp->anticlockwise_rotation = + (uint16_t)rint((angle >= 0.0 ? angle + : angle + 360.0) * + 65536.0 / 360.0); + write = 1; + } + } + + if (has_sps || !ctx->done_first_au) { + if (!isnan(ctx->rotate)) { + disp->anticlockwise_rotation = + (uint16_t)rint((ctx->rotate >= 0.0 ? ctx->rotate + : ctx->rotate + 360.0) * + 65536.0 / 360.0); + write = 1; + } + if (ctx->flip) { + disp->hor_flip = !!(ctx->flip & FLIP_HORIZONTAL); + disp->ver_flip = !!(ctx->flip & FLIP_VERTICAL); + write = 1; + } + } + + if (write) { + disp->display_orientation_repetition_period = 1; + + err = ff_cbs_h264_add_sei_message(ctx->cbc, au, &payload); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to add display orientation " + "SEI message to access unit.\n"); + goto fail; + } + } + } + err = ff_cbs_write_packet(ctx->cbc, out, au); if (err < 0) { av_log(bsf, AV_LOG_ERROR, "Failed to write packet.\n"); @@ -397,9 +524,24 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *out) if (err < 0) goto fail; + if (displaymatrix_side_data) { + err = av_packet_add_side_data(out, AV_PKT_DATA_DISPLAYMATRIX, + displaymatrix_side_data, + displaymatrix_side_data_size); + if (err) { + av_log(bsf, AV_LOG_ERROR, "Failed to attach extracted " + "displaymatrix side data to packet.\n"); + goto fail; + } + displaymatrix_side_data = NULL; + } + + ctx->done_first_au = 1; + err = 0; fail: ff_cbs_fragment_uninit(ctx->cbc, au); + av_freep(&displaymatrix_side_data); av_packet_free(&in); @@ -509,6 +651,25 @@ static const AVOption h264_metadata_options[] = { { "delete_filler", "Delete all filler (both NAL and SEI)", OFFSET(delete_filler), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1 }, + { "display_orientation", "Display orientation SEI", + OFFSET(display_orientation), AV_OPT_TYPE_INT, + { .i64 = PASS }, PASS, EXTRACT, 0, "disp_or" }, + { "pass", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = PASS }, .unit = "disp_or" }, + { "insert", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = INSERT }, .unit = "disp_or" }, + { "remove", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = REMOVE }, .unit = "disp_or" }, + { "extract", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = EXTRACT }, .unit = "disp_or" }, + + { "rotate", "Set rotation in display orientation SEI (anticlockwise angle in degrees)", + OFFSET(rotate), AV_OPT_TYPE_DOUBLE, + { .dbl = NAN }, -360.0, +360.0 }, + { "flip", "Set flip in display orientation SEI", + OFFSET(flip), AV_OPT_TYPE_FLAGS, + { .i64 = 0 }, 0, FLIP_HORIZONTAL | FLIP_VERTICAL, .unit ="flip" }, + { "horizontal", "Set hor_flip", + 0, AV_OPT_TYPE_CONST, { .i64 = FLIP_HORIZONTAL }, .unit ="flip" }, + { "vertical", "Set ver_flip", + 0, AV_OPT_TYPE_CONST, { .i64 = FLIP_VERTICAL }, .unit ="flip" }, + { NULL } };