From patchwork Thu Sep 12 09:23:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Moritz Barsnick X-Patchwork-Id: 15040 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id EB5DA44A18B for ; Thu, 12 Sep 2019 12:23:27 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D76DC688115; Thu, 12 Sep 2019 12:23:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 94E6D688115 for ; Thu, 12 Sep 2019 12:23:19 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1568280199; bh=y1KqNCo9Zy4OiyOIa4KAinoX50lx9C3P46biU/PXmZc=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:In-Reply-To:References; b=iN4IzFzeiBGz63BKvN7jadXvyGvF37khcNv1KxoH96+KMqLLniOH8tchljWxWtQAu E33yW4f6XtKGE9yGQ+AQ+0vac718itWesxfE38pkZzA+v12BJzYJ9/5X2e8syXwmG5 wXRtStp9YdqQd0vqWBnNeJw7toIUP+6WIYKvyOGA= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from paradise.fritz.box ([89.182.225.50]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0MZkv0-1hp9FA3YBq-00LTqz; Thu, 12 Sep 2019 11:23:19 +0200 From: Moritz Barsnick To: FFmpeg development discussions and patches Date: Thu, 12 Sep 2019 11:23:06 +0200 Message-Id: <8e539006873d44a2670b5d55bae06d79cc0eb999.1568279966.git.barsnick@gmx.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Provags-ID: V03:K1:In/ft96BnK1W7uEgdrgja6NbKmTaxw4msDClfnFqwfsQg5Y7ApL qN0R6WQr2dU7I+bNRKoUDHM/e6Wdx35Sv/9vdnGZpBav4r2CXlHtv9WXjS3971WYKeH+BDi cPbOaLj6iPdhu81vsxpG9xX8iWJWMncOgkGUwUFUwE1t8F0Npyizr63x/qSfRoOhp1V+bQX 3IIDw+AvWXkYGO080NxJw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1; V03:K0:JkhNXsx1MQQ=:T2qSRdArVxZjgVvEuMH2fz cUZKUJ+rvEc54tTH2n2qXjgQ53Vye8O6OSVDP+w+nPpOQXiliBvUWjMNd4GMdp/nioWpeZsZ7 bfq6AW4ax9+hMbXJycO8mcX5crVyrvv0tdS5qJv60UtrFPKoXbPYuMmpObWDE8ufjFCPVuzVQ EF2kBrYoxx+1YQbHVta3/WgL3K+wlmH1J6+wdSosIuuB94a5IaQavFfxupV9Ns8vK7CLdONMB cX87ZPyW/0X9QdPrw2yLEsxNtlqT8JLGS5YaUaizl6JZ9wGy8inlU2a+DsX4Q4/LOkOIutPie X0OO1WTrxlieIPaFlzX5va4rG4Fa61zRNa2II3kZHrDFFhvp7xkrpKdNnLnS7hdP0FAwsEuzn n9n2K8KMTIfieAKSqwdhWvDQqIsAbKeUfC53NVj2Z+wAPvI5DXS5M+LsKO26uObByzOxBCHfD 4xR8ARNcV583IF40CVKpM3HiMJcc9RbHD8IvBaeKjvmuswiLULOYSZhEjQTjYpTHfLsr9H4un aIIU/zdOE2chL+XJRAXZVppyHGtH6yytK1Yh07Il/T9YTfhvuO+hoNvqxAauR2T6Knvowu6s8 560sLBVK5zMLZjUlnPrpvSHJrk1iVnEXxX2C+BrW0EfW22ippFYX+xzEpRVWBIcNkBFzF3wwv HB7kUgtTIbjAhX9pU+rm40lozvCZQJ3AQOrWUzZcIJDVnwY5rxMLCOHJUGdUsp1dVSJ3zLJQe mbkzabp15DFn5FJ5rTbL2MtPlRWq2iyK/XkWu2UTZ1iqSQ+MKd42MwINx6VFE3U79lBDlbgKo aGo0lU1wI1rxd8u9rTSh7UwPaCsU4xbbjC8pKN83cQM3Xbz7g6yFFsawlzrXPaD/w1OssUHh9 JiLxojMXmtPhBMhaJFVYks2CwZuIawdn0P+Jjgd47tD6UfHIHFYV4350V9fbwGgZ3dc8S92Da gOMNFFe0t7QuOf+kjOtpPMY01HL5bm3mY0XvlBY1RzfLd3SSrSBjudWGRtGlZPJPh6kb/ATah BwMpHeDHG1eC+JO8BIfcXKkf9t7n0dVf/0UwfVujP8DINDtJ4gaiXRUrEa2aH/i8bxylKSi5Y 9M3gENCzjGjtb4eOSDFKEYZnf0UlOjxKH0zBBmJbp8QEfnv94V7llakfh6cxUnCiNsA9aZDar BaHi+Ktn3QGKK48k/wrpj8jqdqavBq4RkDQIkRNbugHOfc9XSvoJ0F40AIn6CtH70eykmoH3F nx2oorBUDct9K/3Dc2Mrwq0Y/eHGdSHQruwrfcX3G4aURHert0LtuEu1xK1k= Subject: [FFmpeg-devel] [PATCH v3 3/3] avformat/hashenc: add streamhash muxer X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Moritz Barsnick Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Implemented as a variant of the hash muxer, reusing most functions, and making use of the previously introduced array of hashes. Signed-off-by: Moritz Barsnick --- Changelog | 1 + doc/muxers.texi | 47 ++++++++++++++++++ libavformat/Makefile | 1 + libavformat/allformats.c | 1 + libavformat/hashenc.c | 100 +++++++++++++++++++++++++++++++++++---- libavformat/version.h | 4 +- 6 files changed, 142 insertions(+), 12 deletions(-) -- 2.20.1 diff --git a/Changelog b/Changelog index 4b29e015a0..8eeed65680 100644 --- a/Changelog +++ b/Changelog @@ -9,6 +9,7 @@ version : - Supoort AMD AMF encoder on Linux (via Vulkan) - IMM5 video decoder - ZeroMQ protocol +- streamhash muxer version 4.2: diff --git a/doc/muxers.texi b/doc/muxers.texi index 20d31b279c..eb989f888b 100644 --- a/doc/muxers.texi +++ b/doc/muxers.texi @@ -2064,6 +2064,53 @@ Specify whether to remove all fragments when finished. Default 0 (do not remove) @end table +@anchor{streamhash} +@section streamhash + +Per stream hash testing format. + +This muxer computes and prints a cryptographic hash of all the input frames, +on a per-stream basis. This can be used for equality checks without having +to do a complete binary comparison. + +By default audio frames are converted to signed 16-bit raw audio and +video frames to raw video before computing the hash, but the output +of explicit conversions to other codecs can also be used. Timestamps +are ignored. It uses the SHA-256 cryptographic hash function by default, +but supports several other algorithms. + +The output of the muxer consists of one line per stream of the form: +@var{streamindex},@var{streamtype},@var{algo}=@var{hash}, where +@var{streamindex} is the index of the mapped stream, @var{streamtype} is a +single characer indicating the type of stream, @var{algo} is a short string +representing the hash function used, and @var{hash} is a hexadecimal number +representing the computed hash. + +@table @option +@item hash @var{algorithm} +Use the cryptographic hash function specified by the string @var{algorithm}. +Supported values include @code{MD5}, @code{murmur3}, @code{RIPEMD128}, +@code{RIPEMD160}, @code{RIPEMD256}, @code{RIPEMD320}, @code{SHA160}, +@code{SHA224}, @code{SHA256} (default), @code{SHA512/224}, @code{SHA512/256}, +@code{SHA384}, @code{SHA512}, @code{CRC32} and @code{adler32}. + +@end table + +@subsection Examples + +To compute the SHA-256 hash of the input converted to raw audio and +video, and store it in the file @file{out.sha256}: +@example +ffmpeg -i INPUT -f streamhash out.sha256 +@end example + +To print an MD5 hash to stdout use the command: +@example +ffmpeg -i INPUT -f streamhash -hash md5 - +@end example + +See also the @ref{hash} and @ref{framehash} muxers. + @anchor{fifo} @section fifo diff --git a/libavformat/Makefile b/libavformat/Makefile index efa3a112ae..a61d42b721 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -494,6 +494,7 @@ OBJS-$(CONFIG_SRT_DEMUXER) += srtdec.o subtitles.o OBJS-$(CONFIG_SRT_MUXER) += srtenc.o OBJS-$(CONFIG_STL_DEMUXER) += stldec.o subtitles.o OBJS-$(CONFIG_STR_DEMUXER) += psxstr.o +OBJS-$(CONFIG_STREAMHASH_MUXER) += hashenc.o OBJS-$(CONFIG_STREAM_SEGMENT_MUXER) += segment.o OBJS-$(CONFIG_SUBVIEWER1_DEMUXER) += subviewer1dec.o subtitles.o OBJS-$(CONFIG_SUBVIEWER_DEMUXER) += subviewerdec.o subtitles.o diff --git a/libavformat/allformats.c b/libavformat/allformats.c index cd00834807..f7fea32b45 100644 --- a/libavformat/allformats.c +++ b/libavformat/allformats.c @@ -393,6 +393,7 @@ extern AVInputFormat ff_srt_demuxer; extern AVOutputFormat ff_srt_muxer; extern AVInputFormat ff_str_demuxer; extern AVInputFormat ff_stl_demuxer; +extern AVOutputFormat ff_streamhash_muxer; extern AVInputFormat ff_subviewer1_demuxer; extern AVInputFormat ff_subviewer_demuxer; extern AVInputFormat ff_sup_demuxer; diff --git a/libavformat/hashenc.c b/libavformat/hashenc.c index a9be6d0c81..8e090731ae 100644 --- a/libavformat/hashenc.c +++ b/libavformat/hashenc.c @@ -31,6 +31,7 @@ struct HashContext { const AVClass *avclass; struct AVHashContext **hashes; char *hash_name; + int per_stream; int format_version; }; @@ -56,6 +57,13 @@ static const AVOption framehash_options[] = { }; #endif +#if CONFIG_STREAMHASH_MUXER +static const AVOption streamhash_options[] = { + HASH_OPT("sha256"), + { NULL }, +}; +#endif + #if CONFIG_MD5_MUXER static const AVOption md5_options[] = { HASH_OPT("md5"), @@ -76,6 +84,7 @@ static int hash_init(struct AVFormatContext *s) { int res; struct HashContext *c = s->priv_data; + c->per_stream = 0; c->hashes = av_mallocz_array(1, sizeof(c->hashes)); if (!c->hashes) return AVERROR(ENOMEM); @@ -85,24 +94,66 @@ static int hash_init(struct AVFormatContext *s) av_hash_init(c->hashes[0]); return 0; } +#endif + +#if CONFIG_STREAMHASH_MUXER +static int streamhash_init(struct AVFormatContext *s) +{ + int res, i; + struct HashContext *c = s->priv_data; + c->per_stream = 1; + c->hashes = av_mallocz_array(s->nb_streams, sizeof(c->hashes)); + if (!c->hashes) + return AVERROR(ENOMEM); + for (i = 0; i < s->nb_streams; i++) { + res = av_hash_alloc(&c->hashes[i], c->hash_name); + if (res < 0) { + return res; + } + av_hash_init(c->hashes[i]); + } + return 0; +} +#endif + +#if CONFIG_HASH_MUXER || CONFIG_MD5_MUXER || CONFIG_STREAMHASH_MUXER +static char get_media_type_char(enum AVMediaType type) +{ + switch (type) { + case AVMEDIA_TYPE_VIDEO: return 'v'; + case AVMEDIA_TYPE_AUDIO: return 'a'; + case AVMEDIA_TYPE_DATA: return 'd'; + case AVMEDIA_TYPE_SUBTITLE: return 's'; + case AVMEDIA_TYPE_ATTACHMENT: return 't'; + default: return '?'; + } +} static int hash_write_packet(struct AVFormatContext *s, AVPacket *pkt) { struct HashContext *c = s->priv_data; - av_hash_update(c->hashes[0], pkt->data, pkt->size); + av_hash_update(c->hashes[c->per_stream ? pkt->stream_index : 0], pkt->data, pkt->size); return 0; } static int hash_write_trailer(struct AVFormatContext *s) { struct HashContext *c = s->priv_data; - char buf[AV_HASH_MAX_SIZE*2+128]; - snprintf(buf, sizeof(buf) - 200, "%s=", av_hash_get_name(c->hashes[0])); - - av_hash_final_hex(c->hashes[0], buf + strlen(buf), sizeof(buf) - strlen(buf)); - av_strlcatf(buf, sizeof(buf), "\n"); - avio_write(s->pb, buf, strlen(buf)); - avio_flush(s->pb); + int num_hashes = c->per_stream ? s->nb_streams : 1; + for (int i = 0; i < num_hashes; i++) { + char buf[AV_HASH_MAX_SIZE*2+128]; + if (c->per_stream) { + AVStream *st = s->streams[i]; + snprintf(buf, sizeof(buf) - 200, "%d,%c,%s=", i, get_media_type_char(st->codecpar->codec_type), + av_hash_get_name(c->hashes[i])); + } else { + snprintf(buf, sizeof(buf) - 200, "%s=", av_hash_get_name(c->hashes[i])); + } + av_hash_final_hex(c->hashes[i], buf + strlen(buf), sizeof(buf) - strlen(buf)); + av_strlcatf(buf, sizeof(buf), "\n"); + avio_write(s->pb, buf, strlen(buf)); + avio_flush(s->pb); + } return 0; } @@ -110,8 +161,12 @@ static int hash_write_trailer(struct AVFormatContext *s) static void hash_free(struct AVFormatContext *s) { struct HashContext *c = s->priv_data; - if (c->hashes) - av_hash_freep(&c->hashes[0]); + if (c->hashes) { + int num_hashes = c->per_stream ? s->nb_streams : 1; + for (int i = 0; i < num_hashes; i++) { + av_hash_freep(&c->hashes[i]); + } + } av_freep(&c->hashes); } #endif @@ -164,6 +219,30 @@ AVOutputFormat ff_md5_muxer = { }; #endif +#if CONFIG_STREAMHASH_MUXER +static const AVClass streamhashenc_class = { + .class_name = "stream hash muxer", + .item_name = av_default_item_name, + .option = streamhash_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +AVOutputFormat ff_streamhash_muxer = { + .name = "streamhash", + .long_name = NULL_IF_CONFIG_SMALL("Per-stream hash testing"), + .priv_data_size = sizeof(struct HashContext), + .audio_codec = AV_CODEC_ID_PCM_S16LE, + .video_codec = AV_CODEC_ID_RAWVIDEO, + .init = streamhash_init, + .write_packet = hash_write_packet, + .write_trailer = hash_write_trailer, + .deinit = hash_free, + .flags = AVFMT_VARIABLE_FPS | AVFMT_TS_NONSTRICT | + AVFMT_TS_NEGATIVE, + .priv_class = &streamhashenc_class, +}; +#endif + #if CONFIG_FRAMEHASH_MUXER || CONFIG_FRAMEMD5_MUXER static void framehash_print_extradata(struct AVFormatContext *s) { @@ -190,6 +269,7 @@ static int framehash_init(struct AVFormatContext *s) { int res; struct HashContext *c = s->priv_data; + c->per_stream = 0; c->hashes = av_mallocz_array(1, sizeof(c->hashes)); if (!c->hashes) return AVERROR(ENOMEM); diff --git a/libavformat/version.h b/libavformat/version.h index edfa73fb97..bcd0408d28 100644 --- a/libavformat/version.h +++ b/libavformat/version.h @@ -32,8 +32,8 @@ // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium) // Also please add any ticket numbers that you believe might be affected here #define LIBAVFORMAT_VERSION_MAJOR 58 -#define LIBAVFORMAT_VERSION_MINOR 32 -#define LIBAVFORMAT_VERSION_MICRO 104 +#define LIBAVFORMAT_VERSION_MINOR 33 +#define LIBAVFORMAT_VERSION_MICRO 100 #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \ LIBAVFORMAT_VERSION_MINOR, \