[FFmpeg-devel,v2,3/3] avformat/hashenc: add streamhash muxer

Implemented as a variant of the hash muxer, reusing most functions,
and making use of the previously introduced array of hashes.
---
 Changelog                |  1 +
 doc/muxers.texi          | 46 ++++++++++++++++++++++
 libavformat/Makefile     |  1 +
 libavformat/allformats.c |  1 +
 libavformat/hashenc.c    | 82 +++++++++++++++++++++++++++++++++++-----
 libavformat/version.h    |  4 +-
 6 files changed, 124 insertions(+), 11 deletions(-)

--
2.20.1

On 9/11/2019 10:34 AM, Moritz Barsnick wrote:
> Implemented as a variant of the hash muxer, reusing most functions,
> and making use of the previously introduced array of hashes.
> ---
>  Changelog                |  1 +
>  doc/muxers.texi          | 46 ++++++++++++++++++++++
>  libavformat/Makefile     |  1 +
>  libavformat/allformats.c |  1 +
>  libavformat/hashenc.c    | 82 +++++++++++++++++++++++++++++++++++-----
>  libavformat/version.h    |  4 +-
>  6 files changed, 124 insertions(+), 11 deletions(-)
> 
> diff --git a/Changelog b/Changelog
> index 4b29e015a0..8eeed65680 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -9,6 +9,7 @@ version <next>:
>  - Supoort AMD AMF encoder on Linux (via Vulkan)
>  - IMM5 video decoder
>  - ZeroMQ protocol
> +- streamhash muxer
> 
> 
>  version 4.2:
> diff --git a/doc/muxers.texi b/doc/muxers.texi
> index 20d31b279c..014f952d68 100644
> --- a/doc/muxers.texi
> +++ b/doc/muxers.texi
> @@ -2064,6 +2064,52 @@ Specify whether to remove all fragments when finished. Default 0 (do not remove)
> 
>  @end table
> 
> +@anchor{streamhash}
> +@section streamhash
> +
> +Per stream hash testing format.
> +
> +This muxer computes and prints a cryptographic hash of all the input frames,
> +on a per-stream basis. This can be used for equality checks without having
> +to do a complete binary comparison.
> +
> +By default audio frames are converted to signed 16-bit raw audio and
> +video frames to raw video before computing the hash, but the output
> +of explicit conversions to other codecs can also be used. Timestamps
> +are ignored. It uses the SHA-256 cryptographic hash function by default,
> +but supports several other algorithms.
> +
> +The output of the muxer consists of one line per stream of the form:
> +@var{streamindex},@var{algo}=@var{hash}, where @var{streamindex} is the
> +index of the mapped stream, @var{algo} is a short string representing the
> +hash function used, and @var{hash} is a hexadecimal number representing the
> +computed hash.
> +
> +@table @option
> +@item hash @var{algorithm}
> +Use the cryptographic hash function specified by the string @var{algorithm}.
> +Supported values include @code{MD5}, @code{murmur3}, @code{RIPEMD128},
> +@code{RIPEMD160}, @code{RIPEMD256}, @code{RIPEMD320}, @code{SHA160},
> +@code{SHA224}, @code{SHA256} (default), @code{SHA512/224}, @code{SHA512/256},
> +@code{SHA384}, @code{SHA512}, @code{CRC32} and @code{adler32}.
> +
> +@end table
> +
> +@subsection Examples
> +
> +To compute the SHA-256 hash of the input converted to raw audio and
> +video, and store it in the file @file{out.sha256}:
> +@example
> +ffmpeg -i INPUT -f streamhash out.sha256
> +@end example
> +
> +To print an MD5 hash to stdout use the command:
> +@example
> +ffmpeg -i INPUT -f streamhash -hash md5 -
> +@end example
> +
> +See also the @ref{hash} and @ref{framehash} muxers.
> +
>  @anchor{fifo}
>  @section fifo
> 
> diff --git a/libavformat/Makefile b/libavformat/Makefile
> index efa3a112ae..a61d42b721 100644
> --- a/libavformat/Makefile
> +++ b/libavformat/Makefile
> @@ -494,6 +494,7 @@ OBJS-$(CONFIG_SRT_DEMUXER)               += srtdec.o subtitles.o
>  OBJS-$(CONFIG_SRT_MUXER)                 += srtenc.o
>  OBJS-$(CONFIG_STL_DEMUXER)               += stldec.o subtitles.o
>  OBJS-$(CONFIG_STR_DEMUXER)               += psxstr.o
> +OBJS-$(CONFIG_STREAMHASH_MUXER)          += hashenc.o
>  OBJS-$(CONFIG_STREAM_SEGMENT_MUXER)      += segment.o
>  OBJS-$(CONFIG_SUBVIEWER1_DEMUXER)        += subviewer1dec.o subtitles.o
>  OBJS-$(CONFIG_SUBVIEWER_DEMUXER)         += subviewerdec.o subtitles.o
> diff --git a/libavformat/allformats.c b/libavformat/allformats.c
> index cd00834807..f7fea32b45 100644
> --- a/libavformat/allformats.c
> +++ b/libavformat/allformats.c
> @@ -393,6 +393,7 @@ extern AVInputFormat  ff_srt_demuxer;
>  extern AVOutputFormat ff_srt_muxer;
>  extern AVInputFormat  ff_str_demuxer;
>  extern AVInputFormat  ff_stl_demuxer;
> +extern AVOutputFormat ff_streamhash_muxer;
>  extern AVInputFormat  ff_subviewer1_demuxer;
>  extern AVInputFormat  ff_subviewer_demuxer;
>  extern AVInputFormat  ff_sup_demuxer;
> diff --git a/libavformat/hashenc.c b/libavformat/hashenc.c
> index 4fd41e41b6..18dc4dc825 100644
> --- a/libavformat/hashenc.c
> +++ b/libavformat/hashenc.c
> @@ -31,6 +31,7 @@ struct HashContext {
>      const AVClass *avclass;
>      struct AVHashContext **hashes;
>      char *hash_name;
> +    int per_stream;
>      int format_version;
>  };
> 
> @@ -56,6 +57,13 @@ static const AVOption framehash_options[] = {
>  };
>  #endif
> 
> +#if CONFIG_STREAMHASH_MUXER
> +static const AVOption streamhash_options[] = {
> +    HASH_OPT("sha256"),
> +    { NULL },
> +};
> +#endif
> +
>  #if CONFIG_MD5_MUXER
>  static const AVOption md5_options[] = {
>      HASH_OPT("md5"),
> @@ -76,6 +84,7 @@ static int hash_init(struct AVFormatContext *s)
>  {
>      int res;
>      struct HashContext *c = s->priv_data;
> +    c->per_stream = 0;
>      c->hashes = av_mallocz_array(1, sizeof(c->hashes));
>      if (!c->hashes)
>          return AVERROR(ENOMEM);
> @@ -87,24 +96,32 @@ static int hash_init(struct AVFormatContext *s)
>      av_hash_init(c->hashes[0]);
>      return 0;
>  }
> +#endif
> 
> +#if CONFIG_HASH_MUXER || CONFIG_MD5_MUXER || CONFIG_STREAMHASH_MUXER
>  static int hash_write_packet(struct AVFormatContext *s, AVPacket *pkt)
>  {
>      struct HashContext *c = s->priv_data;
> -    av_hash_update(c->hashes[0], pkt->data, pkt->size);
> +    av_hash_update(c->hashes[c->per_stream ? pkt->stream_index : 0], pkt->data, pkt->size);
>      return 0;
>  }
> 
>  static int hash_write_trailer(struct AVFormatContext *s)
>  {
>      struct HashContext *c = s->priv_data;
> -    char buf[AV_HASH_MAX_SIZE*2+128];
> -    snprintf(buf, sizeof(buf) - 200, "%s=", av_hash_get_name(c->hashes[0]));
> -
> -    av_hash_final_hex(c->hashes[0], buf + strlen(buf), sizeof(buf) - strlen(buf));
> -    av_strlcatf(buf, sizeof(buf), "\n");
> -    avio_write(s->pb, buf, strlen(buf));
> -    avio_flush(s->pb);
> +    int num_hashes = c->per_stream ? s->nb_streams : 1;
> +    for (int i = 0; i < num_hashes; i++) {
> +        char buf[AV_HASH_MAX_SIZE*2+128];
> +        if (c->per_stream) {
> +            snprintf(buf, sizeof(buf) - 200, "%d,%s=", i, av_hash_get_name(c->hashes[i]));
> +        } else {
> +            snprintf(buf, sizeof(buf) - 200, "%s=", av_hash_get_name(c->hashes[i]));
> +        }
> +        av_hash_final_hex(c->hashes[i], buf + strlen(buf), sizeof(buf) - strlen(buf));
> +        av_strlcatf(buf, sizeof(buf), "\n");
> +        avio_write(s->pb, buf, strlen(buf));
> +        avio_flush(s->pb);
> +    }
> 
>      return 0;
>  }
> @@ -112,7 +129,10 @@ static int hash_write_trailer(struct AVFormatContext *s)
>  static void hash_free(struct AVFormatContext *s)
>  {
>      struct HashContext *c = s->priv_data;
> -    av_hash_freep(&c->hashes[0]);
> +    int num_hashes = c->per_stream ? s->nb_streams : 1;
> +    for (int i = 0; i < num_hashes; i++) {
> +        av_hash_freep(&c->hashes[i]);
> +    }
>      av_freep(&c->hashes);
>  }
>  #endif
> @@ -165,6 +185,49 @@ AVOutputFormat ff_md5_muxer = {
>  };
>  #endif
> 
> +#if CONFIG_STREAMHASH_MUXER
> +static int streamhash_init(struct AVFormatContext *s)
> +{
> +    int res, i;
> +    struct HashContext *c = s->priv_data;
> +    c->per_stream = 1;
> +    c->hashes = av_mallocz_array(s->nb_streams, sizeof(c->hashes));
> +    if (!c->hashes)
> +        return AVERROR(ENOMEM);
> +    for (i = 0; i < s->nb_streams; i++) {
> +        res = av_hash_alloc(&c->hashes[i], c->hash_name);
> +        if (res < 0) {
> +            hash_free(s);

No need to call this. It's done automatically by the generic code when
this function returns an error.

> +            return res;
> +        }
> +        av_hash_init(c->hashes[i]);
> +    }
> +    return 0;
> +}
> +
> +static const AVClass streamhashenc_class = {
> +    .class_name = "stream hash muxer",
> +    .item_name  = av_default_item_name,
> +    .option     = streamhash_options,
> +    .version    = LIBAVUTIL_VERSION_INT,
> +};
> +
> +AVOutputFormat ff_streamhash_muxer = {
> +    .name              = "streamhash",
> +    .long_name         = NULL_IF_CONFIG_SMALL("Per-stream hash testing"),
> +    .priv_data_size    = sizeof(struct HashContext),
> +    .audio_codec       = AV_CODEC_ID_PCM_S16LE,
> +    .video_codec       = AV_CODEC_ID_RAWVIDEO,
> +    .init              = streamhash_init,
> +    .write_packet      = hash_write_packet,
> +    .write_trailer     = hash_write_trailer,
> +    .deinit            = hash_free,
> +    .flags             = AVFMT_VARIABLE_FPS | AVFMT_TS_NONSTRICT |
> +                         AVFMT_TS_NEGATIVE,
> +    .priv_class        = &streamhashenc_class,
> +};
> +#endif
> +
>  #if CONFIG_FRAMEHASH_MUXER || CONFIG_FRAMEMD5_MUXER
>  static void framehash_print_extradata(struct AVFormatContext *s)
>  {
> @@ -191,6 +254,7 @@ static int framehash_init(struct AVFormatContext *s)
>  {
>      int res;
>      struct HashContext *c = s->priv_data;
> +    c->per_stream = 0;
>      c->hashes = av_mallocz_array(1, sizeof(c->hashes));
>      if (!c->hashes)
>          return AVERROR(ENOMEM);
> diff --git a/libavformat/version.h b/libavformat/version.h
> index edfa73fb97..bcd0408d28 100644
> --- a/libavformat/version.h
> +++ b/libavformat/version.h
> @@ -32,8 +32,8 @@
>  // Major bumping may affect Ticket5467, 5421, 5451(compatibility with Chromium)
>  // Also please add any ticket numbers that you believe might be affected here
>  #define LIBAVFORMAT_VERSION_MAJOR  58
> -#define LIBAVFORMAT_VERSION_MINOR  32
> -#define LIBAVFORMAT_VERSION_MICRO 104
> +#define LIBAVFORMAT_VERSION_MINOR  33
> +#define LIBAVFORMAT_VERSION_MICRO 100
> 
>  #define LIBAVFORMAT_VERSION_INT AV_VERSION_INT(LIBAVFORMAT_VERSION_MAJOR, \
>                                                 LIBAVFORMAT_VERSION_MINOR, \
> --
> 2.20.1
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

On Wed, Sep 11, 2019 at 11:08:01 -0300, James Almer wrote:
> > +static int streamhash_init(struct AVFormatContext *s)
> > +{
> > +    int res, i;
> > +    struct HashContext *c = s->priv_data;
> > +    c->per_stream = 1;
> > +    c->hashes = av_mallocz_array(s->nb_streams, sizeof(c->hashes));
> > +    if (!c->hashes)
> > +        return AVERROR(ENOMEM);
> > +    for (i = 0; i < s->nb_streams; i++) {
> > +        res = av_hash_alloc(&c->hashes[i], c->hash_name);
> > +        if (res < 0) {
> > +            hash_free(s);
>
> No need to call this. It's done automatically by the generic code when
> this function returns an error.

Nice. I actually thought that, from your comment on patch 2/3. Will
resubmit (including 1/3, for completeness' sake) in a moment.

Thanks for the feedback,
Moritz

On 11-09-2019 07:04 PM, Moritz Barsnick wrote:
> Implemented as a variant of the hash muxer, reusing most functions,
> and making use of the previously introduced array of hashes.
> ---
>   Changelog                |  1 +
>   doc/muxers.texi          | 46 ++++++++++++++++++++++
>   libavformat/Makefile     |  1 +
>   libavformat/allformats.c |  1 +
>   libavformat/hashenc.c    | 82 +++++++++++++++++++++++++++++++++++-----
>   libavformat/version.h    |  4 +-
>   6 files changed, 124 insertions(+), 11 deletions(-)
>
> diff --git a/Changelog b/Changelog
> index 4b29e015a0..8eeed65680 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -9,6 +9,7 @@ version <next>:
>   - Supoort AMD AMF encoder on Linux (via Vulkan)
>   - IMM5 video decoder
>   - ZeroMQ protocol
> +- streamhash muxer
>
>
>   version 4.2:
> diff --git a/doc/muxers.texi b/doc/muxers.texi
> index 20d31b279c..014f952d68 100644
> --- a/doc/muxers.texi
> +++ b/doc/muxers.texi
> @@ -2064,6 +2064,52 @@ Specify whether to remove all fragments when finished. Default 0 (do not remove)
>
>   @end table
>
> +@anchor{streamhash}
> +@section streamhash
> +
> +Per stream hash testing format.
> +
> +This muxer computes and prints a cryptographic hash of all the input frames,
> +on a per-stream basis. This can be used for equality checks without having
> +to do a complete binary comparison.
> +
> +By default audio frames are converted to signed 16-bit raw audio and
> +video frames to raw video before computing the hash, but the output
> +of explicit conversions to other codecs can also be used. Timestamps
> +are ignored. It uses the SHA-256 cryptographic hash function by default,
> +but supports several other algorithms.
> +
> +The output of the muxer consists of one line per stream of the form:
> +@var{streamindex},@var{algo}=@var{hash}, where @var{streamindex} is the
> +index of the mapped stream, @var{algo} is a short string representing the
> +hash function used, and @var{hash} is a hexadecimal number representing the
> +computed hash.
> +
> +@table @option
> +@item hash @var{algorithm}
> +Use the cryptographic hash function specified by the string @var{algorithm}.
> +Supported values include @code{MD5}, @code{murmur3}, @code{RIPEMD128},
> +@code{RIPEMD160}, @code{RIPEMD256}, @code{RIPEMD320}, @code{SHA160},
> +@code{SHA224}, @code{SHA256} (default), @code{SHA512/224}, @code{SHA512/256},
> +@code{SHA384}, @code{SHA512}, @code{CRC32} and @code{adler32}.
> +
> +@end table
> +
> +@subsection Examples
> +
> +To compute the SHA-256 hash of the input converted to raw audio and
> +video, and store it in the file @file{out.sha256}:
> +@example
> +ffmpeg -i INPUT -f streamhash out.sha256
> +@end example
> +
> +To print an MD5 hash to stdout use the command:
> +@example
> +ffmpeg -i INPUT -f streamhash -hash md5 -
> +@end example
Since there's no mapping, this will select only one video and audio 
stream, but more importantly the video will be first, which may not be 
the case in the input. Maybe add mapping to the examples and/or add 
stream type to the hash printout.

Gyan

On Wed, Sep 11, 2019 at 19:57:43 +0530, Gyan wrote:
> > +ffmpeg -i INPUT -f streamhash -hash md5 -
> > +@end example
> Since there's no mapping, this will select only one video and audio
> stream, but more importantly the video will be first, which may not be
> the case in the input. Maybe add mapping to the examples and/or add
> stream type to the hash printout.

The original user suggesting this muxer also requested a stream type
indicator. Is that info really useful and not redundant, assuming the
user will need to understand the implications of remuxing anyway?

If I add this, and don't make it optional, what should the format be?
Thus?:

0,v,SHA256=...
1,a,SHA256=...
2,s,SHA256=...

Thanks,
Moritz

On 11-09-2019 08:19 PM, Moritz Barsnick wrote:
> On Wed, Sep 11, 2019 at 19:57:43 +0530, Gyan wrote:
>>> +ffmpeg -i INPUT -f streamhash -hash md5 -
>>> +@end example
>> Since there's no mapping, this will select only one video and audio
>> stream, but more importantly the video will be first, which may not be
>> the case in the input. Maybe add mapping to the examples and/or add
>> stream type to the hash printout.
> The original user suggesting this muxer also requested a stream type
> indicator. Is that info really useful and not redundant, assuming the
> user will need to understand the implications of remuxing anyway?

A bit of redundancy is useful for error-checking.

>
> If I add this, and don't make it optional, what should the format be?
> Thus?:
>
> 0,v,SHA256=...
> 1,a,SHA256=...
> 2,s,SHA256=...
Looks fine.

Thanks,
Gyan

Hi :)

On 11/09/2019 17:57, Gyan wrote:
>
> On 11-09-2019 08:19 PM, Moritz Barsnick wrote:
>> The original user suggesting this muxer also requested a stream type
>> indicator. Is that info really useful and not redundant, assuming the
>> user will need to understand the implications of remuxing anyway?
>
> A bit of redundancy is useful for error-checking.

If I understand this correctly, does it mean that the order of streams
output by the streamhash muxer will be "in order of type" (video, audio,
etc), independent of the order in the source?

So 2 files (source/target) will have matching streamhash output order,
independent of their actual stream order inside their containers?


Since this streamhash is intended for content-validation and not
remux-validation, I would say the way it's now is great!



>>
>> If I add this, and don't make it optional, what should the format be?
>> Thus?:
>>
>> 0,v,SHA256=...
>> 1,a,SHA256=...
>> 2,s,SHA256=...
> Looks fine.

+1 from me, too :)

I guess with multiple A/V tracks it will then be like this?

0,v,SHA256=...
1,v,SHA256=...
2,a,SHA256=...
3,a,SHA256=...
4,a,SHA256=...
5,s,SHA256=...


Thank you very much,
Peter B.

On 12-09-2019 05:14 PM, Peter B. wrote:
> Hi :)
>
> On 11/09/2019 17:57, Gyan wrote:
>> On 11-09-2019 08:19 PM, Moritz Barsnick wrote:
>>> The original user suggesting this muxer also requested a stream type
>>> indicator. Is that info really useful and not redundant, assuming the
>>> user will need to understand the implications of remuxing anyway?
>> A bit of redundancy is useful for error-checking.
> If I understand this correctly, does it mean that the order of streams
> output by the streamhash muxer will be "in order of type" (video, audio,
> etc), independent of the order in the source?
>
> So 2 files (source/target) will have matching streamhash output order,
> independent of their actual stream order inside their containers?

They may not, hence the suggestion to add stream type.

The streamhash muxer will print the streams in order of their index. 
That index, in absence of map, is set automatically, with video always 
being first.

Gyan

On Thu, Sep 12, 2019 at 13:44:57 +0200, Peter B. wrote:
> If I understand this correctly, does it mean that the order of streams
> output by the streamhash muxer will be "in order of type" (video, audio,
> etc), independent of the order in the source?

Since it's a muxer, and not a hashing demuxer (unfortunately - or
rather a different use case[*]), it has the standard ffmpeg muxing
order, unless you specify something else. In other words, best video
and best audio from first input, skipping all else. If you want to
preserve the original mapping, you'll need to specify "-map 0". And
possibly some option to mux unknown streams, and definitely "-c:s copy
-c:d copy". I tried "-copy_unknown", but the unknown data stream seems
to ruin "-t", because of missing timestamps. Perhaps "-ignore_unknown"
should be preferred.

> So 2 files (source/target) will have matching streamhash output order,
> independent of their actual stream order inside their containers?

Yes, if you don't specify otherwise.

> Since this streamhash is intended for content-validation and not
> remux-validation, I would say the way it's now is great!

For content validation, I guess the above recommendation and a general
"-c copy" is advised, to make yourself independent of the decoders.
(E.g. native aac and libfdk_aac can give different decoded output).

> > Looks fine.
>
> +1 from me, too :)

Implemented as such in the v3 patchset.

> I guess with multiple A/V tracks it will then be like this?
>
> 0,v,SHA256=...
> 1,v,SHA256=...
> 2,a,SHA256=...
> 3,a,SHA256=...
> 4,a,SHA256=...
> 5,s,SHA256=...

See above. Not if the input is v,a,a,v,a,a,s.

Moritz

[*] It make make sense as part of a demuxer as well, and as a bitstream
filter, which creates and dumps this data *while* creating other
output. I'm open for requests. ;-)

btw: If you need anything to be tested, just let me know :)


Cheers!
Peter