diff mbox series

[FFmpeg-devel,1/1] avcodec/mpegutils: add motion_vec debug mode

Message ID 20220831230918.1736362-2-chemag@gmail.com
State New
Headers show
Series add motion_vec (motion vector) debug mode | expand

Checks

Context Check Description
yinshiyou/commit_msg_loongarch64 warning Please wrap lines in the body of the commit message between 60 and 72 characters.
andriy/commit_msg_x86 warning Please wrap lines in the body of the commit message between 60 and 72 characters.
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished

Commit Message

Chema Gonzalez Aug. 31, 2022, 11:09 p.m. UTC 
Add a new debug mode ("motion_vec") that prints the values of the
motion vectors (MV).

Note that, in order to give the same weight to MV from bigger
blocks, we may print each MV pair more than once.

Tested:

```
$ ./ffprobe -v quiet -show_frames -debug motion_vec /tmp/in.264 2>&1 >/dev/null
...
[h264 @ 0x2d0d700] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2
[h264 @ 0x2d0d700] New frame, type: I
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2
[h264 @ 0x2d0d700] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0x2d0d700] New frame, type: B
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0  -3    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700]    0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   0    0   ...
[h264 @ 0x2d0d700] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
...
```
---
 libavcodec/avcodec.h       |  1 +
 libavcodec/mpegutils.c     | 45 +++++++++++++++++++++++++++++++++++++-
 libavcodec/options_table.h |  1 +
 3 files changed, 46 insertions(+), 1 deletion(-)

Comments

Ronald S. Bultje Sept. 1, 2022, 1:30 p.m. UTC | #1 
Hi,

On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com> wrote:

> Add a new debug mode ("motion_vec") that prints the values of the
> motion vectors (MV).
>

Isn't this super-hacky? In a well-designed system, a codec might export its
MVs using designated structs (which we already have) and then use these
structs to print it in a codec-agnostic way using a filter (which IIRC we
already have?).

Ronald
Chema Gonzalez Sept. 1, 2022, 3:05 p.m. UTC | #2 
Hi,

All the other debug modes (qp, skip, mb_type) use the same approach.

-Chema


On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <rsbultje@gmail.com> wrote:
>
> Hi,
>
> On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com> wrote:
>>
>> Add a new debug mode ("motion_vec") that prints the values of the
>> motion vectors (MV).
>
>
> Isn't this super-hacky? In a well-designed system, a codec might export its MVs using designated structs (which we already have) and then use these structs to print it in a codec-agnostic way using a filter (which IIRC we already have?).
>
> Ronald
Paul B Mahol Sept. 1, 2022, 5:55 p.m. UTC | #3 
On Thu, Sep 1, 2022 at 5:06 PM Chema Gonzalez <chema@berkeley.edu> wrote:

> Hi,
>
> All the other debug modes (qp, skip, mb_type) use the same approach.
>
>
This adds to much code for something that is already possible.


> -Chema
>
>
> On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <rsbultje@gmail.com>
> wrote:
> >
> > Hi,
> >
> > On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com> wrote:
> >>
> >> Add a new debug mode ("motion_vec") that prints the values of the
> >> motion vectors (MV).
> >
> >
> > Isn't this super-hacky? In a well-designed system, a codec might export
> its MVs using designated structs (which we already have) and then use these
> structs to print it in a codec-agnostic way using a filter (which IIRC we
> already have?).
> >
> > Ronald
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Chema Gonzalez Sept. 1, 2022, 6:43 p.m. UTC | #4 
Hi Paul,

Can you please describe how you can get the MV values?

Thanks,
-Chema


On Thu, Sep 1, 2022 at 10:52 AM Paul B Mahol <onemda@gmail.com> wrote:
>
> On Thu, Sep 1, 2022 at 5:06 PM Chema Gonzalez <chema@berkeley.edu> wrote:
>
> > Hi,
> >
> > All the other debug modes (qp, skip, mb_type) use the same approach.
> >
> >
> This adds to much code for something that is already possible.
>
>
> > -Chema
> >
> >
> > On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <rsbultje@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com> wrote:
> > >>
> > >> Add a new debug mode ("motion_vec") that prints the values of the
> > >> motion vectors (MV).
> > >
> > >
> > > Isn't this super-hacky? In a well-designed system, a codec might export
> > its MVs using designated structs (which we already have) and then use these
> > structs to print it in a codec-agnostic way using a filter (which IIRC we
> > already have?).
> > >
> > > Ronald
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Paul B Mahol Sept. 2, 2022, 7:59 a.m. UTC | #5 
On Thu, Sep 1, 2022 at 8:43 PM Chema Gonzalez <chema@berkeley.edu> wrote:

> Hi Paul,
>
> Can you please describe how you can get the MV values?
>

See how filters do it. They visualize MV but using similar logic one could
dump them to ton of texts.



>
> Thanks,
> -Chema
>
>
> On Thu, Sep 1, 2022 at 10:52 AM Paul B Mahol <onemda@gmail.com> wrote:
> >
> > On Thu, Sep 1, 2022 at 5:06 PM Chema Gonzalez <chema@berkeley.edu>
> wrote:
> >
> > > Hi,
> > >
> > > All the other debug modes (qp, skip, mb_type) use the same approach.
> > >
> > >
> > This adds to much code for something that is already possible.
> >
> >
> > > -Chema
> > >
> > >
> > > On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <rsbultje@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com>
> wrote:
> > > >>
> > > >> Add a new debug mode ("motion_vec") that prints the values of the
> > > >> motion vectors (MV).
> > > >
> > > >
> > > > Isn't this super-hacky? In a well-designed system, a codec might
> export
> > > its MVs using designated structs (which we already have) and then use
> these
> > > structs to print it in a codec-agnostic way using a filter (which IIRC
> we
> > > already have?).
> > > >
> > > > Ronald
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Chema Gonzalez Sept. 2, 2022, 3:12 p.m. UTC | #6 
So is there a filter that already dumps this information?

Thanks,
-Chema

On Fri, Sep 2, 2022 at 12:56 AM Paul B Mahol <onemda@gmail.com> wrote:
>
> On Thu, Sep 1, 2022 at 8:43 PM Chema Gonzalez <chema@berkeley.edu> wrote:
>
> > Hi Paul,
> >
> > Can you please describe how you can get the MV values?
> >
>
> See how filters do it. They visualize MV but using similar logic one could
> dump them to ton of texts.
>
>
>
> >
> > Thanks,
> > -Chema
> >
> >
> > On Thu, Sep 1, 2022 at 10:52 AM Paul B Mahol <onemda@gmail.com> wrote:
> > >
> > > On Thu, Sep 1, 2022 at 5:06 PM Chema Gonzalez <chema@berkeley.edu>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > All the other debug modes (qp, skip, mb_type) use the same approach.
> > > >
> > > >
> > > This adds to much code for something that is already possible.
> > >
> > >
> > > > -Chema
> > > >
> > > >
> > > > On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <rsbultje@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com>
> > wrote:
> > > > >>
> > > > >> Add a new debug mode ("motion_vec") that prints the values of the
> > > > >> motion vectors (MV).
> > > > >
> > > > >
> > > > > Isn't this super-hacky? In a well-designed system, a codec might
> > export
> > > > its MVs using designated structs (which we already have) and then use
> > these
> > > > structs to print it in a codec-agnostic way using a filter (which IIRC
> > we
> > > > already have?).
> > > > >
> > > > > Ronald
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Paul B Mahol Sept. 2, 2022, 4:13 p.m. UTC | #7 
On Fri, Sep 2, 2022 at 5:12 PM Chema Gonzalez <chema@berkeley.edu> wrote:

> So is there a filter that already dumps this information?
>

Not in text but in visual, note that you can write own code that uses
library to dump MV as text, because MV are exported when certain flag is
enabled.

>
> Thanks,
> -Chema
>
> On Fri, Sep 2, 2022 at 12:56 AM Paul B Mahol <onemda@gmail.com> wrote:
> >
> > On Thu, Sep 1, 2022 at 8:43 PM Chema Gonzalez <chema@berkeley.edu>
> wrote:
> >
> > > Hi Paul,
> > >
> > > Can you please describe how you can get the MV values?
> > >
> >
> > See how filters do it. They visualize MV but using similar logic one
> could
> > dump them to ton of texts.
> >
> >
> >
> > >
> > > Thanks,
> > > -Chema
> > >
> > >
> > > On Thu, Sep 1, 2022 at 10:52 AM Paul B Mahol <onemda@gmail.com> wrote:
> > > >
> > > > On Thu, Sep 1, 2022 at 5:06 PM Chema Gonzalez <chema@berkeley.edu>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > All the other debug modes (qp, skip, mb_type) use the same
> approach.
> > > > >
> > > > >
> > > > This adds to much code for something that is already possible.
> > > >
> > > >
> > > > > -Chema
> > > > >
> > > > >
> > > > > On Thu, Sep 1, 2022 at 6:30 AM Ronald S. Bultje <
> rsbultje@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > On Wed, Aug 31, 2022 at 7:10 PM Chema Gonzalez <chemag@gmail.com
> >
> > > wrote:
> > > > > >>
> > > > > >> Add a new debug mode ("motion_vec") that prints the values of
> the
> > > > > >> motion vectors (MV).
> > > > > >
> > > > > >
> > > > > > Isn't this super-hacky? In a well-designed system, a codec might
> > > export
> > > > > its MVs using designated structs (which we already have) and then
> use
> > > these
> > > > > structs to print it in a codec-agnostic way using a filter (which
> IIRC
> > > we
> > > > > already have?).
> > > > > >
> > > > > > Ronald
> > > > > _______________________________________________
> > > > > ffmpeg-devel mailing list
> > > > > ffmpeg-devel@ffmpeg.org
> > > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >
> > > > > To unsubscribe, visit link above, or email
> > > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > >
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Chema Gonzalez Sept. 2, 2022, 6:03 p.m. UTC | #8 
Hi again,

On Fri, Sep 2, 2022 at 9:10 AM Paul B Mahol <onemda@gmail.com> wrote:
>
> On Fri, Sep 2, 2022 at 5:12 PM Chema Gonzalez <chema@berkeley.edu> wrote:
>
> > So is there a filter that already dumps this information?
> >
>
> Not in text but in visual, note that you can write own code that uses
> library to dump MV as text, because MV are exported when certain flag is
> enabled.

Not sure if I follow this. Is the suggestion to use external code that
just uses the ffmpeg libraries (libavcodec in this case)?

Thanks,
-Chema
Paul B Mahol Sept. 2, 2022, 9:08 p.m. UTC | #9 
On Fri, Sep 2, 2022 at 8:04 PM Chema Gonzalez <chema@berkeley.edu> wrote:

> Hi again,
>
> On Fri, Sep 2, 2022 at 9:10 AM Paul B Mahol <onemda@gmail.com> wrote:
> >
> > On Fri, Sep 2, 2022 at 5:12 PM Chema Gonzalez <chema@berkeley.edu>
> wrote:
> >
> > > So is there a filter that already dumps this information?
> > >
> >
> > Not in text but in visual, note that you can write own code that uses
> > library to dump MV as text, because MV are exported when certain flag is
> > enabled.
>
> Not sure if I follow this. Is the suggestion to use external code that
> just uses the ffmpeg libraries (libavcodec in this case)?
>

Yes, no point to add bunch of printfs if code already exports MV
alternative way.


>
> Thanks,
> -Chema
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Chema Gonzalez Sept. 3, 2022, 12:46 a.m. UTC | #10 
Hi again,

On Fri, Sep 2, 2022 at 2:05 PM Paul B Mahol <onemda@gmail.com> wrote:
> > > > So is there a filter that already dumps this information?
> > > Not in text but in visual, note that you can write own code that uses
> > > library to dump MV as text, because MV are exported when certain flag is
> > > enabled.
> > Not sure if I follow this. Is the suggestion to use external code that
> > just uses the ffmpeg libraries (libavcodec in this case)?
> Yes, no point to add bunch of printfs if code already exports MV
> alternative way.

Hmm... IIUC, your suggestion is not to provide the MV vector list
using an ffmpeg debug sub-mode. Instead, have the user write her own
code using one of the ffmpeg libraries. What you mean by "visual" is
something similar to
[this](https://www.ffmpeg.org/doxygen/0.6/api-example_8c-source.html),
right?

Couple of questions:
* is there a (sane) way to have this code included in ffmpeg?
Something like a series of C files/scripts where this type of
functionality would live. FWIW some companies charge thousands of
dollars for h264 stream parsers, where some of the functionality
allows you to access these values using a GUI.
* is there any reason why we have the other ffmpeg debug sub-modes
(mb_type, qp, skip)?

Thanks,
-Chema
Paul B Mahol Sept. 3, 2022, 9:15 a.m. UTC | #11 
On Sat, Sep 3, 2022 at 2:46 AM Chema Gonzalez <chema@berkeley.edu> wrote:

> Hi again,
>
> On Fri, Sep 2, 2022 at 2:05 PM Paul B Mahol <onemda@gmail.com> wrote:
> > > > > So is there a filter that already dumps this information?
> > > > Not in text but in visual, note that you can write own code that uses
> > > > library to dump MV as text, because MV are exported when certain
> flag is
> > > > enabled.
> > > Not sure if I follow this. Is the suggestion to use external code that
> > > just uses the ffmpeg libraries (libavcodec in this case)?
> > Yes, no point to add bunch of printfs if code already exports MV
> > alternative way.
>
> Hmm... IIUC, your suggestion is not to provide the MV vector list
> using an ffmpeg debug sub-mode. Instead, have the user write her own
> code using one of the ffmpeg libraries. What you mean by "visual" is
> something similar to
> [this](https://www.ffmpeg.org/doxygen/0.6/api-example_8c-source.html),
> right?
>
> Couple of questions:
> * is there a (sane) way to have this code included in ffmpeg?
> Something like a series of C files/scripts where this type of
> functionality would live. FWIW some companies charge thousands of
> dollars for h264 stream parsers, where some of the functionality
> allows you to access these values using a GUI.
>

And why is this related at all with this clumsy printfs patch?

There is already functionality in FFmpeg libs to give you MVs.

Nobody blocks you to write sane GUI tool that will import MVs from ffmpeg
API
and sell it for millions to customers.



> * is there any reason why we have the other ffmpeg debug sub-modes
> (mb_type, qp, skip)?
>

They should be ultimately removed and in better way exported.

>
> Thanks,
> -Chema
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Ronald S. Bultje Sept. 3, 2022, 12:07 p.m. UTC | #12 
Hi Chema,

On Fri, Sep 2, 2022 at 11:12 AM Chema Gonzalez <chema@berkeley.edu> wrote:

> So is there a filter that already dumps this information?
>

Yes, it's called codecview. We can help understand how it works if you ask
more specific questions, but something like "git grep EXPORT_DATA_MVS
../libavcodec/mpeg*.c" and checking the complement code in codecview should
explain the basics.

Ronald
Chema Gonzalez Sept. 12, 2022, 11:49 p.m. UTC | #13 
Hi Paul,

I tried to understand the per-frame side-data (and metadata)
mechanism. Adding my notes here in case they help a future reader.

Metadata and side-data seem like similar mechanisms to add auxiliary
information to each frame. Main difference seems to be that metadata
is simpler (a key/value dictionary), while side-data allows adding any
data struct.

Metadata operation is also simpler: There are filters that generate
metadata (e.g. "signalstats" generates key-values such as
`lavfi.signalstats.YMIN=3`, while "silencedetect" generates
audio-related key-values like `lavfi.silence_start=0`). There are also
2x filters that print metadata ("vf_metadata" and "af_ametadata"). So
e.g. to see what signalstats/silencedetect are generating, you can do:

```
$ ffmpeg -i in.264 -vf signalstats,metadata=mode=print -f null -
...
[Parsed_metadata_1 @ ...] frame:0    pts:0       pts_time:0
[Parsed_metadata_1 @ ...] lavfi.signalstats.YMIN=3
...
```

or:
```
$ ffmpeg -y -i in.wav -af "silencedetect=n=-10dB:d=1,ametadata=print"
/tmp/out.wav
...
[Parsed_ametadata_1 @ ...] frame:23   pts:47104   pts_time:0.981333
[Parsed_ametadata_1 @ ...] lavfi.silence_start=0
...
```

Side-data operation is more complicated. There is some side-data
information already generated (e.g. SEI_UNREGISTERED side-data). Some
information requires explicitly asking for it. For example, the
`MOTION_VECTORS` side-data, you need to enable
AV_CODEC_FLAG2_EXPORT_MVS (which means calling ffmpeg/ffplay/ffprobe
with "-flags2 +export_mvs"). The main filter to print side-data
information is showinfo (`vf_showinfo` and `af_ashowinfo`). Now, the
`vf_showinfo` filter only knows how to dump some of the side-data
structs. In particular, it does not know how to dump MOTION_VECTORS
side-data. So, if we add the motion vectors, and then ask showinfo to
print it, we see:

```
$ ffmpeg -hide_banner -flags2 +export_mvs -export_side_data +mvs
-export_side_data +prft -export_side_data +venc_params
-export_side_data +film_grain -i /tmp/in.264 -vf showinfo -f null
/dev/null
...
-- frame 0 is a key frame: We can see SEI_UNREGISTERED and VIDEO_ENC_PARAMS info
[Parsed_showinfo_0 @ 0x308fd40] config in time_base: 1/1200000, frame_rate: 25/1
[Parsed_showinfo_0 @ 0x308fd40] config out time_base: 0/0, frame_rate: 0/0
[Parsed_showinfo_0 @ 0x308fd40] n:   0 pts:      0 pts_time:0
duration:  48000 duration_time:0.04    pos:        0 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:1 type:I checksum:F6BBEA9F
plane_checksum:[AFB1432E 63F2F255 2887B50D] mean:[109 119 138]
stdev:[43.0 12.7 13.3]
[Parsed_showinfo_0 @ 0x308fd40]   side data - User Data Unregistered:
[Parsed_showinfo_0 @ 0x308fd40] UUID=47564adc-5c4c-433f-94ef-c5113cd143a8
[Parsed_showinfo_0 @ 0x308fd40] User Data=01ffccccff0200e4dd42
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40]   side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
-- frame 1 is a P-frame: we can see VIDEO_ENC_PARAMS info, and a
complain about "side-data type 8" (MOTION_VECTORS)
[Parsed_showinfo_0 @ 0x308fd40] n:   1 pts:  48000 pts_time:0.04
duration:  48000 duration_time:0.04    pos:   259304 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:0 type:B checksum:BC4E5C12
plane_checksum:[AEA8857A 34697DA4 805E58E5] mean:[109 119 138]
stdev:[43.0 12.6 13.3]
[Parsed_showinfo_0 @ 0x308fd40]   side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
-- showinfo does not dump MOTION_VECTORS side-data
[Parsed_showinfo_0 @ 0x308fd40]   side data - unknown side data type 8
(547280 bytes)
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
```

So the best way right now to see the MVs is to use
`doc/examples/extract_mvs`, which does exactly that:

```
$ make examples -j
...
$ doc/examples/extract_mvs in.264 | head -40 | \
    csvcut -C framenum,source,flags |csvlook
| blockw | blockh |  srcx | srcy |  dstx | dsty | motion_x | motion_y
| motion_scale |
| ------ | ------ | ----- | ---- | ----- | ---- | -------- | --------
| ------------ |
|     16 |     16 |    20 |   26 |     8 |    8 |       49 |       72
|            4 |
|     16 |     16 |   152 |   15 |   136 |    8 |       65 |       28
|            4 |
|     16 |      8 |   360 |    3 |   360 |    4 |        1 |       -6
|            4 |
|     16 |      8 |   360 |   13 |   360 |   12 |       -1 |        4
|            4 |
|     16 |     16 |   440 |   10 |   440 |    8 |        3 |       10
|            4 |
|      8 |     16 |   829 |    7 |   836 |    8 |      -31 |       -6
|            4 |
|      8 |     16 |   844 |    7 |   844 |    8 |       -1 |       -4
|            4 |
```

> Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics.

codecview will get the motion vectors (assuming you asked for them),
and will overlay arrows showing the MV on top of the original video.

```
$ ffmpeg -flags2 +export_mvs -i input -vf codecview=mv=pf+bf+bb output.%04d.png
```

Adding one of the resulting frames in the next email.

-Chema


On Sat, Sep 3, 2022 at 5:08 AM Ronald S. Bultje <rsbultje@gmail.com> wrote:
>
> Hi Chema,
>
> On Fri, Sep 2, 2022 at 11:12 AM Chema Gonzalez <chema@berkeley.edu> wrote:
>>
>> So is there a filter that already dumps this information?
>
>
> Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics.
>
> Ronald
diff mbox series

Patch

diff --git a/libavcodec/avcodec.h b/libavcodec/avcodec.h
index 65c8535359..4287555a31 100644
--- a/libavcodec/avcodec.h
+++ b/libavcodec/avcodec.h
@@ -1344,6 +1344,7 @@  typedef struct AVCodecContext {
 #define FF_DEBUG_THREADS     0x00010000
 #define FF_DEBUG_GREEN_MD    0x00800000
 #define FF_DEBUG_NOMC        0x01000000
+#define FF_DEBUG_MOTION_VEC  0x02000000
 
     /**
      * Error recognition; may misdetect some more or less valid parts as errors.
diff --git a/libavcodec/mpegutils.c b/libavcodec/mpegutils.c
index ff9418232b..d0a94784fb 100644
--- a/libavcodec/mpegutils.c
+++ b/libavcodec/mpegutils.c
@@ -246,8 +246,11 @@  void ff_print_debug_info2(AVCodecContext *avctx, AVFrame *pict,
         return;
 
 
-    if (avctx->debug & (FF_DEBUG_SKIP | FF_DEBUG_QP | FF_DEBUG_MB_TYPE)) {
+    if (avctx->debug & (FF_DEBUG_SKIP | FF_DEBUG_QP | FF_DEBUG_MB_TYPE | FF_DEBUG_MOTION_VEC)) {
         int x,y;
+        const int mv_sample_log2 = avctx->codec_id == AV_CODEC_ID_H264 || avctx->codec_id == AV_CODEC_ID_SVQ3 ? 2 : 1;
+        const int mv_stride      = (mb_width << mv_sample_log2) +
+                                   (avctx->codec->id == AV_CODEC_ID_H264 ? 0 : 1);
 
         av_log(avctx, AV_LOG_DEBUG, "New frame, type: %c\n",
                av_get_picture_type_char(pict->pict_type));
@@ -271,6 +274,46 @@  void ff_print_debug_info2(AVCodecContext *avctx, AVFrame *pict,
                            get_segmentation_char(mb_type),
                            get_interlacement_char(mb_type));
                 }
+                if (avctx->debug & FF_DEBUG_MOTION_VEC) {
+                    int i, direction, mb_type = mbtype_table[x + y * mb_stride];
+                    if (IS_8X8(mb_type)) {
+                        for (i = 0; i < 4; i++) {
+                            int xy = (x * 2 + (i & 1) +
+                                      (y * 2 + (i >> 1)) * mv_stride) << (mv_sample_log2 - 1);
+                            int mx = motion_val[direction][xy][0];
+                            int my = motion_val[direction][xy][1];
+                            av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        }
+                    } else if (IS_16X8(mb_type)) {
+                        for (i = 0; i < 2; i++) {
+                            int xy = (x * 2 + (y * 2 + i) * mv_stride) << (mv_sample_log2 - 1);
+                            int mx = motion_val[direction][xy][0];
+                            int my = motion_val[direction][xy][1];
+                            if (IS_INTERLACED(mb_type))
+                                my *= 2;
+                            av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                            av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        }
+                    } else if (IS_8X16(mb_type)) {
+                        for (i = 0; i < 2; i++) {
+                            int xy = (x * 2 + i + y * 2 * mv_stride) << (mv_sample_log2 - 1);
+                            int mx = motion_val[direction][xy][0];
+                            int my = motion_val[direction][xy][1];
+                            if (IS_INTERLACED(mb_type))
+                                my *= 2;
+                            av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                            av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        }
+                    } else {
+                        int xy = (x + y * mv_stride) << mv_sample_log2;
+                        int mx = motion_val[direction][xy][0];
+                        int my = motion_val[direction][xy][1];
+                        av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                        av_log(avctx, AV_LOG_DEBUG, "%4d %4d", mx, my);
+                    }
+                }
             }
             av_log(avctx, AV_LOG_DEBUG, "\n");
         }
diff --git a/libavcodec/options_table.h b/libavcodec/options_table.h
index cd02f5096f..0d100069e1 100644
--- a/libavcodec/options_table.h
+++ b/libavcodec/options_table.h
@@ -198,6 +198,7 @@  static const AVOption avcodec_options[] = {
 {"startcode", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_STARTCODE }, INT_MIN, INT_MAX, V|D, "debug"},
 {"er", "error recognition", 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_ER }, INT_MIN, INT_MAX, V|D, "debug"},
 {"mmco", "memory management control operations (H.264)", 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_MMCO }, INT_MIN, INT_MAX, V|D, "debug"},
+{"motion_vec", "macroblock (MB) motion vector values", 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_MOTION_VEC }, INT_MIN, INT_MAX, V|D, "debug"},
 {"bugs", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_BUGS }, INT_MIN, INT_MAX, V|D, "debug"},
 {"buffers", "picture buffer allocations", 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_BUFFERS }, INT_MIN, INT_MAX, V|D, "debug"},
 {"thread_ops", "threading operations", 0, AV_OPT_TYPE_CONST, {.i64 = FF_DEBUG_THREADS }, INT_MIN, INT_MAX, V|A|D, "debug"},