diff mbox series

[FFmpeg-devel,2/4] libavutil: add side data AV_FRAME_DATA_BOUNDING_BOXES

Message ID 20210219065945.15594-2-yejun.guo@intel.com
State New
Headers show
Series [FFmpeg-devel,1/4] libavfilter/bbox.h: add BoundingBoxHeader and BoundingBox
Related show

Checks

Context Check Description
andriy/x86_make success Make finished
andriy/x86_make_fate success Make fate finished
andriy/PPC64_make success Make finished
andriy/PPC64_make_fate success Make fate finished

Commit Message

Guo, Yejun Feb. 19, 2021, 6:59 a.m. UTC
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
---
 doc/APIchanges      | 2 ++
 libavutil/frame.c   | 1 +
 libavutil/frame.h   | 7 +++++++
 libavutil/version.h | 2 +-
 4 files changed, 11 insertions(+), 1 deletion(-)

Comments

Lynne Feb. 19, 2021, 10:53 p.m. UTC | #1
Feb 19, 2021, 07:59 by yejun.guo@intel.com:

> Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
> ---
>  doc/APIchanges      | 2 ++
>  libavutil/frame.c   | 1 +
>  libavutil/frame.h   | 7 +++++++
>  libavutil/version.h | 2 +-
>  4 files changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/doc/APIchanges b/doc/APIchanges
> index c353d2d281..3c6e9e351e 100644
> --- a/doc/APIchanges
> +++ b/doc/APIchanges
> @@ -14,6 +14,8 @@ libavutil:     2017-10-21
>

This won't really work.
Side data must be public but the header it's specified in is neither installed,
neither in lavu. There's no such thing as ffmpeg-only private data.

If we do plan to make this side data public, it'll require a lot more considerations,
and versioning like with the film grain side data, since as you pointed out it may
be used by codecs.

For now, although it's somewhat less appropriate, you can replace
the side data with frame->private_ref. It's an AVBufferRef that's for
use by a single internal library, and we don't use it anywhere except
in lavc for now.
Guo, Yejun Feb. 20, 2021, 2:13 p.m. UTC | #2
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Lynne
> Sent: 2021年2月20日 6:54
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil: add side data
> AV_FRAME_DATA_BOUNDING_BOXES
> 
> Feb 19, 2021, 07:59 by yejun.guo@intel.com:
> 
> > Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
> > ---
> >  doc/APIchanges      | 2 ++
> >  libavutil/frame.c   | 1 +
> >  libavutil/frame.h   | 7 +++++++
> >  libavutil/version.h | 2 +-
> >  4 files changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/doc/APIchanges b/doc/APIchanges
> > index c353d2d281..3c6e9e351e 100644
> > --- a/doc/APIchanges
> > +++ b/doc/APIchanges
> > @@ -14,6 +14,8 @@ libavutil:     2017-10-21
> >
> 
> This won't really work.
> Side data must be public but the header it's specified in is neither installed,
> neither in lavu. There's no such thing as ffmpeg-only private data.
> 
> If we do plan to make this side data public, it'll require a lot more
> considerations,
> and versioning like with the film grain side data, since as you pointed out it
> may
> be used by codecs.

Thanks for your feedback.

Could you share a bit more about the 'versioning' of film grain? thanks.

I checked file film_grain_params.h/c and do not find it is version relative.

As for the bounding box, the BoundingBoxHeader.bbox_size acts like
a version checker. (different bbox_size if the struct is changed later)

> 
> For now, although it's somewhat less appropriate, you can replace
> the side data with frame->private_ref. It's an AVBufferRef that's for
> use by a single internal library, and we don't use it anywhere except
> in lavc for now.

Thanks for pointing this possible way, I'll investigate it.
I would prefer side data which is possible the final solution.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Lynne Feb. 20, 2021, 3:35 p.m. UTC | #3
Feb 20, 2021, 15:13 by yejun.guo@intel.com:

>
>
>> -----Original Message-----
>> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Lynne
>> Sent: 2021年2月20日 6:54
>> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil: add side data
>> AV_FRAME_DATA_BOUNDING_BOXES
>>
>> Feb 19, 2021, 07:59 by yejun.guo@intel.com:
>>
>> > Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
>> > ---
>> >  doc/APIchanges      | 2 ++
>> >  libavutil/frame.c   | 1 +
>> >  libavutil/frame.h   | 7 +++++++
>> >  libavutil/version.h | 2 +-
>> >  4 files changed, 11 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/doc/APIchanges b/doc/APIchanges
>> > index c353d2d281..3c6e9e351e 100644
>> > --- a/doc/APIchanges
>> > +++ b/doc/APIchanges
>> > @@ -14,6 +14,8 @@ libavutil:     2017-10-21
>> >
>>
>> This won't really work.
>> Side data must be public but the header it's specified in is neither installed,
>> neither in lavu. There's no such thing as ffmpeg-only private data.
>>
>> If we do plan to make this side data public, it'll require a lot more
>> considerations,
>> and versioning like with the film grain side data, since as you pointed out it
>> may
>> be used by codecs.
>>
>
> Thanks for your feedback.
>
> Could you share a bit more about the 'versioning' of film grain? thanks.
>
> I checked file film_grain_params.h/c and do not find it is version relative.
>
> As for the bounding box, the BoundingBoxHeader.bbox_size acts like
> a version checker. (different bbox_size if the struct is changed later)
>

That's not really a good solution. I meant having a dedicated and documented
version field which switches structs.
Changing the struct size means we can only add fields, so if we need to alter
the behavior of previous fields we'll end up with a data format more convoluted
than mpeg audio codecs.


>> For now, although it's somewhat less appropriate, you can replace
>> the side data with frame->private_ref. It's an AVBufferRef that's for
>> use by a single internal library, and we don't use it anywhere except
>> in lavc for now.
>>
>
> Thanks for pointing this possible way, I'll investigate it.
> I would prefer side data which is possible the final solution.
>

I'd rather not commit us to a side data format that may change
in the future and has no lavfi-external use. Could you implement it as a
private_ref, at least for the time being? It's lavfi-only, so it makes sense
for now, plus we can keep changing it, and once an external user
appears we can make it proper public side data?
Guo, Yejun Feb. 21, 2021, 1:53 p.m. UTC | #4
> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Lynne
> Sent: Saturday, February 20, 2021 11:36 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil: add side data
> AV_FRAME_DATA_BOUNDING_BOXES
> 
> Feb 20, 2021, 15:13 by yejun.guo@intel.com:
> 
> >
> >
> >> -----Original Message-----
> >> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> >> Lynne
> >> Sent: 2021年2月20日 6:54
> >> To: FFmpeg development discussions and patches
> >> <ffmpeg-devel@ffmpeg.org>
> >> Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil: add side data
> >> AV_FRAME_DATA_BOUNDING_BOXES
> >>
> >> Feb 19, 2021, 07:59 by yejun.guo@intel.com:
> >>
> >> > Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
> >> > ---
> >> >  doc/APIchanges      | 2 ++
> >> >  libavutil/frame.c   | 1 +
> >> >  libavutil/frame.h   | 7 +++++++
> >> >  libavutil/version.h | 2 +-
> >> >  4 files changed, 11 insertions(+), 1 deletion(-)
> >> >
> >> > diff --git a/doc/APIchanges b/doc/APIchanges index
> >> > c353d2d281..3c6e9e351e 100644
> >> > --- a/doc/APIchanges
> >> > +++ b/doc/APIchanges
> >> > @@ -14,6 +14,8 @@ libavutil:     2017-10-21
> >> >
> >>
> >> This won't really work.
> >> Side data must be public but the header it's specified in is neither
> >> installed, neither in lavu. There's no such thing as ffmpeg-only private
> data.
> >>
> >> If we do plan to make this side data public, it'll require a lot more
> >> considerations, and versioning like with the film grain side data,
> >> since as you pointed out it may be used by codecs.
> >>
> >
> > Thanks for your feedback.
> >
> > Could you share a bit more about the 'versioning' of film grain? thanks.
> >
> > I checked file film_grain_params.h/c and do not find it is version relative.
> >
> > As for the bounding box, the BoundingBoxHeader.bbox_size acts like a
> > version checker. (different bbox_size if the struct is changed later)
> >
> 
> That's not really a good solution. I meant having a dedicated and
> documented version field which switches structs.
> Changing the struct size means we can only add fields, so if we need to alter
> the behavior of previous fields we'll end up with a data format more
> convoluted than mpeg audio codecs.

Thanks, got it.

> 
> 
> >> For now, although it's somewhat less appropriate, you can replace the
> >> side data with frame->private_ref. It's an AVBufferRef that's for use
> >> by a single internal library, and we don't use it anywhere except in
> >> lavc for now.
> >>
> >
> > Thanks for pointing this possible way, I'll investigate it.
> > I would prefer side data which is possible the final solution.
> >
> 
> I'd rather not commit us to a side data format that may change in the future
> and has no lavfi-external use. Could you implement it as a private_ref, at
> least for the time being? It's lavfi-only, so it makes sense for now, plus we
> can keep changing it, and once an external user appears we can make it
> proper public side data?

Sure, I'll look at this method, thanks.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org
> with subject "unsubscribe".
diff mbox series

Patch

diff --git a/doc/APIchanges b/doc/APIchanges
index c353d2d281..3c6e9e351e 100644
--- a/doc/APIchanges
+++ b/doc/APIchanges
@@ -14,6 +14,8 @@  libavutil:     2017-10-21
 
 
 API changes, most recent first:
+2021-02-xx - xxxxxxxxxx - lavu 56.xx.100 - frame.h
+  Add AV_FRAME_DATA_BOUNDING_BOXES
 
 2021-02-14 - xxxxxxxxxx - lavd 58.12.100 - avdevice.h
   Deprecated avdevice_capabilities_create() and
diff --git a/libavutil/frame.c b/libavutil/frame.c
index eab51b6a32..53868a2410 100644
--- a/libavutil/frame.c
+++ b/libavutil/frame.c
@@ -852,6 +852,7 @@  const char *av_frame_side_data_name(enum AVFrameSideDataType type)
     case AV_FRAME_DATA_VIDEO_ENC_PARAMS:            return "Video encoding parameters";
     case AV_FRAME_DATA_SEI_UNREGISTERED:            return "H.26[45] User Data Unregistered SEI message";
     case AV_FRAME_DATA_FILM_GRAIN_PARAMS:           return "Film grain parameters";
+    case AV_FRAME_DATA_BOUNDING_BOXES:              return "Bounding boxes";
     }
     return NULL;
 }
diff --git a/libavutil/frame.h b/libavutil/frame.h
index 1aeafef6de..e213e0a646 100644
--- a/libavutil/frame.h
+++ b/libavutil/frame.h
@@ -198,6 +198,13 @@  enum AVFrameSideDataType {
      * Must be present for every frame which should have film grain applied.
      */
     AV_FRAME_DATA_FILM_GRAIN_PARAMS,
+
+    /**
+     * Bounding boxes for object detection and classification, the data is a BoundingBoxHeader
+     * followed with an array of BoudingBox, the number of array element is implied by
+     * (AVFrameSideData.size - sizeof(BoundingBoxHeader)) / BoundingBoxHeader.bbox_size.
+     */
+    AV_FRAME_DATA_BOUNDING_BOXES,
 };
 
 enum AVActiveFormatDescription {
diff --git a/libavutil/version.h b/libavutil/version.h
index b2165754f9..b7c5892a37 100644
--- a/libavutil/version.h
+++ b/libavutil/version.h
@@ -79,7 +79,7 @@ 
  */
 
 #define LIBAVUTIL_VERSION_MAJOR  56
-#define LIBAVUTIL_VERSION_MINOR  65
+#define LIBAVUTIL_VERSION_MINOR  66
 #define LIBAVUTIL_VERSION_MICRO 100
 
 #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \