| Message ID | 20171114152414.18478-2-philipl@overt.org |
|---|---|
| State | Accepted |
| Commit | bb4c9d0a8ead02f7d943c2bae3e36b30e605b30b |
| Headers | show |
On Tue, Nov 14, 2017 at 4:24 PM, Philip Langdale <philipl@overt.org> wrote: > > I didn't bother investigating why, but vdpau avoids this by having > all linesizes set to zero. Cute. Thats probably what most hwaccels do, the hardware surface has its own descriptor, so the linesize is not used. - Hendrik
On Tue, 14 Nov 2017 16:29:06 +0100 Hendrik Leppkes <h.leppkes@gmail.com> wrote: > On Tue, Nov 14, 2017 at 4:24 PM, Philip Langdale <philipl@overt.org> > wrote: > > > > I didn't bother investigating why, but vdpau avoids this by having > > all linesizes set to zero. Cute. > > Thats probably what most hwaccels do, the hardware surface has its own > descriptor, so the linesize is not used. Indeed. I took a look at the get_buffer implementations and cuda is special in that it sets the data and linesize values to reflect the underlying pixfmt. No good deed goes unpunished. --phil
On Tue, Nov 14, 2017 at 07:24:13AM -0800, Philip Langdale wrote: > I'm confused at how none of the previous hwaccels hit this, but > alloc_frame_buffer in ff_alloc_picture asserts that the linesize > of planes 1 and 2 are the same. If the pixfmt has a single uv > plane, like NV12, this won't be true. > > So, let's only do this check if there are more than 2 planes. > > I didn't bother investigating why, but vdpau avoids this by having > all linesizes set to zero. Cute. > > Signed-off-by: Philip Langdale <philipl@overt.org> > --- > libavcodec/mpegpicture.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) LGTM thx [...]
diff --git a/libavcodec/mpegpicture.c b/libavcodec/mpegpicture.c index 9811a778b7..2b72346fb2 100644 --- a/libavcodec/mpegpicture.c +++ b/libavcodec/mpegpicture.c @@ -22,6 +22,7 @@ #include "libavutil/avassert.h" #include "libavutil/common.h" +#include "libavutil/pixdesc.h" #include "avcodec.h" #include "motion_est.h" @@ -155,7 +156,8 @@ static int alloc_frame_buffer(AVCodecContext *avctx, Picture *pic, return -1; } - if (pic->f->linesize[1] != pic->f->linesize[2]) { + if (av_pix_fmt_count_planes(pic->f->format) > 2 && + pic->f->linesize[1] != pic->f->linesize[2]) { av_log(avctx, AV_LOG_ERROR, "get_buffer() failed (uv stride mismatch)\n"); ff_mpeg_unref_picture(avctx, pic);
I'm confused at how none of the previous hwaccels hit this, but alloc_frame_buffer in ff_alloc_picture asserts that the linesize of planes 1 and 2 are the same. If the pixfmt has a single uv plane, like NV12, this won't be true. So, let's only do this check if there are more than 2 planes. I didn't bother investigating why, but vdpau avoids this by having all linesizes set to zero. Cute. Signed-off-by: Philip Langdale <philipl@overt.org> --- libavcodec/mpegpicture.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)