diff mbox

[FFmpeg-devel] libavcodec/zmbvenc: Add support for RGB formats

Message ID 20190307144209.32046-1-matthew.w.fearnley@gmail.com
State Accepted
Commit 0321370601833f4ae47e8e11c44570ea4bd382a4
Headers show

Commit Message

matthew.w.fearnley@gmail.com March 7, 2019, 2:42 p.m. UTC
This consists mostly of the following changes:
- add newly supported pixel formats (RGB555LE, RGB565LE, BGR0)
- select the ZMBV format (c->fmt) and bytes per pixel (c->bypp) based on
  avctx->pix_fmt
- multiply widths/x-values by c->bypp, in places where bytes, not pixels, are
  expected
- disable palette-writing code for non-palette pix_fmts
- make a note about histogram[]'s datatype (it could need increasing if
  ZMBV_BLOCK is increased)
- adjust the c->score_tab length to take up to (and including) 4 times the
  number of pixels in a block
- initialise c->score_tab up to c->bypp * the number of pixels

Note: the ZmbvFormat enum allows for additional bit depths:
- 1,2,4-bit (palette)
- 24-bit (RGB)

At time of writing the specifics of these (e.g. channel order, bit alignment)
are not currently defined, and DOSBox only implements support for 8/15/16/32
bpp.
One might expect the 24-bit format - if implemented - to be BGR24, to have the
same channel order as BGR0.
However, the decoder in zmbv.c has been guessed to use RGB24, so I have chosen
to not contradict this, and omitted specific support for this format.
---
 libavcodec/zmbvenc.c | 104 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 77 insertions(+), 27 deletions(-)

Comments

Tomas Härdin March 8, 2019, 2:04 p.m. UTC | #1
tor 2019-03-07 klockan 14:42 +0000 skrev Matthew Fearnley:
> This consists mostly of the following changes:
> - add newly supported pixel formats (RGB555LE, RGB565LE, BGR0)
> - select the ZMBV format (c->fmt) and bytes per pixel (c->bypp) based on
>   avctx->pix_fmt
> - multiply widths/x-values by c->bypp, in places where bytes, not pixels, are
>   expected
> - disable palette-writing code for non-palette pix_fmts
> - make a note about histogram[]'s datatype (it could need increasing if
>   ZMBV_BLOCK is increased)
> - adjust the c->score_tab length to take up to (and including) 4 times the
>   number of pixels in a block
> - initialise c->score_tab up to c->bypp * the number of pixels
> 
> Note: the ZmbvFormat enum allows for additional bit depths:
> - 1,2,4-bit (palette)
> - 24-bit (RGB)
> 
> At time of writing the specifics of these (e.g. channel order, bit alignment)
> are not currently defined, and DOSBox only implements support for 8/15/16/32
> bpp.
> One might expect the 24-bit format - if implemented - to be BGR24, to have the
> same channel order as BGR0.
> However, the decoder in zmbv.c has been guessed to use RGB24, so I have chosen
> to not contradict this, and omitted specific support for this format.

Sounds good. Maybe we could coordinate 1/2/4/24-bit support with the
dosbox devs? And maybe we should do something about the RGB24 thing in
the decoder..

It seems the decoder performs quite well. I did a little test with
different me_range for the three not-broken zmbv samples in FATE:

  $ for RANGE in 1 2 4 8 16 24 32; do mkdir -p zmbv/$RANGE ; for f in
fate-suite/zmbv/zmbv_*.avi ; do ./ffmpeg -i $f -y -vcodec zmbv
-me_range $RANGE zmbv/$RANGE/$(basename $f) ; done ; done
  $ cd zmbv && du --bytes --max-depth=1 | sort -n
  501874  ./32
  508140  ./24
  513372  ./16
  524550  ./8
  533856  ./4
  548282  ./2
  653010  ./fate  <-- original samples
  735854  ./1

So even -me_range 2 outperforms whatever encoder produced the samples.

Maybe future work could be to try to figure out the optimal block size
based on some heuristic?

Patch looks good to me.

/Tomas
Carl Eugen Hoyos March 8, 2019, 6:07 p.m. UTC | #2
2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> tor 2019-03-07 klockan 14:42 +0000 skrev Matthew Fearnley:
>> This consists mostly of the following changes:
>> - add newly supported pixel formats (RGB555LE, RGB565LE, BGR0)
>> - select the ZMBV format (c->fmt) and bytes per pixel (c->bypp) based on
>>   avctx->pix_fmt
>> - multiply widths/x-values by c->bypp, in places where bytes, not pixels,
>> are
>>   expected
>> - disable palette-writing code for non-palette pix_fmts
>> - make a note about histogram[]'s datatype (it could need increasing if
>>   ZMBV_BLOCK is increased)
>> - adjust the c->score_tab length to take up to (and including) 4 times the
>>   number of pixels in a block
>> - initialise c->score_tab up to c->bypp * the number of pixels
>>
>> Note: the ZmbvFormat enum allows for additional bit depths:
>> - 1,2,4-bit (palette)
>> - 24-bit (RGB)
>>
>> At time of writing the specifics of these (e.g. channel order, bit
>> alignment)
>> are not currently defined, and DOSBox only implements support for
>> 8/15/16/32
>> bpp.
>> One might expect the 24-bit format - if implemented - to be BGR24, to have
>> the
>> same channel order as BGR0.
>> However, the decoder in zmbv.c has been guessed to use RGB24, so I have
>> chosen
>> to not contradict this, and omitted specific support for this format.
>
> Sounds good.

Yes.

> Maybe we could coordinate 1/2/4/24-bit support with the

I believe FFmpeg cannot support 1/2/4 bit for encoding.
(It should be possible to implement decoding to pal8 if
that doesn't work yet and if samples exist.)

> dosbox devs? And maybe we should do something about
> the RGB24 thing in the decoder..

Do I understand correctly that no existing implementation
supports 24bit rgb? If that is correct, I believe FFmpeg
shouldn't add it (but this may only be me).

Carl Eugen
matthew.w.fearnley@gmail.com March 8, 2019, 9:39 p.m. UTC | #3
On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:

> 2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> > tor 2019-03-07 klockan 14:42 +0000 skrev Matthew Fearnley:
> >> This consists mostly of the following changes:
> >> - add newly supported pixel formats (RGB555LE, RGB565LE, BGR0)
> >> - select the ZMBV format (c->fmt) and bytes per pixel (c->bypp) based on
> >>   avctx->pix_fmt
> >> - multiply widths/x-values by c->bypp, in places where bytes, not
> pixels,
> >> are
> >>   expected
> >> - disable palette-writing code for non-palette pix_fmts
> >> - make a note about histogram[]'s datatype (it could need increasing if
> >>   ZMBV_BLOCK is increased)
> >> - adjust the c->score_tab length to take up to (and including) 4 times
> the
> >>   number of pixels in a block
> >> - initialise c->score_tab up to c->bypp * the number of pixels
> >>
> >> Note: the ZmbvFormat enum allows for additional bit depths:
> >> - 1,2,4-bit (palette)
> >> - 24-bit (RGB)
> >>
> >> At time of writing the specifics of these (e.g. channel order, bit
> >> alignment)
> >> are not currently defined, and DOSBox only implements support for
> >> 8/15/16/32
> >> bpp.
> >> One might expect the 24-bit format - if implemented - to be BGR24, to
> have
> >> the
> >> same channel order as BGR0.
> >> However, the decoder in zmbv.c has been guessed to use RGB24, so I have
> >> chosen
> >> to not contradict this, and omitted specific support for this format.
> >
> > Sounds good.
>
> Yes.
>
> > Maybe we could coordinate 1/2/4/24-bit support with the
>
> I believe FFmpeg cannot support 1/2/4 bit for encoding.
>
As far as I can see, FFmpeg has very limited support for bit depths less
than 8.  I think there are basically two formats (plus variants), with
fixed "palettes":
1bpp: MONO_BLACK / MONO_WHITE (black/white only)
4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume red/blue
would be 0,255; green would be 0,85,170,255)

If the ZMBV formats were defined, these might be worth encoder adding
support for.
(Practically speaking though, it would be a slight pain, because the
encoder would do the work in 8bpp and pack/unpack as needed.)
But with PAL8 being the only format allowing a free palette, all sub-8bpp
formats would have to decode to that, so they wouldn't round-trip.

(It should be possible to implement decoding to pal8 if
> that doesn't work yet and if samples exist.)
>
No samples or specifications exist that I know of, so I don't plan to
submit any patches to the decoder unless/until there is something to work
with there.

> dosbox devs? And maybe we should do something about
> > the RGB24 thing in the decoder..
>
Yeah, I think talking with the DOSBox devs sounds like a potentially good
idea.

>
> Do I understand correctly that no existing implementation
> supports 24bit rgb? If that is correct, I believe FFmpeg
> shouldn't add it (but this may only be me).
>
I agree that FFmpeg shouldn't add support for any formats that haven't been
defined.
As far as I know, the specifics of the 24-bit RGB format havn't been
discussed anywhere, and there are no samples I know of.

A likely specification of 24-bit is trivial enough to add support for, that
I was originally planning to add it with an #ifdef (like in the decoder).
But it wouldn't do to have contradictory channel orders proposed in the
decoder and encoder, so I will leave that for now unless DOSBox will commit
to one.

I presume that FFmpeg generally doesn't like to set standards in media
formats, only to implement existing ones.
My personal feelings in this case would be to provide support that's
disabled at compile-time if an official specification can be agreed, and to
have support included by default if an independent implementation - or at
least independent samples - are available that agree with the specification.
,
Matthew
Tomas Härdin March 11, 2019, 10:37 a.m. UTC | #4
fre 2019-03-08 klockan 21:39 +0000 skrev Matthew Fearnley:
> > On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
> > 2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> > > 
> > > Maybe we could coordinate 1/2/4/24-bit support with the
> > 
> > I believe FFmpeg cannot support 1/2/4 bit for encoding.
> > 
> 
> As far as I can see, FFmpeg has very limited support for bit depths less
> than 8.  I think there are basically two formats (plus variants), with
> fixed "palettes":
> 1bpp: MONO_BLACK / MONO_WHITE (black/white only)
> 4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume red/blue
> would be 0,255; green would be 0,85,170,255)
> 
> If the ZMBV formats were defined, these might be worth encoder adding
> support for.
> (Practically speaking though, it would be a slight pain, because the
> encoder would do the work in 8bpp and pack/unpack as needed.)
> But with PAL8 being the only format allowing a free palette, all sub-8bpp
> formats would have to decode to that, so they wouldn't round-trip.

There's some justification for adding sub-8bpp, like BMP. bmp.c
converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4 and 8
don't work at all.

One way to at least allow both the bmp and zmbv encoders to do sub-8bpp 
from PAL8 would be to keep track of the maximum number of colors in
some appropriate struct.

Adding proper sub-8bpp support would involve a lot of libsws headache I
suspect.

> (It should be possible to implement decoding to pal8 if
> > that doesn't work yet and if samples exist.)
> > 
> 
> No samples or specifications exist that I know of, so I don't plan to
> submit any patches to the decoder unless/until there is something to work
> with there.
> 
> > dosbox devs? And maybe we should do something about
> > > the RGB24 thing in the decoder..
> 
> Yeah, I think talking with the DOSBox devs sounds like a potentially good
> idea.
> 
> > 
> > Do I understand correctly that no existing implementation
> > supports 24bit rgb? If that is correct, I believe FFmpeg
> > shouldn't add it (but this may only be me).
> > 
> 
> I agree that FFmpeg shouldn't add support for any formats that haven't been
> defined.
> As far as I know, the specifics of the 24-bit RGB format havn't been
> discussed anywhere, and there are no samples I know of.
> 
> A likely specification of 24-bit is trivial enough to add support for, that
> I was originally planning to add it with an #ifdef (like in the decoder).
> But it wouldn't do to have contradictory channel orders proposed in the
> decoder and encoder, so I will leave that for now unless DOSBox will commit
> to one.
> 
> I presume that FFmpeg generally doesn't like to set standards in media
> formats, only to implement existing ones.
> My personal feelings in this case would be to provide support that's
> disabled at compile-time if an official specification can be agreed, and to
> have support included by default if an independent implementation - or at
> least independent samples - are available that agree with the specification.

Just a small thing to be clear: ZMBV_ENABLE_24BPP is not defined
anywhere, so we're free to do however we want with it. It's not going
to break anyone's workflow unless they were foolish enough to encode
24-bit ZMBVs outside of the non-existing spec.

/Tomas
Carl Eugen Hoyos March 11, 2019, 10:43 a.m. UTC | #5
2019-03-11 11:37 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> fre 2019-03-08 klockan 21:39 +0000 skrev Matthew Fearnley:
>> > On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com>
>> > wrote:
>> > 2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
>> > >
>> > > Maybe we could coordinate 1/2/4/24-bit support with the
>> >
>> > I believe FFmpeg cannot support 1/2/4 bit for encoding.
>>
>> As far as I can see, FFmpeg has very limited support for bit depths less
>> than 8.  I think there are basically two formats (plus variants), with
>> fixed "palettes":
>> 1bpp: MONO_BLACK / MONO_WHITE (black/white only)
>> 4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume red/blue
>> would be 0,255; green would be 0,85,170,255)
>>
>> If the ZMBV formats were defined, these might be worth encoder adding
>> support for.
>> (Practically speaking though, it would be a slight pain, because the
>> encoder would do the work in 8bpp and pack/unpack as needed.)
>> But with PAL8 being the only format allowing a free palette, all sub-8bpp
>> formats would have to decode to that, so they wouldn't round-trip.
>
> There's some justification for adding sub-8bpp, like BMP. bmp.c
> converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4 and 8
> don't work at all.
>
> One way to at least allow both the bmp and zmbv encoders to do sub-8bpp
> from PAL8 would be to keep track of the maximum number of colors in
> some appropriate struct.

bits_per_coded_sample

Carl Eugen
Tomas Härdin March 11, 2019, 11:07 a.m. UTC | #6
mån 2019-03-11 klockan 11:43 +0100 skrev Carl Eugen Hoyos:
> > 2019-03-11 11:37 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> > fre 2019-03-08 klockan 21:39 +0000 skrev Matthew Fearnley:
> > > > On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com>
> > > > wrote:
> > > > > > > > 2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> > > > > 
> > > > > Maybe we could coordinate 1/2/4/24-bit support with the
> > > > 
> > > > I believe FFmpeg cannot support 1/2/4 bit for encoding.
> > > 
> > > As far as I can see, FFmpeg has very limited support for bit depths less
> > > than 8.  I think there are basically two formats (plus variants), with
> > > fixed "palettes":
> > > 1bpp: MONO_BLACK / MONO_WHITE (black/white only)
> > > 4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume red/blue
> > > would be 0,255; green would be 0,85,170,255)
> > > 
> > > If the ZMBV formats were defined, these might be worth encoder adding
> > > support for.
> > > (Practically speaking though, it would be a slight pain, because the
> > > encoder would do the work in 8bpp and pack/unpack as needed.)
> > > But with PAL8 being the only format allowing a free palette, all sub-8bpp
> > > formats would have to decode to that, so they wouldn't round-trip.
> > 
> > There's some justification for adding sub-8bpp, like BMP. bmp.c
> > converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4 and 8
> > don't work at all.
> > 
> > One way to at least allow both the bmp and zmbv encoders to do sub-8bpp
> > from PAL8 would be to keep track of the maximum number of colors in
> > some appropriate struct.
> 
> bits_per_coded_sample

bits_per_raw_sample would seem more appropriate, depending on what part
of the documentation you trust. It's unfortunate that AVCodecContext
and AVCodecParameters have different wording, and possibly different
semantics for these fields..

Whichever is used, the wording would need to become more
precise. bits_per_coded_sample documentation mentions ADPCM which packs
samples, whereas 1/2/4-bit-in-PAL8 would not.

I sent an email to the DOSBox crew about bit ordering

/Tomas
Carl Eugen Hoyos March 11, 2019, 11:34 a.m. UTC | #7
2019-03-11 12:07 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
> mån 2019-03-11 klockan 11:43 +0100 skrev Carl Eugen Hoyos:
>> > 2019-03-11 11:37 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
>> > fre 2019-03-08 klockan 21:39 +0000 skrev Matthew Fearnley:
>> > > > On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com>
>> > > > wrote:
>> > > > > > > > 2019-03-08 15:04 GMT+01:00, Tomas Härdin
>> > > > > > > > <tjoppen@acc.umu.se>:
>> > > > >
>> > > > > Maybe we could coordinate 1/2/4/24-bit support with the
>> > > >
>> > > > I believe FFmpeg cannot support 1/2/4 bit for encoding.
>> > >
>> > > As far as I can see, FFmpeg has very limited support for bit depths
>> > > less
>> > > than 8.  I think there are basically two formats (plus variants), with
>> > > fixed "palettes":
>> > > 1bpp: MONO_BLACK / MONO_WHITE (black/white only)
>> > > 4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume
>> > > red/blue
>> > > would be 0,255; green would be 0,85,170,255)
>> > >
>> > > If the ZMBV formats were defined, these might be worth encoder adding
>> > > support for.
>> > > (Practically speaking though, it would be a slight pain, because the
>> > > encoder would do the work in 8bpp and pack/unpack as needed.)
>> > > But with PAL8 being the only format allowing a free palette, all
>> > > sub-8bpp
>> > > formats would have to decode to that, so they wouldn't round-trip.
>> >
>> > There's some justification for adding sub-8bpp, like BMP. bmp.c
>> > converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4 and 8
>> > don't work at all.
>> >
>> > One way to at least allow both the bmp and zmbv encoders to do sub-8bpp
>> > from PAL8 would be to keep track of the maximum number of colors in
>> > some appropriate struct.
>>
>> bits_per_coded_sample
>
> bits_per_raw_sample would seem more appropriate

Of course!

Sorry, Carl Eugen
matthew.w.fearnley@gmail.com March 12, 2019, 10:27 a.m. UTC | #8
> On 11 Mar 2019, at 10:37, Tomas Härdin <tjoppen@acc.umu.se> wrote:
> 
> fre 2019-03-08 klockan 21:39 +0000 skrev Matthew Fearnley:
>>>> On Fri, 8 Mar 2019 at 18:07, Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
>>>> 2019-03-08 15:04 GMT+01:00, Tomas Härdin <tjoppen@acc.umu.se>:
>>>> 
>>>> Maybe we could coordinate 1/2/4/24-bit support with the
>>> 
>>> I believe FFmpeg cannot support 1/2/4 bit for encoding.
>>> 
>> 
>> As far as I can see, FFmpeg has very limited support for bit depths less
>> than 8.  I think there are basically two formats (plus variants), with
>> fixed "palettes":
>> 1bpp: MONO_BLACK / MONO_WHITE (black/white only)
>> 4bpp: RGB4[_BYTE], BGR4[_BYTE] (RGB 1:2:1, 16 colours - I presume red/blue
>> would be 0,255; green would be 0,85,170,255)
>> 
>> If the ZMBV formats were defined, these might be worth encoder adding
>> support for.
>> (Practically speaking though, it would be a slight pain, because the
>> encoder would do the work in 8bpp and pack/unpack as needed.)
>> But with PAL8 being the only format allowing a free palette, all sub-8bpp
>> formats would have to decode to that, so they wouldn't round-trip.
> 
> There's some justification for adding sub-8bpp, like BMP. bmp.c
> converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4 and 8
> don't work at all.
> 
> One way to at least allow both the bmp and zmbv encoders to do sub-8bpp 
> from PAL8 would be to keep track of the maximum number of colors in
> some appropriate struct.
> 
> Adding proper sub-8bpp support would involve a lot of libsws headache I
> suspect.
It occurs to me that adding sub-8bpp has some implications:

My current understanding (I could be wrong) is that FFmpeg tends to detect the pix_fmt based on the first frame. If FFmpeg detects the first frame as e.g. PAL4, and chooses that as its output, that means the rest of the video will have to be encodable as PAL4, otherwise it (obviously) won’t be encoded properly.

So adding a PAL4 format puts a new constraint on encoders (inside and outside FFmpeg) to not encode frames in a way that looks like PAL4, unless the whole video will be encodable that way.

If FFmpeg supports PAL8 only, then it can be tempting to optimise videos to encode as sub-8bpp whenever possible, knowing they will always (in FFmpeg at least) decode to PAL8. But this could break format detection for tools outside FFmpeg, if they choose to add sub-8bpp support.

The safest thing FFmpeg can do is to always decode sub-8bpp to PAL8, and to emit PAL8 frames as exactly 8bpp (where applicable). It could still offer encoding formats for PAL1/2/4, but these formats could only be detected by scanning the whole video.

The suggestion of bits_per_raw_sample sounds interesting. What would that look like in practice?

>> (It should be possible to implement decoding to pal8 if
>>> that doesn't work yet and if samples exist.)
>>> 
>> 
>> No samples or specifications exist that I know of, so I don't plan to
>> submit any patches to the decoder unless/until there is something to work
>> with there.
>> 
>>> dosbox devs? And maybe we should do something about
>>>> the RGB24 thing in the decoder..
>> 
>> Yeah, I think talking with the DOSBox devs sounds like a potentially good
>> idea.
>> 
>>> 
>>> Do I understand correctly that no existing implementation
>>> supports 24bit rgb? If that is correct, I believe FFmpeg
>>> shouldn't add it (but this may only be me).
>>> 
>> 
>> I agree that FFmpeg shouldn't add support for any formats that haven't been
>> defined.
>> As far as I know, the specifics of the 24-bit RGB format havn't been
>> discussed anywhere, and there are no samples I know of.
>> 
>> A likely specification of 24-bit is trivial enough to add support for, that
>> I was originally planning to add it with an #ifdef (like in the decoder).
>> But it wouldn't do to have contradictory channel orders proposed in the
>> decoder and encoder, so I will leave that for now unless DOSBox will commit
>> to one.
>> 
>> I presume that FFmpeg generally doesn't like to set standards in media
>> formats, only to implement existing ones.
>> My personal feelings in this case would be to provide support that's
>> disabled at compile-time if an official specification can be agreed, and to
>> have support included by default if an independent implementation - or at
>> least independent samples - are available that agree with the specification.
> 
> Just a small thing to be clear: ZMBV_ENABLE_24BPP is not defined
> anywhere, so we're free to do however we want with it. It's not going
> to break anyone's workflow unless they were foolish enough to encode
> 24-bit ZMBVs outside of the non-existing spec.
True. But they might be understandably puzzled if they encode as 24-bit, and then find the channels swapped when they decode.

Thanks for writing the email to the DOSBox crew.
If they choose a channel order, then we have good grounds for fixing the encoder (if need be), and implementing the decoder in the same way.
It occurs to me that they might (in theory) also want to specify 2/4 byte alignment on RGB, like with the MVs.  My gut says there’d be very little benefit though, and it would only be seen with strange video / block widths.

It also occurs to me this will may warrant a version bump in the format, to give an easy error case for decoders that don’t expect it. Particularly if our decoder has to redefine its channel order.

Matthew
Tomas Härdin March 12, 2019, 11:46 a.m. UTC | #9
tis 2019-03-12 klockan 10:27 +0000 skrev Matthew Fearnley:
> > On 11 Mar 2019, at 10:37, Tomas Härdin <tjoppen@acc.umu.se> wrote:
> > 
> > 
> > There's some justification for adding sub-8bpp, like BMP. bmp.c
> > converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4
> > and 8
> > don't work at all.
> > 
> > One way to at least allow both the bmp and zmbv encoders to do sub-
> > 8bpp 
> > from PAL8 would be to keep track of the maximum number of colors in
> > some appropriate struct.
> > 
> > Adding proper sub-8bpp support would involve a lot of libsws
> > headache I
> > suspect.
> 
> It occurs to me that adding sub-8bpp has some implications:
> 
> My current understanding (I could be wrong) is that FFmpeg tends to
> detect the pix_fmt based on the first frame. If FFmpeg detects the
> first frame as e.g. PAL4, and chooses that as its output, that means
> the rest of the video will have to be encodable as PAL4, otherwise it
> (obviously) won’t be encoded properly.
> 
> So adding a PAL4 format puts a new constraint on encoders (inside and
> outside FFmpeg) to not encode frames in a way that looks like PAL4,
> unless the whole video will be encodable that way.

Yes, FFmpeg will probe the initial format of the video and audio.
Nothing says these are constant. There are FATE samples specifically
for files that change resolution. Since ZMBV is a DOS capture codec,
and DOS programs frequently change resolution and colordepth, this is
indeed something we have to think about. Example: DOS boots into mode
3h, 80x25 16-color text. A recording may start in this mode, then
switch to mode 13h (320x200 256 colors graphics), if I understand the
format correctly.

> If FFmpeg supports PAL8 only, then it can be tempting to optimise
> videos to encode as sub-8bpp whenever possible, knowing they will
> always (in FFmpeg at least) decode to PAL8. But this could break
> format detection for tools outside FFmpeg, if they choose to add sub-
> 8bpp support.
> 
> The safest thing FFmpeg can do is to always decode sub-8bpp to PAL8,
> and to emit PAL8 frames as exactly 8bpp (where applicable). It could
> still offer encoding formats for PAL1/2/4, but these formats could
> only be detected by scanning the whole video.
> 
> The suggestion of bits_per_raw_sample sounds interesting. What would
> that look like in practice?

The decoder would set bits_per_raw_sample on every keyframe. It and
resolution might change during the course of a ZMBV file, as explained
above.

I think frivolously changing the bitdepth is a bad idea, at least
inside the encoder. If the encoder is fed with changing bitdepths then
it's a different story.

> > Just a small thing to be clear: ZMBV_ENABLE_24BPP is not defined
> > anywhere, so we're free to do however we want with it. It's not
> > going
> > to break anyone's workflow unless they were foolish enough to
> > encode
> > 24-bit ZMBVs outside of the non-existing spec.
> 
> True. But they might be understandably puzzled if they encode as 24-
> bit, and then find the channels swapped when they decode.

Yeah, that's why we'd need to get this nailed down

> Thanks for writing the email to the DOSBox crew.
> If they choose a channel order, then we have good grounds for fixing
> the encoder (if need be), and implementing the decoder in the same
> way.
> It occurs to me that they might (in theory) also want to specify 2/4
> byte alignment on RGB, like with the MVs.  My gut says there’d be
> very little benefit though, and it would only be seen with strange
> video / block widths.
> 
> It also occurs to me this will may warrant a version bump in the
> format, to give an easy error case for decoders that don’t expect it.
> Particularly if our decoder has to redefine its channel order.

Are there any decoders besides ours and dosbox's?

It also turns out the creator of this codec is Harekiet, who hangs out
in #revision on IRCnet

/Tomas
matthew.w.fearnley@gmail.com March 13, 2019, 5:46 p.m. UTC | #10
> On 12 Mar 2019, at 11:46, Tomas Härdin <tjoppen@acc.umu.se> wrote:
> 
> tis 2019-03-12 klockan 10:27 +0000 skrev Matthew Fearnley:
>>> On 11 Mar 2019, at 10:37, Tomas Härdin <tjoppen@acc.umu.se> wrote:
>>> 
>>> 
>>> There's some justification for adding sub-8bpp, like BMP. bmp.c
>>> converts all of them except GRAY8 to PAL8. Bitdepths besides 1, 4
>>> and 8
>>> don't work at all.
>>> 
>>> One way to at least allow both the bmp and zmbv encoders to do sub-
>>> 8bpp 
>>> from PAL8 would be to keep track of the maximum number of colors in
>>> some appropriate struct.
>>> 
>>> Adding proper sub-8bpp support would involve a lot of libsws
>>> headache I
>>> suspect.
>> 
>> It occurs to me that adding sub-8bpp has some implications:
>> 
>> My current understanding (I could be wrong) is that FFmpeg tends to
>> detect the pix_fmt based on the first frame. If FFmpeg detects the
>> first frame as e.g. PAL4, and chooses that as its output, that means
>> the rest of the video will have to be encodable as PAL4, otherwise it
>> (obviously) won’t be encoded properly.
>> 
>> So adding a PAL4 format puts a new constraint on encoders (inside and
>> outside FFmpeg) to not encode frames in a way that looks like PAL4,
>> unless the whole video will be encodable that way.
> 
> Yes, FFmpeg will probe the initial format of the video and audio.
> Nothing says these are constant. There are FATE samples specifically
> for files that change resolution. Since ZMBV is a DOS capture codec,
> and DOS programs frequently change resolution and colordepth, this is
> indeed something we have to think about. Example: DOS boots into mode
> 3h, 80x25 16-color text. A recording may start in this mode, then
> switch to mode 13h (320x200 256 colors graphics), if I understand the
> format correctly.
DOSBox actually avoids this issue by outputting to a new file whenever it detects a change in colour depth, dimensions or FPS. So in practice, these remain constant over a single video.

I’ve tried stitching two AVI files together (palette + RGB, same dimensions), and found that if a palette-capable format like ‘png’ is chosen, all of the resulting AVI file is encoded with a palette, and RGB sections are dithered.  That possibly suggests FFmpeg doesn’t like changing format halfway through videos, for AVI at least?
> 
>> If FFmpeg supports PAL8 only, then it can be tempting to optimise
>> videos to encode as sub-8bpp whenever possible, knowing they will
>> always (in FFmpeg at least) decode to PAL8. But this could break
>> format detection for tools outside FFmpeg, if they choose to add sub-
>> 8bpp support.
>> 
>> The safest thing FFmpeg can do is to always decode sub-8bpp to PAL8,
>> and to emit PAL8 frames as exactly 8bpp (where applicable). It could
>> still offer encoding formats for PAL1/2/4, but these formats could
>> only be detected by scanning the whole video.
>> 
>> The suggestion of bits_per_raw_sample sounds interesting. What would
>> that look like in practice?
> 
> The decoder would set bits_per_raw_sample on every keyframe. It and
> resolution might change during the course of a ZMBV file, as explained
> above.
> 
> I think frivolously changing the bitdepth is a bad idea, at least
> inside the encoder. If the encoder is fed with changing bitdepths then
> it's a different story.
> 
>>> Just a small thing to be clear: ZMBV_ENABLE_24BPP is not defined
>>> anywhere, so we're free to do however we want with it. It's not
>>> going
>>> to break anyone's workflow unless they were foolish enough to
>>> encode
>>> 24-bit ZMBVs outside of the non-existing spec.
>> 
>> True. But they might be understandably puzzled if they encode as 24-
>> bit, and then find the channels swapped when they decode.
> 
> Yeah, that's why we'd need to get this nailed down
> 
>> Thanks for writing the email to the DOSBox crew.
>> If they choose a channel order, then we have good grounds for fixing
>> the encoder (if need be), and implementing the decoder in the same
>> way.
>> It occurs to me that they might (in theory) also want to specify 2/4
>> byte alignment on RGB, like with the MVs.  My gut says there’d be
>> very little benefit though, and it would only be seen with strange
>> video / block widths.
>> 
>> It also occurs to me this will may warrant a version bump in the
>> format, to give an easy error case for decoders that don’t expect it.
>> Particularly if our decoder has to redefine its channel order.
> 
> Are there any decoders besides ours and dosbox's?
I don’t know of any public implementations.

(That said, I have written a stand-alone tool that encodes/decodes ZMBV, but I’ve not published it anywhere. It’s based heavily on the DOSBox implementation.)
> It also turns out the creator of this codec is Harekiet, who hangs out
> in #revision on IRCnet
Ah ok, do you know him?
It sounds like he’s not concerned too much about what direction we take the format in. But I guess anything we implement may not get made official unless DOSBox adds decoding support.

By the way, I’m happy for this patch to be committed as-is (possibly without the extra note on unsupported bit depths, if that causes any issues). Any new additions outside the existing spec would warrant a new patch I think.

Matthew
Tomas Härdin March 14, 2019, 10:12 a.m. UTC | #11
ons 2019-03-13 klockan 17:46 +0000 skrev Matthew Fearnley:
> > > > On 12 Mar 2019, at 11:46, Tomas Härdin <tjoppen@acc.umu.se> wrote:
> > 
> > tis 2019-03-12 klockan 10:27 +0000 skrev Matthew Fearnley:
> > > > 
> > > It occurs to me that adding sub-8bpp has some implications:
> > > 
> > > My current understanding (I could be wrong) is that FFmpeg tends to
> > > detect the pix_fmt based on the first frame. If FFmpeg detects the
> > > first frame as e.g. PAL4, and chooses that as its output, that means
> > > the rest of the video will have to be encodable as PAL4, otherwise it
> > > (obviously) won’t be encoded properly.
> > > 
> > > So adding a PAL4 format puts a new constraint on encoders (inside and
> > > outside FFmpeg) to not encode frames in a way that looks like PAL4,
> > > unless the whole video will be encodable that way.
> > 
> > Yes, FFmpeg will probe the initial format of the video and audio.
> > Nothing says these are constant. There are FATE samples specifically
> > for files that change resolution. Since ZMBV is a DOS capture codec,
> > and DOS programs frequently change resolution and colordepth, this is
> > indeed something we have to think about. Example: DOS boots into mode
> > 3h, 80x25 16-color text. A recording may start in this mode, then
> > switch to mode 13h (320x200 256 colors graphics), if I understand the
> > format correctly.
> 
> DOSBox actually avoids this issue by outputting to a new file
> whenever it detects a change in colour depth, dimensions or FPS. So
> in practice, these remain constant over a single video.

Ah, convenient

> I’ve tried stitching two AVI files together (palette + RGB, same
> dimensions), and found that if a palette-capable format like ‘png’ is
> chosen, all of the resulting AVI file is encoded with a palette, and
> RGB sections are dithered.  That possibly suggests FFmpeg doesn’t
> like changing format halfway through videos, for AVI at least?

It probably implies AVI can't change palette, or that FFmpeg's support
for changing palettes is spotty at best

> > > Thanks for writing the email to the DOSBox crew.
> > > If they choose a channel order, then we have good grounds for fixing
> > > the encoder (if need be), and implementing the decoder in the same
> > > way.
> > > It occurs to me that they might (in theory) also want to specify 2/4
> > > byte alignment on RGB, like with the MVs.  My gut says there’d be
> > > very little benefit though, and it would only be seen with strange
> > > video / block widths.
> > > 
> > > It also occurs to me this will may warrant a version bump in the
> > > format, to give an easy error case for decoders that don’t expect it.
> > > Particularly if our decoder has to redefine its channel order.
> > 
> > Are there any decoders besides ours and dosbox's?
> 
> I don’t know of any public implementations.
> 
> (That said, I have written a stand-alone tool that encodes/decodes
> ZMBV, but I’ve not published it anywhere. It’s based heavily on the
> DOSBox implementation.)
> > It also turns out the creator of this codec is Harekiet, who hangs
> > out
> > in #revision on IRCnet
> 
> Ah ok, do you know him?
> It sounds like he’s not concerned too much about what direction we
> take the format in. But I guess anything we implement may not get
> made official unless DOSBox adds decoding support.

I suspect he'd be more than happy handing the ZMBV reins to you. Since
24-bit mode isn't official yet I'd say there's room to make
improvements to the format itself, via that flag field. There's also
that unused bit in every MV, which together with the other bit gives
you the possibility of two extra modes beyond copy and XOR. 1-byte MVs
might also be something to investigate, especially for smaller
blocksizes (say 4x4)

> By the way, I’m happy for this patch to be committed as-is (possibly
> without the extra note on unsupported bit depths, if that causes any
> issues). Any new additions outside the existing spec would warrant a
> new patch I think.

Oh yeah, We've gotten a bit sidetracked with this 24-bit business :)
I'll push it once FATE passes. Should be an hour or so.

/Tomas
Tomas Härdin March 14, 2019, 11:25 a.m. UTC | #12
tor 2019-03-14 klockan 11:12 +0100 skrev Tomas Härdin:
> ons 2019-03-13 klockan 17:46 +0000 skrev Matthew Fearnley:
> > > > > 
> > By the way, I’m happy for this patch to be committed as-is (possibly
> > without the extra note on unsupported bit depths, if that causes any
> > issues). Any new additions outside the existing spec would warrant a
> > new patch I think.
> 
> Oh yeah, We've gotten a bit sidetracked with this 24-bit business :)
> I'll push it once FATE passes. Should be an hour or so.

Pushed.

/Tomas
diff mbox

Patch

diff --git a/libavcodec/zmbvenc.c b/libavcodec/zmbvenc.c
index c9d50b6adf..98029de5f6 100644
--- a/libavcodec/zmbvenc.c
+++ b/libavcodec/zmbvenc.c
@@ -34,11 +34,29 @@ 
 
 #include <zlib.h>
 
+/* Frame header flags */
 #define ZMBV_KEYFRAME 1
 #define ZMBV_DELTAPAL 2
 
+/* Motion block width/height (maximum allowed value is 255)
+ * Note: histogram datatype in block_cmp() must be big enough to hold values
+ * up to (4 * ZMBV_BLOCK * ZMBV_BLOCK)
+ */
 #define ZMBV_BLOCK 16
 
+/* Keyframe header format values */
+enum ZmbvFormat {
+    ZMBV_FMT_NONE  = 0,
+    ZMBV_FMT_1BPP  = 1,
+    ZMBV_FMT_2BPP  = 2,
+    ZMBV_FMT_4BPP  = 3,
+    ZMBV_FMT_8BPP  = 4,
+    ZMBV_FMT_15BPP = 5,
+    ZMBV_FMT_16BPP = 6,
+    ZMBV_FMT_24BPP = 7,
+    ZMBV_FMT_32BPP = 8
+};
+
 /**
  * Encoder context
  */
@@ -53,9 +71,11 @@  typedef struct ZmbvEncContext {
     int pstride;
     int comp_size;
     int keyint, curfrm;
+    int bypp;
+    enum ZmbvFormat fmt;
     z_stream zstream;
 
-    int score_tab[ZMBV_BLOCK * ZMBV_BLOCK + 1];
+    int score_tab[ZMBV_BLOCK * ZMBV_BLOCK * 4 + 1];
 } ZmbvEncContext;
 
 
@@ -69,10 +89,11 @@  static inline int block_cmp(ZmbvEncContext *c, uint8_t *src, int stride,
     int sum = 0;
     int i, j;
     uint16_t histogram[256] = {0};
+    int bw_bytes = bw * c->bypp;
 
     /* Build frequency histogram of byte values for src[] ^ src2[] */
     for(j = 0; j < bh; j++){
-        for(i = 0; i < bw; i++){
+        for(i = 0; i < bw_bytes; i++){
             int t = src[i] ^ src2[i];
             histogram[t]++;
         }
@@ -81,7 +102,7 @@  static inline int block_cmp(ZmbvEncContext *c, uint8_t *src, int stride,
     }
 
     /* If not all the xored values were 0, then the blocks are different */
-    *xored = (histogram[0] < bw * bh);
+    *xored = (histogram[0] < bw_bytes * bh);
 
     /* Exit early if blocks are equal */
     if (!*xored) return 0;
@@ -114,7 +135,7 @@  static int zmbv_me(ZmbvEncContext *c, uint8_t *src, int sstride, uint8_t *prev,
 
     /* Try previous block's MV (if not 0,0) */
     if (mx0 || my0){
-        tv = block_cmp(c, src, sstride, prev + mx0 + my0 * pstride, pstride, bw, bh, &txored);
+        tv = block_cmp(c, src, sstride, prev + mx0 * c->bypp + my0 * pstride, pstride, bw, bh, &txored);
         if(tv < bv){
             bv = tv;
             *mx = mx0;
@@ -129,7 +150,7 @@  static int zmbv_me(ZmbvEncContext *c, uint8_t *src, int sstride, uint8_t *prev,
         for(dx = -c->lrange; dx <= c->urange; dx++){
             if(!dx && !dy) continue; // we already tested this block
             if(dx == mx0 && dy == my0) continue; // this one too
-            tv = block_cmp(c, src, sstride, prev + dx + dy * pstride, pstride, bw, bh, &txored);
+            tv = block_cmp(c, src, sstride, prev + dx * c->bypp + dy * pstride, pstride, bw, bh, &txored);
             if(tv < bv){
                  bv = tv;
                  *mx = dx;
@@ -165,9 +186,10 @@  FF_DISABLE_DEPRECATION_WARNINGS
     avctx->coded_frame->key_frame = keyframe;
 FF_ENABLE_DEPRECATION_WARNINGS
 #endif
-    chpal = !keyframe && memcmp(p->data[1], c->pal2, 1024);
 
-    palptr = (uint32_t*)p->data[1];
+    palptr = (avctx->pix_fmt == AV_PIX_FMT_PAL8) ? (uint32_t *)p->data[1] : NULL;
+    chpal = !keyframe && palptr && memcmp(palptr, c->pal2, 1024);
+
     src = p->data[0];
     prev = c->prev;
     if(chpal){
@@ -181,19 +203,21 @@  FF_ENABLE_DEPRECATION_WARNINGS
             c->pal[i * 3 + 1] = tpal[1];
             c->pal[i * 3 + 2] = tpal[2];
         }
-        memcpy(c->pal2, p->data[1], 1024);
+        memcpy(c->pal2, palptr, 1024);
     }
     if(keyframe){
-        for(i = 0; i < 256; i++){
-            AV_WB24(c->pal+(i*3), palptr[i]);
+        if (palptr){
+            for(i = 0; i < 256; i++){
+                AV_WB24(c->pal+(i*3), palptr[i]);
+            }
+            memcpy(c->work_buf, c->pal, 768);
+            memcpy(c->pal2, palptr, 1024);
+            work_size = 768;
         }
-        memcpy(c->work_buf, c->pal, 768);
-        memcpy(c->pal2, p->data[1], 1024);
-        work_size = 768;
         for(i = 0; i < avctx->height; i++){
-            memcpy(c->work_buf + work_size, src, avctx->width);
+            memcpy(c->work_buf + work_size, src, avctx->width * c->bypp);
             src += p->linesize[0];
-            work_size += avctx->width;
+            work_size += avctx->width * c->bypp;
         }
     }else{
         int x, y, bh2, bw2, xored;
@@ -212,16 +236,16 @@  FF_ENABLE_DEPRECATION_WARNINGS
             for(x = 0; x < avctx->width; x += ZMBV_BLOCK, mv += 2) {
                 bw2 = FFMIN(avctx->width - x, ZMBV_BLOCK);
 
-                tsrc = src + x;
-                tprev = prev + x;
+                tsrc = src + x * c->bypp;
+                tprev = prev + x * c->bypp;
 
                 zmbv_me(c, tsrc, p->linesize[0], tprev, c->pstride, x, y, &mx, &my, &xored);
                 mv[0] = (mx << 1) | !!xored;
                 mv[1] = my << 1;
-                tprev += mx + my * c->pstride;
+                tprev += mx * c->bypp + my * c->pstride;
                 if(xored){
                     for(j = 0; j < bh2; j++){
-                        for(i = 0; i < bw2; i++)
+                        for(i = 0; i < bw2 * c->bypp; i++)
                             c->work_buf[work_size++] = tsrc[i] ^ tprev[i];
                         tsrc += p->linesize[0];
                         tprev += c->pstride;
@@ -236,7 +260,7 @@  FF_ENABLE_DEPRECATION_WARNINGS
     src = p->data[0];
     prev = c->prev;
     for(i = 0; i < avctx->height; i++){
-        memcpy(prev, src, avctx->width);
+        memcpy(prev, src, avctx->width * c->bypp);
         prev += c->pstride;
         src += p->linesize[0];
     }
@@ -267,7 +291,7 @@  FF_ENABLE_DEPRECATION_WARNINGS
         *buf++ = 0; // hi ver
         *buf++ = 1; // lo ver
         *buf++ = 1; // comp
-        *buf++ = 4; // format - 8bpp
+        *buf++ = c->fmt; // format
         *buf++ = ZMBV_BLOCK; // block width
         *buf++ = ZMBV_BLOCK; // block height
     }
@@ -303,12 +327,34 @@  static av_cold int encode_init(AVCodecContext *avctx)
     int lvl = 9;
     int prev_size, prev_offset;
 
+    switch (avctx->pix_fmt) {
+    case AV_PIX_FMT_PAL8:
+        c->fmt = ZMBV_FMT_8BPP;
+        c->bypp = 1;
+        break;
+    case AV_PIX_FMT_RGB555LE:
+        c->fmt = ZMBV_FMT_15BPP;
+        c->bypp = 2;
+        break;
+    case AV_PIX_FMT_RGB565LE:
+        c->fmt = ZMBV_FMT_16BPP;
+        c->bypp = 2;
+        break;
+    case AV_PIX_FMT_BGR0:
+        c->fmt = ZMBV_FMT_32BPP;
+        c->bypp = 4;
+        break;
+    default:
+        av_log(avctx, AV_LOG_INFO, "unsupported pixel format\n");
+        return AVERROR(EINVAL);
+    }
+
     /* Entropy-based score tables for comparing blocks.
      * Suitable for blocks up to (ZMBV_BLOCK * ZMBV_BLOCK) bytes.
      * Scores are nonnegative, lower is better.
      */
-    for(i = 1; i <= ZMBV_BLOCK * ZMBV_BLOCK; i++)
-        c->score_tab[i] = -i * log2(i / (double)(ZMBV_BLOCK * ZMBV_BLOCK)) * 256;
+    for(i = 1; i <= ZMBV_BLOCK * ZMBV_BLOCK * c->bypp; i++)
+        c->score_tab[i] = -i * log2(i / (double)(ZMBV_BLOCK * ZMBV_BLOCK * c->bypp)) * 256;
 
     c->avctx = avctx;
 
@@ -331,7 +377,7 @@  static av_cold int encode_init(AVCodecContext *avctx)
 
     // Needed if zlib unused or init aborted before deflateInit
     memset(&c->zstream, 0, sizeof(z_stream));
-    c->comp_size = avctx->width * avctx->height + 1024 +
+    c->comp_size = avctx->width * c->bypp * avctx->height + 1024 +
         ((avctx->width + ZMBV_BLOCK - 1) / ZMBV_BLOCK) * ((avctx->height + ZMBV_BLOCK - 1) / ZMBV_BLOCK) * 2 + 4;
     if (!(c->work_buf = av_malloc(c->comp_size))) {
         av_log(avctx, AV_LOG_ERROR, "Can't allocate work buffer.\n");
@@ -355,8 +401,8 @@  static av_cold int encode_init(AVCodecContext *avctx)
      * - The first row should also be padded with `lrange` pixels before, then
      *   aligned up to a multiple of 16 bytes.
      */
-    c->pstride = FFALIGN(avctx->width + c->lrange, 16);
-    prev_size = FFALIGN(c->lrange, 16) + c->pstride * (c->lrange + avctx->height + c->urange);
+    c->pstride = FFALIGN((avctx->width + c->lrange) * c->bypp, 16);
+    prev_size = FFALIGN(c->lrange * c->bypp, 16) + c->pstride * (c->lrange + avctx->height + c->urange);
     prev_offset = FFALIGN(c->lrange, 16) + c->pstride * c->lrange;
     if (!(c->prev_buf = av_mallocz(prev_size))) {
         av_log(avctx, AV_LOG_ERROR, "Can't allocate picture.\n");
@@ -385,5 +431,9 @@  AVCodec ff_zmbv_encoder = {
     .init           = encode_init,
     .encode2        = encode_frame,
     .close          = encode_end,
-    .pix_fmts       = (const enum AVPixelFormat[]){ AV_PIX_FMT_PAL8, AV_PIX_FMT_NONE },
+    .pix_fmts       = (const enum AVPixelFormat[]) { AV_PIX_FMT_PAL8,
+                                                     AV_PIX_FMT_RGB555LE,
+                                                     AV_PIX_FMT_RGB565LE,
+                                                     AV_PIX_FMT_BGR0,
+                                                     AV_PIX_FMT_NONE },
 };