diff mbox

[FFmpeg-devel,3/3] avformat: set the default whitelist to disable hls

Message ID 20170601114446.11358-3-michael@niedermayer.cc
State New
Headers show

Commit Message

Michael Niedermayer June 1, 2017, 11:44 a.m. UTC
This prevents an exploit leading to an information leak

The existing exploit depends on a specific decoder as well.
It does appear though that the exploit should be possible with any decoder.
The problem is that as long as sensitive information gets into the decoder,
the output of the decoder becomes sensitive as well.
The only obvious solution is to prevent access to sensitive information. Or to
disable hls or possibly some of its feature. More complex solutions like
checking the path to limit access to only subdirectories of the hls path may
work as an alternative. But such solutions are fragile and tricky to implement
portably and would not stop every possible attack nor would they work with all
valid hls files.

Found-by: Emil Lerner and Pavel Cheremushkin
Reported-by: Thierry Foucu <tfoucu@google.com>

Fix inspired by: Tobias Rapp <t.rapp@noa-archive.com>

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavformat/options_table.h | 2 +-
 libavformat/utils.c         | 6 +++++-
 tests/fate/avformat.mak     | 4 ++--
 tests/fate/filter-audio.mak | 4 ++--
 4 files changed, 10 insertions(+), 6 deletions(-)

Comments

Tobias Rapp June 1, 2017, 12:48 p.m. UTC | #1
On 01.06.2017 13:44, Michael Niedermayer wrote:
> This prevents an exploit leading to an information leak
>
> The existing exploit depends on a specific decoder as well.
> It does appear though that the exploit should be possible with any decoder.
> The problem is that as long as sensitive information gets into the decoder,
> the output of the decoder becomes sensitive as well.
> The only obvious solution is to prevent access to sensitive information. Or to
> disable hls or possibly some of its feature. More complex solutions like
> checking the path to limit access to only subdirectories of the hls path may
> work as an alternative. But such solutions are fragile and tricky to implement
> portably and would not stop every possible attack nor would they work with all
> valid hls files.
>
> Found-by: Emil Lerner and Pavel Cheremushkin
> Reported-by: Thierry Foucu <tfoucu@google.com>
>
> Fix inspired by: Tobias Rapp <t.rapp@noa-archive.com>
>
> Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> ---
>  libavformat/options_table.h | 2 +-
>  libavformat/utils.c         | 6 +++++-
>  tests/fate/avformat.mak     | 4 ++--
>  tests/fate/filter-audio.mak | 4 ++--
>  4 files changed, 10 insertions(+), 6 deletions(-)
>
>[...]

I would prefer this patch over the one in 
http://ffmpeg.org/pipermail/ffmpeg-devel/2017-May/211796.html

Regards,
Tobias
Nicolas George June 2, 2017, 7:15 a.m. UTC | #2
Le tridi 13 prairial, an CCXXV, Michael Niedermayer a écrit :
> This prevents an exploit leading to an information leak
> 
> The existing exploit depends on a specific decoder as well.
> It does appear though that the exploit should be possible with any decoder.
> The problem is that as long as sensitive information gets into the decoder,
> the output of the decoder becomes sensitive as well.
> The only obvious solution is to prevent access to sensitive information. Or to
> disable hls or possibly some of its feature. More complex solutions like
> checking the path to limit access to only subdirectories of the hls path may
> work as an alternative. But such solutions are fragile and tricky to implement
> portably and would not stop every possible attack nor would they work with all
> valid hls files.

When you first introduced this protocol white-list to fix a problem with
the concat protocol, I warned that it was not a good fix, it was only
hiding the tip of the iceberg, and in a disruptive way, we would get
other similar issues and trouble brought by the white-list itself. I
told that we should start thinking how to design a proper data
compartmentalization mechanism.

Now, one year and a half after the feature was introduced, the protocol
white-list broke the output of dvd2concat, possibly other features, we
have other security vulnerabilities that it does not fix, and we are
about to disable a feature that, judging by the number of users
questions, many people use.

And we are nowhere near designing a proper data compartmentalization
mechanism.

Notice a pattern?

Regards,
Michael Niedermayer June 2, 2017, noon UTC | #3
On Fri, Jun 02, 2017 at 09:15:25AM +0200, Nicolas George wrote:
> Le tridi 13 prairial, an CCXXV, Michael Niedermayer a écrit :
> > This prevents an exploit leading to an information leak
> > 
> > The existing exploit depends on a specific decoder as well.
> > It does appear though that the exploit should be possible with any decoder.
> > The problem is that as long as sensitive information gets into the decoder,
> > the output of the decoder becomes sensitive as well.
> > The only obvious solution is to prevent access to sensitive information. Or to
> > disable hls or possibly some of its feature. More complex solutions like
> > checking the path to limit access to only subdirectories of the hls path may
> > work as an alternative. But such solutions are fragile and tricky to implement
> > portably and would not stop every possible attack nor would they work with all
> > valid hls files.
> 
> When you first introduced this protocol white-list to fix a problem with
> the concat protocol, I warned that it was not a good fix, it was only
> hiding the tip of the iceberg, and in a disruptive way, we would get
> other similar issues and trouble brought by the white-list itself. I
> told that we should start thinking how to design a proper data
> compartmentalization mechanism.
> 
> Now, one year and a half after the feature was introduced, the protocol
> white-list broke the output of dvd2concat, possibly other features, we
> have other security vulnerabilities that it does not fix, and we are
> about to disable a feature that, judging by the number of users
> questions, many people use.
> 
> And we are nowhere near designing a proper data compartmentalization
> mechanism.
> 
> Notice a pattern?

yes
Security issues are found, i post a fix and people complain,
I ask what solution is prefered and if they want to provide an
alternative fix and get no reply. I post a different fix and other
people complain.

If you knew a year and a half ago about a security issue and about a
great solution to it.
How far is it from completion ?
does this cover the hls vulnerability we discussed in
the last 2 days and Can you post a patch ?

If your solution is just an idea and not implemented after 1 and a half
years then iam not sure its a viable solution, given that security
issues should be fixed quickly.

But the real question still is, how do people prefer us to deal with
this security issue here?

Disable hls by default via whitelist (as in this patch)?
Disable local files in hls by default (as in the previous patch)
Do you or someone else want to submit an alternative fix ?

I of course will only push a patch if people agree to it, thats how
our patch reviews work.
ATM both patches have replies of people complaining with few or
no replies supporting them. And no alternative fixes posted.

thanks

[...]
Nicolas George June 4, 2017, 10:46 a.m. UTC | #4
Le quartidi 14 prairial, an CCXXV, Michael Niedermayer a écrit :
> > Notice a pattern?
> yes
> Security issues are found, i post a fix and people complain,

No. The pattern is: you rush to produce a bad fix.

> If you knew a year and a half ago about a security issue and about a
> great solution to it.
> How far is it from completion ?
> does this cover the hls vulnerability we discussed in
> the last 2 days and Can you post a patch ?

I said that WE needed to look for a solution. We, collective.

I, individual, do not have a solution, I only know that one exists
(Perl, Windows, web browsers all have a similar mechanism) and that
"fixing" the individual issues rather than designing a global solution
is a waste of time.

> But the real question still is, how do people prefer us to deal with
> this security issue here?

This one ? Ignore it but take the opportunity to start designingⁿ: a
proper solution would fix it anyway.

If you do anything else, I will not object to you pushing, but only if
you add "--author=Sysiphus" to your git commit command.

Regards,
Michael Niedermayer June 5, 2017, 12:57 a.m. UTC | #5
Hi

On Sun, Jun 04, 2017 at 12:46:18PM +0200, Nicolas George wrote:
> Le quartidi 14 prairial, an CCXXV, Michael Niedermayer a écrit :
> > > Notice a pattern?
> > yes
> > Security issues are found, i post a fix and people complain,
> 
> No. The pattern is: you rush to produce a bad fix.

thats "ad hominem"

If theres an issue in a change, the center of the discussion should be
the issue so it can be improved.

looking at what you wrote, iam not even sure if you talk
about whitelists, some patch here or something totally different that
you call bad. Its just obvious at who you point not what you talk about
or what you see bad in it.

and that person (being me) is heavily constrained by the wishes of
the rest of the team.

also security issues need to be fixed quickly, the quick fix to
stop an issue and the solution we work toward in the long term
can be very different and a quick fix in the most general sense is
likely quite shit compared to a long term solution.

Still we own our users fixing sec issues quickly, its us who wrote
the vulnerable mess in the first place. We should not let them wait
until we design and implement the perfect long term solution.


> 
> > If you knew a year and a half ago about a security issue and about a
> > great solution to it.
> > How far is it from completion ?
> > does this cover the hls vulnerability we discussed in
> > the last 2 days and Can you post a patch ?
> 
> I said that WE needed to look for a solution. We, collective.
> 
> I, individual, do not have a solution, I only know that one exists
> (Perl, Windows, web browsers all have a similar mechanism) and that
> "fixing" the individual issues rather than designing a global solution
> is a waste of time.

Iam happy to help and work together with you to design and implement
this. Iam not sure what you have in mind though exactly and iam not
sure if its able to fix this.

Can you please explain what you have in mind ?


> 
> > But the real question still is, how do people prefer us to deal with
> > this security issue here?
> 
> This one ? Ignore it but take the opportunity to start designingⁿ: a
> proper solution would fix it anyway.
> 
> If you do anything else, I will not object to you pushing, but only if
> you add "--author=Sysiphus" to your git commit command.

You dont need to convince me that the extension check or changes
within just hls are not a complete solution. Iam quite well aware
of this. This is intended to stop an existing exploit and variants of
it in practice and do so quickly.

A complete solution will also likely add some inconvenience that
some developers object to. I feel that the security outweighs the
inconvenience but others object to it.

Its not the first issue with hls and it likely wont be the last, I
think --author=Sysiphus is quite fitting in fact. Also its really
a change guided by peoples objections ...

Thanks

[...]
Nicolas George June 5, 2017, 3:33 p.m. UTC | #6
Le septidi 17 prairial, an CCXXV, Michael Niedermayer a écrit :
> thats "ad hominem"

I am sorry, I did not realize it was, please forgive me and allow me to
reformulate.

The pattern is: someone is made aware of a minor security exploit in
parts of the code not their direct responsibility. Nonetheless, that
someone rushes and designs and implements a fix that only blocks that
particular exploit and its close cousins without fixing the actual
underlying security issue, and that has a lot of drawbacks on top of
that. I think I am allowed to call that a "bad fix".

We had two instances quoted in this discussion. I think there were
others (possibly not security related) but I cannot pinpoint them.

In both cases, the someone was you. This is not ad-hominem, just a fact,
and part of the pattern.

Maybe you have a tendency to behave that way, but that is not for me to
say, and it would really be ad-hominem. Let us not talk about it.

Maybe your special place in the project pushes you to doing that. I hope
you agree this is not ad-hominem. Maybe ad-positionem.

Maybe people notice it more when you are doing it.

Maybe it was just a coincidence.

Maybe a little bit of each.

Anyway, let us agree to all try and avoid that pattern and not dwell on
it.

> You dont need to convince me that the extension check or changes
> within just hls are not a complete solution. Iam quite well aware
> of this. This is intended to stop an existing exploit and variants of
> it in practice and do so quickly.

It depends on the severity of the threat. This one seems quite minor and
far-fetched, and thus I think we could take our time to fix it properly.
We all have noticed that temporary quick-and-dirty fixes usually stay
here a long time unless whoever implemented them is actively working on
a real fix.

Now, as I said, I have no special knowledge making me a good candidate
for working on a fix. I know how Perl's taint check works, but it will
not suit our needs. I know vaguely how Firefox avoids cross-site
information leak, but not in enough detail to serve as a base for a
design. I know nothing about Windows security zones.

Still, after having spent a few hours in a train without proper network
access, and with someone actually interested in listening, I could give
it some thought.

I will start by explaining why protocol_whitelist is bad.

First, it is a string. Enough with strings used as data structures! That
is inefficient, inconvenient and a maintenance nightmare since extending
or fixing the syntax will likely be considered an API break by some.

Second, the issue never was about protocols. ~/tmp/playlist.m3u8 should
not be allowed to access ../.firefox/cookies.txt (Firefox protects us
from that by adding a salt in the path, but not all sensitive files are
protected that way, this is only an illustrative example), and that
applies to local files, to SMB shares or to SFTP clients;
http://www.example.com/playlist.m3u8 should not be allowed to access
http://localhost:631/, yet they are both HTTP.

The issue is about subsets of the URL space. Files from one URL should
be allowed to access data from URLs in the same relevant subset (same
subdirectory or same web server maybe?), but not outside.

Of course, the protocol is part of the URL. But not all protocols are
like that: file://, http://, ssh:// are part of the URL, while concat:,
crypto:, subfile: are not and should be transparent for the mechanism. I
will call the formers "URL protocols" and the laters "utility
protocols".

So let us get rid of that protocol_whitelist and replace it with an
actual data structure, designed to encode a subset of the URL space. Let
us call it AVSecurityClearance, at least for the rest of this mail.

Now, there is something done right with protocol_whitelist: it is, as it
should be, a property of muxers, demuxers and protocols (and anything
else that would be allowed to access external data) and must be
inherited when a component opens another component (demuxer -> protocol,
or demuxer -> slave demuxer, or protocol -> slave protocol, etc.).
AVSecurityClearance should work the same way.

Now, how should we implement that data structure? I do not know!

Maybe we can restrict it to subsets that are a sub-hierarchy: "anything
below http://www.example.com/music/".

But I suspect AVSecurityClearance will need to be polymorphic: the
structure of data is not necessarily the same for all protocols.

If AVSecurityClearance is polymorphic, then implementing union and
intersection operators is quite easy.

A few considerations about hierarchy. Some protocols have a host part.
The syntax of DNS domains is hierarchical from right to left, unlike the
syntax for file paths. It must be handled separately. The port part
needs a special treatment too: which ones are closer to
http://example.com/: http://www.example.com/ or
http://example.com:8080/? Or maybe they are to be considered all
separate.

Also, URLs with numerical IP addresses require a special treatment; we
do not want to implement a whois client, so we will have to treat them
as completely separate.

Another design question: when a master opens a slave and the
AVSecurityClearance is inherited, should it be copied, like
protocol_whitelist, or shared? If it is shared, later restrictions
caused by data access in one component are immediately propagated to all
the components. I suspect it is the right choice. But then,
AVSecurityClearance must be refcounted.

This is all I have for now. And I will not have time to work on this
anytime soon. But somebody needs to.

Of course, whoever works on it can come up with their own ideas. But I
suspect any working solution will work in the ways I just outlined.

Regards,
Michael Niedermayer June 6, 2017, 2:59 a.m. UTC | #7
On Mon, Jun 05, 2017 at 05:33:29PM +0200, Nicolas George wrote:
> Le septidi 17 prairial, an CCXXV, Michael Niedermayer a écrit :
[...]
> > You dont need to convince me that the extension check or changes
> > within just hls are not a complete solution. Iam quite well aware
> > of this. This is intended to stop an existing exploit and variants of
> > it in practice and do so quickly.
> 
> It depends on the severity of the threat. This one seems quite minor and
> far-fetched, and thus I think we could take our time to fix it properly.
> We all have noticed that temporary quick-and-dirty fixes usually stay
> here a long time unless whoever implemented them is actively working on
> a real fix.

I disagree that the issue is minor and far fetched.

The exploit that i have was successfully used against multiple
companies (it was a demonstration and AFAIK no harm was done).
That same attack works against all recent releases.

My oppinion was and is that a exploit working basically on 100% of
targets and can leak private information is a serious issue.

I think that if argumentation continues to center around the
lack of importance of this issue then we should try to bring some
outside security experts into this discussion.


[...]
> Still, after having spent a few hours in a train without proper network
> access, and with someone actually interested in listening, I could give
> it some thought.
> 
> I will start by explaining why protocol_whitelist is bad.
> 
> First, it is a string. Enough with strings used as data structures! That
> is inefficient, inconvenient and a maintenance nightmare since extending
> or fixing the syntax will likely be considered an API break by some.
> 
> Second, the issue never was about protocols. ~/tmp/playlist.m3u8 should
> not be allowed to access ../.firefox/cookies.txt (Firefox protects us
> from that by adding a salt in the path, but not all sensitive files are
> protected that way, this is only an illustrative example), and that
> applies to local files, to SMB shares or to SFTP clients;

> http://www.example.com/playlist.m3u8 should not be allowed to access
> http://localhost:631/, yet they are both HTTP.

yes but its worse than this
http://www.example.com:631/playlist.m3u8
and
http://www.example.com:631

can already screw you, if you are not carefull. The reason is the DNS
lookups. these 2 uses can lookup to different IPs, one could be on
the LAN the other the attackers server
If you want to prevent this, I think you need to implement a global
DNS cache. Technically that can be done and i belive browsers are
doing this and doing this for the reason above. iam no browser expert
though it just all fits together ...


> 
> The issue is about subsets of the URL space. Files from one URL should
> be allowed to access data from URLs in the same relevant subset (same
> subdirectory or same web server maybe?), but not outside.

What percentage of hls files out in the wild does this work with ?


> 
> Of course, the protocol is part of the URL. But not all protocols are
> like that: file://, http://, ssh:// are part of the URL, while concat:,
> crypto:, subfile: are not and should be transparent for the mechanism. I
> will call the formers "URL protocols" and the laters "utility
> protocols".
> 
> So let us get rid of that protocol_whitelist and replace it with an
> actual data structure, designed to encode a subset of the URL space. Let
> us call it AVSecurityClearance, at least for the rest of this mail.
> 
> Now, there is something done right with protocol_whitelist: it is, as it
> should be, a property of muxers, demuxers and protocols (and anything
> else that would be allowed to access external data) and must be
> inherited when a component opens another component (demuxer -> protocol,
> or demuxer -> slave demuxer, or protocol -> slave protocol, etc.).
> AVSecurityClearance should work the same way.
> 
> Now, how should we implement that data structure? I do not know!
> 

> Maybe we can restrict it to subsets that are a sub-hierarchy: "anything
> below http://www.example.com/music/".

a file /home/jane/fromweb.hls
should not be allowed to access /home/jane/.ssh/
which would be a sudirectory


> 
> But I suspect AVSecurityClearance will need to be polymorphic: the
> structure of data is not necessarily the same for all protocols.
> 
> If AVSecurityClearance is polymorphic, then implementing union and
> intersection operators is quite easy.
> 
> A few considerations about hierarchy. Some protocols have a host part.
> The syntax of DNS domains is hierarchical from right to left, unlike the
> syntax for file paths. It must be handled separately. The port part
> needs a special treatment too: which ones are closer to
> http://example.com/: http://www.example.com/ or
> http://example.com:8080/? Or maybe they are to be considered all
> separate.
> 
> Also, URLs with numerical IP addresses require a special treatment; we
> do not want to implement a whois client, so we will have to treat them
> as completely separate.
> 
> Another design question: when a master opens a slave and the
> AVSecurityClearance is inherited, should it be copied, like
> protocol_whitelist, or shared? If it is shared, later restrictions
> caused by data access in one component are immediately propagated to all
> the components. I suspect it is the right choice. But then,
> AVSecurityClearance must be refcounted.
> 
> This is all I have for now. And I will not have time to work on this
> anytime soon. But somebody needs to.
> 

> Of course, whoever works on it can come up with their own ideas. But I

> suspect any working solution will work in the ways I just outlined.

yes this sounds plausible.

Though if you dont have time to work on this, i doubt i will work on it
alone anytime soon. I dont want to work on this just to have someone
reject it because it either breaks too many hls files or because of
a global DNS cache if that turns out to be needed ...
Said differently iam happy to help but there need to be more people
than me behind this pushing this forward ...

[...]
Hendrik Leppkes June 6, 2017, 4:55 a.m. UTC | #8
On Tue, Jun 6, 2017 at 4:59 AM, Michael Niedermayer
<michael@niedermayer.cc> wrote:
>>
>> The issue is about subsets of the URL space. Files from one URL should
>> be allowed to access data from URLs in the same relevant subset (same
>> subdirectory or same web server maybe?), but not outside.
>
> What percentage of hls files out in the wild does this work with ?
>
>

If you implement this over HTTP, you also need to implement CORS
(Cross Origin Resouce Sharing), which is a defined mechanism to break
these static rules when you need/want to load content from other
domains.
Many web-based HLS players basically require this so the browser can
access the files.

- Hendrik
wm4 June 6, 2017, 1:47 p.m. UTC | #9
On Tue, 6 Jun 2017 04:59:58 +0200
Michael Niedermayer <michael@niedermayer.cc> wrote:

> I disagree that the issue is minor and far fetched.
> 
> The exploit that i have was successfully used against multiple
> companies (it was a demonstration and AFAIK no harm was done).
> That same attack works against all recent releases.
> 
> My oppinion was and is that a exploit working basically on 100% of
> targets and can leak private information is a serious issue.

Until I see actual proof, I call bullshit. It also might be that there
are better solutions, but we can't know, because you withhold
information.

I'm sick of these "security" fixes, which just make everything
trickier, worse, and not necessarily much more secure.
Marton Balint June 6, 2017, 7:50 p.m. UTC | #10
On Tue, 6 Jun 2017, Michael Niedermayer wrote:

> On Mon, Jun 05, 2017 at 05:33:29PM +0200, Nicolas George wrote:
>> Le septidi 17 prairial, an CCXXV, Michael Niedermayer a écrit :
> [...]
>>> You dont need to convince me that the extension check or changes
>>> within just hls are not a complete solution. Iam quite well aware
>>> of this. This is intended to stop an existing exploit and variants of
>>> it in practice and do so quickly.
>>
>> It depends on the severity of the threat. This one seems quite minor and
>> far-fetched, and thus I think we could take our time to fix it properly.
>> We all have noticed that temporary quick-and-dirty fixes usually stay
>> here a long time unless whoever implemented them is actively working on
>> a real fix.
>
> I disagree that the issue is minor and far fetched.

Do we really want to impelment a whole security framework inside an AV 
library? Can't we decouple this from libav*? E.g. let the user implement 
his security framework via callbacks or something?

We can provide a good enough reference implementation for the command line 
tools (outside the libraries), so things won't break too much, but if you 
ask me, by default, all reference openings should be disabled, that is the 
only truly secure thing, anything else can be insecure based on your use 
case.

Regards,
Marton
diff mbox

Patch

diff --git a/libavformat/options_table.h b/libavformat/options_table.h
index 0c1915d6d4..f33e126838 100644
--- a/libavformat/options_table.h
+++ b/libavformat/options_table.h
@@ -104,7 +104,7 @@  static const AVOption avformat_options[] = {
 {"make_zero",         "shift timestamps so they start at 0",       0, AV_OPT_TYPE_CONST, {.i64 = AVFMT_AVOID_NEG_TS_MAKE_ZERO },         INT_MIN, INT_MAX, E, "avoid_negative_ts"},
 {"dump_separator", "set information dump field separator", OFFSET(dump_separator), AV_OPT_TYPE_STRING, {.str = ", "}, CHAR_MIN, CHAR_MAX, D|E},
 {"codec_whitelist", "List of decoders that are allowed to be used", OFFSET(codec_whitelist), AV_OPT_TYPE_STRING, { .str = NULL },  CHAR_MIN, CHAR_MAX, D },
-{"format_whitelist", "List of demuxers that are allowed to be used", OFFSET(format_whitelist), AV_OPT_TYPE_STRING, { .str = NULL },  CHAR_MIN, CHAR_MAX, D },
+{"format_whitelist", "List of demuxers that are allowed to be used", OFFSET(format_whitelist), AV_OPT_TYPE_STRING, { .str = "-hls,ALL" },  CHAR_MIN, CHAR_MAX, D },
 {"protocol_whitelist", "List of protocols that are allowed to be used", OFFSET(protocol_whitelist), AV_OPT_TYPE_STRING, { .str = NULL },  CHAR_MIN, CHAR_MAX, D },
 {"protocol_blacklist", "List of protocols that are not allowed to be used", OFFSET(protocol_blacklist), AV_OPT_TYPE_STRING, { .str = NULL },  CHAR_MIN, CHAR_MAX, D },
 {"max_streams", "maximum number of streams", OFFSET(max_streams), AV_OPT_TYPE_INT, { .i64 = 1000 }, 0, INT_MAX, D },
diff --git a/libavformat/utils.c b/libavformat/utils.c
index 7dd6084f27..23160a89cc 100644
--- a/libavformat/utils.c
+++ b/libavformat/utils.c
@@ -144,8 +144,9 @@  void av_format_inject_global_side_data(AVFormatContext *s)
 
 int ff_copy_whiteblacklists(AVFormatContext *dst, const AVFormatContext *src)
 {
+    char *old_format_whitelist = dst->format_whitelist; // This has a non NULL default
+
     av_assert0(!dst->codec_whitelist &&
-               !dst->format_whitelist &&
                !dst->protocol_whitelist &&
                !dst->protocol_blacklist);
     dst-> codec_whitelist = av_strdup(src->codec_whitelist);
@@ -157,8 +158,11 @@  int ff_copy_whiteblacklists(AVFormatContext *dst, const AVFormatContext *src)
         || (src->protocol_whitelist && !dst->protocol_whitelist)
         || (src->protocol_blacklist && !dst->protocol_blacklist)) {
         av_log(dst, AV_LOG_ERROR, "Failed to duplicate black/whitelist\n");
+        av_free(dst->format_whitelist);
+        dst->format_whitelist = old_format_whitelist;
         return AVERROR(ENOMEM);
     }
+    av_free(old_format_whitelist);
     return 0;
 }
 
diff --git a/tests/fate/avformat.mak b/tests/fate/avformat.mak
index 82a531c7a5..77021b793e 100644
--- a/tests/fate/avformat.mak
+++ b/tests/fate/avformat.mak
@@ -119,12 +119,12 @@  tests/data/adts-to-mkv-cated-%.mkv: tests/data/adts-to-mkv-header.mkv tests/data
 
 FATE_SEGMENT += fate-segment-mp4-to-ts
 fate-segment-mp4-to-ts: tests/data/mp4-to-ts.m3u8
-fate-segment-mp4-to-ts: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/mp4-to-ts.m3u8 -c copy
+fate-segment-mp4-to-ts: CMD = framecrc -flags +bitexact -format_whitelist ALL -i $(TARGET_PATH)/tests/data/mp4-to-ts.m3u8 -c copy
 FATE_SEGMENT-$(call ALLYES, MOV_DEMUXER H264_MP4TOANNEXB_BSF MPEGTS_MUXER MATROSKA_DEMUXER SEGMENT_MUXER HLS_DEMUXER) += fate-segment-mp4-to-ts
 
 FATE_SEGMENT += fate-segment-adts-to-mkv
 fate-segment-adts-to-mkv: tests/data/adts-to-mkv.m3u8
-fate-segment-adts-to-mkv: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/adts-to-mkv.m3u8 -c copy
+fate-segment-adts-to-mkv: CMD = framecrc -flags +bitexact -format_whitelist ALL -i $(TARGET_PATH)/tests/data/adts-to-mkv.m3u8 -c copy
 fate-segment-adts-to-mkv: REF = $(SRC_PATH)/tests/ref/fate/segment-adts-to-mkv-header-all
 FATE_SEGMENT-$(call ALLYES, AAC_DEMUXER AAC_ADTSTOASC_BSF MATROSKA_MUXER MATROSKA_DEMUXER SEGMENT_MUXER HLS_DEMUXER) += fate-segment-adts-to-mkv
 
diff --git a/tests/fate/filter-audio.mak b/tests/fate/filter-audio.mak
index 5d15b31e0b..58f8a71dfe 100644
--- a/tests/fate/filter-audio.mak
+++ b/tests/fate/filter-audio.mak
@@ -150,7 +150,7 @@  tests/data/hls-list.m3u8: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
 
 FATE_AFILTER-$(call ALLYES, HLS_DEMUXER MPEGTS_MUXER MPEGTS_DEMUXER AEVALSRC_FILTER LAVFI_INDEV MP2FIXED_ENCODER) += fate-filter-hls
 fate-filter-hls: tests/data/hls-list.m3u8
-fate-filter-hls: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/hls-list.m3u8
+fate-filter-hls: CMD = framecrc -flags +bitexact -format_whitelist hls,mpegts -i $(TARGET_PATH)/tests/data/hls-list.m3u8
 
 tests/data/hls-list-append.m3u8: TAG = GEN
 tests/data/hls-list-append.m3u8: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
@@ -164,7 +164,7 @@  tests/data/hls-list-append.m3u8: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
 
 FATE_AFILTER-$(call ALLYES, HLS_DEMUXER MPEGTS_MUXER MPEGTS_DEMUXER AEVALSRC_FILTER LAVFI_INDEV MP2FIXED_ENCODER) += fate-filter-hls-append
 fate-filter-hls-append: tests/data/hls-list-append.m3u8
-fate-filter-hls-append: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/hls-list-append.m3u8 -af asetpts=RTCTIME
+fate-filter-hls-append: CMD = framecrc -flags +bitexact -format_whitelist hls,mpegts -i $(TARGET_PATH)/tests/data/hls-list-append.m3u8 -af asetpts=RTCTIME
 
 FATE_AMIX += fate-filter-amix-simple
 fate-filter-amix-simple: CMD = ffmpeg -filter_complex amix -i $(SRC) -ss 3 -i $(SRC1) -f f32le -