From patchwork Fri Aug 23 23:03:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niki Bowe X-Patchwork-Id: 14684 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id B2AE4448880 for ; Sat, 24 Aug 2019 02:08:57 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9745E68ABA6; Sat, 24 Aug 2019 02:08:57 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0AD2F68AB83 for ; Sat, 24 Aug 2019 02:08:52 +0300 (EEST) Received: by mail-wm1-f66.google.com with SMTP id g67so10512784wme.1 for ; Fri, 23 Aug 2019 16:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=iBGwpcbA3dB4zwCRWR02AkL3oCW/CoScOd8PnjmOYig=; b=HpAF9prXIlLz8iF7Rctwz7LAGwH3AVg8alWKKhQp9x2mrxPGmNlCH+bVCCXNZ+MfmT aeWCXeCgXtlWR2RWWiYEsSi/WNbT27a0xNz1MJ14sSvZr8uz26pp92OF44Wjp4QwMDJ1 t797X147YoWFftrf6wSmHJPYp/syrlTBilMW+cc4UbT4a9KT5dzKoIhEBAAzLgCfcsIR 4iDuKAuqxbsCWsclriyuT9eJPAgjDvWLAxkumlKa4HnRDJbnKB6XSIKpR6dprhlPJU8A IaBzsTrarR7Qq4LqnWYZZklYKPAttmuv1uGkq7AJu9ivnknEDL7xxvmAzaPhB9dzfiHT YlZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=iBGwpcbA3dB4zwCRWR02AkL3oCW/CoScOd8PnjmOYig=; b=aiSu52QnA0MdvYrW/xTvrYBlOcNSONnMq42QngWUNL8SEWeguojlTY3dzASDxFKMLE 3+XlXbv/0L4tCak7/nn9NOn1UmvhLkjQmbURUSF/Kdysawz9rBHx4Uv0Ixm7/OBc6pSr LuMIeYajL9nI5UVB/H2H0IwCjXp8vcl17romc6SZ/XtMOa9aRwq8Gz0FtxLjWZEn4R8j Hr7pUm2SB7bKJ+m8WUYvhZAPslnOQ0cHo5tiIDiGIk3IEIjJzrJgwx5+PCylUX1flMsj 4R2+nnwPyx982dCHz/fcy0buIVph9E7Pmc3+ylpsNq0Qm1F/qJMCChzMfKDHXDoB5sEG IqYA== X-Gm-Message-State: APjAAAWrlayHlNUO06LjulyyQzYUPopoekOp2FCeATOFMUHHOLdeTgfg k+uaCtODMZOK4f9ZrtI4FqT8mtpzj767KcNhR0wRSAKTz5cHmw== X-Google-Smtp-Source: APXvYqziJmXpYzck5FuLm5tOKLmKYLUwjJhTi/y+rbYE+Fg3R/5Nh8OUr2nmDZipY48HOR2lpOCFajCkIxGMbSm601Q= X-Received: by 2002:a1c:2e8d:: with SMTP id u135mr7694879wmu.101.1566601402658; Fri, 23 Aug 2019 16:03:22 -0700 (PDT) MIME-Version: 1.0 References: <20190820013648.161805-1-nbowe@google.com> In-Reply-To: From: Niki Bowe Date: Fri, 23 Aug 2019 16:03:10 -0700 Message-ID: To: FFmpeg development discussions and patches X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: Re: [FFmpeg-devel] [PATCH] avformat: Fix probing on some JPEGs X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On Thu, Aug 22, 2019 at 2:30 AM Paul B Mahol wrote: > On Thu, Aug 22, 2019 at 11:19 AM Carl Eugen Hoyos > wrote: > > > Am Mi., 21. Aug. 2019 um 23:05 Uhr schrieb Niki Bowe > > : > > > > > > On Mon, Aug 19, 2019 at 7:22 PM Carl Eugen Hoyos > > wrote: > > > > > > > > > > > This score would mean that mjpeg can never be detected. > > > > I suspect you have to reduce one of the demuxers to "- 1". > > > > > > > > > > > Thanks Carl. > > > Attached patch to reduce mpeg probe by -1, which also fixes the issue. > > > > Sorry, I misread the original report, it looked to me as if mJpeg was > > the culprit. > > > > Imo, the mpeg probing should be fixed (return a smaller value) for your > > sample > > by detecting that it is not mpeg, not by returning a smaller value for > > all samples. > > > > 1000% agree. > > It didn't return a smaller value for all samples, only the "invalid VDR files and short PES streams" case. Most mpegps files still return 26 immediately, because they have pack headers. However, here is another patch where I try to limit it to only changing score for these jpegs. I noticed that these jpegs had a lot of 0x00000100 sequences, which matches mpeg picture header start code. I added another heuristic which matches these jpegs, but not any mpegps files I could find. Alternatively I could make reduce score if it doesn't start with a start code? At the moment its happy to search until it finds start codes. Is everyone really sure the best approach is to modify mpegps_probe for this? The mpegps_probe function returns 25 in many instances where it may not be mpegps. It does only minimal structural checking, and allows invalid data to still classify as mpegps. jpeg probing returns 25 in some cases where it is almost certainly a jpeg (Has to go through multiple tags to get to SOS, many of which early out if they find invalid data). Note that 25 is still treated as "low confidence" for jpeg. It logs "Format jpeg_pipe detected only with low score of 25, misdetection possible!" for these jpegs. So I still think a score of 25 is too low for these jpegs, and that a better fix would be to return 26 for jpeg_pipe and mjpeg if it makes it past multiple tags to SOS. From 168485f954452de1e3ff78932c896cff01c2a303 Mon Sep 17 00:00:00 2001 From: Nikolas Bowe Date: Thu, 8 Aug 2019 15:32:51 -0700 Subject: [PATCH] avformat: Fix probing on some JPEGs Fixes "Invalid data found when processing input" on some JPEGs. Some large extensionless JPEGs can get probe score collisions with mpeg eg $ ffprobe -loglevel trace /tmp/foo [NULL @ 0x55c130ab04c0] Opening '/tmp/foo' for reading [file @ 0x55c130ab0f40] Setting default whitelist 'file,crypto' Probing jpeg_pipe score:6 size:2048 Probing jpeg_pipe score:6 size:4096 Probing jpeg_pipe score:6 size:8192 Probing jpeg_pipe score:6 size:16384 Probing jpeg_pipe score:25 size:32768 Probing jpeg_pipe score:25 size:65536 Probing jpeg_pipe score:25 size:131072 Probing jpeg_pipe score:25 size:262144 Probing jpeg_pipe score:25 size:524288 Probing mpeg score:25 size:1048576 Probing jpeg_pipe score:25 size:1048576 [AVIOContext @ 0x55c130ab9300] Statistics: 1048576 bytes read, 0 seeks /tmp/foo: Invalid data found when processing input This patch fixes this by reducing probe score for mpeg --- libavformat/mpeg.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/libavformat/mpeg.c b/libavformat/mpeg.c index 3205f209e6..6dcaefb5de 100644 --- a/libavformat/mpeg.c +++ b/libavformat/mpeg.c @@ -68,7 +68,7 @@ static int mpegps_probe(const AVProbeData *p) { uint32_t code = -1; int i; - int sys = 0, pspack = 0, priv1 = 0, vid = 0; + int sys = 0, pspack = 0, priv1 = 0, vid = 0, pic = 0; int audio = 0, invalid = 0, score = 0; int endpes = 0; @@ -92,18 +92,23 @@ static int mpegps_probe(const AVProbeData *p) else if ((code & 0xe0) == AUDIO_ID && pes) {audio++; i+=len;} else if (code == PRIVATE_STREAM_1 && pes) {priv1++; i+=len;} else if (code == 0x1fd && pes) vid++; //VC1 - else if ((code & 0xf0) == VIDEO_ID && !pes) invalid++; else if ((code & 0xe0) == AUDIO_ID && !pes) invalid++; else if (code == PRIVATE_STREAM_1 && !pes) invalid++; + else if (code == 0x100) pic++; } } + // If we don't find a pack header, and we have much more picture headers codes than + // mpeg video stream codes, then its probably not mpeg. + if (pspack == 0 && vid > 0 && vid * 10 < pic) + return AVPROBE_SCORE_EXTENSION / 2 - 1; + if (vid + audio > invalid + 1) /* invalid VDR files nd short PES streams */ score = AVPROBE_SCORE_EXTENSION / 2; -// av_log(NULL, AV_LOG_ERROR, "vid:%d aud:%d sys:%d pspack:%d invalid:%d size:%d \n", -// vid, audio, sys, pspack, invalid, p->buf_size); +// av_log(NULL, AV_LOG_ERROR, "vid:%d aud:%d sys:%d pspack:%d pic:%d invalid:%d size:%d \n", +// vid, audio, sys, pspack, pic, invalid, p->buf_size); if (sys > invalid && sys * 9 <= pspack * 10) return (audio > 12 || vid > 3 || pspack > 2) ? AVPROBE_SCORE_EXTENSION + 2 -- 2.23.0.187.g17f5b7556c-goog