From patchwork Wed Jul 17 08:34:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Asaf Kave X-Patchwork-Id: 13975 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 35EB9449F12 for ; Wed, 17 Jul 2019 11:42:02 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 15A1868ADE3; Wed, 17 Jul 2019 11:42:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B4BDF68AD87 for ; Wed, 17 Jul 2019 11:41:55 +0300 (EEST) Received: by mail-wm1-f68.google.com with SMTP id h19so18595596wme.0 for ; Wed, 17 Jul 2019 01:41:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=2CZh+VsJNvD9n/2HK6MjfmwmI8KAXcFb+g2Y1jWVWUg=; b=rt3Dyc1RzAXFGDyOqZr+61FfKpKxYtZHjptpp1mODFfHA04FutELoQ1jX5EmG/+MdV d0ZmqZtBshmMdyr1Y2Rw9FADpSe/KboG7iT7GnSE2umiE4wlMaO55uXBjuybgW7eadRN jGueSMNmRcjvZd/SIHaCr4DDDmO6aqSNlc3lzusYNhhzpRZiZKvmjylFzgTAl50ZwGHN 7QZnwTZ0JvwUux+vXUIc5B2KA2V6UN2fq74ly5vAv/8f6wrF872so0T32KaMDK2SS9JC PocIRGq9m9JGxzpBErEcuZifZ94PAWwhAKaPHwY3JoTg/6YigarI9yIf26IkEmVlUHsY Ipsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=2CZh+VsJNvD9n/2HK6MjfmwmI8KAXcFb+g2Y1jWVWUg=; b=sp6uTQCl59E0kqz/j4ZnXQ6oU7hyWNUqjIaC+4hmEYLyPh9gZYjztknvXH2PSHzXbq BNhyoCnWdnhd5juKb0UkmcTJ3r41kude5eolEWT/9WHXjRXGsxQBdL2FkPoIUJs93scf o5KWhFj+fs62QwPLuC8FL/Lv9wJaHrbm54RGvQ0y5e/TZynti/mNSlpaWA5shbsKAtem V3dTG3C+ezzQZJE47CYIXQCHJPctUtId4yqxliwc2jCEcozLcOYaGabb+BXh/E2HOoaE mxebo7gO7HoIABGpCSLk41oDMqbSfXGzjU7TFD4u+XdOXqxOhpveffz8VseJOcGo7hWt B+7A== X-Gm-Message-State: APjAAAVJTauJpUBlKCh60U7FvByngxZDdp8alVPme58PYhOOX2UZe087 QsticINx3RRJdrBOb373eeMFRfhSJ4c= X-Google-Smtp-Source: APXvYqyP80y3DHQnnEYnfxRnZpMYBeHZfi9FF8YL09dBCdFEmmSIzoV5mwKwveV8K03GZISGqPFd2g== X-Received: by 2002:a1c:e009:: with SMTP id x9mr34548567wmg.5.1563352520968; Wed, 17 Jul 2019 01:35:20 -0700 (PDT) Received: from ubuntu.localdomain ([31.154.132.210]) by smtp.googlemail.com with ESMTPSA id f204sm31094293wme.18.2019.07.17.01.35.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Jul 2019 01:35:20 -0700 (PDT) From: Asaf Kave To: ffmpeg-devel@ffmpeg.org Date: Wed, 17 Jul 2019 11:34:21 +0300 Message-Id: <20190717083421.9387-1-kaveasaf@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 1/1] libavcodec/h264: Handle ENOMEM error when decoding slice nal unit. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Asaf Kave MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Populate the real error when alloc_picture failed, during decoding h264 slice. When this error occurred, need to break the decoding process of the rest nal's, and need to rise this error back to the user, to indicate that there is decoder memory issue. --- libavcodec/h264_slice.c | 5 +++-- libavcodec/h264dec.c | 4 ++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/libavcodec/h264_slice.c b/libavcodec/h264_slice.c index 5ceee107a0..76353fe85b 100644 --- a/libavcodec/h264_slice.c +++ b/libavcodec/h264_slice.c @@ -1644,9 +1644,10 @@ static int h264_field_start(H264Context *h, const H264SliceContext *sl, } if (!FIELD_PICTURE(h) || h->first_field) { - if (h264_frame_start(h) < 0) { + ret = h264_frame_start(h); + if (ret < 0) { h->first_field = 0; - return AVERROR_INVALIDDATA; + return ret; } } else { int field = h->picture_structure == PICT_BOTTOM_FIELD; diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c index 8d1bd16a8e..0a4d4b8c62 100644 --- a/libavcodec/h264dec.c +++ b/libavcodec/h264dec.c @@ -754,6 +754,10 @@ static int decode_nal_units(H264Context *h, const uint8_t *buf, int buf_size) if (err < 0) { av_log(h->avctx, AV_LOG_ERROR, "decode_slice_header error\n"); + if (err == AVERROR(ENOMEM)) { + ret = err; + goto end; + } } }