From patchwork Tue Dec 1 20:19:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 24227 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 39EF044BA85 for ; Tue, 1 Dec 2020 22:31:57 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1EED9689BD5; Tue, 1 Dec 2020 22:31:57 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.red.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 833DA689AA7 for ; Tue, 1 Dec 2020 22:31:48 +0200 (EET) Received: from localhost (localhost [IPv6:::1]) by mail.red.khirnov.net (Postfix) with ESMTP id DE6FD296B62 for ; Tue, 1 Dec 2020 21:31:43 +0100 (CET) Received: from mail.red.khirnov.net ([IPv6:::1]) by localhost (mail.red.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id R9LeO_faCP3Q for ; Tue, 1 Dec 2020 21:31:43 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail.red.khirnov.net (Postfix) with ESMTPS id D39AB296B61 for ; Tue, 1 Dec 2020 21:31:37 +0100 (CET) Received: by libav.khirnov.net (Postfix, from userid 1000) id 52E433A04F4; Tue, 1 Dec 2020 21:20:15 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Tue, 1 Dec 2020 21:19:49 +0100 Message-Id: <20201201201949.18834-9-anton@khirnov.net> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201201201949.18834-1-anton@khirnov.net> References: <20201201201949.18834-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 9/9] smvjpegdec: merge into mjpegdec X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" SMVJPEG stores frames as slices of a big JPEG image. The decoder is implemented as a wrapper that instantiates a full internal MJPEG decoder, then forwards the decoded frames with offset data pointers. This is unnecessarily complex and fragile, not supporting useful decoder capabilities like direct rendering. Re-implement the decoder inside the MJPEG decoder, which is accomplished by returning each decoded frame multiple times, setting cropping information appropriately on each instance. One peculiar aspect of the previous design is that since - the smvjpeg decoder returns one frame per input packet - there are multiple frames in each packets (the aformentioned slices) the demuxer needs to return each packet multiple times. This is now also eliminated - the demuxer now returns each packet exactly once, with the duration set to the number of frames it decodes to. This also removes one of the last remaining internal uses of the old video decoding API. --- MAINTAINERS | 1 - configure | 1 + libavcodec/Makefile | 1 - libavcodec/mjpegdec.c | 88 +++++++++++++++- libavcodec/mjpegdec.h | 4 + libavcodec/smvjpegdec.c | 223 ---------------------------------------- libavformat/wavdec.c | 12 +-- tests/ref/fate/smvjpeg | 2 +- 8 files changed, 96 insertions(+), 236 deletions(-) delete mode 100644 libavcodec/smvjpegdec.c diff --git a/MAINTAINERS b/MAINTAINERS index 2464f82be3..00c9007671 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -235,7 +235,6 @@ Codecs: rv10.c Michael Niedermayer s3tc* Ivo van Poorten smc.c Mike Melanson - smvjpegdec.c Ash Hughes snow* Michael Niedermayer, Loren Merritt sonic.c Alex Beregszaszi speedhq.c Steinar H. Gunderson diff --git a/configure b/configure index 51e43fbf66..277c2d699a 100755 --- a/configure +++ b/configure @@ -2838,6 +2838,7 @@ rv40_decoder_select="golomb h264pred h264qpel mpegvideo rv34dsp" screenpresso_decoder_deps="zlib" shorten_decoder_select="bswapdsp" sipr_decoder_select="lsp" +smvjpeg_decoder_select="mjpeg_decoder" snow_decoder_select="dwt h264qpel hpeldsp me_cmp rangecoder videodsp" snow_encoder_select="dwt h264qpel hpeldsp me_cmp mpegvideoenc rangecoder" sonic_decoder_select="golomb rangecoder" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index a6435c9e85..1f6e929a42 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -616,7 +616,6 @@ OBJS-$(CONFIG_SIREN_DECODER) += siren.o OBJS-$(CONFIG_SMACKAUD_DECODER) += smacker.o OBJS-$(CONFIG_SMACKER_DECODER) += smacker.o OBJS-$(CONFIG_SMC_DECODER) += smc.o -OBJS-$(CONFIG_SMVJPEG_DECODER) += smvjpegdec.o OBJS-$(CONFIG_SNOW_DECODER) += snowdec.o snow.o snow_dwt.o OBJS-$(CONFIG_SNOW_ENCODER) += snowenc.o snow.o snow_dwt.o \ h263.o h263data.o ituh263enc.o diff --git a/libavcodec/mjpegdec.c b/libavcodec/mjpegdec.c index 96dfc3dc19..7cc5f2a32e 100644 --- a/libavcodec/mjpegdec.c +++ b/libavcodec/mjpegdec.c @@ -203,7 +203,19 @@ av_cold int ff_mjpeg_decode_init(AVCodecContext *avctx) s->interlace_polarity = 1; } - if ( avctx->extradata_size > 8 + if (avctx->codec_id == AV_CODEC_ID_SMVJPEG) { + if (avctx->extradata_size >= 4) + s->smv_frames_per_jpeg = AV_RL32(avctx->extradata); + + if (s->smv_frames_per_jpeg <= 0) { + av_log(avctx, AV_LOG_ERROR, "Invalid number of frames per jpeg.\n"); + return AVERROR_INVALIDDATA; + } + + s->smv_frame = av_frame_alloc(); + if (!s->smv_frame) + return AVERROR(ENOMEM); + } else if (avctx->extradata_size > 8 && AV_RL32(avctx->extradata) == 0x2C && AV_RL32(avctx->extradata+4) == 0x18) { parse_avid(s, avctx->extradata, avctx->extradata_size); @@ -477,6 +489,12 @@ int ff_mjpeg_decode_sof(MJpegDecodeContext *s) size_change = 0; } + if (s->avctx->codec_id == AV_CODEC_ID_SMVJPEG) { + s->avctx->height = s->avctx->coded_height / s->smv_frames_per_jpeg; + if (s->avctx->height <= 0) + return AVERROR_INVALIDDATA; + } + if (s->got_picture && s->interlaced && (s->bottom_field == !s->interlace_polarity)) { if (s->progressive) { avpriv_request_sample(s->avctx, "progressively coded interlaced picture"); @@ -2341,6 +2359,42 @@ static void reset_icc_profile(MJpegDecodeContext *s) s->iccnum = 0; } +// SMV JPEG just stacks several output frames into one JPEG picture +// we handle that by setting up the cropping parameters appropriately +static int smv_process_frame(AVCodecContext *avctx, AVFrame *frame) +{ + MJpegDecodeContext *s = avctx->priv_data; + int ret; + + if (s->smv_next_frame > 0) { + av_assert0(s->smv_frame->buf[0]); + av_frame_unref(frame); + ret = av_frame_ref(frame, s->smv_frame); + if (ret < 0) + return ret; + } else { + av_assert0(frame->buf[0]); + av_frame_unref(s->smv_frame); + ret = av_frame_ref(s->smv_frame, frame); + if (ret < 0) + return ret; + } + + av_assert0((s->smv_next_frame + 1) * avctx->height <= avctx->coded_height); + + frame->width = avctx->coded_width; + frame->height = avctx->coded_height; + frame->crop_top = FFMIN(s->smv_next_frame * avctx->height, frame->height); + frame->crop_bottom = frame->height - (s->smv_next_frame + 1) * avctx->height; + + s->smv_next_frame = (s->smv_next_frame + 1) % s->smv_frames_per_jpeg; + + if (s->smv_next_frame == 0) + av_frame_unref(s->smv_frame); + + return 0; +} + static int mjpeg_get_packet(AVCodecContext *avctx) { MJpegDecodeContext *s = avctx->priv_data; @@ -2377,6 +2431,9 @@ int ff_mjpeg_receive_frame(AVCodecContext *avctx, AVFrame *frame) int ret = 0; int is16bit; + if (avctx->codec_id == AV_CODEC_ID_SMVJPEG && s->smv_next_frame > 0) + return smv_process_frame(avctx, frame); + av_dict_free(&s->exif_metadata); av_freep(&s->stereo3d); s->adobe_transform = -1; @@ -2837,6 +2894,14 @@ the_end: av_dict_copy(&frame->metadata, s->exif_metadata, 0); av_dict_free(&s->exif_metadata); + if (avctx->codec_id == AV_CODEC_ID_SMVJPEG) { + ret = smv_process_frame(avctx, frame); + if (ret < 0) { + av_frame_unref(frame); + return ret; + } + } + ret = 0; the_end_no_picture: @@ -2863,6 +2928,8 @@ av_cold int ff_mjpeg_decode_end(AVCodecContext *avctx) av_packet_free(&s->pkt); + av_frame_free(&s->smv_frame); + av_freep(&s->buffer); av_freep(&s->stereo3d); av_freep(&s->ljpeg_buffer); @@ -2889,6 +2956,9 @@ static void decode_flush(AVCodecContext *avctx) { MJpegDecodeContext *s = avctx->priv_data; s->got_picture = 0; + + s->smv_next_frame = 0; + av_frame_unref(s->smv_frame); } #if CONFIG_MJPEG_DECODER @@ -2950,3 +3020,19 @@ AVCodec ff_thp_decoder = { .caps_internal = FF_CODEC_CAP_INIT_THREADSAFE, }; #endif + +#if CONFIG_SMVJPEG_DECODER +AVCodec ff_smvjpeg_decoder = { + .name = "smvjpeg", + .long_name = NULL_IF_CONFIG_SMALL("SMV JPEG"), + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_SMVJPEG, + .priv_data_size = sizeof(MJpegDecodeContext), + .init = ff_mjpeg_decode_init, + .close = ff_mjpeg_decode_end, + .receive_frame = ff_mjpeg_receive_frame, + .flush = decode_flush, + .capabilities = AV_CODEC_CAP_DR1, + .caps_internal = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_EXPORTS_CROPPING, +}; +#endif diff --git a/libavcodec/mjpegdec.h b/libavcodec/mjpegdec.h index 66fb0ddcba..d9844adf4e 100644 --- a/libavcodec/mjpegdec.h +++ b/libavcodec/mjpegdec.h @@ -142,6 +142,10 @@ typedef struct MJpegDecodeContext { int iccnum; int iccread; + AVFrame *smv_frame; + int smv_frames_per_jpeg; + int smv_next_frame; + // Raw stream data for hwaccel use. const uint8_t *raw_image_buffer; size_t raw_image_buffer_size; diff --git a/libavcodec/smvjpegdec.c b/libavcodec/smvjpegdec.c deleted file mode 100644 index 973a9117f2..0000000000 --- a/libavcodec/smvjpegdec.c +++ /dev/null @@ -1,223 +0,0 @@ -/* - * SMV JPEG decoder - * Copyright (c) 2013 Ash Hughes - * - * This file is part of FFmpeg. - * - * FFmpeg is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * FFmpeg is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public - * License along with FFmpeg; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA - */ - -/** - * @file - * SMV JPEG decoder. - */ - -// #define DEBUG -#include "avcodec.h" -#include "libavutil/opt.h" -#include "libavutil/imgutils.h" -#include "mjpegdec.h" -#include "internal.h" - -typedef struct SMVJpegDecodeContext { - MJpegDecodeContext jpg; - AVFrame *picture[2]; /* pictures array */ - AVCodecContext* avctx; - int frames_per_jpeg; - int mjpeg_data_size; -} SMVJpegDecodeContext; - -static inline void smv_img_pnt_plane(uint8_t **dst, uint8_t *src, - int src_linesize, int height, int nlines) -{ - if (!dst || !src) - return; - src += (nlines) * src_linesize * height; - *dst = src; -} - -static inline void smv_img_pnt(uint8_t *dst_data[4], uint8_t *src_data[4], - const int src_linesizes[4], - enum AVPixelFormat pix_fmt, int width, int height, - int nlines) -{ - const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(pix_fmt); - int i, planes_nb = 0; - - if (desc->flags & AV_PIX_FMT_FLAG_HWACCEL) - return; - - for (i = 0; i < desc->nb_components; i++) - planes_nb = FFMAX(planes_nb, desc->comp[i].plane + 1); - - for (i = 0; i < planes_nb; i++) { - int h = height; - if (i == 1 || i == 2) { - h = AV_CEIL_RSHIFT(height, desc->log2_chroma_h); - } - smv_img_pnt_plane(&dst_data[i], src_data[i], - src_linesizes[i], h, nlines); - } - if (desc->flags & AV_PIX_FMT_FLAG_PAL || - desc->flags & FF_PSEUDOPAL) - dst_data[1] = src_data[1]; -} - -static av_cold int smvjpeg_decode_end(AVCodecContext *avctx) -{ - SMVJpegDecodeContext *s = avctx->priv_data; - MJpegDecodeContext *jpg = &s->jpg; - - jpg->picture_ptr = NULL; - av_frame_free(&s->picture[0]); - av_frame_free(&s->picture[1]); - avcodec_free_context(&s->avctx); - return 0; -} - -static av_cold int smvjpeg_decode_init(AVCodecContext *avctx) -{ - SMVJpegDecodeContext *s = avctx->priv_data; - AVCodec *codec; - AVDictionary *thread_opt = NULL; - int ret = 0, r; - - s->frames_per_jpeg = 0; - - s->picture[0] = av_frame_alloc(); - if (!s->picture[0]) - return AVERROR(ENOMEM); - - s->picture[1] = av_frame_alloc(); - if (!s->picture[1]) { - av_frame_free(&s->picture[0]); - return AVERROR(ENOMEM); - } - - s->jpg.picture_ptr = s->picture[0]; - - if (avctx->extradata_size >= 4) - s->frames_per_jpeg = AV_RL32(avctx->extradata); - - if (s->frames_per_jpeg <= 0) { - av_log(avctx, AV_LOG_ERROR, "Invalid number of frames per jpeg.\n"); - ret = AVERROR_INVALIDDATA; - } - - codec = avcodec_find_decoder(AV_CODEC_ID_MJPEG); - if (!codec) { - av_log(avctx, AV_LOG_ERROR, "MJPEG codec not found\n"); - smvjpeg_decode_end(avctx); - return AVERROR_DECODER_NOT_FOUND; - } - - s->avctx = avcodec_alloc_context3(codec); - - av_dict_set(&thread_opt, "threads", "1", 0); - s->avctx->refcounted_frames = 1; - s->avctx->flags = avctx->flags; - s->avctx->idct_algo = avctx->idct_algo; - if ((r = ff_codec_open2_recursive(s->avctx, codec, &thread_opt)) < 0) { - av_log(avctx, AV_LOG_ERROR, "MJPEG codec failed to open\n"); - ret = r; - } - av_dict_free(&thread_opt); - - if (ret < 0) - smvjpeg_decode_end(avctx); - return ret; -} - -static int smvjpeg_decode_frame(AVCodecContext *avctx, void *data, int *data_size, - AVPacket *avpkt) -{ - const AVPixFmtDescriptor *desc; - SMVJpegDecodeContext *s = avctx->priv_data; - AVFrame* mjpeg_data = s->picture[0]; - int i, cur_frame = 0, ret = 0; - - cur_frame = avpkt->pts % s->frames_per_jpeg; - - /* cur_frame is later used to calculate the buffer offset, so it mustn't be negative */ - if (cur_frame < 0) - cur_frame += s->frames_per_jpeg; - - /* Are we at the start of a block? */ - if (!cur_frame) { - av_frame_unref(mjpeg_data); - ret = avcodec_decode_video2(s->avctx, mjpeg_data, &s->mjpeg_data_size, avpkt); - if (ret < 0) { - s->mjpeg_data_size = 0; - return ret; - } - } else if (!s->mjpeg_data_size) - return AVERROR(EINVAL); - - desc = av_pix_fmt_desc_get(s->avctx->pix_fmt); - av_assert0(desc); - - if (mjpeg_data->height % (s->frames_per_jpeg << desc->log2_chroma_h)) { - av_log(avctx, AV_LOG_ERROR, "Invalid height\n"); - return AVERROR_INVALIDDATA; - } - - /*use the last lot... */ - *data_size = s->mjpeg_data_size; - - avctx->pix_fmt = s->avctx->pix_fmt; - - /* We shouldn't get here if frames_per_jpeg <= 0 because this was rejected - in init */ - ret = ff_set_dimensions(avctx, mjpeg_data->width, mjpeg_data->height / s->frames_per_jpeg); - if (ret < 0) { - av_log(s, AV_LOG_ERROR, "Failed to set dimensions\n"); - return ret; - } - - if (*data_size) { - s->picture[1]->extended_data = NULL; - s->picture[1]->width = avctx->width; - s->picture[1]->height = avctx->height; - s->picture[1]->format = avctx->pix_fmt; - smv_img_pnt(s->picture[1]->data, mjpeg_data->data, mjpeg_data->linesize, - avctx->pix_fmt, avctx->width, avctx->height, cur_frame); - for (i = 0; i < AV_NUM_DATA_POINTERS; i++) - s->picture[1]->linesize[i] = mjpeg_data->linesize[i]; - - ret = av_frame_ref(data, s->picture[1]); - if (ret < 0) - return ret; - } - - return avpkt->size; -} - -static const AVClass smvjpegdec_class = { - .class_name = "SMVJPEG decoder", - .item_name = av_default_item_name, - .version = LIBAVUTIL_VERSION_INT, -}; - -AVCodec ff_smvjpeg_decoder = { - .name = "smvjpeg", - .long_name = NULL_IF_CONFIG_SMALL("SMV JPEG"), - .type = AVMEDIA_TYPE_VIDEO, - .id = AV_CODEC_ID_SMVJPEG, - .priv_data_size = sizeof(SMVJpegDecodeContext), - .init = smvjpeg_decode_init, - .close = smvjpeg_decode_end, - .decode = smvjpeg_decode_frame, - .priv_class = &smvjpegdec_class, -}; diff --git a/libavformat/wavdec.c b/libavformat/wavdec.c index df6030a42d..f49c25c48d 100644 --- a/libavformat/wavdec.c +++ b/libavformat/wavdec.c @@ -58,7 +58,6 @@ typedef struct WAVDemuxContext { int ignore_length; int max_size; int spdif; - int smv_cur_pt; int smv_given_first; int unaligned; // e.g. if an odd number of bytes ID3 tag was prepended int rifx; // RIFX: integer byte order for parameters is big endian @@ -497,7 +496,6 @@ static int wav_read_header(AVFormatContext *s) return AVERROR_INVALIDDATA; } AV_WL32(vst->codecpar->extradata, wav->smv_frames_per_jpeg); - wav->smv_cur_pt = 0; goto break_loop; case MKTAG('L', 'I', 'S', 'T'): case MKTAG('l', 'i', 's', 't'): @@ -714,12 +712,9 @@ smv_retry: if (ret < 0) goto smv_out; pkt->pos -= 3; - pkt->pts = wav->smv_block * wav->smv_frames_per_jpeg + wav->smv_cur_pt; - wav->smv_cur_pt++; - if (wav->smv_frames_per_jpeg > 0) - wav->smv_cur_pt %= wav->smv_frames_per_jpeg; - if (!wav->smv_cur_pt) - wav->smv_block++; + pkt->pts = wav->smv_block * wav->smv_frames_per_jpeg; + pkt->duration = wav->smv_frames_per_jpeg; + wav->smv_block++; pkt->stream_index = 1; smv_out: @@ -781,7 +776,6 @@ static int wav_read_seek(AVFormatContext *s, timestamp = av_rescale_q(smv_timestamp, s->streams[1]->time_base, s->streams[0]->time_base); if (wav->smv_frames_per_jpeg > 0) { wav->smv_block = smv_timestamp / wav->smv_frames_per_jpeg; - wav->smv_cur_pt = smv_timestamp % wav->smv_frames_per_jpeg; } } diff --git a/tests/ref/fate/smvjpeg b/tests/ref/fate/smvjpeg index 360fad6595..0ab4f21309 100644 --- a/tests/ref/fate/smvjpeg +++ b/tests/ref/fate/smvjpeg @@ -2,7 +2,7 @@ #media_type 0: video #codec_id 0: rawvideo #dimensions 0: 128x160 -#sar 0: 0/1 +#sar 0: 72/72 0, 0, 0, 1, 30720, 0x3a821807 0, 1, 1, 1, 30720, 0x95168e5d 0, 2, 2, 1, 30720, 0xd4d98e45