From patchwork Sun Feb 2 00:29:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Klimov X-Patchwork-Id: 17656 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 6182D44B9DD for ; Sun, 2 Feb 2020 02:29:44 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 21AA468813B; Sun, 2 Feb 2020 02:29:44 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail.uralweb.ru (mail.uralweb.ru [213.142.46.7]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D2D8E688121 for ; Sun, 2 Feb 2020 02:29:37 +0200 (EET) Received: from uralweb.ru (unknown [91.238.137.108]) by mail.uralweb.ru (Postfix) with ESMTPSA id 54BCE600074 for ; Sun, 2 Feb 2020 05:29:36 +0500 (+05) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=uralweb.ru; s=uralweb.ru; t=1580603376; bh=l1do7tERq/9OiE5i7XMuTax3UkYULydyTbF6YyEWCWk=; h=From:To:Subject:Date:From; b=bwlj4FmdkaNSOnW8EBK5Sx0HlK4neEm36lyAWl8njqp5PCi+BYtQPmZq8In71BD6f IbCEAAkp+xotIQPDYvQwq538j9NZ47AdFNGVL9/Hyzbxv/KO+yjCbA+auIvEui6YBc 4CDMMTM8nJ2BFcQMpPISY3Naj1UU6TeJv4v0VClA= From: Pavel Klimov To: ffmpeg-devel@ffmpeg.org Date: Sun, 2 Feb 2020 05:29:12 +0500 Message-Id: <20200202002913.4864-1-r57shell@uralweb.ru> X-Mailer: git-send-email 2.17.0.windows.1 Subject: [FFmpeg-devel] [PATCH 0/1] libswscale: add area upscale X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" I often encode pixelated videos from old game consoles emulators. Usual workaround is upscale with neighbor, or exceeding upscale followed by lanczos downscale. I knew that area upscale should give similar effect, but didn't take effort to find out why ffmpeg with area method doesn't give what I expected to see. Up until recently. Recently I did check sources to find out what is going on. And I found in sources comment that area method falls into bilinear when it's used for upscale. First thing I thought was: not implemented? So, just for fun I tried to implement it. Why not? - I thought. It was a bit challenging, because main function that makes coefficients is using fixed point arithmetics and has no info about its units. Also srcPos and dstPos initial values are not described. Eventually, I just used approach where I don't really need to know much. Idea is following. In area representation, any point is covered by color of nearest pixel of source image. So, I just need to integrate nearest neighbor formula over pixel width. Taking into assumption that nearest neighborhood is correct, I integrated it. And it worked. At least as I thought. Then, I made python script using fractions with built-in big integers to have perfect area upscale for reference. And made test images. Also I made script for image comparison. It turned out that image comming from ffmpeg is off by 4 for some pixels. So I made log of resulting coefficients, and also made script that does same in python. It turned out that only 4 most significant decimal digits of integer coefficients were right. (by the way they are 53 bits of int64_t) During some of this, I figured out that xInc is rounded fixed point 16.16 width of pixel. So, assuming that xInc is always calculated same, I made similar dInc but using type double, and changed formulas taking it into account. After change into doubles, only two least significant decimal digits were wrong. Next, when it was working fine I thought about sharing patch and read about fate. I've made fate, and it failed. I didn't mention before, but I was changing existing area upscale to not fall into bilinear and make it upscale using area method. Here is list of failed testcases using -k switch: hevc-paramchange-yuv420p-yuv420p10 vsynth_lena-dv-411 vsynth_lena-dnxhd-edge3-hr vsynth_lena-dnxhd-edge2-hr vsynth_lena-dnxhd-edge1-hr vsynth2-dv-411 vsynth2-dnxhd-edge3-hr vsynth2-dnxhd-edge2-hr vsynth2-dnxhd-edge1-hr vsynth1-dv-411 vsynth1-dnxhd-edge3-hr vsynth1-dnxhd-edge2-hr vsynth1-dnxhd-edge1-hr First thoughts: why do you incorporate rescale into other stuff? Why don't you use sources as is. But after a bit of thinking, it didn't matter anymore. Yes you could change references and behavior of scaling, but it doesn't fit usual cases. Area scaling falling into bilinear upscale makes sense. Usually you don't know for sure will you do upscale or downscale. Or, at least you don't want to care about it. And current behavior with area scale is win-win: if downscale - area is better, if upscale - bilinear is better for normal video that you don't want to be pixelated. Also, image often has planes of different size. So, I switched to separate option falling into area downscale. Oh, by the way. Area upscale is identical to theoretical upscale to infinity by nearest neighbor followed by downscale by area. But neither holds for neighbor upscale to infinity followed by lanczos/neighbor downscale to target. In short. I have: 1) fixed arithmetic version with bad precision 2) python perfect version, coefficients generator 3) floating point version with good precision I don't insist on addition of area upscale method. But if you like the idea of having it, then I could probably assist. I provide patch that modify libswscale, because it was easiest approach for me. Also, it naturally fits in there. Any suggestions appreciated. Things to consider: Appropriate place: libswscale / .gitignore ? Name: area-only, area-all, area-both, area-forced, area-up... Fixed arithmetic / floating point version Should I include change in docs at once, or in separate commit? Should I add test in fate? I don't see lanczos or gauss tests. Pavel Klimov (1): libswscale: add area upscale libswscale/options.c | 1 + libswscale/swscale.h | 1 + libswscale/utils.c | 30 +++++++++++++++++++++++++++++- 3 files changed, 31 insertions(+), 1 deletion(-)