From patchwork Wed Aug 23 19:28:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ashish Singh X-Patchwork-Id: 4811 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.15.201 with SMTP id 70csp12244jao; Wed, 23 Aug 2017 12:34:35 -0700 (PDT) X-Received: by 10.223.165.138 with SMTP id g10mr2244302wrc.167.1503516875580; Wed, 23 Aug 2017 12:34:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1503516875; cv=none; d=google.com; s=arc-20160816; b=SuSIKoGcWRbwq2bUgk+8R0zLp0kcrehO4TElSveh8u/fIzoxk2uWnUmF/+j8huC2m0 s5YP+ggTCCJ2z5AQ69MrOwkeVQxlDx+NxDM8O+553is7iWT3o5OXC5vJeBwCzfjZrwqz ESNKsLHSqP2I4y9dMFikb7pRb8I+D9kmPYoldI+YPzzM4R4+9pHLMxIofMmjD7pRuXWd Sza2FoEzTF9tPXJYwB5csnd5IMZfJjdB8AJWnGCQ+05pP/kVJB5Tn+aWdErs7FaI9HRC dpO6T22lgYBEATL5WX4WOEyHCyrWDqgeZEQN9dBo9wTY4icujMT7zXUDTR8PLkaOGB9f OmJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=iuj14MHCFtxugrVpmFcjRhY9tZyIhgim4VquUEBRAPs=; b=SiIhv44oG2Uko3VGsFtwxGKWaeMPYk38ETNJDO8OKbgC54sQTnDL4sLEohotOQgpTb KzV1LBbWpbeFZJBJiVUkrzSMAGpIJLocTAFrdvIkH0znwS0oVSWKShfY3RQRE1FnUj+M 6PL0ltzEhDpARcBHrEAbVGcdsddQAAFlAclKIXz2xs5+a5qrP9lu+S3++zmbbmMrEEv3 i2hw8UoZ9z9gSw4Ds/B7AldL+1OAAWJhrG+EVbHBuN+MRTEAdnaWJkWMK2GFjumg0u6O Lisco6qxz7EWFug1FN919W71BDFDhUc0ORN+rhRfJvL+q+XCj/KFViJ6CRwiq3FsECzK 6F/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KoPvF7nD; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q75si252007wme.139.2017.08.23.12.34.34; Wed, 23 Aug 2017 12:34:35 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KoPvF7nD; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CE1E5688253; Wed, 23 Aug 2017 22:34:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f67.google.com (mail-pg0-f67.google.com [74.125.83.67]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4D7AA6807B8 for ; Wed, 23 Aug 2017 22:34:16 +0300 (EEST) Received: by mail-pg0-f67.google.com with SMTP id 189so502757pgj.0 for ; Wed, 23 Aug 2017 12:34:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=WHp5/sYYnx+GWiMvrUWw+YuL+ne8dSI6NJD3kahSGww=; b=KoPvF7nD9d9dbkAiTAiLmrpie0QQFYdspac9nTBX0nea8aW62jjI0hMLSzfMsSHDHo ik8CLd/k9Ri4eeMptQLGHtApW+nWqE4emV75TnwWM9TX3KbPKuvmwJN+SFzAc58sNA6I IVd0hF18NYYmiaxSpUqffO++CyD+O6D9EJxoprGYYk9fJc6WWgrhG47JMaDsxRzUiZjn DkrlchUDAmXXRDp8gaGaRAta23tqqb50Vq+rYLS9BZ0HI36MRpDs2zVrU0iA/WYeshZb FAISJP9ozn9BtqFp3Q4mVfESwvv9UAC3yKhDxqLSFcwf6Vu7Cir7aJl6z7lEOpStiIG0 5KMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=WHp5/sYYnx+GWiMvrUWw+YuL+ne8dSI6NJD3kahSGww=; b=jDoSEHhhbiLfAbbgRZb1/ZAvyedyVgJe7rV7uWBa6t2DBLMe29b4Dqnija9sPF8S/o Ef29AJGw8yiQpqlsnccuzh+rQhpVZdz9WfhGwxnMbn5ByiJ/xq3UCzRur/y3zM+g4Cwu Ub4j4HdmuTIWl/mdmUmpReefjXkwykLepO/f8LJdVF7dS8mUvnpAKmkvba1m8fdESxV0 2XlkOwuFMedHIu4k8ay9fSO6D042QD7jgAolMXs1PBc3D5wJteruc2R1s/pyadBb95B+ KDJhOsfyM2ETHm+tJtjrGVFYIGx4RRnwxRulGbhhiSBaML8+XUHfVbWgA/Dgysim5Q/z +BDw== X-Gm-Message-State: AHYfb5gGHHAhh4OKkSzRGIVGnhWrvY0DdcWcxp3WKwD+RA6vTaoPvtYl 0UEuuy1uy+wVC2b/zl8= X-Received: by 10.98.16.203 with SMTP id 72mr3874905pfq.290.1503516550370; Wed, 23 Aug 2017 12:29:10 -0700 (PDT) Received: from localhost.localdomain ([49.44.51.66]) by smtp.gmail.com with ESMTPSA id v187sm3841010pgv.67.2017.08.23.12.29.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 23 Aug 2017 12:29:09 -0700 (PDT) From: Ashish Pratap Singh To: ffmpeg-devel@ffmpeg.org Date: Thu, 24 Aug 2017 00:58:53 +0530 Message-Id: <1503516533-7623-1-git-send-email-ashk43712@gmail.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH] avfilter: Add vmaf filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Ashish Singh MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Ashish Singh Hi, this is vmaf filter. It fuses the scores of previous metrics adm, motion and vif using svm algorithm. It's different from libvmaf filter because it has a very little external dependency (only one svm model file). Currently it supports only one model which can be extended later for other models. I have added the model file inside libavfilter/data/ so that it can run successfully. There is still a bit of work left to do like changing each filter from float to integer. It's in progress along with SIMD optimizations. Signed-off-by: Ashish Singh --- Changelog | 1 + doc/filters.texi | 38 ++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/data/vmaf_v0.6.1.pkl.model | 218 ++++++++ libavfilter/vf_vmaf.c | 945 +++++++++++++++++++++++++++++++++ libavfilter/vmaf.h | 138 +++++ 7 files changed, 1342 insertions(+) create mode 100644 libavfilter/data/vmaf_v0.6.1.pkl.model create mode 100644 libavfilter/vf_vmaf.c create mode 100644 libavfilter/vmaf.h diff --git a/Changelog b/Changelog index 7a6987a..b33f0b2 100644 --- a/Changelog +++ b/Changelog @@ -33,6 +33,7 @@ version : - tlut2 video filter - floodfill video filter - pseudocolor video filter +- vmaf video filter version 3.3: - CrystalHD decoder moved to new decode API diff --git a/doc/filters.texi b/doc/filters.texi index 3b5a38f..2396d96 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -15375,6 +15375,44 @@ vignette='PI/4+random(1)*PI/50':eval=frame @end itemize +@section vmaf + +Obtain the VMAF (Video Multi-Method Assessment Fusion) +score between two input videos. + +Both video inputs must have the same resolution and pixel format. + +The obtained VMAF score is printed through the logging system. + +If no model path is specified it uses the default model: @code{vmaf_v0.6.1.pkl}. + +The filter has following options: + +@table @option +@item model_path +Set the model path which is to be used for SVM. +Default value: @code{"vmaf_v0.6.1.pkl.model"} + +@item enable_transform +Enables transform for computing vmaf. + +@item pool +Set the pool method to be used for computing vmaf (mean, min or harmonic mean). +@end table + +On the below examples the input file @file{main.mpg} being processed is +compared with the reference file @file{ref.mpg}. + +For example: +@example +ffmpeg -i main.mpg -i ref.mpg -lavfi vmaf -f null - +@end example + +Example with options: +@example +ffmpeg -i main.mpg -i ref.mpg -lavfi vmaf="pool=min" -f null - +@end example + @section vstack Stack input videos vertically. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 1d92dc1..068b29f 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -327,6 +327,7 @@ OBJS-$(CONFIG_VFLIP_FILTER) += vf_vflip.o OBJS-$(CONFIG_VIDSTABDETECT_FILTER) += vidstabutils.o vf_vidstabdetect.o OBJS-$(CONFIG_VIDSTABTRANSFORM_FILTER) += vidstabutils.o vf_vidstabtransform.o OBJS-$(CONFIG_VIGNETTE_FILTER) += vf_vignette.o +OBJS-$(CONFIG_VMAF_FILTER) += vf_vmaf.o dualinput.o framesync.o OBJS-$(CONFIG_VSTACK_FILTER) += vf_stack.o framesync2.o OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 8b9b9a4..39c3265 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -338,6 +338,7 @@ static void register_all(void) REGISTER_FILTER(VIDSTABDETECT, vidstabdetect, vf); REGISTER_FILTER(VIDSTABTRANSFORM, vidstabtransform, vf); REGISTER_FILTER(VIGNETTE, vignette, vf); + REGISTER_FILTER(VMAF, vmaf, vf); REGISTER_FILTER(VSTACK, vstack, vf); REGISTER_FILTER(W3FDIF, w3fdif, vf); REGISTER_FILTER(WAVEFORM, waveform, vf); diff --git a/libavfilter/data/vmaf_v0.6.1.pkl.model b/libavfilter/data/vmaf_v0.6.1.pkl.model new file mode 100644 index 0000000..fe157df --- /dev/null +++ b/libavfilter/data/vmaf_v0.6.1.pkl.model @@ -0,0 +1,218 @@ +svm_type nu_svr +kernel_type rbf +gamma 0.04 +nr_class 2 +total_sv 211 +rho -1.33133 +SV +-4 1:0.65734273 2:0.34681232 3:0.093755557 4:0.60913934 5:0.69117362 6:0.73495824 +4 1:0.8727433 2:0.49612229 3:0.59146724 4:0.78105663 5:0.84916292 6:0.8882561 +4 1:0.89890005 2:0.49612229 3:0.66823667 4:0.86050887 5:0.90873162 6:0.93335071 +4 1:0.20371751 2:0.49612229 3:0.10534315 4:-1.110223e-16 6:2.220446e-16 +4 1:0.33913836 2:0.49612229 3:0.14024497 4:0.074708413 5:0.10231651 6:0.1259153 +4 1:0.66426757 2:0.49612229 3:0.35268026 4:0.4805681 5:0.59603341 6:0.67408692 +4 1:0.59561632 2:0.49612229 3:0.27561601 4:0.33977371 5:0.4325213 6:0.50244952 +4 1:0.50821444 2:0.49612229 3:0.20276685 4:0.2004308 5:0.25758651 6:0.30054029 +4 1:0.77877298 2:0.49612229 3:0.444392 4:0.61630491 5:0.71210086 6:0.77386496 +4 1:0.71666017 2:0.49612229 3:0.35967401 4:0.47825205 5:0.57045236 6:0.63752441 +4 1:0.64025669 2:0.49612229 3:0.27766156 4:0.33407105 5:0.40732401 6:0.46359154 +4 1:0.88343983 2:0.23066177 3:0.65873851 4:0.86090402 5:0.90661213 6:0.93008753 +4 1:0.90822691 2:0.23066177 3:0.71439481 4:0.90904598 5:0.94146542 6:0.95674338 +-4 1:0.49037399 2:0.23066177 3:0.32329421 4:0.33686197 5:0.39456977 6:0.44944683 +-4 1:0.69044383 2:0.23066177 3:0.43933868 4:0.56327049 5:0.65339511 6:0.71348696 +-4 1:0.62390093 2:0.23066177 3:0.3800888 4:0.44927578 5:0.52327759 6:0.57907725 +4 1:0.81887942 2:0.23066177 3:0.56208506 4:0.76164281 5:0.83176644 6:0.86914911 +4 1:0.77189471 2:0.23066177 3:0.50145055 4:0.66525882 5:0.74327951 6:0.79017822 +4 1:0.71405433 2:0.23066177 3:0.43952897 4:0.55736023 5:0.63319876 6:0.68402869 +4 1:0.92114073 3:0.45198963 4:0.97703695 5:0.9907273 6:0.99510256 +4 1:1 3:0.83319067 4:0.98956086 5:0.99577089 6:0.99784595 +4 4:0.10344019 5:0.34323945 6:0.63855969 +4 1:0.19531482 3:0.034330388 4:0.25480402 5:0.54197045 6:0.78020579 +4 1:0.48394064 3:0.11866359 4:0.58816959 5:0.86435738 6:0.96191842 +4 1:0.47628079 3:0.11185039 4:0.56180003 5:0.83415721 6:0.93617329 +4 1:0.46278632 3:0.10308547 4:0.52247575 5:0.78583924 6:0.89392193 +4 1:0.7038079 3:0.2174879 4:0.84423613 5:0.9662906 6:0.98430594 +4 1:0.69596686 3:0.20657211 4:0.81196884 5:0.94140702 6:0.96680805 +4 1:0.68404358 3:0.19261438 4:0.76066415 5:0.89973293 6:0.93660362 +4 1:0.84073022 2:0.34681232 3:0.22411304 4:0.88845644 5:0.94169671 6:0.96221395 +-4 1:0.33900937 2:0.34681232 3:0.027607294 4:0.40659646 5:0.45456869 6:0.48256597 +-4 1:0.44593129 2:0.34681232 3:0.041939301 4:0.45284872 5:0.5157613 6:0.55335821 +-4 1:0.67301747 2:0.34681232 3:0.11526222 4:0.68549511 5:0.78556255 6:0.83507583 +-4 1:0.62833533 2:0.34681232 3:0.092281981 4:0.61278125 5:0.70626575 6:0.75613977 +-4 1:0.57196879 2:0.34681232 3:0.067548447 4:0.53383404 5:0.61287548 6:0.65468717 +-0.3312466607741135 1:0.75125028 2:0.34681232 3:0.1457048 4:0.75791308 5:0.84155109 6:0.88132116 +-4 1:0.71121936 2:0.34681232 3:0.12095689 4:0.68834617 5:0.77453583 6:0.81892861 +-4 1:0.80269544 2:0.25207203 3:0.3681723 4:0.80658472 5:0.8702283 6:0.90583519 +-4 1:0.86095387 2:0.25207203 3:0.52475418 4:0.85053413 5:0.90454501 6:0.93093678 +-4 1:0.5008963 2:0.25207203 3:0.2005129 4:0.41516485 5:0.45282017 6:0.47396143 +-4 1:0.56977992 2:0.25207203 3:0.21631076 4:0.45848604 5:0.51102137 6:0.53823055 +-4 1:0.72779828 2:0.25207203 3:0.3051639 4:0.67537297 5:0.75767261 6:0.80327187 +-4 1:0.68848569 2:0.25207203 3:0.27393051 4:0.60399854 5:0.68000038 6:0.72275152 +-4 1:0.64121401 2:0.25207203 3:0.23994344 4:0.52538719 5:0.5891732 6:0.62164073 +-4 1:0.76673633 2:0.25207203 3:0.33053889 4:0.73085549 5:0.80341439 6:0.84546456 +-4 1:0.73041172 2:0.25207203 3:0.29691153 4:0.66166141 5:0.73408074 6:0.77757209 +-4 1:0.68529047 2:0.25207203 3:0.26283557 4:0.58611788 5:0.65192525 6:0.69015011 +4 1:0.86902267 2:0.48885268 3:0.5143645 4:0.8587242 5:0.91841685 6:0.94498293 +4 1:0.89266106 2:0.48885268 3:0.55208861 4:0.89938377 5:0.94642982 6:0.96615102 +-4 1:0.42554844 2:0.48885268 3:0.2554221 4:0.36916892 5:0.43100226 6:0.50888404 +-4 1:0.52520274 2:0.48885268 3:0.27824915 4:0.42915458 5:0.50850476 6:0.58585271 +-4 1:0.69357445 2:0.48885268 3:0.35289928 4:0.61359907 5:0.7217863 6:0.78790011 +-4 1:0.64679648 2:0.48885268 3:0.31268451 4:0.5167094 5:0.61224976 6:0.68477529 +4 1:0.80595874 2:0.48885268 3:0.44075432 4:0.7803455 5:0.86328719 6:0.90222545 +-4 1:0.7715192 2:0.48885268 3:0.4012577 4:0.70792536 5:0.80063653 6:0.85083872 +4 1:0.82199966 2:0.20629643 3:0.30562098 4:0.80541317 5:0.89285836 6:0.92907353 +-4 1:0.84774006 2:0.20629643 3:0.36755712 4:0.8681203 5:0.93297792 6:0.95700049 +4 1:0.26631905 2:0.20629643 3:0.076468978 4:0.29833807 5:0.37989948 6:0.4576277 +-4 1:0.65439648 2:0.20629643 3:0.19487894 4:0.63045155 5:0.76931142 6:0.83706632 +4 1:0.55295603 2:0.20629643 3:0.13877412 4:0.4724047 5:0.59295828 6:0.66834832 +4 1:0.75448924 2:0.20629643 3:0.24707248 4:0.72284103 5:0.83178838 6:0.88053503 +4 1:0.83852041 2:0.15600331 3:0.1625414 4:0.81948421 5:0.90185357 6:0.9347395 +4 1:0.85805266 2:0.15600331 3:0.19693206 4:0.86294641 5:0.92990351 6:0.95498998 +-4 1:0.43384835 2:0.15600331 3:0.030541611 4:0.37279112 5:0.4588284 6:0.52004828 +-4 1:0.72588966 2:0.48885268 3:0.35394597 4:0.61189191 5:0.70897304 6:0.77099691 +-4 1:0.65865915 2:0.20629643 3:0.1796405 4:0.56432133 5:0.68049028 6:0.74616621 +-4 1:0.53095193 2:0.15600331 3:0.046271684 4:0.4328793 5:0.5309142 6:0.59282089 +-4 1:0.71891465 2:0.15600331 3:0.11085278 4:0.68794624 5:0.80350923 6:0.85660483 +-4 1:0.68635753 2:0.15600331 3:0.091457045 4:0.60849701 5:0.72282659 6:0.78137183 +-4 1:0.64162333 2:0.15600331 3:0.068820233 4:0.51732819 5:0.62198733 6:0.67977328 +4 1:0.78395225 2:0.15600331 3:0.13401869 4:0.75274384 5:0.8506531 6:0.89321405 +-4 1:0.75276337 2:0.15600331 3:0.11289462 4:0.67598462 5:0.78117168 6:0.83259364 +-4 1:0.71345342 2:0.15600331 3:0.089218917 4:0.58797907 5:0.69284768 6:0.74971699 +4 1:0.93500967 2:0.08765484 3:0.72226864 4:0.93291747 5:0.960644 6:0.97304054 +4 1:0.95150668 2:0.08765484 3:0.77391346 4:0.95596295 5:0.97544784 6:0.98405871 +-4 1:0.48148634 2:0.08765484 3:0.36628046 4:0.45852823 5:0.56005228 6:0.65708595 +-4 1:0.59853216 2:0.08765484 3:0.42071301 4:0.56376512 5:0.66454599 6:0.741236 +-4 1:0.79297271 2:0.08765484 3:0.5597726 4:0.80653689 5:0.88996341 6:0.92691132 +-4 1:0.76798941 2:0.08765484 3:0.52069978 4:0.74484555 5:0.83431246 6:0.87935204 +-4 1:0.73225133 2:0.08765484 3:0.47011786 4:0.66069877 5:0.75226598 6:0.80539407 +-4 1:0.87240592 2:0.08765484 3:0.62680052 4:0.88208508 5:0.93041565 6:0.9505376 +-4 1:0.84834872 2:0.08765484 3:0.58154998 4:0.82429855 5:0.8858516 6:0.91563291 +-4 1:0.84365382 2:0.93973481 3:0.36718425 4:0.81512123 5:0.88887359 6:0.92320992 +-4 1:0.89242364 2:0.93973481 3:0.41336953 4:0.88038833 5:0.93688884 6:0.95992879 +-4 1:0.31373571 2:0.93973481 3:0.18757116 4:0.34864297 5:0.3777168 6:0.38922611 +-4 1:0.42490775 2:0.93973481 3:0.20295859 4:0.39290035 5:0.43632323 6:0.45871216 +-4 1:0.66865444 2:0.93973481 3:0.28594627 4:0.63969879 5:0.73360583 6:0.78380069 +-4 1:0.62642524 2:0.93973481 3:0.26141889 4:0.56602175 5:0.64775366 6:0.69263211 +-4 1:0.57430455 2:0.93973481 3:0.23537634 4:0.48984694 5:0.55363885 6:0.5853905 +-4 1:0.76178555 2:0.93973481 3:0.32205372 4:0.7176044 5:0.80237787 6:0.84588741 +-4 1:0.72282163 2:0.93973481 3:0.29554025 4:0.64471949 5:0.72634443 6:0.77062686 +-4 1:0.67693861 2:0.93973481 3:0.2669659 4:0.56720118 5:0.63868728 6:0.67673331 +4 1:0.86023804 2:0.49739676 3:0.53966638 4:0.77392585 5:0.84784447 6:0.89031641 +1.296591709971377 1:0.31779385 2:0.49739676 3:0.17094319 4:0.12195679 5:0.13277563 6:0.14165413 +4 1:0.68317784 2:0.49739676 3:0.37192301 4:0.52750491 5:0.62426522 6:0.6929947 +4 1:0.55611181 2:0.49739676 3:0.24752355 4:0.28326524 5:0.33261781 6:0.37104424 +4 1:0.7772257 2:0.49739676 3:0.43832146 4:0.63397606 5:0.7240692 6:0.78367237 +4 1:0.66186286 2:0.49739676 3:0.30599867 4:0.39201262 5:0.45927759 6:0.51239284 +4 1:0.94601776 2:0.04579546 3:0.69472114 4:0.97790884 5:0.9891237 6:0.993277 +4 1:0.98838404 2:0.04579546 3:0.90293444 4:0.99181622 5:0.99642641 6:0.9978864 +4 1:0.30006056 2:0.04579546 3:0.31879 4:0.45852885 5:0.59717781 6:0.71487885 +-4 1:0.44902891 2:0.04579546 3:0.35412414 4:0.55926446 5:0.70175505 6:0.79649177 +-4 1:0.69856222 2:0.04579546 3:0.45989947 4:0.82115248 5:0.92520734 6:0.9594384 +-4 1:0.67730161 2:0.04579546 3:0.44400319 4:0.77920819 5:0.88713866 6:0.92903178 +-4 1:0.64419192 2:0.04579546 3:0.42297435 4:0.72390263 5:0.83364665 6:0.88344569 +-4 1:0.80781899 2:0.04579546 3:0.52334234 4:0.88859427 5:0.94013924 6:0.95946903 +-4 1:0.78080761 2:0.04579546 3:0.499439 4:0.84012074 5:0.90229375 6:0.92936693 +4 1:0.97128596 2:0.014623935 3:0.90135809 4:0.99584619 5:0.9970631 6:0.99757649 +4 1:0.99645027 2:0.014623935 3:1 4:1 5:1 6:1 +-4 1:0.5326065 2:0.014623935 3:0.75468972 4:0.76017077 5:0.83753774 6:0.92265059 +-4 1:0.62757004 2:0.014623935 3:0.77708563 4:0.84258654 5:0.91016348 6:0.95440359 +-4 1:0.79306842 2:0.014623935 3:0.78900741 4:0.90386551 5:0.96905764 6:0.98466408 +-4 1:0.77722867 2:0.014623935 3:0.78701408 4:0.89679281 5:0.96056131 6:0.977629 +-4 1:0.75934622 2:0.014623935 3:0.78422805 4:0.88268036 5:0.94383829 6:0.96596858 +-4 1:0.8878718 2:0.014623935 3:0.81445984 4:0.96615706 5:0.98858241 6:0.99176534 +-4 1:0.88211614 2:0.014623935 3:0.81253935 4:0.95982371 5:0.98309178 6:0.9870796 +4 1:0.83805466 2:0.22767235 3:0.31750162 4:0.85145925 5:0.9121085 6:0.93772147 +4 1:0.86620985 2:0.22767235 3:0.35742938 4:0.89821492 5:0.94339974 6:0.96076173 +4 1:0.39289606 2:0.22767235 3:0.12019254 4:0.3951559 5:0.44657802 6:0.46771549 +4 1:0.48692411 2:0.22767235 3:0.13362033 4:0.43434224 5:0.49900609 6:0.53177669 +4 1:0.69743918 2:0.22767235 3:0.2263303 4:0.68859985 5:0.78706365 6:0.83662428 +4 1:0.65237548 2:0.22767235 3:0.19328493 4:0.60107975 5:0.69684945 6:0.74949279 +4 1:0.59461718 2:0.22767235 3:0.15963705 4:0.51010642 5:0.59283393 6:0.63883591 +4 1:0.77302727 2:0.22767235 3:0.26078021 4:0.76359704 5:0.8470807 6:0.8858359 +4 1:0.72953038 2:0.22767235 3:0.22331233 4:0.67735915 5:0.77029889 6:0.81802539 +4 1:0.87210923 2:0.16787772 3:0.69408521 4:0.91495146 5:0.94890261 6:0.96269344 +-4 1:0.81595959 2:0.08765484 3:0.52947327 4:0.7501341 5:0.82294191 6:0.86264385 +4 1:0.72562415 2:0.49739676 3:0.37130724 4:0.51472366 5:0.59961357 6:0.66258291 +-4 1:0.87135693 2:0.014623935 3:0.80905852 4:0.94637428 5:0.97242826 6:0.97946694 +-4 1:0.48910215 2:0.16787772 3:0.49792761 4:0.59161372 5:0.62979552 6:0.64254584 +-4 1:0.5685964 2:0.16787772 3:0.5149767 4:0.63026581 5:0.67890679 6:0.69964851 +-4 1:0.75935478 2:0.16787772 3:0.60695536 4:0.80906778 5:0.87125816 6:0.89810007 +-4 1:0.71788601 2:0.16787772 3:0.57600091 4:0.75310216 5:0.81471966 6:0.84249923 +-4 1:0.66516668 2:0.16787772 3:0.54473368 4:0.69254626 5:0.74796983 6:0.77177867 +4 1:0.81880869 2:0.16787772 3:0.64309172 4:0.86078024 5:0.90892223 6:0.92908907 +-4 1:0.78054558 2:0.16787772 3:0.60849279 4:0.80724494 5:0.86183239 6:0.88618408 +4 1:0.95353512 2:0.055921852 3:0.61526026 4:0.94655706 5:0.97211195 6:0.98210701 +4 1:0.98368527 2:0.055921852 3:0.7405327 4:0.96928567 5:0.9853799 6:0.99080378 +4 1:0.11318821 2:0.055921852 3:0.1590151 4:0.30536689 5:0.48614515 6:0.64344462 +4 1:0.30298819 2:0.055921852 3:0.19401703 4:0.41679982 5:0.61495039 6:0.74140301 +4 1:0.60614412 2:0.055921852 3:0.31791569 4:0.72365433 5:0.88324129 6:0.93484545 +4 1:0.58738733 2:0.055921852 3:0.29301498 4:0.67070014 5:0.83429953 6:0.89348041 +4 1:0.79496816 2:0.055921852 3:0.42192974 4:0.86711004 5:0.94030868 6:0.96084539 +4 1:0.77749763 2:0.055921852 3:0.38714172 4:0.81340799 5:0.90059649 6:0.93006702 +4 1:0.75215882 2:0.055921852 3:0.34721658 4:0.73960747 5:0.84370247 6:0.88485372 +4 1:0.89732805 2:0.58937038 3:0.58823535 4:0.80035053 5:0.86988422 6:0.90533033 +-4 1:0.9228759 2:0.58937038 3:0.65797705 4:0.87169952 5:0.92200942 6:0.94454256 +4 1:0.19504362 2:0.58937038 3:0.21585801 4:0.1754362 5:0.20844015 6:0.23846443 +4 1:0.34425894 2:0.58937038 3:0.24672569 4:0.24188506 5:0.29544562 6:0.33843061 +4 1:0.66407117 2:0.58937038 3:0.40045124 4:0.55415203 5:0.66628031 6:0.73418465 +4 1:0.60780044 2:0.58937038 3:0.34931828 4:0.4519606 5:0.54893247 6:0.61355219 +4 1:0.53476258 2:0.58937038 3:0.29851601 4:0.34826788 5:0.42168642 6:0.47203603 +4 1:0.79195776 2:0.58937038 3:0.47493233 4:0.66775916 5:0.76196439 6:0.81489875 +4 1:0.7415564 2:0.58937038 3:0.41507439 4:0.56413083 5:0.65815516 6:0.7166999 +4 1:0.82021207 2:1 3:0.37381485 4:0.7891612 5:0.87031145 6:0.90944281 +-3.795805084530972 1:0.85903236 2:1 3:0.43235998 4:0.86707094 5:0.92632217 6:0.95151451 +-4 1:0.25243046 2:1 3:0.084027451 4:0.15537936 5:0.17410072 6:0.17212333 +-4 1:0.35643487 2:1 3:0.10644455 4:0.21484368 5:0.25587544 6:0.27527817 +-4 1:0.57605414 2:1 3:0.19031962 4:0.43030863 5:0.5277316 6:0.59069772 +-4 1:0.49071444 2:1 3:0.14452095 4:0.31406915 5:0.38353445 6:0.42653517 +4 1:0.73255545 2:1 3:0.28883701 4:0.65284485 5:0.75623242 6:0.81297442 +0.4082706381617505 1:0.67015395 2:1 3:0.2367756 4:0.5367057 5:0.64063877 6:0.70451767 +-4 1:0.84450653 2:0.083369236 3:0.57279245 4:0.85249389 5:0.91751611 6:0.94621989 +-4 1:0.39559773 2:0.083369236 3:0.28184137 4:0.37025203 5:0.46733936 6:0.53517338 +-4 1:0.70621493 2:0.083369236 3:0.42718441 4:0.69347659 5:0.81124449 6:0.87136343 +-4 1:0.65615861 2:0.083369236 3:0.37833052 4:0.59301482 5:0.71772587 6:0.7905538 +-4 1:0.58837863 2:0.083369236 3:0.33229353 4:0.48675881 5:0.60141743 6:0.67458413 +-4 1:0.77687144 2:0.083369236 3:0.48094343 4:0.76665994 5:0.86191893 6:0.90760934 +-1.966116876631112 1:0.72849768 2:0.083369236 3:0.42082971 4:0.66591147 5:0.77995959 6:0.84260661 +-3.906831378063804 1:0.66320082 2:0.083369236 3:0.36350305 4:0.54888271 5:0.66506794 6:0.73685112 +4 1:0.84500499 2:0.42532178 3:0.43562507 4:0.80721931 5:0.87934044 6:0.91434143 +4 1:0.8874543 2:0.42532178 3:0.50912639 4:0.87959883 5:0.93223488 6:0.95450335 +4 1:0.31032192 2:0.42532178 3:0.18976794 4:0.30662908 5:0.34637104 6:0.3661022 +4 1:0.41026349 2:0.42532178 3:0.20589097 4:0.35241209 5:0.40358156 6:0.42577381 +4 1:0.67552108 2:0.42532178 3:0.30879992 4:0.60375124 5:0.70097073 6:0.75507206 +4 1:0.62772585 2:0.42532178 3:0.27349745 4:0.5196735 5:0.60339149 6:0.65103342 +4 1:0.5741386 2:0.42532178 3:0.24033766 4:0.43855753 5:0.50243186 6:0.53322825 +4 1:0.7629976 2:0.42532178 3:0.35347476 4:0.69239941 5:0.78245146 6:0.83117443 +4 1:0.71746409 2:0.42532178 3:0.31296983 4:0.60525302 5:0.69243388 6:0.7432587 +-4 1:0.73137955 2:0.16787772 3:0.57222383 4:0.74405775 5:0.79993424 6:0.82484891 +4 1:0.67383121 2:0.58937038 3:0.35481019 4:0.45269287 5:0.53578336 6:0.59116487 +4 1:0.5905971 2:1 3:0.18559792 4:0.41535212 5:0.50422336 6:0.56173557 +4 1:0.66157018 2:0.42532178 3:0.27479904 4:0.51802649 5:0.59270541 6:0.63560969 +-4 1:0.66827754 2:0.54342577 3:0.18169339 4:0.50290989 5:0.59875259 6:0.65332628 +4 1:0.85027066 2:0.20820673 3:0.40997978 4:0.82462749 5:0.89794736 6:0.93142825 +4 1:0.87892054 2:0.20820673 3:0.45891267 4:0.87823329 5:0.93535353 6:0.95883927 +4 1:0.3986268 2:0.20820673 3:0.17753958 4:0.33495583 5:0.39777832 6:0.44399359 +-4 1:0.48997993 2:0.20820673 3:0.20172681 4:0.39715881 5:0.47368229 6:0.52781628 +4 1:0.7022939 2:0.20820673 3:0.31094767 4:0.6676259 5:0.77726116 6:0.83518027 +-4 1:0.65773092 2:0.20820673 3:0.27420721 4:0.57889989 5:0.68485118 6:0.74837036 +0.2951376518668717 1:0.60031736 2:0.20820673 3:0.23419121 4:0.48018865 5:0.57200972 6:0.63197473 +4 1:0.77623676 2:0.20820673 3:0.3510016 4:0.74206651 5:0.83508543 6:0.88101902 +4 1:0.73562396 2:0.20820673 3:0.31004997 4:0.6557112 5:0.75585014 6:0.81164989 +-4 1:0.67923081 2:0.20820673 3:0.26679137 4:0.55816547 5:0.65579282 6:0.71593631 +4 1:0.83968539 2:0.54342577 3:0.32439292 4:0.78747769 5:0.87303614 6:0.91271252 +4 1:0.86656342 2:0.54342577 3:0.37898741 4:0.85252726 5:0.92049615 6:0.94848246 +-4 1:0.42728303 2:0.54342577 3:0.10123262 4:0.31581962 5:0.38571265 6:0.42827036 +-4 1:0.63194526 2:0.54342577 3:0.18169045 4:0.51611903 5:0.62179755 6:0.68216176 +4 1:0.56954706 2:0.54342577 3:0.14271477 4:0.41491191 5:0.50173488 6:0.55220392 +4 1:0.76753176 2:0.54342577 3:0.26295318 4:0.6905031 5:0.79291823 6:0.84469464 +-4 1:0.72348649 2:0.54342577 3:0.22334634 4:0.60145902 5:0.70573225 6:0.76318544 +4 1:0.83584492 2:0.047285912 3:0.53826775 4:0.933335 5:0.95948954 6:0.96870909 +4 1:0.85530855 2:0.047285912 3:0.55323777 4:0.95113339 5:0.97249918 6:0.9795177 +-4 1:0.53835734 2:0.047285912 3:0.41965074 4:0.71632669 5:0.73953043 6:0.73487553 +-4 1:0.59175144 2:0.047285912 3:0.43113594 4:0.74141738 5:0.76929188 6:0.77018949 +-4 1:0.75962366 2:0.047285912 3:0.49613729 4:0.87838146 5:0.91688438 6:0.93150362 +-4 1:0.72043129 2:0.047285912 3:0.47217411 4:0.83138845 5:0.8704229 6:0.88419439 +-4 1:0.67287449 2:0.047285912 3:0.44652268 4:0.77691812 5:0.81043483 6:0.8177009 +-4 1:0.8023177 2:0.047285912 3:0.51559706 4:0.90512389 5:0.93743101 6:0.9492968 +-4 1:0.76751376 2:0.047285912 3:0.49225957 4:0.86357299 5:0.89948127 6:0.91221155 +-4 1:0.72124785 2:0.047285912 3:0.46606653 4:0.81323145 5:0.84847474 6:0.85892657 diff --git a/libavfilter/vf_vmaf.c b/libavfilter/vf_vmaf.c new file mode 100644 index 0000000..4cb93eb --- /dev/null +++ b/libavfilter/vf_vmaf.c @@ -0,0 +1,945 @@ +/* + * Copyright (c) 2017 Ronald S. Bultje + * Copyright (c) 2017 Ashish Pratap Singh + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Calculate the VMAF between two input videos. + */ + +#include "libavutil/avstring.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" +#include "avfilter.h" +#include "dualinput.h" +#include "drawutils.h" +#include "formats.h" +#include "internal.h" +#include "video.h" +#include "adm.h" +#include "vmaf_motion.h" +#include "vif.h" +#include "vmaf.h" + +typedef struct VMAFContext { + const AVClass *class; + FFDualInputContext dinput; + const AVPixFmtDescriptor *desc; + int width; + int height; + uint8_t called; + double score; + double scores[8]; + double score_num; + double score_den; + int conv_filter[5]; + float *ref_data; + float *main_data; + float *adm_data_buf; + float *adm_temp_lo; + float *adm_temp_hi; + uint16_t *prev_blur_data; + uint16_t *blur_data; + uint16_t *temp_data; + float *vif_data_buf; + float *vif_temp; + double prev_motion_score; + double vmaf_score; + uint64_t nb_frames; + char *model_path; + int enable_transform; + char *pool; + svm_model *svm_model_ptr; + svm_node* nodes; + void (*pool_method)(double *score, double curr); + double prediction; +} VMAFContext; + +#define OFFSET(x) offsetof(VMAFContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption vmaf_options[] = { + {"model_path", "Set the model to be used for computing vmaf.", OFFSET(model_path), AV_OPT_TYPE_STRING, {.str="libavfilter/data/vmaf_v0.6.1.pkl.model"}, 0, 1, FLAGS}, + {"enable_transform", "Enables transform for computing vmaf.", OFFSET(enable_transform), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS}, + {"pool", "Set the pool method to be used for computing vmaf.", OFFSET(pool), AV_OPT_TYPE_STRING, {.str="mean"}, 0, 1, FLAGS}, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(vmaf); + +#define MAX_ALIGN 32 +#define ALIGN_CEIL(x) ((x) + ((x) % MAX_ALIGN ? MAX_ALIGN - (x) % MAX_ALIGN : 0)) + +enum { C_SVC, NU_SVC, ONE_CLASS, EPSILON_SVR, NU_SVR }; /** svm_type */ +enum { LINEAR, POLY, RBF, SIGMOID, PRECOMPUTED }; /** kernel_type */ + +#define swap(type, x, y) { type t=x; x=y; y=t; } + +static inline double power(double base, int times) +{ + double tmp = base, ret = 1.0; + + for(int t = times; t > 0; t /= 2) { + if(t % 2 == 1) { + ret *= tmp; + } + tmp = tmp * tmp; + } + return ret; +} + +static double dot(const svm_node *px, const svm_node *py) +{ + double sum = 0; + while(px->index != -1 && py->index != -1) { + if(px->index == py->index) { + sum += px->value * py->value; + px++; + py++; + } else { + if(px->index > py->index) { + py++; + } else { + px++; + } + } + } + return sum; +} + +static double k_function(const svm_node *x, const svm_node *y, + const svm_parameter *param) +{ + switch(param->kernel_type) { + case LINEAR: + return dot(x, y); + case POLY: + return power(param->gamma * dot(x, y) + param->coef0, param->degree); + case RBF: { + double sum = 0; + while(x->index != -1 && y->index !=-1) { + if(x->index == y->index) { + double d = x->value - y->value; + sum += d * d; + x++; + y++; + } else { + if(x->index > y->index) { + sum += y->value * y->value; + y++; + } else { + sum += x->value * x->value; + x++; + } + } + } + + while(x->index != -1) { + sum += x->value * x->value; + x++; + } + + while(y->index != -1) { + sum += y->value * y->value; + y++; + } + + return exp(-param->gamma * sum); + } + case SIGMOID: + return tanh(param->gamma * dot(x, y) + param->coef0); + case PRECOMPUTED: + return x[(int)(y->value)].value; + default: + return 0; + } +} + +#define INF HUGE_VAL +#define TAU 1e-12 +#define Malloc(type,n) (type *)malloc((n)*sizeof(type)) + +static double svm_predict_values(const svm_model *model, const svm_node *x, double* dec_values) +{ + int i, j; + if(model->param.svm_type == ONE_CLASS || + model->param.svm_type == EPSILON_SVR || + model->param.svm_type == NU_SVR) { + double *sv_coef = model->sv_coef[0]; + double sum = 0; + for(i = 0; i < model->l; i++) { + sum += sv_coef[i] * k_function(x, model->SV[i], &model->param); + } + sum -= model->rho[0]; + *dec_values = sum; + + if(model->param.svm_type == ONE_CLASS) { + return (sum > 0) ? 1 : -1; + } else { + return sum; + } + } else { + int nr_class = model->nr_class; + int l = model->l; + int *start; + int *vote; + int p; + int vote_max_idx; + double *kvalue = Malloc(double,l); + for(i = 0; i < l; i++) { + kvalue[i] = k_function(x, model->SV[i], &model->param); + } + + start = Malloc(int,nr_class); + start[0] = 0; + for(i = 1; i < nr_class; i++) { + start[i] = start[i - 1] + model->nSV[i - 1]; + } + + vote = Malloc(int,nr_class); + for(i = 0; i < nr_class; i++) { + vote[i] = 0; + } + + p=0; + for(i = 0; i < nr_class; i++) { + for(j = i + 1; j < nr_class; j++) { + double sum = 0; + int si = start[i]; + int sj = start[j]; + int ci = model->nSV[i]; + int cj = model->nSV[j]; + + int k; + double *coef1 = model->sv_coef[j - 1]; + double *coef2 = model->sv_coef[i]; + for(k = 0; k < ci; k++) { + sum += coef1[si + k] * kvalue[si + k]; + } + for(k = 0; k < cj; k++) { + sum += coef2[sj + k] * kvalue[sj + k]; + } + sum -= model->rho[p]; + dec_values[p] = sum; + + if(dec_values[p] > 0) { + vote[i]++; + } else { + vote[j]++; + } + p++; + } + } + + vote_max_idx = 0; + for(i = 1; i < nr_class; i++) { + if(vote[i] > vote[vote_max_idx]) { + vote_max_idx = i; + } + } + + av_free(kvalue); + av_free(start); + av_free(vote); + return model->label[vote_max_idx]; + } +} + +static double svm_predict(const svm_model *model, const svm_node *x) +{ + int nr_class = model->nr_class; + double *dec_values; + double pred_result; + if(model->param.svm_type == ONE_CLASS || + model->param.svm_type == EPSILON_SVR || + model->param.svm_type == NU_SVR) { + dec_values = Malloc(double, 1); + } else { + dec_values = Malloc(double, nr_class * (nr_class - 1) / 2); + } + pred_result = svm_predict_values(model, x, dec_values); + av_free(dec_values); + return pred_result; +} + +static const char *svm_type_table[] = +{ + "c_svc","nu_svc","one_class","epsilon_svr","nu_svr",NULL +}; + +static const char *kernel_type_table[]= +{ + "linear","polynomial","rbf","sigmoid","precomputed",NULL +}; + +static char *line = NULL; +static int max_line_len; + +static char* readline(FILE *input) +{ + int len; + + if(fgets(line,max_line_len,input) == NULL) { + return NULL; + } + + while(strrchr(line, '\n') == NULL) { + max_line_len *= 2; + line = (char *) realloc(line, max_line_len); + len = (int) strlen(line); + if(fgets(line+len, max_line_len-len, input) == NULL) { + break; + } + } + return line; +} + +/** FSCANF helps to handle fscanf failures. + * Its do-while block avoids the ambiguity when + * if (...) + * FSCANF(); + * is used + */ +#define FSCANF(_stream, _format, _var) do{ if (fscanf(_stream, _format, _var) != 1) return 0; }while(0) +static int read_model_header(FILE *fp, svm_model* model, AVFilterContext *ctx) +{ + svm_parameter* param = &model->param; + char cmd[81]; + int i; + while(1) { + FSCANF(fp, "%80s", cmd); + + if(av_strcasecmp(cmd, "svm_type") == 0) { + FSCANF(fp, "%80s", cmd); + for(i = 0; svm_type_table[i]; i++) { + if(av_strcasecmp(svm_type_table[i], cmd) == 0) { + param->svm_type = i; + break; + } + } + if(svm_type_table[i] == NULL) { + av_log(ctx, AV_LOG_ERROR, "unknown svm type.\n"); + return 0; + } + } else if(av_strcasecmp(cmd, "kernel_type") == 0) { + FSCANF(fp, "%80s", cmd); + for(i = 0; kernel_type_table[i]; i++) { + if(av_strcasecmp(kernel_type_table[i], cmd) == 0) { + param->kernel_type = i; + break; + } + } + if(kernel_type_table[i] == NULL) { + av_log(ctx, AV_LOG_ERROR, "unknown kernel function.\n"); + return 0; + } + } else if(av_strcasecmp(cmd, "degree") == 0) { + FSCANF(fp, "%d", ¶m->degree); + } else if(av_strcasecmp(cmd, "gamma") == 0) { + FSCANF(fp, "%lf", ¶m->gamma); + } else if(av_strcasecmp(cmd, "coef0") == 0) { + FSCANF(fp, "%lf", ¶m->coef0); + } else if(av_strcasecmp(cmd, "nr_class") == 0) { + FSCANF(fp,"%d",&model->nr_class); + } else if(av_strcasecmp(cmd, "total_sv") == 0) { + FSCANF(fp, "%d", &model->l); + } else if(av_strcasecmp(cmd, "rho")==0) { + int n = model->nr_class * (model->nr_class-1) / 2; + model->rho = Malloc(double, n); + for(i = 0; i < n; i++) { + FSCANF(fp, "%lf", &model->rho[i]); + } + } else if(av_strcasecmp(cmd, "label") == 0) { + int n = model->nr_class; + model->label = Malloc(int, n); + for(i = 0; i < n; i++) { + FSCANF(fp, "%d", &model->label[i]); + } + } else if(av_strcasecmp(cmd, "probA") == 0) { + int n = model->nr_class * (model->nr_class - 1) / 2; + model->probA = Malloc(double, n); + for(i = 0;i < n; i++) { + FSCANF(fp, "%lf", &model->probA[i]); + } + } else if(av_strcasecmp(cmd, "probB") == 0) { + int n = model->nr_class * (model->nr_class - 1) / 2; + model->probB = Malloc(double,n); + for(i = 0; i < n; i++) { + FSCANF(fp, "%lf", &model->probB[i]); + } + } else if(av_strcasecmp(cmd, "nr_sv") == 0) { + int n = model->nr_class; + model->nSV = Malloc(int, n); + for(i = 0; i < n; i++) { + FSCANF(fp, "%d", &model->nSV[i]); + } + } else if(av_strcasecmp(cmd, "SV") == 0) { + while(1) { + int c = getc(fp); + if(c == EOF || c == '\n') { + break; + } + } + break; + } else { + av_log(ctx, AV_LOG_ERROR, "unknown text in model file: [%s]\n", cmd); + return 0; + } + } + + return 1; + +} + +static svm_model *svm_load_model(const char *model_file_name, AVFilterContext *ctx) +{ + FILE *fp = fopen(model_file_name, "rb"); + int i, j, k, l, m; + char *p, *endptr, *idx, *val; + svm_model *model; + + int elements; + long pos; + svm_node *x_space; + + if(fp == NULL) { + return NULL; + } + + /** read parameters */ + model = Malloc(svm_model,1); + model->rho = NULL; + model->probA = NULL; + model->probB = NULL; + model->sv_indices = NULL; + model->label = NULL; + model->nSV = NULL; + + /** read header */ + if (!read_model_header(fp, model, ctx)) { + av_log(ctx, AV_LOG_ERROR, "ERROR: fscanf failed to read model\n"); + av_free(model->rho); + av_free(model->label); + av_free(model->nSV); + av_free(model); + return NULL; + } + + /** read sv_coef and SV */ + elements = 0; + pos = ftell(fp); + + max_line_len = 1024; + line = Malloc(char, max_line_len); + + while(readline(fp) != NULL) { + p = strtok(line, ":"); + while(1) { + p = strtok(NULL, ":"); + if(p == NULL) { + break; + } + elements++; + } + } + elements += model->l; + + fseek(fp, pos, SEEK_SET); + + m = model->nr_class - 1; + l = model->l; + model->sv_coef = Malloc(double *,m); + for(i = 0; i < m; i++) { + model->sv_coef[i] = Malloc(double,l); + } + model->SV = Malloc(svm_node*, l); + x_space = NULL; + if(l > 0) { + x_space = Malloc(svm_node, elements); + } + + j=0; + for(i = 0; i < l; i++) { + readline(fp); + model->SV[i] = &x_space[j]; + + p = strtok(line, " \t"); + model->sv_coef[0][i] = strtod(p, &endptr); + for(k = 1; k < m; k++) { + p = strtok(NULL, " \t"); + model->sv_coef[k][i] = strtod(p, &endptr); + } + + while(1) { + idx = strtok(NULL, ":"); + val = strtok(NULL, " \t"); + + if(val == NULL) { + break; + } + x_space[j].index = (int) strtol(idx, &endptr, 10); + x_space[j].value = strtod(val, &endptr); + + j++; + } + x_space[j++].index = -1; + } + av_free(line); + + if (ferror(fp) != 0 || fclose(fp) != 0) { + return NULL; + } + + model->free_sv = 1; + return model; +} + +static void svm_free_model_content(svm_model* model_ptr) +{ + int i; + if(model_ptr->free_sv && model_ptr->l > 0 && model_ptr->SV != NULL) { + av_free((void *) (model_ptr->SV[0])); + } + if(model_ptr->sv_coef) { + for(i = 0; i < model_ptr->nr_class - 1; i++) { + av_free(model_ptr->sv_coef[i]); + } + } + + av_free(model_ptr->SV); + model_ptr->SV = NULL; + + av_free(model_ptr->sv_coef); + model_ptr->sv_coef = NULL; + + av_free(model_ptr->rho); + model_ptr->rho = NULL; + + av_free(model_ptr->label); + model_ptr->label= NULL; + + av_free(model_ptr->probA); + model_ptr->probA = NULL; + + av_free(model_ptr->probB); + model_ptr->probB= NULL; + + av_free(model_ptr->sv_indices); + model_ptr->sv_indices = NULL; + + av_free(model_ptr->nSV); + model_ptr->nSV = NULL; +} + +static void svm_free_and_destroy_model(svm_model** model_ptr_ptr) +{ + if(model_ptr_ptr != NULL && *model_ptr_ptr != NULL) { + svm_free_model_content(*model_ptr_ptr); + av_free(*model_ptr_ptr); + *model_ptr_ptr = NULL; + } +} + +static void mean(double *score, double curr) +{ + *score += curr; +} + +static void min(double *score, double curr) +{ + *score = FFMIN(*score, curr); +} + +static void harmonic_mean(double *score, double curr) +{ + *score += 1.0 / (curr + 1.0); +} + +#define offset_fn(type, bits) \ + static void offset_##bits##bit(VMAFContext *s, const AVFrame *ref, AVFrame *main, int stride) \ +{ \ + int w = s->width; \ + int h = s->height; \ + int i,j; \ + \ + ptrdiff_t ref_stride = ref->linesize[0]; \ + ptrdiff_t main_stride = main->linesize[0]; \ + \ + const type *ref_ptr = (const type *) ref->data[0]; \ + const type *main_ptr = (const type *) main->data[0]; \ + \ + float *ref_ptr_data = s->ref_data; \ + float *main_ptr_data = s->main_data; \ + \ + for(i = 0; i < h; i++) { \ + for(j = 0; j < w; j++) { \ + ref_ptr_data[j] = (float) ref_ptr[j]; \ + main_ptr_data[j] = (float) main_ptr[j]; \ + } \ + ref_ptr += ref_stride / sizeof(type); \ + ref_ptr_data += stride / sizeof(float); \ + main_ptr += main_stride / sizeof(type); \ + main_ptr_data += stride / sizeof(float); \ + } \ +} + +offset_fn(uint8_t, 8); +offset_fn(uint16_t, 10); + +static int compute_vmaf(const AVFrame *ref, AVFrame *main, void *ctx) +{ + VMAFContext *s = (VMAFContext *) ctx; + + size_t motion_data_sz; + int i,j; + ptrdiff_t ref_stride; + ptrdiff_t ref_px_stride; + ptrdiff_t stride; + ptrdiff_t motion_stride; + ptrdiff_t motion_px_stride; + int w = s->width; + int h = s->height; + + ref_stride = ref->linesize[0]; + + stride = ALIGN_CEIL(w * sizeof(float)); + motion_stride = ALIGN_CEIL(w * sizeof(uint16_t)); + motion_px_stride = motion_stride / sizeof(uint16_t); + + /** Offset ref and main pixel by OPT_RANGE_PIXEL_OFFSET */ + if (s->desc->comp[0].depth <= 8) { + offset_8bit(s, ref, main, stride); + } else { + offset_10bit(s, ref, main, stride); + } + + motion_data_sz = (size_t)motion_stride * s->height; + + compute_adm2(s->ref_data, s->main_data, w, h, stride, stride, &s->score, + &s->score_num, &s->score_den, s->scores, s->adm_data_buf, + s->adm_temp_lo, s->adm_temp_hi); + s->nodes[0].index = 1; + s->nodes[0].value = (double)(slopes[1]) * (double)(s->score_num / s->score_den) + (double)(intercepts[1]); + + if (s->desc->comp[0].depth <= 8) { + ref_px_stride = ref_stride / sizeof(uint8_t); + convolution_f32(s->conv_filter, 5, (const uint8_t *) ref->data[0], + s->blur_data, s->temp_data, s->width, s->height, + ref_px_stride, motion_px_stride, 8); + } else { + ref_px_stride = ref_stride / sizeof(uint16_t); + convolution_f32(s->conv_filter, 5, (const uint16_t *) ref->data[0], + s->blur_data, s->temp_data, s->width, s->height, + ref_px_stride, motion_px_stride, 10); + } + + if(!s->nb_frames) { + s->score = 0.0; + } else { + compute_vmafmotion(s->prev_blur_data, s->blur_data, s->width, s->height, + motion_stride, motion_stride, &s->score); + } + + memcpy(s->prev_blur_data, s->blur_data, motion_data_sz); + + s->nodes[1].index = 2; + s->nodes[1].value = (double)(slopes[2]) * (double)FFMIN(s->prev_motion_score, s->score) + (double)(intercepts[2]); + s->prev_motion_score = s->score; + + compute_vif2(s->ref_data, s->main_data, w, h, stride, stride, &s->score, + &s->score_num, &s->score_den, s->scores, s->vif_data_buf, + s->vif_temp); + + j = 0; + for(i = 0; j < 4; i += 2) { + s->nodes[j+2].index = j+3; + s->nodes[j+2].value = (double)(slopes[j+3]) * (double)((s->scores[i]) / (s->scores[i+1])) + (double)(intercepts[j+3]); + j++; + } + + s->prediction = svm_predict(s->svm_model_ptr, s->nodes); + + if (!av_strcasecmp(norm_type, "linear_rescale")) { + /** denormalize */ + s->prediction = (s->prediction - (double)(intercepts[0])) / (double)(slopes[0]); + } + + /** score transform */ + if (s->enable_transform) { + double value = 0.0; + + /** quadratic transform */ + value += (double)(score_transform[0]); + value += (double)(score_transform[1]) * s->prediction; + value += (double)(score_transform[2]) * s->prediction * s->prediction; + + /** rectification */ + if (value < s->prediction) { + value = s->prediction; + } + + s->prediction = value; + } + + s->pool_method(&s->vmaf_score, s->prediction); + return 0; +} + +static AVFrame *do_vmaf(AVFilterContext *ctx, AVFrame *main, const AVFrame *ref) +{ + VMAFContext *s = ctx->priv; + + compute_vmaf(ref, main, s); + + s->nb_frames++; + + return main; +} + +static av_cold int init(AVFilterContext *ctx) +{ + VMAFContext *s = ctx->priv; + + if(!s->called) { + int i; + for(i = 0; i < 5; i++) { + s->conv_filter[i] = lrint(FILTER_5[i] * (1 << N)); + } + + s->svm_model_ptr = svm_load_model(s->model_path, ctx); + s->nodes = (svm_node *) av_malloc(sizeof(svm_node) * (6 + 1)); + s->nodes[6].index = -1; + if(!av_strcasecmp(s->pool, "mean")) { + s->pool_method = mean; + } else if(!av_strcasecmp(s->pool, "min")) { + s->vmaf_score = INT_MAX; + s->pool_method = min; + } else if(!av_strcasecmp(s->pool, "harmonic")) { + s->pool_method = harmonic_mean; + } + } + + s->called = 1; + s->dinput.process = do_vmaf; + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P10LE, AV_PIX_FMT_YUV422P10LE, AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NONE + }; + + AVFilterFormats *fmts_list = ff_make_format_list(pix_fmts); + if (!fmts_list) + return AVERROR(ENOMEM); + return ff_set_common_formats(ctx, fmts_list); +} + +static int config_input_ref(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + VMAFContext *s = ctx->priv; + ptrdiff_t stride; + size_t data_sz; + ptrdiff_t adm_buf_stride; + size_t adm_buf_sz; + ptrdiff_t vif_buf_stride; + size_t vif_buf_sz; + + if (ctx->inputs[0]->w != ctx->inputs[1]->w || + ctx->inputs[0]->h != ctx->inputs[1]->h) { + av_log(ctx, AV_LOG_ERROR, "Width and height of input videos must be same.\n"); + return AVERROR(EINVAL); + } + if (ctx->inputs[0]->format != ctx->inputs[1]->format) { + av_log(ctx, AV_LOG_ERROR, "Inputs must be of same pixel format.\n"); + return AVERROR(EINVAL); + } + + s->desc = av_pix_fmt_desc_get(inlink->format); + s->width = ctx->inputs[0]->w; + s->height = ctx->inputs[0]->h; + + + stride = ALIGN_CEIL(s->width * sizeof(float)); + data_sz = (size_t)stride * s->height; + + if (!(s->ref_data = av_malloc(data_sz))) { + return AVERROR(ENOMEM); + } + + if (!(s->main_data = av_malloc(data_sz))) { + return AVERROR(ENOMEM); + } + + adm_buf_stride = ALIGN_CEIL(((s->width + 1) / 2) * sizeof(float)); + adm_buf_sz = (size_t)adm_buf_stride * ((s->height + 1) / 2); + + if (SIZE_MAX / adm_buf_sz < 35) { + av_log(ctx, AV_LOG_ERROR, "error: SIZE_MAX / buf_sz_one < 35.\n"); + return AVERROR(EINVAL); + } + + if (!(s->adm_data_buf = av_malloc(adm_buf_sz * 35))) { + return AVERROR(ENOMEM); + } + + if (!(s->adm_temp_lo = av_malloc(stride))) { + return AVERROR(ENOMEM); + } + if (!(s->adm_temp_hi = av_malloc(stride))) { + return AVERROR(ENOMEM); + } + + stride = ALIGN_CEIL(s->width * sizeof(uint16_t)); + data_sz = (size_t)stride * s->height; + + if (!(s->prev_blur_data = av_mallocz(data_sz))) { + return AVERROR(ENOMEM); + } + + if (!(s->blur_data = av_mallocz(data_sz))) { + return AVERROR(ENOMEM); + } + + if (!(s->temp_data = av_mallocz(data_sz))) { + return AVERROR(ENOMEM); + } + + vif_buf_stride = ALIGN_CEIL(s->width * sizeof(float)); + vif_buf_sz = (size_t)vif_buf_stride * s->height; + + if (SIZE_MAX / data_sz < 15) { + av_log(ctx, AV_LOG_ERROR, "error: SIZE_MAX / buf_sz < 15\n"); + return AVERROR(EINVAL); + } + + if (!(s->vif_data_buf = av_malloc(vif_buf_sz * 16))) { + return AVERROR(ENOMEM); + } + + if (!(s->vif_temp = av_malloc(s->width * sizeof(float)))) { + return AVERROR(ENOMEM); + } + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + VMAFContext *s = ctx->priv; + AVFilterLink *mainlink = ctx->inputs[0]; + int ret; + + outlink->w = mainlink->w; + outlink->h = mainlink->h; + outlink->time_base = mainlink->time_base; + outlink->sample_aspect_ratio = mainlink->sample_aspect_ratio; + outlink->frame_rate = mainlink->frame_rate; + if ((ret = ff_dualinput_init(ctx, &s->dinput)) < 0) + return ret; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *inpicref) +{ + VMAFContext *s = inlink->dst->priv; + return ff_dualinput_filter_frame(&s->dinput, inlink, inpicref); +} + +static int request_frame(AVFilterLink *outlink) +{ + VMAFContext *s = outlink->src->priv; + return ff_dualinput_request_frame(&s->dinput, outlink); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + VMAFContext *s = ctx->priv; + + if (s->nb_frames > 0) { + if(!av_strcasecmp(s->pool, "mean")) { + s->vmaf_score = s->vmaf_score / s->nb_frames; + } else if(!av_strcasecmp(s->pool, "min")) { + s->vmaf_score = s->vmaf_score; + } else if(!av_strcasecmp(s->pool, "harmonic")) { + s->vmaf_score = 1.0 / (s->vmaf_score / s->nb_frames) - 1.0; + } + + av_log(ctx, AV_LOG_INFO, "VMAF Score: %.3f\n", s->vmaf_score); + + svm_free_and_destroy_model((svm_model **)&s->svm_model_ptr); + + av_free(s->ref_data); + av_free(s->main_data); + av_free(s->adm_data_buf); + av_free(s->adm_temp_lo); + av_free(s->adm_temp_hi); + av_free(s->prev_blur_data); + av_free(s->blur_data); + av_free(s->temp_data); + av_free(s->vif_data_buf); + av_free(s->vif_temp); + } + + ff_dualinput_uninit(&s->dinput); +} + +static const AVFilterPad vmaf_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + },{ + .name = "reference", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input_ref, + }, + { NULL } +}; + +static const AVFilterPad vmaf_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + .request_frame = request_frame, + }, + { NULL } +}; + +AVFilter ff_vf_vmaf = { + .name = "vmaf", + .description = NULL_IF_CONFIG_SMALL("Calculate the VMAF between two video streams."), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(VMAFContext), + .priv_class = &vmaf_class, + .inputs = vmaf_inputs, + .outputs = vmaf_outputs, +}; diff --git a/libavfilter/vmaf.h b/libavfilter/vmaf.h new file mode 100644 index 0000000..6d16f60 --- /dev/null +++ b/libavfilter/vmaf.h @@ -0,0 +1,138 @@ +/* + * Copyright (c) 2017 Ronald S. Bultje + * Copyright (c) 2017 Ashish Pratap Singh + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Calculate the VMAF between two input videos. + */ + +/** Normalization type */ +const char *norm_type = "linear_rescale"; + +/** cliping to be applied on vmaf score */ +const double score_clip[2] = { + 0.0, + 100.0 +}; + +/** feature vector */ +const char *feature_names[6] = { + "VMAF_feature_adm2_score", + "VMAF_feature_motion2_score", + "VMAF_feature_vif_scale0_score", + "VMAF_feature_vif_scale1_score", + "VMAF_feature_vif_scale2_score", + "VMAF_feature_vif_scale3_score" +}; + +const double intercepts[7] = { + -0.3092981927591963, + -1.7993968597186747, + -0.003017198086831897, + -0.1728125095425364, + -0.5294309090081222, + -0.7577185792093722, + -1.083428597549764 +}; + +const double slopes[7] = { + 0.012020766332648465, + 2.8098077502505414, + 0.06264407466686016, + 1.222763456258933, + 1.5360318811084146, + 1.7620864995501058, + 2.08656468286432 +}; + +/** transform constants */ +const double score_transform[3] = { + 1.70674692, + 1.72643844, + -0.00705305 +}; + +typedef struct { + int index; + double value; +} svm_node; + +typedef struct { + int l; + double *y; + svm_node **x; +} svm_problem; + +typedef struct { + int svm_type; + int kernel_type; + int degree; /** for poly */ + double gamma; /** for poly/rbf/sigmoid */ + double coef0; /** for poly/sigmoid */ + + /** these are for training only */ + double cache_size; /** in MB */ + double eps; /** stopping criteria */ + double C; /** for C_SVC, EPSILON_SVR and NU_SVR */ + int nr_weight; /** for C_SVC */ + int *weight_label; /** for C_SVC */ + double* weight; /** for C_SVC */ + double nu; /** for NU_SVC, ONE_CLASS, and NU_SVR */ + double p; /** for EPSILON_SVR */ + int shrinking; /** use the shrinking heuristics */ + int probability; /** do probability estimates */ +} svm_parameter; + +/** + * svm_model + */ +typedef struct { + svm_parameter param; /** parameter */ + int nr_class; /** number of classes, = 2 in regression/one class svm */ + int l; /** total #SV */ + svm_node **SV; /** SVs (SV[l]) */ + double **sv_coef; /** coefficients for SVs in decision functions (sv_coef[k-1][l]) */ + double *rho; /** constants in decision functions (rho[k*(k-1)/2]) */ + double *probA; /** pariwise probability information */ + double *probB; + int *sv_indices; /** sv_indices[0,...,nSV-1] are values in [1,...,num_traning_data] to indicate SVs in the training set */ + + /** for classification only */ + + int *label; /** label of each class (label[k]) */ + int *nSV; /** number of SVs for each class (nSV[k]) */ + /** nSV[0] + nSV[1] + ... + nSV[k-1] = l */ + int free_sv; /** 1 if svm_model is created by svm_load_model*/ + /** 0 if svm_model is created by svm_train */ +} svm_model; + + +typedef struct { + const svm_node **x; + double *x_square; + + // svm_parameter + const int kernel_type; + const int degree; + const double gamma; + const double coef0; +} Kernel; +