From patchwork Sat Mar 4 20:55:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ashish Singh X-Patchwork-Id: 2753 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.31.14 with SMTP id f14csp797630vsf; Sat, 4 Mar 2017 13:21:22 -0800 (PST) X-Received: by 10.28.182.7 with SMTP id g7mr8113332wmf.108.1488662482820; Sat, 04 Mar 2017 13:21:22 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d62si8213057wmc.54.2017.03.04.13.21.22; Sat, 04 Mar 2017 13:21:22 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E7F546882BA; Sat, 4 Mar 2017 23:21:06 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f68.google.com (mail-pg0-f68.google.com [74.125.83.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 90CE86882A7 for ; Sat, 4 Mar 2017 23:20:59 +0200 (EET) Received: by mail-pg0-f68.google.com with SMTP id r1so281553pgr.1 for ; Sat, 04 Mar 2017 13:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=VLXfF6SI9rHpCoArw7nkb+yQhor6UjZlGQUi/KqxSkg=; b=ZhnRyMlFnoVFm0l16BiStc5fab36SKltrFDIFpd4DV1EwYVLoQY/iVni6eHKb8Ruxn 8W4/DFJtnLT8PwVPhM40oQs4TOEP7ax/jBgIXGfwTxzBPiCfT08d2ty6yQjofPahxOPX 6Wthh5/bGME7jEw/nGhoN05I4VPqgGHkF+WCXkUmi4yUgjhH1VVV2GPX9M9lYmwH9srJ zfMcy6lBP0OzxpjOyHEose2cMnTx1JLRNfrlG1+WO6kWI7DK1z/DPv0nEYw989ASmlBc /cXRzH2i/QjUTQQFmGiC0GoGqqp84srqNI7DpEwJdLkJ2fPohRNH/2WiKpzZnJJy/L/G 3Frw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=VLXfF6SI9rHpCoArw7nkb+yQhor6UjZlGQUi/KqxSkg=; b=GhMt+k36V6WZx6cV/vezzXKXC3jKGC6nE2xjJyDuQqd2EmEgC7YB0jWe86SUfH29SE HaXj7QwXO+nMwHL2++8zLdmOWcFqY1N4zAa2JeOo4rpigW5d+BarEjaJwsJCbFttgg1u wbCqqi3K/9E7OGzE9mYXV01tGsgkb525juL2agfs7cHVkf1jhOw4bAY/iZi/DBlYrN7U r4kIFwwG1MrFXviUedS8Xth/e8XVRd/qr/9u9zqRyvQbbpbOiEjgg6WFf5ea/LxhbLMK aiv6FCvoCSEUs/FQlnrlOVTYHQdTQpYd2NbUGngd5ikuX25fQUkvHvuGDkj2egkWPYTT 9OHw== X-Gm-Message-State: AMke39kYzb0fXoAedCGbNtJtDrOef/p8nlml7D3XQvQLhI73OaT15MGx2IO/X/vDfZEftw== X-Received: by 10.84.174.131 with SMTP id r3mr11135081plb.30.1488660909889; Sat, 04 Mar 2017 12:55:09 -0800 (PST) Received: from localhost.localdomain ([27.7.21.206]) by smtp.gmail.com with ESMTPSA id s4sm30724722pfi.74.2017.03.04.12.55.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 04 Mar 2017 12:55:08 -0800 (PST) From: Ashish Singh To: ffmpeg-devel@ffmpeg.org Date: Sun, 5 Mar 2017 02:25:03 +0530 Message-Id: <1488660903-26784-1-git-send-email-ashk43712@gmail.com> X-Mailer: git-send-email 2.7.4 Subject: [FFmpeg-devel] [PATCH] Added vmaf filter linking to netflix's vmaf repo. It may not necessarily be suitable for adding into mainline since it uses fork/pipe and exec for calling vmaf scripts. It is mostly a Proof Of Concept to prepare for adding actual vmaf as GSOC 17 project. The code may nevertheless be useful for other people wanting to play around with vmaf in ffmpeg. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: ashk43712 MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: ashk43712 Compiled Netflix's vmaf repo needs to be present inside libavfilter directory for it to run successfully. --- libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_vmaf.c | 319 +++++++++++++++++++++++++++++++++++++++++++++++ libavfilter/vmaf.h | 33 +++++ 4 files changed, 354 insertions(+) create mode 100644 libavfilter/vf_vmaf.c create mode 100644 libavfilter/vmaf.h diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 0ba1c74..f456ea5 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -305,6 +305,7 @@ OBJS-$(CONFIG_VFLIP_FILTER) += vf_vflip.o OBJS-$(CONFIG_VIDSTABDETECT_FILTER) += vidstabutils.o vf_vidstabdetect.o OBJS-$(CONFIG_VIDSTABTRANSFORM_FILTER) += vidstabutils.o vf_vidstabtransform.o OBJS-$(CONFIG_VIGNETTE_FILTER) += vf_vignette.o +OBJS-$(CONFIG_VMAF_FILTER) += vf_vmaf.o dualinput.o framesync.o OBJS-$(CONFIG_VSTACK_FILTER) += vf_stack.o framesync.o OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 15a74c4..6fe6b86 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -320,6 +320,7 @@ void avfilter_register_all(void) REGISTER_FILTER(VIDSTABDETECT, vidstabdetect, vf); REGISTER_FILTER(VIDSTABTRANSFORM, vidstabtransform, vf); REGISTER_FILTER(VIGNETTE, vignette, vf); + REGISTER_FILTER(VMAF, vmaf, vf); REGISTER_FILTER(VSTACK, vstack, vf); REGISTER_FILTER(W3FDIF, w3fdif, vf); REGISTER_FILTER(WAVEFORM, waveform, vf); diff --git a/libavfilter/vf_vmaf.c b/libavfilter/vf_vmaf.c new file mode 100644 index 0000000..a62aad1 --- /dev/null +++ b/libavfilter/vf_vmaf.c @@ -0,0 +1,319 @@ +/* + * Copyright (c) 2017 Ashish Singh + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Caculate the VMAF between two input videos. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "libavutil/avstring.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" +#include "avfilter.h" +#include "dualinput.h" +#include "drawutils.h" +#include "formats.h" +#include "internal.h" +#include "vmaf.h" +#include "video.h" + + +typedef struct VMAFContext { + const AVClass *class; + FFDualInputContext dinput; + double vmaf_sum; + uint64_t nb_frames; + FILE *stats_file; + char *stats_file_str; + char vmaf_dir[501]; + int stats_version; + int stats_header_written; + int stats_add_max; + int is_rgb; + uint8_t rgba_map[4]; + char comps[4]; + int nb_components; + int planewidth[4]; + int planeheight[4]; + double planeweight[4]; + VMAFDSPContext dsp; +} VMAFContext; + +#define OFFSET(x) offsetof(VMAFContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption vmaf_options[] = { + {"stats_file", "Set file where to store per-frame difference information", OFFSET(stats_file_str), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, + {"f", "Set file where to store per-frame difference information", OFFSET(stats_file_str), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, + {"stats_version", "Set the format version for the stats file.", OFFSET(stats_version), AV_OPT_TYPE_INT, {.i64=1}, 1, 2, FLAGS }, + {"output_max", "Add raw stats (max values) to the output log.", OFFSET(stats_add_max), AV_OPT_TYPE_BOOL, {.i64=0}, 0, 1, FLAGS}, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(vmaf); + +static inline double get_vmaf(double vmaf_sum, uint64_t nb_frames) +{ + return vmaf_sum/nb_frames; +} + +static void set_meta(AVDictionary **metadata, const char *key, float d) +{ + char value[128]; + snprintf(value, sizeof(value), "%0.7f", d); + av_dict_set(metadata,key,value,0); +} + +static AVFrame *do_vmaf(AVFilterContext *ctx, AVFrame *main, + const AVFrame *ref) +{ + VMAFContext *s = ctx->priv; + + AVDictionary **metadata = avpriv_frame_get_metadatap(main); + + int pipefd[2]; + char buf[1001]; + char str[20]; + int w = main->width; + int h = main->height; + + char *format = av_get_pix_fmt_name(main->format); + + char *fifo1 = "t1.yuv"; + char *fifo2 = "t2.yuv"; + + char width[5],height[5]; + sprintf(width, "%d", main->width); + sprintf(height, "%d", main->height); + + + if(pipe(pipefd)==-1){ + av_log(ctx, AV_LOG_ERROR, "Pipe creation failed.\n"); + AVERROR(EINVAL); + } + + int pid = fork(); + + if(pid == -1){ + av_log(ctx, AV_LOG_ERROR, "Process creation failed.\n"); + AVERROR(EINVAL); + } + else if ( pid == 0 ) { + close(pipefd[0]); + dup2(pipefd[1], 1); + + int ret = execlp(s->vmaf_dir,"python",format,width,height,fifo1,fifo2,"--out-fmt","text",(char*)NULL); + av_log(ctx, AV_LOG_ERROR, "No such file or directory.\n"); + exit(0); + } else { + + FILE *fd1,*fd2; + + fd1 = fopen(fifo1, "wb"); + uint8_t *ptr=main->data[0]; + int y; + for (y=0; ylinesize[0]; + } + fclose(fd1); + + fd2 = fopen(fifo2, "wb"); + ptr=ref->data[0]; + for (y=0; ylinesize[0]; + } + fclose(fd2); + + wait(NULL); + close(pipefd[1]); + read(pipefd[0], &buf, sizeof(buf)); + + close(pipefd[0]); + + int len= strlen(buf); + char *ned = "VMAF_score"; + char *find = strstr(buf,ned); + int i=0; + find += 11; + while(*find!=' '&&*find!='\n'){ + str[i++]=*find; + ++find; + } + str[i]='\0'; + + } + double d; + + sscanf(str, "%lf", &d); + + set_meta(metadata, "lavfi.vmaf.score.",d); + av_log(ctx, AV_LOG_INFO, "vmaf score for frame %d is %lf.\n",s->nb_frames,d); + + s->vmaf_sum += d; + + s->nb_frames++; + + return main; +} + +static av_cold int init(AVFilterContext *ctx) +{ + VMAFContext *s = ctx->priv; + + s->dinput.process = do_vmaf; + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_YUV444P, AV_PIX_FMT_YUV422P, AV_PIX_FMT_YUV420P, + AV_PIX_FMT_YUV444P10LE, AV_PIX_FMT_YUV422P10LE, AV_PIX_FMT_YUV420P10LE, + AV_PIX_FMT_NONE + }; + + AVFilterFormats *fmts_list = ff_make_format_list(pix_fmts); + if (!fmts_list) + return AVERROR(ENOMEM); + return ff_set_common_formats(ctx, fmts_list); +} + +static int config_input_ref(AVFilterLink *inlink) +{ + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(inlink->format); + AVFilterContext *ctx = inlink->dst; + VMAFContext *s = ctx->priv; + double average_max; + unsigned sum; + int j; + + s->nb_components = desc->nb_components; + if (ctx->inputs[0]->w != ctx->inputs[1]->w || + ctx->inputs[0]->h != ctx->inputs[1]->h) { + av_log(ctx, AV_LOG_ERROR, "Width and height of input videos must be same.\n"); + return AVERROR(EINVAL); + } + if (ctx->inputs[0]->format != ctx->inputs[1]->format) { + av_log(ctx, AV_LOG_ERROR, "Inputs must be of same pixel format.\n"); + return AVERROR(EINVAL); + } + + + getcwd(s->vmaf_dir, sizeof(s->vmaf_dir)); + int len = strlen(s->vmaf_dir); + char rel_path[30] = "/libavfilter/vmaf/run_vmaf"; + + for(j=0; jvmaf_dir[len++]=rel_path[j]; + } + s->vmaf_dir[len] = '\0'; + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + VMAFContext *s = ctx->priv; + AVFilterLink *mainlink = ctx->inputs[0]; + int ret; + outlink->w = mainlink->w; + outlink->h = mainlink->h; + outlink->time_base = mainlink->time_base; + outlink->sample_aspect_ratio = mainlink->sample_aspect_ratio; + outlink->frame_rate = mainlink->frame_rate; + if ((ret = ff_dualinput_init(ctx, &s->dinput)) < 0) + return ret; + + return 0; +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *inpicref) +{ + VMAFContext *s = inlink->dst->priv; + return ff_dualinput_filter_frame(&s->dinput, inlink, inpicref); +} + +static int request_frame(AVFilterLink *outlink) +{ + VMAFContext *s = outlink->src->priv; + return ff_dualinput_request_frame(&s->dinput, outlink); +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + VMAFContext *s = ctx->priv; + + if (s->nb_frames > 0) { + av_log(ctx, AV_LOG_INFO, "VMAF average:%f\n", + get_vmaf(s->vmaf_sum, s->nb_frames)); + } + + ff_dualinput_uninit(&s->dinput); + +} + +static const AVFilterPad vmaf_inputs[] = { + { + .name = "main", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + },{ + .name = "reference", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input_ref, + }, + { NULL } +}; + +static const AVFilterPad vmaf_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + .request_frame = request_frame, + }, + { NULL } +}; + +AVFilter ff_vf_vmaf = { + .name = "vmaf", + .description = NULL_IF_CONFIG_SMALL("Calculate the VMAF between two video streams."), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(VMAFContext), + .priv_class = &vmaf_class, + .inputs = vmaf_inputs, + .outputs = vmaf_outputs, +}; diff --git a/libavfilter/vmaf.h b/libavfilter/vmaf.h new file mode 100644 index 0000000..20ada43 --- /dev/null +++ b/libavfilter/vmaf.h @@ -0,0 +1,33 @@ +/* + * Copyright (c) 2017 Ashish Singh + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVFILTER_VMAF_H +#define AVFILTER_VMAF_H + +#include +#include + +typedef struct VMAFDSPContext { + uint64_t (*sse_line)(const uint8_t *buf, const uint8_t *ref, int w); +} VMAFDSPContext; + +void ff_vmaf_init_x86(VMAFDSPContext *dsp, int bpp); + +#endif /* AVFILTER_VMAF_H */