From patchwork Thu Oct 27 18:08:27 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pallavi Kumari X-Patchwork-Id: 1204 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.103.140.133 with SMTP id o127csp700834vsd; Thu, 27 Oct 2016 11:08:38 -0700 (PDT) X-Received: by 10.194.220.232 with SMTP id pz8mr9382876wjc.154.1477591718199; Thu, 27 Oct 2016 11:08:38 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 20si4617093wmv.135.2016.10.27.11.08.37; Thu, 27 Oct 2016 11:08:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 76063689ED1; Thu, 27 Oct 2016 21:08:32 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-yw0-f169.google.com (mail-yw0-f169.google.com [209.85.161.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7478F689E5B for ; Thu, 27 Oct 2016 21:08:26 +0300 (EEST) Received: by mail-yw0-f169.google.com with SMTP id u124so53490636ywg.3 for ; Thu, 27 Oct 2016 11:08:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=dX33NsklSh80DbtsBOvHGIHF1JdKll8+Q8PDQxSN2yU=; b=QGWJlx9DBptXmhUraBDyfWLJjYjNzt0Q6pKXMcpXhP/7epdG7KRHAAD6WPjfoXkc+V PjPykgEOP8UvasvKTtHXtt1N6PDKIYYkKiK6JHObE7QW6tWz2VTfOaZGIWvo2qFZSxW0 dtiA9C+05KnUA/fSoVObVYqqOOp3WLrS8VAgWfolGHb5zHW7t4D6jU3pA3L24xiDA4VH UD44byZw7jJDaq2rvGRrfnR7+ZiGukSRox5AdUqqtsgr+NJ8kJoW2O58qOHLUfo9xR4J rQD61QvHb+35GclqC2G7m3eca6k1jWTdpg0LZcnoGMH/K+8kPr20hUesrqQ0BVBgkhNz VR1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=dX33NsklSh80DbtsBOvHGIHF1JdKll8+Q8PDQxSN2yU=; b=hQyi9KVtZfGmWvYt1mGjPYUHyPR/XexKzSNTK1Mm3zxKHmHwy4ynuDQEGhx4EETF2i RUF1pNEBtG00Xd06vo73ovo3JOzAWctHjguh6o1s87tniR7e956izQK81JJdxBPNBD1R UHsO+P2/Rp6j4sjSfdKhaeA4x0zDwxSEQjaSxuZiGtzIrEM1E4g3HZ4ORE7b2acIHzdK W8cm3GoP28m1dv/SLR+tR9ZhbHNrHWrurIOx/lD7MEkyEElUVfJEMWrdouArsyzYEB5b iUfgCdayOnzKQXGly8pxTsYb01OCbRjGnsosp/A4/cSSkJi68ePogAOIc/rLUkuTeTHS lZag== X-Gm-Message-State: ABUngvet+IJKs6fxnBNbc0Q3aJ4vyGMRgN6MWY3izqfZUcgOf7U7Ouh84lK8Jx6YbaK3oLkdKChqPTmvzPczsg== X-Received: by 10.107.9.16 with SMTP id j16mr7790630ioi.230.1477591708364; Thu, 27 Oct 2016 11:08:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.170.220 with HTTP; Thu, 27 Oct 2016 11:08:27 -0700 (PDT) In-Reply-To: <20161026215428.GN4602@nb4> References: <20161011215404.GF4602@nb4> <20161012005637.GH4602@nb4> <20161015001558.GN4602@nb4> <20161026123111.GZ4602@nb4> <20161026215428.GN4602@nb4> From: Pallavi Kumari Date: Thu, 27 Oct 2016 23:38:27 +0530 Message-ID: To: FFmpeg development discussions and patches X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: Re: [FFmpeg-devel] [OPW] OPW Project Proposal X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Hi Michael, I have attached a patch with the mail. With `avcodec_get_frame_defaults (&frame)` in the function readAudio program was working fine with old version in my system. But this function is not in the latest git repo so I used `av_frame_unref(&frame)` instead and the program segfaults. I am not sure why or how to fix this. Also, I haven't properly understood the input and output flow of libavfilters (how to write input and output links? and overall flow). Could you explain it to me? On Thu, Oct 27, 2016 at 3:24 AM, Michael Niedermayer wrote: > On Thu, Oct 27, 2016 at 12:40:53AM +0530, Pallavi Kumari wrote: > > I mean deciding a timeline for the opw project. Which is to be mentioned > in > > application > > its your application, you can choose whatever timeline you feel makes > sense. > As far as iam concered whats important is to finish the qualification > task well before we have to choose the applicant(s) we accept this > year. > > The timeline for the main project in dec-mar can be segmented as you > like but leave plenty of time toward the end, problems always occur > and things get delayed. > Also both for qualification and the main project submit code early > if you have questions ask early, > if you dont get satisfactory reply from me on something ask again and > louder/clearer ... i do sometimes also forget to reply until i then > remember again days later ... > > also i think the architecture wasnt discussed much yet > do you have some plans about it ? > > I mean for example there could be a avfilter that passes audio through > and adds fingerint(s) as metadata into thf AVFrame > (as with avpriv_frame_get_metadatap in vf_idet.c) and also print the > fingerprints via av_log() > > a 2nd filter could then look these fingerprints from the metadata > up in some file/database > > a 3rd one could add the fingerprints to the file/database > > or these also could all be in one filter of course > > this can be extended in the future to for example have a filter > delay the audio until lookup succeeds and then once identified > lookup lyrics online and add them or such (this of course would be a > future thing after outreachy/opw) > > but maybe you have a different plan on how the components would > interact ? > > thx > > > > > On Wed, Oct 26, 2016 at 6:01 PM, Michael Niedermayer > > > wrote: > > > > > Hi > > > > > > On Mon, Oct 17, 2016 at 02:06:26PM +0530, Pallavi Kumari wrote: > > > > Hi Michael, > > > > > > > > I figured out the use of fft. Help me with the time line setting. > Thanks > > > > > > I dont understand the question, > > > if its about AVFILTER_FLAG_SUPPORT_TIMELINE*, please ignore this for > > > now, its not needed > > > > > > [...] > > > -- > > > Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC7 > 87040B0FAB > > > > > > Rewriting code that is poorly written but fully understood is good. > > > Rewriting code that one doesnt understand is a sign that one is less > smart > > > then the original author, trying to rewrite it will not make it better. > > > > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > -- > Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB > > He who knows, does not speak. He who speaks, does not know. -- Lao Tsu > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > From 60429fb7d83d11d6dd733c3477950a940bd0d84a Mon Sep 17 00:00:00 2001 From: Atana Date: Mon, 24 Oct 2016 17:16:09 +0530 Subject: [PATCH] avfilter: add peakpoints filter --- libavfilter/Makefile | 1 + libavfilter/af_peakpoints.c | 263 ++++++++++++++++++++++++++++++++++++++++++++ libavfilter/allfilters.c | 1 + libavfilter/version.h | 2 +- 4 files changed, 266 insertions(+), 1 deletion(-) create mode 100644 libavfilter/af_peakpoints.c diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 7ed4696..1a18902 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -96,6 +96,7 @@ OBJS-$(CONFIG_LADSPA_FILTER) += af_ladspa.o OBJS-$(CONFIG_LOUDNORM_FILTER) += af_loudnorm.o OBJS-$(CONFIG_LOWPASS_FILTER) += af_biquads.o OBJS-$(CONFIG_PAN_FILTER) += af_pan.o +OBJS-$(CONFIG_PEAKPOINTS_FILTER) += af_peakpoints.o OBJS-$(CONFIG_REPLAYGAIN_FILTER) += af_replaygain.o OBJS-$(CONFIG_RESAMPLE_FILTER) += af_resample.o OBJS-$(CONFIG_RUBBERBAND_FILTER) += af_rubberband.o diff --git a/libavfilter/af_peakpoints.c b/libavfilter/af_peakpoints.c new file mode 100644 index 0000000..84b51da --- /dev/null +++ b/libavfilter/af_peakpoints.c @@ -0,0 +1,263 @@ +/* + * Copyright (c) 2016 Pallavi Kumari + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavcodec/avcodec.h" +#include "libavcodec/avfft.h" +#include "libavformat/avformat.h" +#include "libswscale/swscale.h" +#include "avfilter.h" +#include "libavutil/opt.h" + + +/* Structure to contain peak points context */ +typedef struct { + const AVClass *class; + double *data; + int nsamples; + double *peaks; + int size; // number of peaks + int windowSize; + char *inputFile; +} PeakPointsContext; + +/* returns maximum value from an array conditioned on start and end index */ +static double getMax(double *res_arr, int startIndex, int endIndex) { + int i; + double max = res_arr[startIndex]; + for (i = startIndex; i <= endIndex; i++) { + if (res_arr[i] > max) { + max = res_arr[i]; + } + } + return max; +} + +/* Stores peak frequency for each window(of chunkSize) in peaks array */ +static void getPeakPointInChunk(int chunkSize, double *res_arr, int size, double *peaks) { + int endIndex, i = 0, peakIndex = 0; + int startIndex = 0; + double max; + // get a chunk and find max value in it + while (i < size) { + if (i % chunkSize-1 == 0) { + max = getMax(res_arr, startIndex, i); + peaks[peakIndex++] = max; + startIndex = startIndex + chunkSize; + } + i += 1; + } +} + +/* read audio stream from input file */ +static void readAudio (int *sampleRate, PeakPointsContext *ppc) { + // Initialize FFmpeg + av_register_all (); + + AVFormatContext *formatContext = NULL; + AVCodecContext *codecContext = NULL; + AVCodec *codec = NULL; + AVPacket packet; + AVFrame frame; + + ssize_t nSamples = 0; + + // open the file and autodetect the format + if (avformat_open_input (&formatContext, ppc->inputFile, NULL, NULL) < 0) + printf("Can not open input file %s", ppc->inputFile); + + if (avformat_find_stream_info (formatContext, NULL) < 0) + printf("Can not find stream information"); + + // find audio stream (between video/audio/subtitles/.. streams) + int audioStreamId = av_find_best_stream (formatContext, AVMEDIA_TYPE_AUDIO, + -1, -1, &codec, 0); + if (audioStreamId < 0) + printf("Can not find audio stream in the input file"); + + codecContext = formatContext->streams[audioStreamId]->codec; + + // init the audio decoder + if (avcodec_open2 (codecContext, codec, NULL) < 0) + printf("Can not open audio decoder"); + + // read all packets + int i = 0; + while (av_read_frame (formatContext, &packet) == 0) { + if (packet.stream_index == audioStreamId) { + //avcodec_get_frame_defaults (&frame); + av_frame_unref (&frame); + int gotFrame = 0; + if (avcodec_decode_audio4 (codecContext,&frame,&gotFrame,&packet) < 0){ + printf("Error decoding audio"); + } + if (gotFrame) { + // audio frame has been decoded + int size = av_samples_get_buffer_size (NULL, + codecContext->channels, + frame.nb_samples, + codecContext->sample_fmt, + 1); + if (size < 0) { + printf("av_samples_get_buffer_size invalid value"); + } + + // printf("%d\n", *frame.data[0]); + ppc->data[i] = (double)*frame.data[0]; + i += 1; + nSamples += frame.nb_samples; + } + } + av_free_packet (&packet); + } + ppc->nsamples = i; + + if (nSamples < 1) + printf("Decoded audio data is empty"); + + int sampleSize = av_get_bytes_per_sample (codecContext->sample_fmt); + if (sampleSize < 1) + printf("Invalid sample format"); + + // optional, return value with sample rate + if (sampleRate) *sampleRate = codecContext->sample_rate; + + if (codecContext) avcodec_close (codecContext); + + avformat_close_input (&formatContext); +} + +/* Get peaks points from windowed frequency domain data*/ +static void getPeakPoints(PeakPointsContext *ppc) { + int i, k, size, chunkSize, chunkSampleSize, resSize, sample_rate; + double *fft_res; + int m; + + ppc->data = malloc(sizeof(double)*10000); + readAudio (&sample_rate, ppc); + + size = ppc->nsamples; + m = log2(ppc->windowSize); + chunkSize = ppc->windowSize; + chunkSampleSize = size/chunkSize; + resSize = chunkSize * chunkSampleSize; + fft_res = malloc(sizeof(double) * resSize); + + RDFTContext *rdftC = av_rdft_init(m, DFT_R2C); + FFTSample *data; + data = malloc(sizeof(FFTSample)*chunkSize); + // FFT transform for windowed time domain data + // window is of size chunkSize + k = 0; + while (k < resSize) { + //copy data + for (i = 0; i < chunkSize; i++) { + data[i] = ppc->data[i+k]; + } + //calculate FFT + av_rdft_calc(rdftC, data); + for (i = 0; i < chunkSize; i++) { + fft_res[i+k] = data[i]; + } + k = k + chunkSize; + } + + av_rdft_end(rdftC); + + int pSize = resSize/chunkSize; + ppc->size = pSize; + ppc->peaks = malloc(sizeof(double)*pSize); + getPeakPointInChunk(chunkSize, fft_res, resSize, ppc->peaks); +} + + +#define OFFSET(x) offsetof(PeakPointsContext, x) + +static const AVOption peakpoints_options[] = { + { "input", "set input audio file", OFFSET(inputFile), AV_OPT_TYPE_STRING, {.str="sample.mp3"}, CHAR_MIN, CHAR_MAX }, + { "wsize", "set window size", OFFSET(windowSize), AV_OPT_TYPE_INT, {.i64=16}, 0, INT_MAX}, + { NULL }, +}; + +static const char *peakpoints_get_name(void *ctx) { + return "peakpoints"; +} + +static const AVClass peakpoints_class = { + .class_name = "PeakPointsContext", + .item_name = peakpoints_get_name, + .option = peakpoints_options, +}; + + +AVFILTER_DEFINE_CLASS(peakpoints); + +static av_cold int init(AVFilterContext *ctx) +{ + PeakPointsContext *p = ctx->priv; + + if (!(p->inputFile)) { + av_log(ctx, AV_LOG_ERROR, "Input audio file must be passed\n"); + return AVERROR(EINVAL); + } + + if (p->windowSize < 16) { + av_log(ctx, AV_LOG_ERROR, "window size must be greater than or equal to 16"); + return AVERROR(EINVAL); + } + //printf("Getting peak points.."); + // get peaks + getPeakPoints(p); + //printf("Peak points collected."); + // print peaks + int i; + for (i = 0; i < p->size; i++) { + av_log(ctx, AV_LOG_DEBUG, "%f", p->peaks[i]); + } + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + PeakPointsContext *p = ctx->priv; + int i; + + // free memory allocated for audio data + for (i = 0; p->nsamples; i++) { + free(&p->data[i]); + } + + // free memory allocated for peaks + for (i = 0; p->size; i++) { + free(&p->data[i]); + } +} + + +AVFilter ff_af_peakpoints = { + .name = "peakpoints", + .description = NULL_IF_CONFIG_SMALL("peak points from frequency domain windowed data."), + .init = init, + .uninit = uninit, + //.query_formats = query_formats, + .priv_size = sizeof(PeakPointsContext), + //.inputs = inputs, + //.outputs = outputs, + .priv_class = &peakpoints_class, +}; diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 82a65ee..759e63a 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -114,6 +114,7 @@ void avfilter_register_all(void) REGISTER_FILTER(LOUDNORM, loudnorm, af); REGISTER_FILTER(LOWPASS, lowpass, af); REGISTER_FILTER(PAN, pan, af); + REGISTER_FILTER(PEAKPOINTS, peakpoints, af); REGISTER_FILTER(REPLAYGAIN, replaygain, af); REGISTER_FILTER(RESAMPLE, resample, af); REGISTER_FILTER(RUBBERBAND, rubberband, af); diff --git a/libavfilter/version.h b/libavfilter/version.h index 8c6cf06..adf8c71 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -30,7 +30,7 @@ #include "libavutil/version.h" #define LIBAVFILTER_VERSION_MAJOR 6 -#define LIBAVFILTER_VERSION_MINOR 64 +#define LIBAVFILTER_VERSION_MINOR 65 #define LIBAVFILTER_VERSION_MICRO 100 #define LIBAVFILTER_VERSION_INT AV_VERSION_INT(LIBAVFILTER_VERSION_MAJOR, \ -- 1.9.1