| Message ID | tencent_6126C367DFAA82931D1C594BA9661A509406@qq.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel,1/2] avutil/executor: Allowing thread_count be zero | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/make_loongarch64 | success | Make finished |
| yinshiyou/make_fate_loongarch64 | success | Make fate finished |
| andriy/make_x86 | success | Make finished |
| andriy/make_fate_x86 | success | Make fate finished |
Quoting Zhao Zhili (2024-06-17 07:19:26) > From: Zhao Zhili <zhilizhao@tencent.com> > > When thread_count be zero, it will be run on current thread like > !HAVE_THREADS. Other APIs treat zero to mean "auto".
On Mon, Jun 17, 2024 at 9:05 AM Anton Khirnov <anton@khirnov.net> wrote: > Quoting Zhao Zhili (2024-06-17 07:19:26) > > From: Zhao Zhili <zhilizhao@tencent.com> > > > > When thread_count be zero, it will be run on current thread like > > !HAVE_THREADS. > > Other APIs treat zero to mean "auto". > Agreed, this approach for 0 to mean do it on same thread is questionable at best. And no real reasoning for such approach was given as explanation in this patch. > > -- > Anton Khirnov > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: > > Quoting Zhao Zhili (2024-06-17 07:19:26) >> From: Zhao Zhili <zhilizhao@tencent.com> >> >> When thread_count be zero, it will be run on current thread like >> !HAVE_THREADS. > > Other APIs treat zero to mean "auto". executor don’t detect cpu cores by itself. It’s more low level than libavcodec. Zero thread is zero thread, literally. If we use thread_count one to mean run on current thread, how to create a single thread then? What’s the API do you suggest? Some negative value? > > -- > Anton Khirnov > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> wrote: > > > > > On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: > > > > Quoting Zhao Zhili (2024-06-17 07:19:26) > >> From: Zhao Zhili <zhilizhao@tencent.com> > >> > >> When thread_count be zero, it will be run on current thread like > >> !HAVE_THREADS. > > > > Other APIs treat zero to mean "auto". > > executor don’t detect cpu cores by itself. It’s more low level than libavcodec. > > Zero thread is zero thread, literally. If we use thread_count one to mean > run on current thread, how to create a single thread then? Whats the point of creating a single thread? Does the main thread ever do something else in the meantime, or does it just wait for the job anyway? - Hendrik
> On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> wrote: > > On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> wrote: >> >> >> >>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: >>> >>> Quoting Zhao Zhili (2024-06-17 07:19:26) >>>> From: Zhao Zhili <zhilizhao@tencent.com> >>>> >>>> When thread_count be zero, it will be run on current thread like >>>> !HAVE_THREADS. >>> >>> Other APIs treat zero to mean "auto". >> >> executor don’t detect cpu cores by itself. It’s more low level than libavcodec. >> >> Zero thread is zero thread, literally. If we use thread_count one to mean >> run on current thread, how to create a single thread then? > > Whats the point of creating a single thread? Does the main thread ever > do something else in the meantime, or does it just wait for the job > anyway? Executor as a basic infrastructure should support such usage. The caller don’t need to wait for the job to finish. > > - Hendrik > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> wrote: > > > > On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> wrote: > > > > On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> > wrote: > >> > >> > >> > >>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: > >>> > >>> Quoting Zhao Zhili (2024-06-17 07:19:26) > >>>> From: Zhao Zhili <zhilizhao@tencent.com> > >>>> > >>>> When thread_count be zero, it will be run on current thread like > >>>> !HAVE_THREADS. > >>> > >>> Other APIs treat zero to mean "auto". > >> > >> executor don’t detect cpu cores by itself. It’s more low level than > libavcodec. > >> > >> Zero thread is zero thread, literally. If we use thread_count one to > mean > >> run on current thread, how to create a single thread then? > > > > Whats the point of creating a single thread? Does the main thread ever > > do something else in the meantime, or does it just wait for the job > > anyway? > > Executor as a basic infrastructure should support such usage. The caller > don’t need to wait for the job to finish. > Hi Zhili, Thank you for the patch. Could you explain more about its usage? Why do we need to run everything in the main thread when we have pthread in the system? Thank you.
Nuo Mi <nuomi2021@gmail.com> 于2024年6月18日周二 19:51写道： > > On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> wrote: > > > > > > > > On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> wrote: > > > > > > On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> > > wrote: > > >> > > >> > > >> > > >>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: > > >>> > > >>> Quoting Zhao Zhili (2024-06-17 07:19:26) > > >>>> From: Zhao Zhili <zhilizhao@tencent.com> > > >>>> > > >>>> When thread_count be zero, it will be run on current thread like > > >>>> !HAVE_THREADS. > > >>> > > >>> Other APIs treat zero to mean "auto". > > >> > > >> executor don’t detect cpu cores by itself. It’s more low level than > > libavcodec. > > >> > > >> Zero thread is zero thread, literally. If we use thread_count one to > > mean > > >> run on current thread, how to create a single thread then? > > > > > > Whats the point of creating a single thread? Does the main thread ever > > > do something else in the meantime, or does it just wait for the job > > > anyway? > > > > Executor as a basic infrastructure should support such usage. The caller > > don’t need to wait for the job to finish. > > > Hi Zhili, > Thank you for the patch. > Could you explain more about its usage? > Why do we need to run everything in the main thread when we have pthread in > the system? Just control CPU resource be used by decoder, perhaps the computer is running some other applications. There are running more application in one computer, and the ffmpeg decoder is need not realtime. For example running machine learning traning some model, and use ffmpeg transcoding vod videos, but transcode functions is borrow machine learning's team computers :D. > > Thank you. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". Thanks Steven
> On Jun 18, 2024, at 19:50, Nuo Mi <nuomi2021@gmail.com> wrote: > > On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> wrote: > >> >> >>> On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> wrote: >>> >>> On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> >> wrote: >>>> >>>> >>>> >>>>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: >>>>> >>>>> Quoting Zhao Zhili (2024-06-17 07:19:26) >>>>>> From: Zhao Zhili <zhilizhao@tencent.com> >>>>>> >>>>>> When thread_count be zero, it will be run on current thread like >>>>>> !HAVE_THREADS. >>>>> >>>>> Other APIs treat zero to mean "auto". >>>> >>>> executor don’t detect cpu cores by itself. It’s more low level than >> libavcodec. >>>> >>>> Zero thread is zero thread, literally. If we use thread_count one to >> mean >>>> run on current thread, how to create a single thread then? >>> >>> Whats the point of creating a single thread? Does the main thread ever >>> do something else in the meantime, or does it just wait for the job >>> anyway? >> >> Executor as a basic infrastructure should support such usage. The caller >> don’t need to wait for the job to finish. >> > Hi Zhili, > Thank you for the patch. > Could you explain more about its usage? > Why do we need to run everything in the main thread when we have pthread in > the system? In addition to match other decodes behavior, there are some usecases which require the decoder to not create a thread at runtime. For example, when build libavcodec with wasm and run in web browser, the runtime may or may not support threads, threads number should be decided at runtime, and can be zero. > > Thank you. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Tue, Jun 18, 2024 at 8:06 PM Steven Liu <lingjiujianke@gmail.com> wrote: > Nuo Mi <nuomi2021@gmail.com> 于2024年6月18日周二 19:51写道： > > > > On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> > wrote: > > > > > > > > > > > > On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> > wrote: > > > > > > > > On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> > > > wrote: > > > >> > > > >> > > > >> > > > >>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> > wrote: > > > >>> > > > >>> Quoting Zhao Zhili (2024-06-17 07:19:26) > > > >>>> From: Zhao Zhili <zhilizhao@tencent.com> > > > >>>> > > > >>>> When thread_count be zero, it will be run on current thread like > > > >>>> !HAVE_THREADS. > > > >>> > > > >>> Other APIs treat zero to mean "auto". > > > >> > > > >> executor don’t detect cpu cores by itself. It’s more low level than > > > libavcodec. > > > >> > > > >> Zero thread is zero thread, literally. If we use thread_count one to > > > mean > > > >> run on current thread, how to create a single thread then? > > > > > > > > Whats the point of creating a single thread? Does the main thread > ever > > > > do something else in the meantime, or does it just wait for the job > > > > anyway? > > > > > > Executor as a basic infrastructure should support such usage. The > caller > > > don’t need to wait for the job to finish. > > > > > Hi Zhili, > > Thank you for the patch. > > Could you explain more about its usage? > > Why do we need to run everything in the main thread when we have pthread > in > > the system? > Just control CPU resource be used by decoder, perhaps the computer is > running some other applications. > There are running more application in one computer, and the ffmpeg > decoder is need not realtime. > For example running machine learning traning some model, and use > ffmpeg transcoding vod videos, > but transcode functions is borrow machine learning's team computers :D. > You can use "taskset" for CPU affinity. It will constrain the CPU resources used by a process. > > > > > Thank you. > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > > Thanks > Steven > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
On Tue, Jun 18, 2024 at 10:24 PM Zhao Zhili <quinkblack@foxmail.com> wrote: > > > > On Jun 18, 2024, at 19:50, Nuo Mi <nuomi2021@gmail.com> wrote: > > > > On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> > wrote: > > > >> > >> > >>> On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> > wrote: > >>> > >>> On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> > >> wrote: > >>>> > >>>> > >>>> > >>>>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> wrote: > >>>>> > >>>>> Quoting Zhao Zhili (2024-06-17 07:19:26) > >>>>>> From: Zhao Zhili <zhilizhao@tencent.com> > >>>>>> > >>>>>> When thread_count be zero, it will be run on current thread like > >>>>>> !HAVE_THREADS. > >>>>> > >>>>> Other APIs treat zero to mean "auto". > >>>> > >>>> executor don’t detect cpu cores by itself. It’s more low level than > >> libavcodec. > >>>> > >>>> Zero thread is zero thread, literally. If we use thread_count one to > >> mean > >>>> run on current thread, how to create a single thread then? > >>> > >>> Whats the point of creating a single thread? Does the main thread ever > >>> do something else in the meantime, or does it just wait for the job > >>> anyway? > >> > >> Executor as a basic infrastructure should support such usage. The caller > >> don’t need to wait for the job to finish. > >> > > Hi Zhili, > > Thank you for the patch. > > Could you explain more about its usage? > > Why do we need to run everything in the main thread when we have pthread > in > > the system? > > In addition to match other decodes behavior, there are some usecases which > require the decoder to not create a thread at runtime. For example, when > build > libavcodec with wasm and run in web browser, the runtime may or may not > support threads, threads number should be decided at runtime, and can be > zero. Thank you for the details Sounds reasonable if other code has the same behavior and we have real use cases for this. Perhaps we can explain why we need this in the commit log. Hi @Anton Khirnov <anton@khirnov.net> and @Paul B Mahol <onemda@gmail.com> Do you prefer -1? We need to return an invalid parameter for 0 if we use -1 to indicate zero threads. Thank you. > > > > > > Thank you. > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". >
Nuo Mi <nuomi2021@gmail.com> 于2024年6月19日周三 16:40写道： > > On Tue, Jun 18, 2024 at 8:06 PM Steven Liu <lingjiujianke@gmail.com> wrote: > > > Nuo Mi <nuomi2021@gmail.com> 于2024年6月18日周二 19:51写道： > > > > > > On Mon, Jun 17, 2024 at 5:28 PM Zhao Zhili <quinkblack@foxmail.com> > > wrote: > > > > > > > > > > > > > > > > On Jun 17, 2024, at 16:45, Hendrik Leppkes <h.leppkes@gmail.com> > > wrote: > > > > > > > > > > On Mon, Jun 17, 2024 at 10:03 AM Zhao Zhili <quinkblack@foxmail.com> > > > > wrote: > > > > >> > > > > >> > > > > >> > > > > >>> On Jun 17, 2024, at 15:05, Anton Khirnov <anton@khirnov.net> > > wrote: > > > > >>> > > > > >>> Quoting Zhao Zhili (2024-06-17 07:19:26) > > > > >>>> From: Zhao Zhili <zhilizhao@tencent.com> > > > > >>>> > > > > >>>> When thread_count be zero, it will be run on current thread like > > > > >>>> !HAVE_THREADS. > > > > >>> > > > > >>> Other APIs treat zero to mean "auto". > > > > >> > > > > >> executor don’t detect cpu cores by itself. It’s more low level than > > > > libavcodec. > > > > >> > > > > >> Zero thread is zero thread, literally. If we use thread_count one to > > > > mean > > > > >> run on current thread, how to create a single thread then? > > > > > > > > > > Whats the point of creating a single thread? Does the main thread > > ever > > > > > do something else in the meantime, or does it just wait for the job > > > > > anyway? > > > > > > > > Executor as a basic infrastructure should support such usage. The > > caller > > > > don’t need to wait for the job to finish. > > > > > > > Hi Zhili, > > > Thank you for the patch. > > > Could you explain more about its usage? > > > Why do we need to run everything in the main thread when we have pthread > > in > > > the system? > > Just control CPU resource be used by decoder, perhaps the computer is > > running some other applications. > > There are running more application in one computer, and the ffmpeg > > decoder is need not realtime. > > For example running machine learning traning some model, and use > > ffmpeg transcoding vod videos, > > but transcode functions is borrow machine learning's team computers :D. > > > You can use "taskset" for CPU affinity. It will constrain the CPU resources > used by a process. it is one way, only ok out of containers, and that should can be control by ffmpeg self. maybe cpu_count is ok, but not only this one scene, as zhili said, that is the other scene.
diff --git a/libavutil/executor.c b/libavutil/executor.c index 26691fe157..fb20104b58 100644 --- a/libavutil/executor.c +++ b/libavutil/executor.c @@ -82,9 +82,11 @@ static int run_one_task(AVExecutor *e, void *lc) /* nothing */; if (*prev) { AVTask *t = remove_task(prev, *prev); - ff_mutex_unlock(&e->lock); + if (e->thread_count > 0) + ff_mutex_unlock(&e->lock); cb->run(t, lc, cb->user_data); - ff_mutex_lock(&e->lock); + if (e->thread_count > 0) + ff_mutex_lock(&e->lock); return 1; } return 0; @@ -146,14 +148,17 @@ AVExecutor* av_executor_alloc(const AVTaskCallbacks *cb, int thread_count) return NULL; e->cb = *cb; - e->local_contexts = av_calloc(thread_count, e->cb.local_context_size); + e->local_contexts = av_calloc(FFMAX(thread_count, 1), e->cb.local_context_size); if (!e->local_contexts) goto free_executor; - e->threads = av_calloc(thread_count, sizeof(*e->threads)); + e->threads = av_calloc(FFMAX(thread_count, 1), sizeof(*e->threads)); if (!e->threads) goto free_executor; + if (!thread_count) + return e; + has_lock = !ff_mutex_init(&e->lock, NULL); has_cond = !ff_cond_init(&e->cond, NULL); @@ -175,9 +180,12 @@ free_executor: void av_executor_free(AVExecutor **executor) { + int thread_count; + if (!executor || !*executor) return; - executor_free(*executor, 1, 1); + thread_count = (*executor)->thread_count; + executor_free(*executor, thread_count, thread_count); *executor = NULL; } @@ -195,9 +203,9 @@ void av_executor_execute(AVExecutor *e, AVTask *t) ff_cond_signal(&e->cond); ff_mutex_unlock(&e->lock); -#if !HAVE_THREADS - // We are running in a single-threaded environment, so we must handle all tasks ourselves - while (run_one_task(e, e->local_contexts)) - /* nothing */; -#endif + if (!e->thread_count || !HAVE_THREADS) { + // We are running in a single-threaded environment, so we must handle all tasks ourselves + while (run_one_task(e, e->local_contexts)) + /* nothing */; + } }
From: Zhao Zhili <zhilizhao@tencent.com> When thread_count be zero, it will be run on current thread like !HAVE_THREADS. --- libavutil/executor.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-)