[FFmpeg-devel] doc: Add infra.txt

---
 doc/infra.txt | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 doc/infra.txt

Michael Niedermayer:
> ---
>  doc/infra.txt | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 99 insertions(+)
>  create mode 100644 doc/infra.txt
> 
> diff --git a/doc/infra.txt b/doc/infra.txt
> new file mode 100644
> index 0000000000..5947a2c715
> --- /dev/null
> +++ b/doc/infra.txt
> @@ -0,0 +1,99 @@
> +FFmpeg Infrastructure:
> +======================
> +
> +
> +
> +
> +Servers:
> +~~~~~~~~
> +
> +
> +Main Server:
> +------------
> +Our Main server is hosted at telepoint.bg
> +for more details see: https://www.ffmpeg.org/#thanks_sponsor_0001
> +Nothing runs on our main server directly, instead several VMs run on it.
> +
> +
> +ffmpeg.org VM:
> +--------------
> +Web, mail, and public facing git, also website git
> +
> +
> +fftrac VM:
> +----------
> +trac.ffmpeg.org         Issue tracking
> +
> +
> +ffaux VM:
> +---------
> +patchwork.ffmpeg.org    Patch tracking
> +vote.ffmpeg.org         Condorcet voting
> +
> +
> +fate:
> +-----
> +fate.ffmpeg.org         FFmpeg automated testing environment
> +
> +
> +All servers currently run ubuntu
> +
> +
> +
> +Cronjobs:
> +~~~~~~~~~
> +Part of the docs is in the main ffmpeg repository as texi files, this part is build by a cronjob. So is the
> +doxygen stuff as well as the FFmpeg git snapshot.
> +These 3 scripts are under the ffcron user
> +
> +
> +
> +Git:
> +~~~~
> +Public facing git is provided by our infra, (https://git.ffmpeg.org/gitweb)
> +main developer ffmpeg git repository for historic reasons is provided by (git@source.ffmpeg.org:ffmpeg)
> +Other developer git repositories are provided via git@git.ffmpeg.org:<NAME_OF_REPOSITORY>
> +git mirrors are available on https://github.com/FFmpeg
> +(there are some exceptions where primary repositories are on github or elsewhere instead of the mirrors)
> +
> +Github mirrors are redundantly synced by multiple people
> +
> +You need a new git repository related to FFmpeg ? contact root at ffmpeg.org
> +
> +
> +Fate:
> +~~~~~
> +fatesamples are provided via rsync. Every FFmpeg developer who has a shell account in ffmepg.org

ffmpeg.org

> +should be in the samples group and be able to upload samples.
> +See https://www.ffmpeg.org/fate.html#Uploading-new-samples-to-the-fate-suite
> +
> +
> +
> +Accounts:
> +~~~~~~~~~
> +You need an account for some FFmpeg work? Send mail to root at ffmpeg.org
> +
> +
> +
> +VMs:
> +~~~~
> +You need a VM, docker container for FFmpeg? contact root at ffmpeg.org
> +(for docker, CC Andriy)
> +
> +
> +
> +IRC:
> +~~~~
> +irc channels are at https://libera.chat/
> +irc channel archieves are at https://libera.irclog.whitequark.org

archives

> +
> +
> +
> +Currently open positions:
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +volunteer postfix / mailman expert.
> +    You will need to do a mailman2 -> 3 update, old archieve links must continue to work

archive

> +    Some evaluation and improvments of DMARC/DKIM/... handling

improvements

> +    general long term maintaince of the mailman lists

maintenance

> +volunteer fateserver maintainer (perl)
> +    our general reaction speed to issues like cross site scripting issues is too slow

On Thu, Feb 22, 2024 at 10:05:36PM +0100, Andreas Rheinhardt wrote:
> Michael Niedermayer:
> > ---
> >  doc/infra.txt | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 99 insertions(+)
> >  create mode 100644 doc/infra.txt
> > 
> > diff --git a/doc/infra.txt b/doc/infra.txt
> > new file mode 100644
> > index 0000000000..5947a2c715
> > --- /dev/null
> > +++ b/doc/infra.txt
> > @@ -0,0 +1,99 @@
> > +FFmpeg Infrastructure:
> > +======================
> > +
> > +
> > +
> > +
> > +Servers:
> > +~~~~~~~~
> > +
> > +
> > +Main Server:
> > +------------
> > +Our Main server is hosted at telepoint.bg
> > +for more details see: https://www.ffmpeg.org/#thanks_sponsor_0001
> > +Nothing runs on our main server directly, instead several VMs run on it.
> > +
> > +
> > +ffmpeg.org VM:
> > +--------------
> > +Web, mail, and public facing git, also website git
> > +
> > +
> > +fftrac VM:
> > +----------
> > +trac.ffmpeg.org         Issue tracking
> > +
> > +
> > +ffaux VM:
> > +---------
> > +patchwork.ffmpeg.org    Patch tracking
> > +vote.ffmpeg.org         Condorcet voting
> > +
> > +
> > +fate:
> > +-----
> > +fate.ffmpeg.org         FFmpeg automated testing environment
> > +
> > +
> > +All servers currently run ubuntu
> > +
> > +
> > +
> > +Cronjobs:
> > +~~~~~~~~~
> > +Part of the docs is in the main ffmpeg repository as texi files, this part is build by a cronjob. So is the
> > +doxygen stuff as well as the FFmpeg git snapshot.
> > +These 3 scripts are under the ffcron user
> > +
> > +
> > +
> > +Git:
> > +~~~~
> > +Public facing git is provided by our infra, (https://git.ffmpeg.org/gitweb)
> > +main developer ffmpeg git repository for historic reasons is provided by (git@source.ffmpeg.org:ffmpeg)
> > +Other developer git repositories are provided via git@git.ffmpeg.org:<NAME_OF_REPOSITORY>
> > +git mirrors are available on https://github.com/FFmpeg
> > +(there are some exceptions where primary repositories are on github or elsewhere instead of the mirrors)
> > +
> > +Github mirrors are redundantly synced by multiple people
> > +
> > +You need a new git repository related to FFmpeg ? contact root at ffmpeg.org
> > +
> > +
> > +Fate:
> > +~~~~~
> > +fatesamples are provided via rsync. Every FFmpeg developer who has a shell account in ffmepg.org
> 
> ffmpeg.org
> 
> > +should be in the samples group and be able to upload samples.
> > +See https://www.ffmpeg.org/fate.html#Uploading-new-samples-to-the-fate-suite
> > +
> > +
> > +
> > +Accounts:
> > +~~~~~~~~~
> > +You need an account for some FFmpeg work? Send mail to root at ffmpeg.org
> > +
> > +
> > +
> > +VMs:
> > +~~~~
> > +You need a VM, docker container for FFmpeg? contact root at ffmpeg.org
> > +(for docker, CC Andriy)
> > +
> > +
> > +
> > +IRC:
> > +~~~~
> > +irc channels are at https://libera.chat/
> > +irc channel archieves are at https://libera.irclog.whitequark.org
> 
> archives
> 
> > +
> > +
> > +
> > +Currently open positions:
> > +~~~~~~~~~~~~~~~~~~~~~~~~~
> > +volunteer postfix / mailman expert.
> > +    You will need to do a mailman2 -> 3 update, old archieve links must continue to work
> 
> archive
> 
> > +    Some evaluation and improvments of DMARC/DKIM/... handling
> 
> improvements
> 
> > +    general long term maintaince of the mailman lists
> 
> maintenance

will apply with fixes and without the "open positions" because that
seems not to fit in the ffmpeg-git that well.

thx

[...]

Hi,

On Thu, Feb 22, 2024 at 3:54 PM Michael Niedermayer <michael@niedermayer.cc>
wrote:

> +Fate:

+~~~~~
> +fatesamples are provided via rsync. Every FFmpeg developer who has a
> shell account in ffmepg.org
> +should be in the samples group and be able to upload samples.
> +See
> https://www.ffmpeg.org/fate.html#Uploading-new-samples-to-the-fate-suite
>

Is there a public list of "who has a shell account on ffmpeg.org" that can
be linked to here? The goal here is to help people on IRC who ask "I'd like
to add a sample to fate, how do I do that?" in some other way than "ask
Michael". (I know the list, but I'd like a public place so everyone can
find it, not just me.)

It would be useful to also add info on who the roots are. This regularly
comes up and having public disclosure means we don't have to argue about it
or resort to "ask Michael" :)

Thanks for working on this!

Ronald

Hi

On Sat, Feb 24, 2024 at 10:56:49AM -0500, Ronald S. Bultje wrote:
> Hi,
> 
> On Thu, Feb 22, 2024 at 3:54 PM Michael Niedermayer <michael@niedermayer.cc>
> wrote:
> 
> > +Fate:
> 
> +~~~~~
> > +fatesamples are provided via rsync. Every FFmpeg developer who has a
> > shell account in ffmepg.org
> > +should be in the samples group and be able to upload samples.
> > +See
> > https://www.ffmpeg.org/fate.html#Uploading-new-samples-to-the-fate-suite
> >
> 
> Is there a public list of "who has a shell account on ffmpeg.org" that can
> be linked to here? The goal here is to help people on IRC who ask "I'd like
> to add a sample to fate, how do I do that?" in some other way than "ask
> Michael". (I know the list, but I'd like a public place so everyone can
> find it, not just me.)

Thats a valid reason to have an account
try rsbultje on ffmpeg.org
your git ssh key should work to authenticate
see /home/* for who has an account
and /etc/group for who is in samples
i just now noticed some people who recently got accounts where forgotten to be
added top samples.
you can now monitor exactly who has accounts and who is in samples.
Ask them, make it public, remind me if i forget adding someone to samlpes
which it seems i have severl times. Its quite easy to forget a
adduser user samples
so if i had to maintain a public list, that list would never be reliable
i would keep forgetting either adding people to it or add people and then
forget actually opening their account :)


> 
> It would be useful to also add info on who the roots are. This regularly
> comes up and having public disclosure means we don't have to argue about it
> or resort to "ask Michael" :)

the roots should be listed in MAINTAINERs
in addition ubitux, tim nicholochson, roberto togni and thresh seem to have
access too. I dont know why they are not listed in MAINTAINERs, maybe they
removed themselfs, or wheer never added ...

thx

[...]

Hi,

On Sat, Feb 24, 2024 at 1:28 PM Michael Niedermayer <michael@niedermayer.cc>
wrote:

> so if i had to maintain a public list, that list would never be reliable
>

I can continue updating it. The current list is:

$ ssh ffmpeg.org cat /etc/group|grep samples|tr ':' '\n'|grep ,|tr ',' ' '
compn cehoyos reimar beastd llogan ubitux durandal nevcairiel daemon404
jamrial martinvignali thilo atomnuker pross elenril rsbultje lynne jdek
andriy rathann rtogni reynaldo

I volunteer to run that script once a year and update doc/infra.txt
accordingly.

Ronald

Hi, I can't pull from main developer git server. Patchwork is slow to loading.
I have tried two ISP. Anything happened?

On Thu, Feb 29, 2024, at 16:27, Zhao Zhili wrote:
> Hi, I can't pull from main developer git server. Patchwork is slow to loading.
> I have tried two ISP. Anything happened?

All seem okay for me.

> On Feb 29, 2024, at 7:27 AM, Zhao Zhili <quinkblack@foxmail.com> wrote:
> 
> Hi, I can't pull from main developer git server. Patchwork is slow to loading.
> I have tried two ISP. Anything happened?

Not sure if it's related but trac has also been very unreliable for me, with frequent "Service Unavailable" errors from Apache. 

- Cosmin

On Thu, Feb 29, 2024 at 06:25:20PM +0000, Cosmin Stejerean via ffmpeg-devel wrote:
> 
> 
> > On Feb 29, 2024, at 7:27 AM, Zhao Zhili <quinkblack@foxmail.com> wrote:
> > 
> > Hi, I can't pull from main developer git server. Patchwork is slow to loading.
> > I have tried two ISP. Anything happened?

developer git is provided by videolan, patchwork is on a VM on our main server
so there should be no correlation its different hw different places hosted


> 
> Not sure if it's related but trac has also been very unreliable for me, with frequent "Service Unavailable" errors from Apache. 

i see these occasionally, but so far i have been unable to fully understand the issue

when i login after one of these
on the server "top" shows nothing hogging the cpu,
free shows most memory available
theres double digit gb free disk space
/server-status
looks liek this:

        Parent Server MPM Generation: 12
        Server uptime: 15 days 20 hours 13 minutes 44 seconds
        Server load: 0.05 0.07 0.17
        Total accesses: 4221997 - Total Traffic: 159.9 GB - Total Duration: 818395622
        CPU Usage: u806.2 s216.76 cu246864 cs10657.5 - 18.9% CPU load
        3.08 requests/sec - 122.5 kB/second - 39.7 kB/request - 193.841 ms/request
        2 requests currently being processed, 23 idle workers
        Slot	PID	Stopping	Connections	          Threads	Async   connections
                                        total	accepting	busy	idle	writing	keep-alive	closing
        2	360340	no (old gen)	0	yes	        0	25	0	0	        0
        3	360418	no (old gen)	0	yes	        0	25	0	0	        0
        5	295117	no	        3	yes	        2	23	0	0	        0
        Sum	3	0	        3	 	        2	73	0	0	        0
        ..................................................______________
        ____________________________________.........................___
        R__W__________________
        Scoreboard Key:
        "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
        "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
        "C" Closing connection, "L" Logging, "G" Gracefully finishing,
        "I" Idle cleanup of worker, "." Open slot with no current process

I tried downloading a random page 10 times sequentially and multiple of them
came up as 503

restarting apache and i can download the page 100 times in parallel with
no issues now
I suspect a relation to the "no (old gen)" processes
but i dont know where these came from. This is 2.4.41 and that seems to
have bugs with not always stoping processes on a reload or something
at least thats what i found, it doesnt seem to re-occur ATM when intentionally
doing a
service apache2 reload

tips welcome, also if another root admin wants to look at this, please do
iam not a web server admin nor is this ma favorite job :)
but at least ive today figured out the apparent relation to "no (old gen)"

thx

[...]