From patchwork Wed Jul 14 18:51:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Niedermayer X-Patchwork-Id: 28929 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp685638ios; Wed, 14 Jul 2021 11:59:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzaS30TLil5dKcx8ZbsGfCWH+ClLOV3aj8P6WfyRdK5JbDBFqvMP4pgqnwUc6+zrsu1a9Sc X-Received: by 2002:a17:906:4745:: with SMTP id j5mr13944282ejs.75.1626289157097; Wed, 14 Jul 2021 11:59:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1626289157; cv=none; d=google.com; s=arc-20160816; b=JgQlXXqFGHJXld4TwlIxzv0fd+vLb/V4jNxswFpPfN/4GhXFK4A/i13Tj1AsR0RQIo YdB0JS8rtMtI6xU8EjvLQ5iudZxqxbi1+l8fVZVobUdTB54Dfuhuvs8bGbNlZyRXj5zB GgGXEsPWa1ZWtHAXrbDRLmIMbQGpi3isFV1OxfzKCkMxDyVgW2bMS5PfXLh+jkd5oIak ZWSutx2j9XdDrlVvzX+/L08ofosRAy6TZYnUMet5mRESQr2h91u5VIczDnlROcXpkQhF bni3+2AGUjlRz1vd0h689gx/FPEWdVR3PEHcopiDDMijbXfa7KjvAekwgDDJYinqYdTC GRgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:delivered-to; bh=r5T1ow9UFddjWdcllmqTC1aqgmD7Ahe3lO/6AbDsPeU=; b=zbL1JzUruPGX0kxaVYUTEkLgY+Vupda/EQ5oCk4k5GqYcsXnXaFqtuas/RWBZ27Rkn jvpAQX7eS6VhIRpBMCj3JmzJ2BSID3P4cKrmbIwqUUPRnpBaguGDOb92/yn0UWa+ZEog BC5KL6uOQhW5+0tb1Y6aGypVnEsovYALFo+NYJlWHG984VE2N5UFK3nNhMSP6/cBR1uE 64+m2Y4yW/+2tPpQbUN8SXdCElSpNWWSz/SabmMN9+g9mciRuQC5yk/TPWBKEghldoR7 0Othoa58MK+zLDPx9npnYTinwz30ZBvzpNRAIBNTfOwUzFhg+7bfS58mvKZL8t3dHfiS 5RgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id rn4si4419509ejb.0.2021.07.14.11.59.16; Wed, 14 Jul 2021 11:59:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 313FC68A976; Wed, 14 Jul 2021 21:59:14 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vie01a-dmta-pe05-3.mx.upcmail.net (vie01a-dmta-pe05-3.mx.upcmail.net [84.116.36.13]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AC9CD68A87F for ; Wed, 14 Jul 2021 21:59:06 +0300 (EEST) Received: from [172.31.216.235] (helo=vie01a-pemc-psmtp-pe12.mail.upcmail.net) by vie01a-dmta-pe05.mx.upcmail.net with esmtp (Exim 4.92) (envelope-from ) id 1m3jza-000rYz-03 for ffmpeg-devel@ffmpeg.org; Wed, 14 Jul 2021 20:52:22 +0200 Received: from localhost ([213.47.68.29]) by vie01a-pemc-psmtp-pe12.mail.upcmail.net with ESMTP id 3jycmJeBFljeH3jycmRrXS; Wed, 14 Jul 2021 20:51:22 +0200 X-Env-Mailfrom: michael@niedermayer.cc X-Env-Rcptto: ffmpeg-devel@ffmpeg.org X-SourceIP: 213.47.68.29 X-CNFS-Analysis: v=2.3 cv=BoHjPrf5 c=1 sm=1 tr=0 a=2hcxjKEKjp0CzLx6oWAm4g==:117 a=2hcxjKEKjp0CzLx6oWAm4g==:17 a=MKtGQD3n3ToA:10 a=1oJP67jkp3AA:10 a=GEAsPZ9sns4A:10 a=ZZnuYtJkoWoA:10 a=J25TCDAvEiyeXJs4RMwA:9 From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Wed, 14 Jul 2021 20:51:21 +0200 Message-Id: <20210714185121.24646-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 X-CMAE-Envelope: MS4wfJl1QrOIepEw63ckENzoZ5TUrz5kUhMXypA/25uMYeN9dhVQvHjz2gXrmYN9bEXtYCt/lcCbxZmYEx6hixy2VCcfNaorVHGs5zUIYAXKNIvUgK8RZgbO SqONYpzwJGWAhc/nRMjlNte6OvSKncA9uRUa0RMaPGx/5x3/otTD18tR Subject: [FFmpeg-devel] [PATCH] ffmpeg-web/robots.txt: attempt to keep spiders out of dynamically generated git content X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: qEoQ+674kmmR Signed-off-by: Michael Niedermayer --- htdocs/robots.txt | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/htdocs/robots.txt b/htdocs/robots.txt index eb05362..4bbc395 100644 --- a/htdocs/robots.txt +++ b/htdocs/robots.txt @@ -1,2 +1,13 @@ User-agent: * -Disallow: +Crawl-delay: 10 +Disallow: /gitweb/ +Disallow: /*a=search* +Disallow: /*/search/* +Disallow: /*a=blobdiff* +Disallow: /*/blobdiff/* +Disallow: /*a=commitdiff* +Disallow: /*/commitdiff/* +Disallow: /*a=snapshot* +Disallow: /*/snapshot/* +Disallow: /*a=blame* +Disallow: /*/blame/*