Unverified Commit e09cda74 authored by Mattia Rizzolo's avatar Mattia Rizzolo
Browse files

Only block the bots on jenkins.d.n and tests.r-b.o, not the other websites 



This should let the crawlers crawl r-b.o, diffoscope.org, etc, which we
really want to be indexed everywhere.

Signed-off-by: Mattia Rizzolo's avatarMattia Rizzolo <mattia@debian.org>
parent 23442fc2

hosts/jenkins/etc/apache2/sites-available/jenkins.debian.net.conf

+8 −5
Original line number Diff line number Diff line
@@ -15,10 +15,7 @@ 
	Header always add Strict-Transport-Security "max-age=15552000"

</Macro>



<Macro common-directives $name>

	ServerName $name

	ServerAdmin holger@layer-acht.org



<Macro block-bots>

	#block bad bots with a 403

	SetEnvIf User-Agent "AhrefsBot" bad_bot

	SetEnvIf User-Agent "Baiduspider" bad_bot
@@ -39,7 +36,11 @@ 
	SetEnvIf User-Agent "Twitterbot" bad_bot

	SetEnvIf User-Agent "yacybot" bad_bot

	SetEnvIf User-Agent "ZoomBot" bad_bot

	#end block bad bots

</Macro>



<Macro common-directives $name>

	ServerName $name

	ServerAdmin holger@layer-acht.org



	<Directory />

		Options FollowSymLinks
@@ -129,6 +130,7 @@ Use https-redirect www.diffoscope.org 
<VirtualHost *:443>

	Use common-directives jenkins.debian.net

	Use common-directives-ssl jenkins.debian.net

	Use block-bots



	DocumentRoot /var/www

	AddDefaultCharset utf-8
@@ -189,6 +191,7 @@ Use https-redirect www.diffoscope.org 
<VirtualHost *:443>

	Use common-directives tests.reproducible-builds.org

	Use common-directives-ssl tests.reproducible-builds.org

	Use block-bots



	DocumentRoot /var/lib/jenkins/userContent/reproducible

	AddDefaultCharset utf-8