86314238748
Dynamically Generated Robots.txt file by Stuart Geiger
Stuart Geiger is fascinated by the search engine bots that crawl the web to index pages, and the time-honoured protocol of them having to first read the robot.txt file that grants them access to your website. Interestingly it is something that “a systems administrator will actually edit and maintain by hand”. Geiger has written a simple PHP script that will generate a unique robot.txt file when solicited by a bot crawler. The crawler would receive a custom-made entry pass that would only allow its kind to enter. So if its a Google crawler, it will receive a robot.txt that only lets Google crawlers enter and so on. Its a simple process but I can see potential in expanding on Geiger’s work here to generate interesting interactions between these otherwise invisible bots, perhaps with the goal of visualising them, or getting indexed differently across search engines. If you do, please let me know.