MENU
A robots.txt file tells the search engine crawlers about the URLs which the crawler can access on your site. Which of the following is true about robots.txt?
The robots.txt file is a text file placed in a website's root directory to communicate with web crawlers (e.g., Googlebot) about which pages or resources should not be accessed or indexed. It uses directives like Disallow to specify restricted areas (e.g., Disallow: /admin/). However, robots.txt is not a security mechanism; it is only a request to crawlers, and malicious bots or users can ignore it.
Option A ('Developers must not list any sensitive files and directories in this file'): Correct. Listing sensitive files or directories (e.g., Disallow: /secret/) in robots.txt can inadvertently expose their existence to attackers, who can then attempt to access them directly. The best practice is to avoid mentioning sensitive paths and rely on proper access controls (e.g., authentication, authorization) instead.
Option B ('Developers must list all sensitive files and directories in this file to secure them'): Incorrect. Listing sensitive paths in robots.txt does not secure them; it only informs crawlers to avoid them, and it can serve as a roadmap for attackers.
Option C ('Both A and B'): Incorrect, as A and B are contradictory; B is false.
Option D ('None of the above'): Incorrect, as A is true.
The correct answer is A, aligning with the CAP syllabus under 'Web Crawler Security' and 'Information Disclosure Prevention.'