Different search engines keep the cache of web pages for different durations. Because of search engine cache, a visitor may reach a web page of a recently closed website during his/her search with relevant keywords. For example, while searching for say, escort marketing services, a visitor may land up on the "About Us" page although the original website was closed sometimes back. The cached version usually displays different site elements like, CSS, images and such other information to the visitor.
Any organization using Search engine Optimization to promote their website may always want to see all search engine activities including the views of cached pages. The web server log files can show the referrer details of the cached pages. Very often, the visitor may get linked back to the original website through the cached web pages.
To stop search engines from indexing a certain page or the entire website and thereby to prevent caching, one can use the "robots.txt" file. The robots.txt file is stored in the web server that actually instructs the search engine about indexing or archiving. If the user does not want a page to be indexed, (for instance, a developer engaged in an escort agency web design process) he/she can use a small code segment to prevent search engines from indexing the web page.
