# # Robots.txt file # # All robots will spider the domain User-agent: * Disallow: # blocks Internet Archive wayback machine User-agent: ia_archiver Disallow: /