The spiders crawl the URLs systematically. Simultaneously, they check with the robots.txt file to examine whether or not they are permitted to crawl any particular URL. I'd be remiss if I did not provide a shout-out to https://www.moneyrobot.com/