Norconex Web Crawler is use for in-depth crawling of web sites of any size. It aims to be very flexible, easy to extend, and portable