Very simple alternative to "Apache Nutch":http://nutch.apache.org/ created in Grails. Crawled data could be stored to files, saved to database or sent to Apache Solr server for indexing. Use "Apache Camel":http://camel.apache.org/ as integration framework and "Apache ActiveMQ":http://activemq.apache.org/ as source messaging and integration patterns server.