Heritrix是一个开源,可扩展的web爬虫项目。Heritrix设计成严格按照robots.txt文件的排除指示和META robots标签。-Heritrix is an open source, scalable web reptiles project. Heritrix is designed in strict accordance with the robots.txt file to exclude directives and META robots tags.