我使用apachenutch以编程方式在emr集群中以6个周期对大约7000个url进行爬网（在爬网过程中几乎没有定制的map reduce作业）。版本是：nutch=v1.15 hadoop=2.7.3我在amazonemr集群上运行它，有20个ec2 m4.1大型spot示例。爬网的代码是：

public crawl(Folder seeds, Folder output) 
        throws IOException, InterruptedException {
        final Folder crawldb = output.folder("crawldb");
        try {
            new Injector(this.conf).inject(
                crawldb.path(), seeds.path(),
                true, true
            );
        } catch (final ClassNotFoundException err) {
            throw new IOException("Failed to inject URLs", err);
        }
        final Folder segments = output.mkdir("segments");
        // cycles = 6 in my case
        for (int idx = 0; idx < cycles; ++idx) {
            this.cycle(crawldb, segments);
        }
    }

    private void cycle(final Folder crawldb, final Folder segments)
        throws IOException, InterruptedException {
        try {
            Logger.info(this, "Generating...");
            // configured as 1_000_000 in EMR cluster
            final int topn = this.conf.getInt("yc.gen.topn", 1000);
            // configured as 40 (2 x slave_nodes) in EMR cluster
            final int nfetch = this.conf.getInt("yc.gen.nfetch", 1);
            new Generator(this.conf).generate(
                crawldb.path(),
                segments.path(),
                nfetch, topn, System.currentTimeMillis()
            );
            // the latest segment
            final Optional<Folder> next = Batch.nextSegment(segments);
            if (next.isPresent()) {
                final Path sgmt = next.get().path();
                Logger.info(this, "Fetching %s...", sgmt);
                new Fetcher(this.conf).fetch(
                    // @checkstyle MagicNumber (1 line)
                    sgmt, 10
                );
                Logger.info(this, "Parsing %s...", sgmt);
                new ParseSegment(this.conf).parse(sgmt);
            }
            new CrawlDb(this.conf).update(
                crawldb.path(),
                // all segments paths
                segments.subfolders().stream()
                    .toArray(Path[]::new),
                true, true
            );
        } catch (final ClassNotFoundException err) {
            throw new IOException(
                "Failed to generate/fetch/parse segment", err
            );
        }
    }

当我用7000个种子url和6个运行周期运行它时，nutch会变得非常慢 FetchData 工作：它运行了大约3个小时，它似乎是等待最后一个Map完成约2.5个小时（见截图附件）。这项工作可能有什么问题以及如何加快fetchdata阶段的速度，也许我可以将其配置为跳过缓慢的fetcher（如果我错过几个url，这不是一个大问题）。