java—在hadoop的mapreduce框架中运行“SeleniumWebDriver”,在map步骤中冻结

ddhy6vgd  于 2021-06-04  发布在  Hadoop
关注(0)|答案(0)|浏览(275)

几天前,我决定让seleniumwebdriver(第三方包)在hadoop的mapreduce框架中运行。我遇到了一个问题。Map冻结 new FirefoxDriver(); . firefoxdriver类位于名为 selenium-server-standalone-2.38.0.jar . 如果有人有一些经验或兴趣,我需要你的帮助!

一些细节:

问题详细信息
为了在命令行中运行代码,我使用“xvfb”停止firefox图形界面。然后我一开始说的问题就出现了。我查了任务追踪者的日志,发现代码冻结了 this.driver = new FirefoxDriver(ffprofile); 虽然代码冻结了,但是firefox已经安装好了,我使用 ps -A | grep firefox 环境:
ubuntu 10.04 32位;hadoop-1.2.0;mozilla firefox 17.0.5版本;selenium-server-standalone-2.38.0.jar;xvfb;
提示
(1) hadoop在pesudo分布式模式下运行;
(2) 当我在eclipse中运行代码时,一切正常。firefox会按计划弹出(最后我会展示演示代码);
(3) 如果你跑进去 org.openqa.selenium.WebDriverException: Unable to bind to locking port 7054 within 45000 ms ,使用逗号 ps -A | grep firefox 检查是否安装了firefox,并使用 killall firefox .
(4) 让代码在命令行中运行。也许你会遇到 Error no display specified 您可以安装 xvfb 和设置 xvfb 使用 Xvfb :99 -ac 2>/dev/null & . 安装前 xvfb 追加一行 export DISPLAY=:99 到年底 HADOOP_HOME/conf/hadoop-env.sh 代码演示

public class MapRunnerNewFirefox extends Configured implements Tool, MapRunnable{
        public static final Logger LOG = LoggerFactory.getLogger(MapRunnerNewFirefox.class);
        @Override
        public void configure(JobConf conf) {
        }
        @Override
        public void run(RecordReader recordReader,
            OutputCollector output, Reporter reporter) throws IOException {
            LongWritable key = new LongWritable(-1);// shouldn't be null ,otherwise the recordReader will report nullpointer err;
            Text val = new Text("begin text");      // same as up line;
            int i = 0;
            reporter.progress();
            while(recordReader.next(key, val)){         
                if(LOG.isInfoEnabled()){
                    LOG.info("key: "+key.toString()+" val: "+val.toString());
                }
                String temp = "ao";
                NewFirefox ff = new NewFirefox("/home/cc/firefox/firefox/firefox");
                output.collect(new Text("get-"+key.toString()), new Text(temp));
            }
        }
        @Override
        public int run(String[] args) throws Exception {
            if(LOG.isInfoEnabled()) {
                LOG.info("set maprunner conf");
            }
            Path urlDir = new Path(args[0]);
            Path resultDir = new Path(args[1] + System.currentTimeMillis());
            JobConf job = new JobConf(getConf());
            job.setNumMapTasks(1);
            job.setJobName("hello maprunners");
            job.setInputFormat(TextInputFormat.class);
            FileInputFormat.addInputPath(job, urlDir);      
            job.setMapRunnerClass(MapRunnerNewFirefox.class);
            FileOutputFormat.setOutputPath(job, resultDir);
            job.setOutputFormat(TextOutputFormat.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(Text.class);
            JobClient.runJob(job);      
            return 0;
        }
        public static void main(String[] args) throws Exception{    
            Configuration conf = new Configuration();       
            int res = ToolRunner.run(conf, new MapRunnerNewFirefox(), args);
            System.exit(res);   
        }
    }
    public class NewFirefox {
        private WebDriver driver;
        private static final Logger LOG = LoggerFactory.getLogger(NewFirefox.class);
        public NewFirefox(String firefoxPath){
            if(LOG.isInfoEnabled()){
                LOG.info("firefox****0");
            }
            System.setProperty("webdriver.firefox.bin", firefoxPath);
            if(LOG.isInfoEnabled()){
                LOG.info("firefox****1");
            }
            ProfilesIni profile = new ProfilesIni();
            FirefoxProfile ffprofile = profile.getProfile("default");
            if(LOG.isInfoEnabled()){
                LOG.info("firefox****2");
            }
            this.driver = new FirefoxDriver(ffprofile);
            if(LOG.isInfoEnabled()){
                LOG.info("firefox****3");
            }
            this.driver.quit();
            if(LOG.isInfoEnabled()){
                LOG.info("firefox quit");
            }
        }
    }

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题