几天前,我决定让seleniumwebdriver(第三方包)在hadoop的mapreduce框架中运行。我遇到了一个问题。Map冻结 new FirefoxDriver();
. firefoxdriver类位于名为 selenium-server-standalone-2.38.0.jar
. 如果有人有一些经验或兴趣,我需要你的帮助!
一些细节:
问题详细信息
为了在命令行中运行代码,我使用“xvfb”停止firefox图形界面。然后我一开始说的问题就出现了。我查了任务追踪者的日志,发现代码冻结了 this.driver = new FirefoxDriver(ffprofile);
虽然代码冻结了,但是firefox已经安装好了,我使用 ps -A | grep firefox
环境:
ubuntu 10.04 32位;hadoop-1.2.0;mozilla firefox 17.0.5版本;selenium-server-standalone-2.38.0.jar;xvfb;
提示
(1) hadoop在pesudo分布式模式下运行;
(2) 当我在eclipse中运行代码时,一切正常。firefox会按计划弹出(最后我会展示演示代码);
(3) 如果你跑进去 org.openqa.selenium.WebDriverException: Unable to bind to locking port 7054 within 45000 ms
,使用逗号 ps -A | grep firefox
检查是否安装了firefox,并使用 killall firefox
.
(4) 让代码在命令行中运行。也许你会遇到 Error no display specified
您可以安装 xvfb
和设置 xvfb
使用 Xvfb :99 -ac 2>/dev/null &
. 安装前 xvfb
追加一行 export DISPLAY=:99
到年底 HADOOP_HOME/conf/hadoop-env.sh
代码演示
public class MapRunnerNewFirefox extends Configured implements Tool, MapRunnable{
public static final Logger LOG = LoggerFactory.getLogger(MapRunnerNewFirefox.class);
@Override
public void configure(JobConf conf) {
}
@Override
public void run(RecordReader recordReader,
OutputCollector output, Reporter reporter) throws IOException {
LongWritable key = new LongWritable(-1);// shouldn't be null ,otherwise the recordReader will report nullpointer err;
Text val = new Text("begin text"); // same as up line;
int i = 0;
reporter.progress();
while(recordReader.next(key, val)){
if(LOG.isInfoEnabled()){
LOG.info("key: "+key.toString()+" val: "+val.toString());
}
String temp = "ao";
NewFirefox ff = new NewFirefox("/home/cc/firefox/firefox/firefox");
output.collect(new Text("get-"+key.toString()), new Text(temp));
}
}
@Override
public int run(String[] args) throws Exception {
if(LOG.isInfoEnabled()) {
LOG.info("set maprunner conf");
}
Path urlDir = new Path(args[0]);
Path resultDir = new Path(args[1] + System.currentTimeMillis());
JobConf job = new JobConf(getConf());
job.setNumMapTasks(1);
job.setJobName("hello maprunners");
job.setInputFormat(TextInputFormat.class);
FileInputFormat.addInputPath(job, urlDir);
job.setMapRunnerClass(MapRunnerNewFirefox.class);
FileOutputFormat.setOutputPath(job, resultDir);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception{
Configuration conf = new Configuration();
int res = ToolRunner.run(conf, new MapRunnerNewFirefox(), args);
System.exit(res);
}
}
public class NewFirefox {
private WebDriver driver;
private static final Logger LOG = LoggerFactory.getLogger(NewFirefox.class);
public NewFirefox(String firefoxPath){
if(LOG.isInfoEnabled()){
LOG.info("firefox****0");
}
System.setProperty("webdriver.firefox.bin", firefoxPath);
if(LOG.isInfoEnabled()){
LOG.info("firefox****1");
}
ProfilesIni profile = new ProfilesIni();
FirefoxProfile ffprofile = profile.getProfile("default");
if(LOG.isInfoEnabled()){
LOG.info("firefox****2");
}
this.driver = new FirefoxDriver(ffprofile);
if(LOG.isInfoEnabled()){
LOG.info("firefox****3");
}
this.driver.quit();
if(LOG.isInfoEnabled()){
LOG.info("firefox quit");
}
}
}
暂无答案!
目前还没有任何答案,快来回答吧!