博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
eclipse 配置mapreduce环境出错
阅读量:4658 次
发布时间:2019-06-09

本文共 4840 字,大约阅读时间需要 16 分钟。

初学mapreduce,想在eclipse上配置mapreduce的环境,网上之类的教程,很多但是按照教程配之后,并不能正常运行。

碰到下面的错误:

15/10/17 20:10:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

15/10/17 20:10:39 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
15/10/17 20:10:40 INFO input.FileInputFormat: Total input paths to process : 2
15/10/17 20:10:40 INFO mapred.JobClient: Running job: job_local_0001
15/10/17 20:10:40 INFO input.FileInputFormat: Total input paths to process : 2
15/10/17 20:10:41 INFO mapred.MapTask: io.sort.mb = 100
15/10/17 20:10:41 INFO mapred.MapTask: data buffer = 79691776/99614720
15/10/17 20:10:41 INFO mapred.MapTask: record buffer = 262144/327680
15/10/17 20:10:41 INFO mapred.MapTask: Starting flush of map output
15/10/17 20:10:41 INFO mapred.MapTask: Finished spill 0
15/10/17 20:10:41 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
15/10/17 20:10:41 INFO mapred.LocalJobRunner:
15/10/17 20:10:41 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
15/10/17 20:10:41 INFO mapred.MapTask: io.sort.mb = 100
15/10/17 20:10:41 INFO mapred.MapTask: data buffer = 79691776/99614720
15/10/17 20:10:41 INFO mapred.MapTask: record buffer = 262144/327680
15/10/17 20:10:41 INFO mapred.MapTask: Starting flush of map output
15/10/17 20:10:41 INFO mapred.MapTask: Finished spill 0
15/10/17 20:10:41 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
15/10/17 20:10:41 INFO mapred.LocalJobRunner:
15/10/17 20:10:41 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
15/10/17 20:10:41 INFO mapred.LocalJobRunner:
15/10/17 20:10:41 INFO mapred.Merger: Merging 2 sorted segments
15/10/17 20:10:41 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 52 bytes
15/10/17 20:10:41 INFO mapred.LocalJobRunner:
15/10/17 20:10:41 INFO mapred.JobClient: map 100% reduce 0%
15/10/17 20:10:42 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
15/10/17 20:10:42 INFO mapred.LocalJobRunner:
15/10/17 20:10:42 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
15/10/17 20:10:42 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://master:9000/user/hadoop/out99
15/10/17 20:10:42 INFO mapred.LocalJobRunner: reduce > reduce
15/10/17 20:10:42 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
15/10/17 20:10:42 INFO mapred.JobClient: map 100% reduce 100%
15/10/17 20:10:42 INFO mapred.JobClient: Job complete: job_local_0001
15/10/17 20:10:42 INFO mapred.JobClient: Counters: 14
15/10/17 20:10:42 INFO mapred.JobClient: FileSystemCounters
15/10/17 20:10:42 INFO mapred.JobClient: FILE_BYTES_READ=50343
15/10/17 20:10:42 INFO mapred.JobClient: HDFS_BYTES_READ=59
15/10/17 20:10:42 INFO mapred.JobClient: FILE_BYTES_WRITTEN=102356
15/10/17 20:10:42 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=24
15/10/17 20:10:42 INFO mapred.JobClient: Map-Reduce Framework
15/10/17 20:10:42 INFO mapred.JobClient: Reduce input groups=3
15/10/17 20:10:42 INFO mapred.JobClient: Combine output records=4
15/10/17 20:10:42 INFO mapred.JobClient: Map input records=2
15/10/17 20:10:42 INFO mapred.JobClient: Reduce shuffle bytes=0
15/10/17 20:10:42 INFO mapred.JobClient: Reduce output records=3
15/10/17 20:10:42 INFO mapred.JobClient: Spilled Records=8
15/10/17 20:10:42 INFO mapred.JobClient: Map output bytes=40
15/10/17 20:10:42 INFO mapred.JobClient: Combine input records=4
15/10/17 20:10:42 INFO mapred.JobClient: Map output records=4
15/10/17 20:10:42 INFO mapred.JobClient: Reduce input records=4

运行程序为hadoop自带的WordCount.java源代码

  1.在WordCount.java上右键导出jar文件到工程的根目录下。

  2.将导出的wordcount.jar文件,右键加入到buildpath。

  3.在源代码中加入

.................. public static void main(String[] args) throws Exception {    Configuration conf = new Configuration();    conf.set("mapred.job.tracker", "192.168.2.1:9001");    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();    if (otherArgs.length != 2) {      System.err.println("Usage: wordcount 
"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } .....................

转载于:https://www.cnblogs.com/wxquare/p/4888130.html

你可能感兴趣的文章
Java重写《C经典100题》 --21
查看>>
【Android基础】Fragment 详解之Fragment生命周期
查看>>
链表(裸题)
查看>>
11运算符重载
查看>>
磁盘系统的管理
查看>>
C/S
查看>>
Http Get/Post请求的区别
查看>>
STM32一键下载电路设计原理
查看>>
C语言中函数返回字符串的四种方法
查看>>
10月区块链领域投融资事件盘点
查看>>
Mybatis缓存策略
查看>>
卷积的意义【转】
查看>>
android图形系统详解五:Android绘制模式
查看>>
[剑指offer] 23. 二叉搜索树的后序遍历序列
查看>>
canvas绘画交叉波浪
查看>>
Linux 内核分析
查看>>
试一下:XP ( SP2 ) 本身就支持查杀流氓软件!
查看>>
centos6(7) minimal 基本环境配置
查看>>
P2837晚餐队列安排
查看>>
DP专题
查看>>