分析需求：

WordCount的主要需求就是统计单词出现的数量，处理过程分为Map阶段和Reduce阶段。

代码分析：

Mapper类：

public class WordCountMapper extends Mapper<LongWritable, Text,Text, IntWritable> {
Text txt = new Text();
IntWritable intWritable = new IntWritable();
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
System.out.println("WordCountMapper
key"+key+"
value:" +value);
//分割字符串
String[] words = value.toString().split(" ");//[hello ,jdk ,hello ,java]
for(String word:words)
{
txt.set(word);
intWritable.set(1);
context.write(txt,intWritable);
}
}
}

Reduce类

public class WordCountReduce extends Reducer<Text, IntWritable,Text, LongWritable> {
@Override
// hello
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int count=0;
for(IntWritable intWritable:values)
{
count+=intWritable.get();
}
LongWritable longWritable = new LongWritable(count);
System.out.println("WordCountReduce key:"+key+"
value:"+longWritable);
context.write(key,longWritable);
}
}

Driver类(启动类)

public class WordCountDriver {
public static void main(String[] args) throws Exception {
System.setProperty("HADOOP_USER_NAME","root");
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
//设置启动类
job.setJarByClass(WordCountDriver.class);
//配置当前job 执行的mapper类
job.setMapperClass(WordCountMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
//配置当前job，执行的reduce类
job.setReducerClass(WordCountReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
//指定读取文件路径
// Path path = new Path("hdfs://nnode1:9000/input");
Path path = new Path(args[0]);
FileInputFormat.setInputPaths(job,path);
//指定执行任务完成后的输出路径
// Path patout = new Path("hdfs://nnode1:9000/output");
Path pathout = new Path(args[1]);
FileSystem fs = FileSystem.get(pathout.toUri(),conf);
if(fs.exists(pathout))
{
fs.delete(pathout,true);
}
FileOutputFormat.setOutputPath(job,pathout);
//表示将运行进度等信息及时输出给用户
job.waitForCompletion(true);
//System.out.println(args[0]+"---"+args[1]);
}
}

最后

以上就是灵巧战斗机为你收集整理的使用IDEA实现MapReduce的WordCount程序分析需求：代码分析：的全部内容，希望文章能够帮你解决使用IDEA实现MapReduce的WordCount程序分析需求：代码分析：所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。