Hadoop实战任务-----编写MapReduce进行数据清洗

标题项目需求:

一个英文书籍包含成千上万个单词或者短语,在大量的单词中,找出相同字母组成的所有单词.

策略:

为加快数据处理的速度,借助Hadoop中的MapReduce编程模型的特点,编写出并行计算程序.

解决方案:

1、在 Map
阶段，对每个word（单词）按字母进行排序生成sortedWord，然后输出key/value键值对(sortedWord,word) 。
2、在 Reduce 阶段，统计出每组相同字母组成的所有文本

代码如下: Map代码

package com.itstar.partitiontest;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;import java.util.Arrays;/** 项目需求:*一个英文书籍包含成千上万个单词或者短语,在大量的单词中,找出相同字母组成的所有单词.* 策略:*为加快数据处理的速度,借助Hadoop中的MapReduce编程模型的特点,编写出并行计算程序.**解决方案:*1、在 Map 阶段，对每个word（单词）按字母进行排序生成sortedWord，然后输出key/value键值对(sortedWord,word) 。*2、在 Reduce 阶段，统计出每组相同字母组成的所有文本 。**** */public class wordMap extends Mapper {Text sortedText =new Text();Text orginalText =new Text();@Overrideprotected void map(Object key, Text value, Context context) throws IOException, InterruptedException {String line = value.toString();char[] wordline = line.toCharArray();Arrays.sort(wordline);String sortword = new String(wordline);sortedText.set(sortword);orginalText.set(line);context.write(sortedText, orginalText);}}

【Hadoop实战任务-----编写MapReduce进行数据清洗】Reduce代码

package com.itstar.partitiontest;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;import java.util.StringTokenizer;public class wordReduce extends Reducer {Text outputKey = new Text();Text outputValue = https://tazarkount.com/read/new Text();@Overrideprotected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {String output ="";for (Text text:values) {if (!output.equals("")){output =output+",";}output=output+text.toString();}StringTokenizer outputTokemizer = new StringTokenizer(output, ",");if (outputTokemizer.countTokens() >=2){outputKey.set(key.toString());outputValue.set(output);context.write(outputKey, outputValue);}}}

package com.itstar.partitiontest;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;public class wordDriver {public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {args =new String[]{"F:\\Software\\Hadoop\\input\\word.txt","F:\\Software\\Hadoop\\output\\outworld"};Configuration conf = new Configuration();Job job = Job.getInstance(conf);job.setJarByClass(wordDriver.class);job.setMapperClass(wordMap.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(Text.class);job.setNumReduceTasks(0);FileInputFormat.setInputPaths(job, new Path(args[0]));FileOutputFormat.setOutputPath(job,new Path(args[1]));job.waitForCompletion(true);}}

运行测试

该代码没有在Hadoop集群运行,在windows本地运行的.如需在Hadoop集群运行,将wordDriver里面的代码进行修改.