Spark 统计各学科排名前3-未分区一、元数据如下：二、目标：三、思路：四、代码如下五、运行结果如下

87 阅读 0 评论 58 点赞

我是靠谱客的博主玩命红牛，最近开发中收集的这篇文章主要介绍Spark 统计各学科排名前3-未分区一、元数据如下：二、目标：三、思路：四、代码如下五、运行结果如下，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

2019独角兽企业重金招聘Python工程师标准>>>

一、元数据如下：

20161123101523 http://h5.learn.com/h5/teacher.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://bigdatalearn.com/bigdata/teacher.shtml
20161123101523 http://java.learn.com/java/video.shtml
20161123101523 http://bigdata.learn.com/bigdata/teacher.shtml
20161123101523 http://ui.learn.com/ui/course.shtml
20161123101523 http://bigdata.learn.com/bigdata/teacher.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://java.learn.com/java/video.shtml
20161123101523 http://ui.learn.com/ui/video.shtml
20161123101523 http://h5.learn.com/h5/course.shtml
20161123101523 http://h5.learn.com/h5/teacher.shtml
20161123101523 http://bigdatalearn.com/bigdata/teacher.shtml
20161123101523 http://bigdata.learn.com/bigdata/video.shtml
20161123101523 http://ui.learn.com/ui/teacher.shtml
20161123101523 http://java.learn.com/java/video.shtml

二、目标：

统计出每个学科排名前3的，如：

三、思路：

1、先将所有Key相同的合并，得到如下结果

java-jdbc-8

java-js2e 18

java-高并发-100

java-中间件 15

h5-画布-18

h5-js-184

h5-事件驱动-20

h5-css-192

h5-easyui-3

大数据-kudu-38

大数据-Spark-300

大数据-Sqoop-9

大数据-Hbase-150

大数据-Hadoop 280

2、按照Key分组，然后排序取前3

四、代码如下

package scalapackage.testspark
import java.net.URL
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
/**
* Created by Germmy on 2018/5/12.
*/
object TestTop3 {
def main(args: Array[String]) {
val sparkConf: SparkConf = new SparkConf().setAppName("SparkTop3").setMaster("local[*]")
val sc: SparkContext = new SparkContext(sparkConf)
val file: RDD[String] = sc.textFile("D:\temp\course.txt")
//1.先reduceByKey
val res=file.map(x=>{
val split: Array[String] = x.split(" ")
val url: String = split(1)
(url,1)
})
val sumedUrls: RDD[(String, Int)] = res.reduceByKey(_+_)
//取出学科
val res2=sumedUrls.map(x=>{
val url=x._1
val count=x._2
val project=new URL(url).getHost
(project,url,count)
})
//分组统计
val values: RDD[(String, List[(String, String, Int)])] = res2.groupBy(_._1).mapValues(_.toList.sortBy(_._3).reverse.take(3))
println(values.collect().toBuffer)
}
}