运行Spark GraphX Pregel出现Issue communicating with driver in heartbeater异常
最近基于Spark GraphX Pregel开发了一个ETL任务,运行过程中会报Issue communicating with driver in heartbeater,然后就是Connection refused错误。经分析,Spark GraphX Pregel在每次迭代都会产生新的job, 如果迭代的次数多,会产生大量的job, driver会被阻塞造成通讯失败,任务中断。解决办法:调整driver-memory参数。spark.driver.memory 1g Amou