YNAO OpenIR  > 其他
Alternative TitleReal-Time Data Processing in Mingamtu UItrawide Spectral Radio Heliograph Based on Spark Streaming
卫守林1,2; 刘鹏翔1; 王锋1,2; 邓辉1; 梁波1; 戴伟1,2
Source Publication天文研究与技术(Astronomical Research & Technology)
Contribution Rank第2完成单位
Indexed ByCSCD
KeywordMuser 射电天文 Spark 流式计算 实时计算

目前天文观测中对数据的实时处理需求越来越多,性能要求也越来越高,我国明安图射电频谱日像仪(Mingant U Sp Ectral Radioheliograph,MUSER)是同时以高时间、高空间和高频率分辨率对太阳进行射电频谱成像的设备。在低频部分的日常观测中,包含了两方面的需求:(1)对历史数据的处理;(2)5秒钟抽样观测数据的处理。抽样观测数据需要实时处理,并在监控终端显示,数据处理过程包含了数据校验、修正、成图、洁化等多个步骤,传统的单机处理模式已无法满足大数据量下的实时性要求。因此,实时数据计算中,使用Spark Streaming流式计算这一新兴的分布式计算方法,设计了自定义的接收器,并将多个图形处理器节点加入到分布式集群中。通过实验对性能进行评估,结果证明基于内存的高速执行引擎的特点能显著提高性能。期待能通过实验进一步优化算法和配置,获得更好的结果,并最终运用到实际环境中。目前天文观测中对数据的实时处理需求越来越多,性能要求也越来越高,我国明安图射电频谱日像仪(Mingant U Sp Ectral Radioheliograph,MUSER)是同时以高时间、高空间和高频率分辨率对太阳进行射电频谱成像的设备。在低频部分的日常观测中,包含了两方面的需求:(1)对历史数据的处理;(2)5秒钟抽样观测数据的处理。抽样观测数据需要实时处理,并在监控终端显示,数据处理过程包含了数据校验、修正、成图、洁化等多个步骤,传统的单机处理模式已无法满足大数据量下的实时性要求。因此,实时数据计算中,使用Spark Streaming流式计算这一新兴的分布式计算方法,设计了自定义的接收器,并将多个图形处理器节点加入到分布式集群中。通过实验对性能进行评估,结果证明基于内存的高速执行引擎的特点能显著提高性能。期待能通过实验进一步优化算法和配置,获得更好的结果,并最终运用到实际环境中。

Other Abstract

There is a growing demand for real-time processing in astronomical observations in recent years, meanwhile, the requirement for performance is also increasing dramatically.  Mingantu Ultrawide Spectral Radio Heliograph (MUSER) is a synthetic aperture radio interferometer with high temporal, spatial and spectral resolution. In daily observation of low frequency, MUSER contains two aspects of data processing, historical data processing and sampling observational data which is produced every 5 seconds and processed in real-time mode. The procedure of raw data processing contains validation, correction, clean and other processing steps, then the results need to be transmitted in real-time mode to monitoring end without user constantly refreshing or sending a request. The traditional stand-alone processing mode has been unable to meet the requirements of large amounts of data in real-time mode. In this paper, we explored the use of Spark Streaming in a new approach for MUSER real-time calculations across multiple machines and evaluated its effectiveness and efficiency. A customized receiver was created for real-time binary stream of MUSER. We also extended the Spark cluster by adding multiple GPU's nodes. The experiments have shown that Spark Streaming can significantly improve MUSER real-time processing performance for its memory-based execution engine. We might look forward to optimize the algorithm through experiments and configurations so as to obtain better results, and apply it to the actual environment of MUSER finally.

Funding Project国家自然科学基金[11403009] ; 国家自然科学基金[U1231205]
Funding Organization国家自然科学基金[11403009, U1231205]
Subject Area天文学 ; 天文学其他学科 ; 计算机科学技术 ; 计算机应用
MOST Discipline Catalogue理学 ; 理学::天文学 ; 工学 ; 工学::计算机科学与技术(可授工学、理学学位)
Citation statistics
Cited Times:1[CSCD]   [CSCD Record]
Document Type期刊论文
Affiliation1.昆明理工大学云南省计算机技术应用重点实验室, 云南, 昆明, 650500
2.中国科学院云南天文台, 云南, 昆明, 650011
First Author AffilicationYunnan Observatories, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
卫守林,刘鹏翔,王锋,等. 基于SparkStreaming的明安图射电频谱日像仪实时数据处理[J]. 天文研究与技术(Astronomical Research & Technology),2017,14(4):421-428.
APA 卫守林,刘鹏翔,王锋,邓辉,梁波,&戴伟.(2017).基于SparkStreaming的明安图射电频谱日像仪实时数据处理.天文研究与技术(Astronomical Research & Technology),14(4),421-428.
MLA 卫守林,et al."基于SparkStreaming的明安图射电频谱日像仪实时数据处理".天文研究与技术(Astronomical Research & Technology) 14.4(2017):421-428.
Files in This Item:
File Name/Size DocType Version Access License
基于SparkStreaming的明安图(1076KB)期刊论文出版稿开放获取CC BY-NC-SAView Application Full Text
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[卫守林]'s Articles
[刘鹏翔]'s Articles
[王锋]'s Articles
Baidu academic
Similar articles in Baidu academic
[卫守林]'s Articles
[刘鹏翔]'s Articles
[王锋]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[卫守林]'s Articles
[刘鹏翔]'s Articles
[王锋]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: 基于SparkStreaming的明安图射电频谱日像仪实时数据处理.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.