YNAO OpenIR  > 其他
DALiuGE: A graph execution framework for harnessing the astronomical data deluge
Wu, C.1; Tobar, R.1; Vinsen, K.1; Wicenec, A.1; Pallot, D.1; Lao, B.2; Wang, R.3; An, T.2; Boulton, M.1; Cooper, I.1; Dodson, R.1; Dolensky, M.1; Mei Y(梅盈)4,5; Wang F(王锋)4,5
发表期刊ASTRONOMY AND COMPUTING
2017-07-01
卷号20页码:1-15
DOI10.1016/j.ascom.2017.03.007
产权排序第5完成单位
收录类别EI
关键词Dataflow Graph Execution Engine Data Driven Square Kilometre Array Many-task Computing
摘要The Data Activated Liu(1) Graph Engine - DALiuGE(2) - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities. (C) 2017 Elsevier B.V. All rights reserved.
项目资助者Australian Government ; Government of Western Australia ; Amazon Web Services ; AstroCompute project ; National Natural Science Foundation of China (NSFC)(61221491, U1231205, U1631129) ; Open Fund from HPCL(201401-01) ; State Government of Western Australia
语种英语
学科领域天文学
文章类型Article
ISSN2213-1337
URL查看原文 ; 查看原文
WOS记录号WOS:000412033600001
WOS研究方向Astronomy & Astrophysics ; Computer Science
WOS类目Astronomy & Astrophysics ; Computer Science, Interdisciplinary Applications
引用统计
文献类型期刊论文
条目标识符http://ir.ynao.ac.cn/handle/114a53/10042
专题其他
通讯作者Wu, C.
作者单位1.International Centre for Radio Astronomy Research (ICRAR), The University of Western Australia, M468, 35 Stirling Highway, Crawley, Perth, WA, 6009, Australia
2.Shanghai Astronomical Observatory, Shanghai, China
3.Oak Ridge National Laboratory, TN, 37831, United States
4.Kunming University of Science and Technology Kunming, Yunnan, China
5.Yunnan Observatories Chinese Academy of Sciences Kunming, Yunnan, China
推荐引用方式
GB/T 7714
Wu, C.,Tobar, R.,Vinsen, K.,et al. DALiuGE: A graph execution framework for harnessing the astronomical data deluge[J]. ASTRONOMY AND COMPUTING,2017,20:1-15.
APA Wu, C..,Tobar, R..,Vinsen, K..,Wicenec, A..,Pallot, D..,...&Wang F.(2017).DALiuGE: A graph execution framework for harnessing the astronomical data deluge.ASTRONOMY AND COMPUTING,20,1-15.
MLA Wu, C.,et al."DALiuGE: A graph execution framework for harnessing the astronomical data deluge".ASTRONOMY AND COMPUTING 20(2017):1-15.
条目包含的文件
文件名称/大小 文献类型 版本类型 开放类型 使用许可
DALiuGE_ A graph exe(3011KB)期刊论文作者接受稿开放获取CC BY-NC-SA浏览 请求全文
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Wu, C.]的文章
[Tobar, R.]的文章
[Vinsen, K.]的文章
百度学术
百度学术中相似的文章
[Wu, C.]的文章
[Tobar, R.]的文章
[Vinsen, K.]的文章
必应学术
必应学术中相似的文章
[Wu, C.]的文章
[Tobar, R.]的文章
[Vinsen, K.]的文章
相关权益政策
暂无数据
收藏/分享
文件名: DALiuGE_ A graph execution framework for harnessing the astronomical data deluge.pdf
格式: Adobe PDF
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。