YNAO OpenIR  > 其他
DALiuGE: A graph execution framework for harnessing the astronomical data deluge
Wu, C.1; Tobar, R.1; Vinsen, K.1; Wicenec, A.1; Pallot, D.1; Lao, B.2; Wang, R.3; An, T.2; Boulton, M.1; Cooper, I.1; Dodson, R.1; Dolensky, M.1; Mei Y(梅盈)4,5; Wang F(王锋)4,5
Source PublicationASTRONOMY AND COMPUTING
2017-07-01
Volume20Pages:1-15
DOI10.1016/j.ascom.2017.03.007
Contribution Rank第5完成单位
Indexed ByEI
KeywordDataflow Graph Execution Engine Data Driven Square Kilometre Array Many-task Computing
Abstract

The Data Activated Liu(1) Graph Engine - DALiuGE(2) - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities. (C) 2017 Elsevier B.V. All rights reserved.

Funding ProjectAustralian Government ; Government of Western Australia ; Amazon Web Services ; AstroCompute project ; National Natural Science Foundation of China (NSFC)[61221491] ; National Natural Science Foundation of China (NSFC)[U1231205] ; National Natural Science Foundation of China (NSFC)[U1631129] ; Open Fund from HPCL[201401-01] ; State Government of Western Australia
Funding OrganizationAustralian Government ; Government of Western Australia ; Amazon Web Services ; AstroCompute project ; National Natural Science Foundation of China (NSFC)[61221491, U1231205, U1631129] ; Open Fund from HPCL[201401-01] ; State Government of Western Australia
Language英语
Subject Area天文学 ; 天文学其他学科 ; 计算机科学技术 ; 计算机应用
MOST Discipline Catalogue理学 ; 理学::天文学 ; 工学 ; 工学::计算机科学与技术(可授工学、理学学位)
SubtypeArticle
PublisherELSEVIER SCIENCE BV
Publication PlacePO BOX 211, 1000 AE AMSTERDAM, NETHERLANDS
ISSN2213-1337
URL查看原文
WOS IDWOS:000412033600001
WOS Research AreaAstronomy & Astrophysics ; Computer Science
WOS SubjectAstronomy & Astrophysics ; Computer Science, Interdisciplinary Applications
EI Accession Number20172203712472
EI KeywordsPipeline processing systems
EI Classification Number619.1Pipe, Piping and Pipelines - 722.4Digital Computers and Systems - 723.2Data Processing and Image Processing
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ynao.ac.cn/handle/114a53/10042
Collection其他
Corresponding AuthorWu, C.
Affiliation1.International Centre for Radio Astronomy Research (ICRAR), The University of Western Australia, M468, 35 Stirling Highway, Crawley, Perth, WA, 6009, Australia
2.Shanghai Astronomical Observatory, Shanghai, China
3.Oak Ridge National Laboratory, TN, 37831, United States
4.Kunming University of Science and Technology Kunming, Yunnan, China
5.Yunnan Observatories Chinese Academy of Sciences Kunming, Yunnan, China
Recommended Citation
GB/T 7714
Wu, C.,Tobar, R.,Vinsen, K.,et al. DALiuGE: A graph execution framework for harnessing the astronomical data deluge[J]. ASTRONOMY AND COMPUTING,2017,20:1-15.
APA Wu, C..,Tobar, R..,Vinsen, K..,Wicenec, A..,Pallot, D..,...&Wang F.(2017).DALiuGE: A graph execution framework for harnessing the astronomical data deluge.ASTRONOMY AND COMPUTING,20,1-15.
MLA Wu, C.,et al."DALiuGE: A graph execution framework for harnessing the astronomical data deluge".ASTRONOMY AND COMPUTING 20(2017):1-15.
Files in This Item:
File Name/Size DocType Version Access License
DALiuGE_ A graph exe(3011KB)期刊论文出版稿开放获取CC BY-NC-SAView Application Full Text
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Baidu academic
Similar articles in Baidu academic
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: DALiuGE_ A graph execution framework for harnessing the astronomical data deluge.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.