Author ORCID Identifier
https://orcid.org/0000-0003-3877-7432
Document Type
Conference Paper
Rights
Available under a Creative Commons Attribution Non-Commercial Share Alike 4.0 International Licence
Disciplines
Computer Sciences
Abstract
Since the 1970s the CCD has been the principle method of measuring flux to calculate the apparent magnitude of celestial objects within astronomical photometry. Each CCD image must be digitally cleaned and calibrated prior to its use. As data archives increase in size to Petabytes, the data processing challenge requires image processing techniques to continue to exceed the rate of data capture.
This paper describes NIMBUS, a rapidly scalable, failure resilient distributed network architecture capable of processing CCD image data at a rate of hundreds of Terabytes per day. NIMBUS is implemented using a decentralized web queue to control the compression of data, the uploading of data to distributed web servers, and the creation of web messages to identify the location of the processed data. This paper demonstrates the horizontal scalability of NIMBUS which has demonstrated a processing rate of 192 Terabytes per day with clear indications that higher processing rates are possible
DOI
https://doi.org/10.21427/qe5y-zy88
Recommended Citation
Doyle, P. (2019). High-Speed Distributed Data Process of Photometric Astronomical Data. Technological University Dublin. DOI: 10.21427/QE5Y-ZY88
Publication Details
Proceedings of the 2019 International Conference on Global Entrepreneurial Talent
Management & Social Collaboration. Ko, I. Gwangju: Chonnam National University, p. 22-32, South Korea
Original language English Title of host publication Proceedings of the 2019 International Conference on Global Entrepreneurial Talent Management & Social Collaboration Editors Ilsang Ko Place of Publication Gwangju Publisher Chonnam National University Pages 22-32 Number of pages 10 Publication status Published - 24 Apr 2019 Event 2019 International Conference on Global Entrepreneurial Talent Management & Social Collaboration - Chonnam National University, Gwangju, Korea, Republic of
Duration: 23 Apr 2019 → 27 Apr 2019
Conference
Conference 2019 International Conference on Global Entrepreneurial Talent Management & Social Collaboration Abbreviated title ICGETMSC 2019 Country/Territory Korea, Republic of City Gwangju Period 23/04/19 → 27/04/19