Инд. авторы:	Потапов В.П., Попов С.Е., Костылев М.А.
Заглавие:	Метод обработки радарных данных на базе системы массово-параллельного исполнения заданий Apache Spark
Библ. ссылка:	Потапов В.П., Попов С.Е., Костылев М.А. Метод обработки радарных данных на базе системы массово-параллельного исполнения заданий Apache Spark // Вычислительные технологии. - 2017. - Т.22. - № S1. - С.60-74. - ISSN 1560-7534. - EISSN 2313-691X.
Внешние системы:	РИНЦ: 29221476;
Реферат:	rus: Представлен современный подход к созданию распределенного программного комплекса для потоковой пред- и постобработки радарных снимков. Отличительной особенностью системы является применение массово-параллельной технологии Apache Spark, что позволило использовать аппаратное обеспечение общего назначения, а также возможность исполнения существующих алгоритмов, не предназначенных для распределенной обротки на множестве узлов без изменения реализации алгоритма. В работе приводится сравнение технологий распределенных вычислений, представлено общее описание кластера и механизма выполнения задач пре- и постпроцессинга радарных данных, также приведены особенности имплементации конкретных задач в рамках предложенного подхода. Представлены результаты тестирования разработанных алгоритмов на демонстрационном кластере. eng: The paper presents a novel approach to development of distributed software for streaming pre- and post-processing of radar images. A distinctive feature of the system is employment of Apache Spark mass-parallel processing, which allowed operations with general-purpose hardware as well as the ability to execute existing algorithms that are not designed for distributed processing on multiple nodes without changing the implementation of the algorithm. The paper compares distributed computing technologies and presents a general description of the cluster and the mechanism for performing the tasks of pre- and post-processing of radar data, as well as specifies the implementation of tasks within the framework of the proposed approach. The parallel computations methods based on GPUs, Java-multithreading and the application development based on a massively parallel architecture are used. In particular, in the phase unwrapping algorithm, the construction of the growth ring, the estimating of absolute phase values and the performance of reliability tests performed on the side of the GPU are presented. The number map of the regions which is monitored and updated after each iteration allows monitoring all regions along the optimal unwrapping path simultaneously and tracking the moments of their encounters and intersections. In the massively parallel algorithm execution, we apply a pairing deployment scheme for the input data in separate HDFS directories. As result of reading of the input data, a Resilient Distributed Datasets object is created. It, consists of pairs which include the file name and the binary data stream. The binary sequence is passed to the map-function, which executes the phase unwrapping procedure. The implementation of the map-function checks the node hardware and runs the code on the CPU or GPU. The interferometric image is processed entirely locally at the computational node without dividing into parts. Parallelization is achieved by processing a large number of images on a set of nodes. In conclusion, the test results of cluster execution of the developed algorithms are presented. 7х speedup performance is shown with parallel start of eight computational tasks in the phase unwrapping algorithm. Compared to traditional approaches to radar data processing, in which parallel computations either do not apply, or are used only to improve the performance of calculations, the proposed solution is focused on processing a large number of packet data.
Ключевые слова:	Apache spark; Apache Hadoop; распределенные информационные системы; радарная интерферометрия; алгоритмы обработки; distributed information systems; sar interfometry; processing algorithms;
Издано:	2017
Физ. характеристика:	с.60-74
Цитирование:	1. Елизаветин И.В., Шувалов Р.И., Буш В.А. Принципы и методы радиолокационной съемки для целей формирования цифровой модели местности // Геодезия и картография. 2009. № 1. C. 39-45. 2. Ferretti, A., Monti-Guarnieri A., Prati, C., Rocca, F., Massonnet, D. InSAR Principles: Guidelines for SAR Interferometry Processing and Interpretation. ESA Publications, 2007. TM-19. Available at: http://www.esa.int/esapub/tm/tm19/TM-19_ptA.pdf (accessed 02.08.2016). 3. Zhengxiao Li, James Bethel. Image coregistration in sar interferometry // The Intern. Archives of the Photogrammetry. Remote Sensing and Spatial Inform. Sci. Vol. XXXVII. Pt B1. Beijing, 2008. P. 433-438. 4. Massonnet, D., Feigl, K.L. Radar interferometry and its application to changes in the earth’s surface // Reviews of Geophysics. 1998. Vol. 36(4). P. 441-500. 5. Costantini, M., Farina, A., Zirilli, F. A fast phase unwrapping algorithm for SAR interferometry // IEEE Trans. GARS. 2002. Vol. 37, No. 1. P. 452-460. 6. Mistry, P., Braganza, S., Kaeli, D., Leeser, M. Accelerating phase unwrapping and affine transformations for optical quadrature microscopy using CUDA // Proc. of 2nd Workshop on General Purpose Proc. on Graphics Proc. Units, GPGPU 2009. USA, Washington, DC: ACM, 2009. P. 28-37. 7. Karasev, P.A., Campbell, D.P., Richards, M.A. Obtaining a 35x speedup in 2D phase unwrapping using commodity graphics processors // Radar Conference. IEEE, 2007. P. 574- 578. 8. Верба В.С., Неронский Л.Б., Осипов И.Г., Турук В.Э. Радиолокационные системы землеобзора космического базирования. М.: Радиотехника, 2010. 675 с. 9. Zhenhua Wu, Wenjing Ma, Guoping Long, Yucheng Li High Performance Two-Dimensional Phase Unwrapping on GPUs // Proc. of the 11th ACM Conf. on Computing Frontiers - CF ’14. 2014. New York, USA: ACM, 2014. P. 35:1-35:10. 10. Shi Xin-Liang, Xie Xiao-Chun. GPU acceleration of range alignment based on minimum entropy criterion // Radar Conference 2013, IET Intern. 14-16 April 2013. P. 1-4. 11. Guerriero, A., Anelli, V. W., Pagliara, A., Nutricato, R., Nitti, D.O. High performance GPU implementation of InSAR time-consuming algorithm kernels // Proc. of the 1st Workshop on the State of the art and Challenges of Research Efforts at POLIBA. Bari, Italy: Politecnico di Bari, 2014. 383 p. 12. Zhang, F., Wang, B., Xiang, M. Accelerating InSAR raw data simulation on GPU using CUDA // Geoscience and Remote Sensing Symp. (IGARSS). 2010 IEEE Intern. 25-30 July 2010. Bari, Italy: Politecnico di Bari, 2010. P. 2932-2935. 13. Marinkovic, P.S., Hanssen, R.F., Kampes, B.M. Utilization of Parallelization Algorithms in InSAR/PS-InSAR Processing // Proc. of the 2004 Envisat ERS Symposium (ESA SP-572). 6-10 Sept. 2004. Salzburg, Austria: ESA, 2004. P. 1-7 14. Sheng, G., Qi-Ming, Z., Jian, J., Cun-Ren, L., Qing-xi, T. Parallel processing of InSAR interferogram filtering with CUDA programming // Science of Surveying and Mapping. 2015. No. 1. P. 54-68. 15. Gabriel, E., Fagg, G.E., Bosilca, G., Angskun, Th., Dongarra, J.J., Squyres, J.M., Sahay, V., Kambadur, Pr., Barrett, B., Lumsdaine, A., Castain, R.H., Daniel, D.J., Graham, R.L., Woodall, T.S. Open MPI: Goals, concept, and design of a next generation MPI implementation. Available at: https://www.open-mpi.org/papers/europvmmpi-2004-overview/euro-pvmmpi-2004-overview.pdf (accessed 23.09.2014). 16. Kampes, B., Hanssen, R., Perski, Z. Radar interferometry with public domain tools presentation. Available at: http://doris.tudelft.nl/Literature/kampes_fringe03.pdf (accessed 23.09.2014). 17. Frigo, M., Johnson, S.G. FFTW: An adaptive software architecture for the FFT // ICASSP Conf. Proceedings. 15 May 1998. Seattle, Washington, USA: IEEE, 1998. Vol. 3. P. 1381-1384. 18. Larkin, J. Fast GPU development with CUDA libraries. Available at: https://www.olcf.ornl.gov/wp-content/uploads/2013/02/GPU_libraries-JL.pdf (accessed 23.09.2014). 19. Demmel, J., Dongarra, J. ST-HEC: Reliable and scalable software for linear algebra computations on high end computers. Available at: https://people.eecs.berkeley.edu/ demmel/Sca-LAPACK-Proposal.pdf (accessed 23.09.2014). 20. Феоктистов, А.А,. Захаров, А.И., Гусев, М.А., Денисов, П.В. Исследование возможностей метода малых базовых линий на примере модуля SBaS программного пакета SARScape и данных РСА ASAR/ENVISat и PALSAR/ALOS. Ч. 1. Ключевые моменты метода // Журн. радиоэлектроники. 2015. № 9. С. 1-26. 21. Zinno, I., Mossucca, L., Elefante, S., De Luca, C., Casola, V., Terzo, O., Casu, F., Lanari, R. Cloud computing for earth surface deformation analysis via spaceborne radar imaging: a case study // IEEE Trans. Cloud Computing. 2016. No. 4. P. 104-118. 22. Zinno, I., Mossucca, L., Elefante, S., Mossucca, L., De Luca, C., Manunta, M., Terzo, O., Lanari, R., Casu, F. First assessment of the P-SBAS DInSAR algorithm performances within a cloud computing environment // J. of Selected Topics in Applied Earth Jbservations and Remote Sensing. 2015. Vol. 8. P. 4675-4686. 23. Kannan, P. Beyond Hadoop MapReduce Apache Tez and Apache Spark. San Jose State Univ. Available at: http://www.sjsu.edu/people/robert.chun/courses/CS259Fall2013/s3/F.pdf 24. Nathan P. Real-time analytics with Spark Streaming. Available at: http://viva-lab.ece.virginia.edu/foswiki/pub/InSAR/RitaEducation/InSAR%20Technology% 20Literature%20Search.pdf (accessed 02.08.2016). 25. Nagler, E. Introduction to Oozie. Apache Oozie Documentation. Available at: http://www.cse.buffalo.edu/ bina/cse487/fall2011/Oozie.pdf (accessed 02.08.2016). 26. Jhajj, R. Apache Hadoop Hue Tutorial. Available at: https://examples.javacodegeeks.com/enterprise-java/apache-hadoop/apache-hadoop-huetutorial/ (accessed 02.08.2016).