Некоторые программные среды аналитики больших данных

Abstract

Изложены концептуальные вопросы построения сред обработки данных – кластерных систем на программной платформе Hadoop. Описана инфраструктура HadoopMapReduce для организации параллельных распределенных вычислений над данны ми и показано эволюционное преобразование платформы Hadoop применительно к интерактивным и потоковым динамиче ским нагрузкам.

Викладено концептуальні питання побудови середовищ обробки даних – кластерных систем на програмній платформі Hadoop. Описано інфраструктуру HadoopMapReduce для організації паралельних розподілених обчислень над даними і показано ево люційне перетворення платформи Hadoop стосовно інтерактивних і потокових динамічних навантажень.

improvement of the traditional processing technology and to create the advanced analytics environments. The conceptual issues of data media construction, in particular, on the Hadoop cluster system software platform is presented. The HadoopMapReduce infrastructure is described for the parallel distributed computing on the data and the evolutionary transformation of Hadoop platform using the infrastructure and streaming dynamic loads, as well as HadoopMapReduce infrastructure constraints. It is shown that an introduction of YARN (Yet Another Resource Negotiator) on the computing Hadoop platform allows to perform the different workloads in a linearly scalable cluster Hadoop YARN (Hadoop 2.0), achieving calculations of the high efficiency. Frameworks, Spark, Tez and Storm use the possibility of YARN . The components that make a total Hadoop 2.0 de facto the standard technology for working with Big Data are analyzed. These are the constructions Hive for design-oriented interactive queries to SQL-like language HQL (Hive query language) and working with large data storage; Pig – a high-level procedure language Pig Latin, designed for accessing the semidistributed lennym datasets; HBase – distributed non-relational DBMS, working effectively with the individual records in real time; Apache Accumulo – oriented on a high level of safety distributed, scalable data repository with the strict requirements of the information and personal data protection. The problems of large data efficiently various types download of Hadoop ecosystem using Hive and Pig. A comparative analysis of ELT (extract-load-transform) and ETL