spark streaming (real-time stream word frequency statistics)

First in idea Import maven dependency package <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>2.0.0</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kaf ...

Posted by ym_chaitu on Sun, 01 May 2022 03:48:44 +0300

Hive chapter of big data development 5-Hive data query language

Remark: Hive version 2.1.1 1. Overview of Hive SELECT (Data Query Language) The select statement is the most frequently used statement in Hive, and it is also the statement with the most complex syntax. Many syntaxes of select statements are similar to traditional relational databases, which also facilitates the transition from tradition ...

Posted by sarika on Sat, 30 Apr 2022 08:55:28 +0300

Record local installation oushudb - install zk and hdfs

Because the company is busy recently, I forgot to update it. Then I talked about it in the last article, The last article said This article describes some preparations for installing oushudb. This article mainly talks about the installation of corresponding components before installing oushudb. 1, Zookeper installation 1. Create a zkhostfile c ...

Posted by Chips on Sat, 30 Apr 2022 00:19:07 +0300

Develop a big data SQL Engine that does not need to be rewritten as Hive QL

This article is shared from Huawei cloud community "developing big data SQL Engine from scratch", author: JavaEdge. Learn the core principles of big data technology, master some efficient ways of thinking and thinking, and build your own technical knowledge system. We can deduce and even realize various principles without understand ...

Posted by simonyuriko on Fri, 29 Apr 2022 17:24:06 +0300

An article takes you to understand SVG conversion knowledge

Svg transforms shapes created in SVG images. For example, move, scale, and rotate shapes. This is a convenient way to display vertical or diagonal text. 1, Simple example of conversion Example: <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <rect x="50" y="50" height="110" width="110" ...

Posted by veroaero on Fri, 29 Apr 2022 04:10:26 +0300

CDH6.3.2 installing elasticsearch7 9.0 (super detailed, with my own bag)

Xiaobai installed some custom services on cdh for the first time, and stepped on a lot of holes in the process. Some posts on the Internet are the same. I feel that everyone is copying and pasting and then publishing directly. I have summarized the pitfalls and solutions encountered in the process. I hope this blog can help you. At the same ...

Posted by Unholy Prayer on Thu, 28 Apr 2022 10:20:45 +0300

Building ELK log collection system in simple terms

Take a look at the catalog first background Imagine such a scenario: Nginx loads two Tomcat s, so it is very troublesome to check the log. Each time you check the log, you have to log in to two servers and search one by one. Two are OK. What if five? What about 10? It's hard to view logs, so we need a log collection system to centrally mana ...

Posted by pixelfish on Wed, 27 Apr 2022 09:44:33 +0300

Installation and application of offline digital warehouse maxwell

maxwell synchronize incremental data (1) Overview Maxwell will monitor the data change operations of Mysql database in real time (including insert, update and delete), and send the changed data to Kafka, Kinesi and other stream data processing platforms in JSON format. Maxwell's working principle is to read the binary log of MySQL database i ...

Posted by mass on Tue, 26 Apr 2022 05:27:50 +0300

Apache NiFi custom Processor

demand When Apache NiFi is used to distribute the enterprise master data to the downstream business system in real time, the downstream systems include MySQL, PostgreSQL, Oracle and other business systems. Among them, NiFi does not directly support Oracle Upsert semantics, resulting in a large number of updates to the master data of products, ...

Posted by betportal on Tue, 26 Apr 2022 01:25:01 +0300

Hadoop3.1.4 compiling on Linux platform

HDFS core source code analysis catalogue Hadoop source code compilationHDFS source code structure analysisHDFS core source code analysis Learning objectives Master the scene of compiling source codeMaster Hadoop source code and compile it on Linux platformUnderstand the compilation of Hadoop source code on Windows platformUnderstand the sou ...

Posted by cesar_ser on Tue, 26 Apr 2022 00:32:06 +0300