Flink Table API & SQL Programming Guide

Apache Flink provides two top-level relational APIs: Table API and SQL. Flink realizes batch flow unification through Table API & SQL. Among them, Table API is a language integrated query API for Scala and Java, which allows the query of relational operators (such as select, where and join) to be combined in a very intuitive way. Flink SQL ...

Posted by rtconner on Tue, 24 May 2022 19:33:05 +0300

Correctly use Flink broadcast stream and record the failure of Flink to make checkpoints

Recently, when I was working on a project, I was involved in such a scenario that a relatively small table that will not be changed often should be used as a dimension table to match with the real-time flow. This table is a table in MySQL. My first reaction is to read this table for broadcasting. Inelegant use of broadcast streams The brief c ...

Posted by omanush on Sat, 21 May 2022 23:43:47 +0300

Viewing successfully submitted offset indicator in Flink WebUI is negative

1. Problems After submitting a flink task, I went to WebUi to check the offset indicator, and found that the value turned out to be a negative number (as shown below): The definition given on the official website of this indicator is: For each partition, the offset of the last successful submission to Kafka; so this value is not normal anyway. ...

Posted by jefrat72 on Wed, 18 May 2022 15:44:05 +0300

02. Implementation principle and source code analysis of Flink Client

Personal official account: Implementation principle of Flink Client Before we interpret the source code of Flink Client, we must first understand its implementation principle. Main functions of Flink Client We can see that the FLink Client mainly has three tasks. The first is to run the Application, the second is to operate and ma ...

Posted by statrat on Tue, 17 May 2022 00:17:30 +0300

Table API and Flink SQL - Chapter 2 API call

2.1 basic program structure The program structure of Table API and SQL is similar to that of streaming processing; It can also be roughly considered that there are several steps: first create the execution environment, and then define source, transform and sink. The specific operation process is as follows: val tableEnv = ... // Execution e ...

Posted by Sentosa on Mon, 16 May 2022 13:15:06 +0300

Flink state programming

concept In Flink architecture, stateful computing is one of the very important features of Flink Flink benefits: Support high throughput, low latency and high performance Support event time Event_time concept Support stateful computing Stateful calculation refers to: In the process of program calculation, the intermediate results generated by ...

Posted by beeman000 on Mon, 16 May 2022 09:39:24 +0300

flink checkpoint source code analysis II

Overview: the last article generally received the overall concept and process of source code analysis of flycheckpoint. Combined with the code, it introduces the initiation and task execution process of checkpoint Detailed reference: https://blog.csdn.net/weixin_40809627/article/details/108537480 This article will follow the previous article a ...

Posted by freddyw on Sun, 15 May 2022 15:26:01 +0300

Alink ramble: cluster evaluation of source code analysis

Alink ramble (22): cluster evaluation of source code analysis 0x00 summary Alink is a new generation of machine learning algorithm platform developed by Alibaba based on Flink, a real-time computing engine. It is the first machine learning platform in the industry to support batch algorithm and streaming algorithm at the same time. This a ...

Posted by chenci on Sat, 14 May 2022 16:31:35 +0300

Flink heartbeat service mechanism

Heartbeat mechanism is a mechanism used to detect whether the client or server is alive. By regularly sending requests to each other, there are two common heartbeat detection methods: Socket socket SO_KEEPALIVE has its own heartbeat mechanism. It sends heartbeat packets to the other party regularly, and the other party will reply automaticall ...

Posted by ozone1 on Fri, 13 May 2022 07:01:12 +0300

Flink learning from 0 to 1 - Chapter 5 Flink stream processing API

1. Flink stream processing API 1.1 Environment 1.1.1 getExecutionEnvironment Create an execution environment that represents the context of the current executing program. If the program is called independently, this method returns to the local execution environment; If the program is called from the command-line client to submit to the clu ...

Posted by abhishekphp6 on Wed, 11 May 2022 16:00:44 +0300