1, shell concise notes


1 shell introduction

  • What is a shell

    shell is a program written in c language. It is a bridge between users and linux kernel. It is not only a command statement, but also an interpretative programming language

    kernel <----> shell <----> user

    • kernel: serves software, receives user or software instructions, drives hardware and completes work
    • shell: command interpreter
    • User: user interface, connecting users
  • Functions of shell

    • Command line interpretation function
    • Start program
    • Input / output redirection
    • Pipe connection
    • File name replacement (echo / *)
    • Variable maintenance
    • environmental control
    • shell programming

2 shell syntax

shell script is to write all commands that complete a task into a text file from top to bottom according to the order of execution, and then give execution permission

  • How to write a shell script

    • Naming of shell scripts

      The name should be meaningful. It is recommended to use it SH at the end, and see the name and meaning, such as check_memory.sh

    • shell script format

      • The script running environment must be specified at the beginning of the shell script to #! This special symbol combination

      ​ #!/ bin/bash # specifies that the script is run and parsed by / bin/bash

      • Comments in the shell use # numbers

        #Define script execution environment
        #In the shell script, it is better to add the script description field
        #! /usr/bin/bash
        #Author: xxx
        #Created Time: 2020/09/06 21:52
        #Script Description: first shell study script
        #Script composition
        # Interpretation environment #!/usr/bin/bash
        # Note description
        # Execute code
  • shell script running

    • The script needs execution permission to run. After giving execution permission to a file, the script can run

      chmod 700 filename

    • If you don't want to give the script execution permission, you can use the bash command to run the script without execution permission

      bash filename

  • Special symbols in shell

    Special symbols describe
    ~ Home directory, # cd ~ represents the user's home directory
    ! Execute historical command,!! Execute the previous command
    $ Get content character from variable
    +,-,*,/,% Corresponding mathematical operations, addition, subtraction, multiplication, division and remainder
    & Background execution
    * Asterisks are wildcards in the shell that match all
    A question mark is a wildcard in the shell that matches a character other than carriage return
    ; Semicolons can execute multiple commands on one line in the shell, separated by semicolons
    | The output of the previous command of the pipeline character is used as the input of the next command cat filename | gerp "abc"
    \ Escape character
    `` Backquote, execute the command echo "today is date +%F" in the command
    ' Single quotation marks. Strings in scripts should be enclosed in single quotation marks, but unlike double quotation marks, single quotation marks do not interpret variables
    " Double quotation marks. Strings appearing in the script can be enclosed in double quotation marks
  • The Conduit

    | Pipe symbol in shell Most used,Many combined commands need to be output through combined commands
      The pipe symbol is actually the output of the previous command as the input of the next command
  • redirect

    >   Redirect input overwrite original data
    >>  Redirect additional input,Add at the end of the original data
    <   redirect output         #wc -l < /etc/passwd
    >>  Redirect additional output      #fdisk /dev/sdb <<EOF ...... EOF
  • Mathematical operation in shell

    expr command:Can only do integer operations,The format is rather old-fashioned,Pay attention to the spaces
    # expr 1 + 1
    # expr 1 \* 2 -- note that * should be escaped when it appears, otherwise it is considered as a wildcard
    #echo 7 + 1
    #echo $?   --$? It can judge whether the execution of the previous command is successful. If it is successful, it will return 0, otherwise it will return other numbers
    #expr 7 + 1.1 &>/dev/null ;  echo $?  --/ Dev / null stands for recycle bin
    --Compound command
    #Echo "current memory usage rate: ` echo" scale = 2; 141 * 100 / 7988 "|bc `%" usage rate "
    --Double parenthesis operation,stay shell in(()),It can also be used to do mathematical operations
    #echo $((100+3))
  • Script exit

    exit NUM Exit script,Free system resources,NUM Represents an integer,Represents the return value.

3 shell formatted output

A program needs 0 or more inputs and one or more outputs

  • Introduction to echo command

    • **Function: * * output content to default display device

      echo The function of the command is to display a text on the display,Generally, it plays a role of reminder.
    • **Syntax: * * echo [-ne] [string]

      --Supplementary notes
      echo The input string is sent to standard output.The output strings are separated by white space characters,And add the line number at the end.
      --Command options
      -n Don't wrap at the end
      -e If the following characters appear in the string,Special treatment,It will not be output as general text:
         Escape character
             \a Sound a warning tone
             \b Delete previous character
             \c No line break at the end
             \f Line breaks, but the cursor remains in its original position
             \n Wrap a line and move the cursor to the beginning of the line
             \r Move the cursor to the beginning of the line,But no line breaks
             \t insert tab
             \v And\f identical
             \nnn insert nnn(octal number system)Represented by ASCII character
             -help Show help
             -version display version information
      for time in `seq 9 -1 0`;do
      	echo -n -e "\b$time"
      	sleep 1

4 shell basic input

  • read command

    Keyboard input is accepted by default, and the carriage return indicates the end of input

    --read Command options
    -p  Print information
    -t  Limited time
    -s  Do not echo
    -n  Enter the number of characters
    echo -n -e "Login: "
    read login
    echo -n -e "Password: "
    read -t5 -s -n6 password
    echo "account: $login password: $password"

5 variables

  • Variable introduction

    Computer unit:
    • **Definition: * * variable is the most commonly used way to access data temporarily in memory in programming
  • Variable classification

    1. Local variable: user private variable, which can only be used by local users and saved in the home directory bash_profile,. In bashrc file
    2. Global variables: all users can use them and save them in / etc/profile and / etc/bashrc files
    3. User defined variables: user defined, such as variables in scripts
  • Define variables

    • Variable format: variable name = value

      In shell programming, there must be no space between the variable name and the equal sign

      Variable name naming rules:
      	1.Only English letters can be used for naming,Numbers and underscores,The first letter cannot begin with a number
      	2.There must be no spaces in the middle,You can use underscores"_"
      	3.Punctuation cannot be used
      	4.out of commission bash Keywords in(available help Command view reserved keywords)
      be careful:
      	1.Strings should be enclosed in single or double quotation marks
  • Read variable contents

    Read variable contents:$

    Read method: $variable name

    echo $A
  • Cancel variable unset

    #unset A
  • Define global variable export

    #export name='test_name'
    If you want local or global variables to be used permanently,You can write the variables to be set into the variable file
  • Define permanent variables

    Local variable: user private variable. Only this user can use

    Global variables: all users can use

6 array

  • Array introduction

  • Basic array

    Array allows users to assign multiple values at a time. When you need to read data, you can easily read it through index call

    • Array syntax

      Array name=(Element 1 element 2 element 3 ...)
    • Array readout

      ${Array name[Indexes]}
      By default, the index is the queued number of elements in the array,By default, the first one starts from 0
    • Array assignment

      • Method 1: assign one value at a time

      • Method 2: assign multiple values at one time

        #array_0=(tom jack alice)
        #array_0=(`cat /etc/passwd `) -- you want to assign each line in the file to the array as an element_ 0
        #array_0=(`ls /usr/bin`)
    • View array

      #declare -a -- view the array defined by the current bash
    • Accessing array elements

      #echo ${array_0[0]} -- access the first element in the array
      #Echo ${array_0 [@]} -- access all elements in the array, equivalent to echo ${array_0 [*]}
      #echo ${#array_0 [@]} -- count the number of array elements
      #echo ${!array_0 [@]} -- get the index of array elements
      #echo ${array_0[@]:1} -- start with array subscript 1
      #echo ${array_0[@]:1:2} -- access two elements starting with the array subscript 1
    • Traversal array

      Method 1: the default array is traversed by the number of array elements

      Method 2: for associative arrays, you can traverse through the index of array elements

  • Associative array

    Associative array allows users to customize the index of the array, which is more convenient and efficient

    • Associative array definition

      Declare associative array variables
      #declare -A ass_array_1
      #declare -A ass_array_2
    • Associative array assignment

      Method 1:Assign one value at a time
      	Array name[Indexes]=Variable name
       Method 2:Assign multiple values at a time
      	#ass_array_2=([index_name_1]=tom [index_name_2]=jack ...)
    • View array

      # declare -A

7 shell process control if judgment statement

  • Operation in shell

    • Mathematical comparison operation

      Operator interpretation:
      	-eq   be equal to
      	-ge   Greater than or equal to
      	-gt   greater than
      	-le   Less than or equal to
      	-lt   less than
      	-ne   Not equal to
      be careful:
      	1.Only integers can be compared
      	2.Encountered a floating point type that needs to be compared,You can enlarge it 10 times 
    • String comparison operation

      Operator interpretation,Pay attention to the string and don't forget to use quotation marks
      	==	be equal to
      	!=	Not equal to
      	-n	Check whether the length of the string is greater than 0
      	-z	Check whether the length of the string is 0
    • Compare with documents

      test	-d	Check that the file exists and is a directory
      		-e	Check whether the file exists
      		-f	Check whether the file exists and is a file
      		-r	Check that the file exists and is readable
      		-s  Check whether the file exists and is not empty
      		-w	Check that the file exists and is writable
      		-x	Check that the file exists and is executable
      		-O  Check whether the file exists and is owned by the current user
      		-G	Check whether the file exists and the default group is the current user group
      		file1 -nt file2	inspect file1 Whether than file2 new
      		file1 -ot file2 inspect file1 Whether than file2 used
    • Logical operation

      	Logic and operation	&&
      	Logical or operation	||
      	Logical non operation	|
      Precautions for logical operation:
      	Both logical and or operations require two or more conditions,Logical non operation can only have one condition
    • Assignment operation

      	=		Assignment Operators 
  • if syntax

    • Syntax 1: Single if statement

      Scope of application: only one step judgment is needed, and the condition returns true or false

      Statement format:

      if [condition] 	#The condition value is true or false
    • Syntax 2: If then else

      Scope of application: two-step judgment: what if the condition is true and what if the condition is false

      if [condition]
      if [ $1 -eq $2]
      		echo "$1 = $2"
      	if [ $1 -gt $2]
      			echo "$1 > $2"
      			echo "$1 < $2"
    • Syntax 3: If then elif statement

      Scope of application: more than two or more judgment results, that is, more than one judgment condition

      if [condition 1]
      elif [condition 2]
      if [ $1 -eq $2]
      		echo "$1 = $2"
      elif [ $1 -gt $2 ]
      		echo "$1 > $2"
      		echo "$1 < $2"
  • if advanced application

    1. The condition symbol uses double parentheses, which can implant a mathematical expression in the condition

      if (( 100%3+1>10 ));then
      	echo "yes"
      	echo "no"
    2. Using double brackets, you can use wildcards in conditions

      for i in r1 rr2 cc rr3
      		if [[ $i == r* ]];then
      			echo $i

8 shell process control for loop statement

  • for loop introduction

  • for syntax

    • for syntax i

      for var in value1 value2 ......
      for i in `seq 1 9`
      		echo $i
    • for syntax II

      C Stylized for command
      for ((variable;condition;Self increasing and decreasing operation))
      		Code block
      for ((i=1;i<10;i++))
      		echo $i

9 loop control statement

  • sleep N script sleeps for N seconds after executing this step

  • continue skips a loop in a loop

    for ((i=1;i<10;i++))
    		if [ $i -eq 5 ];then
    	echo $i
  • **break jumps out of the loop and continues to execute subsequent code**

10 shell process control while loop statement

  • Introduction to while loop

    If you know the number of cycles, you can use for. If you don't know how many times the code needs to cycle, you can use while

  • while loop syntax

    while [ condition ]	#Note that if the condition is true, while will cycle, and if the condition is false, while will stop the cycle
  • while loop actual combat

    read -p "login: " account
    while [ $account != 'root' ]
    		read -p "login: " account

11 until statement

  • until introduction

    In contrast to while, until starts execution when the condition is false and stops execution when the condition is true

  • until syntax

    until [ condition ]  #Note that if the condition is false, until will cycle, and if the condition is true, until will stop the cycle
    	commands Code block

12 case multi conditional branch statement

  • case introduction

    Implement different plans according to different conditions

  • case syntax

    case variable in
     Condition 1)
    	Execute code block 1
    Condition 2)
    	Execute code block 2
    be careful:After each code block is executed;;The end represents the end,case The ending should be written upside down esac Come to an end

13 shell special variables

  • Special parameters

    $0	Script name
    $*	Represents all parameters,The interval is IFS The first character of the intrinsic parameter
    $@	And*Asterisks are similar,The difference is that there is no reference IFS
    $#	Represents the number of parameters
    $	The return value of the last instruction executed
    $- 	Recently implemented foreground pipeline Option parameters for
    $$	Per se Process ID
    $:	Execute previous background PID
    $_	Displays the last command executed
    $N	shell What are the external parameters of

14 shell function

  • Function introduction

    Modularize code,One module implements one function.
    Advantages of function:
    	1.Code modularization,Convenient call,Save memory
    	2.Code modularization,Less code,Troubleshooting is simple
    	3.Code modularization,You can change the execution order of the code
  • Function syntax

    Grammar one:
    Function name () {
    	Code block
    	return N
    Grammar II:
    function Function name {
    	Code block
    	return N
  • Function application

    #Define a function
    print_str () {
    	echo "welcome to my home"
    function hello {
    	echo "hello world"
    	1.print and hello Is the name of the function,Naming rules for function names and reference variables
    	2.After defining the function,If you want to call this function,Just call through the function name
    #Variable area
    #Function library provided by the system
    #if [ -f /etc/init.d/functions ];then
    #    . /etc/init.d/functions
    #    echo "not found file /etc/init.d/functions"
    #    exit
    start() {
        # If nginx is not started, start it directly. Otherwise, an error is reported and it has been started
        if [ -f $pid_file ];then
            # Get process id
            nginx_process_id=`cat $pid_file`
            # Query the status of related processes through process id and count the quantity
            nginx_process_num=`ps aux | grep $nginx_process_id | grep -v "grep" | wc -l`
        #If the file exists and the number of processes is equal to 1, it means nginx is running
        if [ -f $pid_file ]&&[ $nginx_process_num -ge 1 ];then
            echo " nginx running... "
            #If the file exists and the number of processes is less than 1, nginx is not running
            if [ -f $pid_file ]&&[ $nginx_process_num -lt 1 ];then
                #Delete process
                rm -f $pid_file
                #Start process
                echo " nginx start `daemon $nginxd` "
                #Another way to start
                #action "nginx start" $nginxd
            echo " nginx start `daemon $nginxd` "
    stop() {
        if [ -f $pid_file ]&&[ $nginx_process_num -ge 1];then
            action "nginx stop" killall -s QUIT $proc
            rm -f $pid_file
            action "nginx stop" killall -s QUIT $proc 2>/dev/null
    restart () {
        sleep 1
    reload () {
        if [ -f $pid_file ]&&[ $nginx_process_num -ge 1 ];then
            action "nginx reload" killall -s HUP $proc
            action "nginx reload" killall -s HUP $proc 2>/dev/null
    status () {
        if [ -f $pid_file ]&&[ $nginx_process_num -ge 1 ];then
            echo "nginx running..."
            echo "nginx stop"
    case $1 in
    start) start;;
    stop) stop;;
    restart) restart;;
    reload) reload;;
    status) status;;
    *) ehco "USAGE: $0 start|stop|restart|reload|status";;

15 special characters of regular expressions

  • Introduction to regular expressions

    ​ A regular expression is a text pattern matching that includes ordinary characters (for example, letters between a and z) and special characters (called metacharacters) It is a pattern of string matching, which can be used to check whether a string contains a substring, replace the matching substring, or take the substring of a condition from a string

    ​ shell supports regular expressions, but not all commands support regular expressions. Among common commands, only grep, sed and awk commands support regular expressions

  • Locator

    How to use locator: anchor the beginning and end at the same time to make accurate matching; Single anchor the beginning and end to make fuzzy matching

    Locator explain
    ^ Anchor start ^ a starts with a, which is one character by default
    $ Anchor end a $ends with a, which is one character by default
    egrep Equivalent to grep -E
    #egrep "^ac$" file -- exactly match the contents of the file
    #egrep "^a" file -- matches the contents of the file beginning with a
  • Matcher: match string

    Match character explain
    . Matches any character except carriage return
    ( ) String grouping
    [ ] Defines a character class that matches a character in parentheses
    [ ^ ] Indicates that the character in the character class in the negative bracket is reversed
    \ Escape character
    | (a|b) match a or B
  • Qualifier: a qualified description of the preceding character or string

    qualifier explain
    * An asterisk after a character indicates that the character does not appear or appears more than once
    ? Similar to the asterisk, it indicates that the character appears once or does not appear
    + Similar to the asterisk, it indicates that the character before it appears one or more times, but it must appear once
    {n, m} Appears after a character, indicating that the character has at least n times and at most m times
    {m} It happened m times
  • POSIX special characters

    Special characters explain
    [:alnum:] Match any alphabetic character 0-9 a-z A-Z
    [:alpha:] Match any letter, uppercase or lowercase
    [:digit:] Number 0-9
    [:graph:] Non empty character (non space control character)
    [:lower:] Lowercase characters a-z
    [:upper:] Uppercase characters A-Z
    [:cntrl:] Control character
    [:print:] Non empty characters (including spaces)
    [:punct:] punctuation
    [:blank:] Spaces and TAB characters
    [:xdigit:] Hexadecimal digit
    [:space:] All white space characters (new lines, spaces, tabs)
    --be careful
    [[]] The meaning of double brackets:The first bracket is a match[],Match any character in brackets,the second[]Yes format such as[:digit:]
    #egrep "^a[[:alnum:]]c$" file

16 shell operations on files

  • brief introduction

    In the process of shell script writing, the related operations of files are often used, such as adding content, modifying content, deleting part of content, viewing part of content, etc. the above functions can be realized through perl, sed and other commands

  • sed command

    • sed introduction

      sed is an external command provided in Linux. It is a line (stream) editor. It is a non interactive operation to add, delete, modify and query the file content. Users can only enter the editing command on the command line, specify the file name, and then view the output on the screen

      sed And text editor:
      	text editor:The editing object is a file
      	Line editor:The editing object is a line in the file
    • sed data processing principle

      				   sed Principle of data processing
       A line in text————————————>Cache————————————>Computer screen
      		    Data is processed in the cache,Then output to the screen by default
    • sed command

      **Syntax: * * sed [options] '{command}[flags]' [filename]

      --Command options
      -e script Add the command specified in the script to the command executed when processing input,There must be multiple operations in a row
      -f script Adds the command specified in the file to the command executed when processing input
      -n		  Suppress automatic output
      -i		  Edit file content
      -i.bak	  Create at the same time when modifying.bak Backup files
      -r		  Use extended regular expressions
      !		  Reverse, (Following mode conditions and shell Different)
      --sed Common internal commands
      a	Add after match
      i	Add before match
      p	Print
      d	delete
      s	Find replace
      c	change
      y	transformation  N D P
       number		  Represents the pattern of new text replacement
      g:		 	Represents replacing all instances of existing text with new text
      p:		 	Indicates that the original content is printed
      w filename: Write the replacement result to the file
      #sed 'a\hello world' data
      #sed '3a\hello world' data
      #sed '2,4a\hello world' data
      #sed '/3 the/a\hello world' data -- the first two /, indicating the matching pattern

17 shell_awk basic syntax

  • awk introduction

    awk is a language that can generate and process report data

    awk believes that each line in the file is a record, the separator between records is a newline character, each column is a field, and the separator between fields is one or more spaces or tab by default

    The working mode of awk is to read data, treat each row of data as a record, divide each record into several fields with field separator, and then output the value of each field

    --awk grammar
    awk [options] [BEGIN]{program}[END][file]
    --Common command options
    -F fs      Specifies that the file separator that depicts the data field in a row defaults to a space
    -f file    Specifies the file name of the reader
    -v var=value  definition awk Variables and default values used in the program
    --be careful
    awk The program script is defined by the left parenthesis and the right brace.Script commands must be placed between two curly braces.
    --awk Program running priority is
    	1.BEGIN:   Execute before starting processing the data stream,Optional
    	2.program: How to handle data flow,Required
    	3.END:     Execute after processing the data stream,Optional
  • Basic usage of awk

    1. awk extraction of fields (columns)

      Field extraction:Extract a column of data in a text and print it out
      --Field related built-in variables
      $0   Represents the entire line of text
      $1   Represents the first data field in a text line
      $2   Represents the second data field in a text line
      $N   Represents the second line in a text line N Data fields
      $NF  Represents the last data field in a text line
      #awk '{print $0}' test
    2. Detailed explanation of command options

      -F   Specifies the separator between fields
      When the output data stream field format is not awk Default field format,We can use-F Command option to redefine the data flow field separator
      #awk -F ":" '{print $1,$3,$NF}' passwd 
    3. awk extraction of records (rows)

      Record extraction: Extract a line in a text and print it out
      There are two methods to extract records: 
      	a.Pass line number
      	b.By regular matching
      Record related built-in variables
      NR:Specify line number
      #awk 'NR==3{print $0}' test
    4. awk string extraction

      The point where records and fields meet is a string
      --Print test The sixth field in the third row
      #awk 'NR==3{print $6}' test
    5. Priority of awk program

      BEGIN Is the highest priority block of code,It's execution PROGRAM Previously executed,No data source is required,Because it does not involve any data processing,Nor does it depend on PROGRAM Code block;
      PROGRAM What does data flow do,Is a required code block,It is also the default code block,Therefore, the data source must be provided during execution;
      END It is the operation after processing the data stream,If necessary END Code block,Must PROGRAM Support of,Single cannot be executed
      #awk 'BEGIN{print "hello world"}{print $0}END{print "bye world"}' test
  • awk advanced usage

    awk is a language. In addition to defining variables, it can also define arrays for operation and process control

    • awk definition array

      Array definition method: array name [index] = value

      - Define array array,There are two elements,100, respectively,200,Print array elements
      # awk 'BEGIN{array[0]=100;array[1]=200};print array[0],array[1]'
    • awk operation

      1.Assignment operation =
      2.Comparison operation >,>=,==,<,<=,!=
      3.Mathematical operation +,-,*,/,%,**,++,--
      4.Logical matching &&,||
      5.Matching operation ~,!~
    • awk environment variable

      variable describe
      FIELDWIDTHS A list of numbers separated by spaces that define the exact width of each data field
      FS Input field separator
      OFS Output field separator
      RS Enter record separator
      ORS Output record separator
      # awk 'BEGIN{FS=":"}$1 !~ "ro"{print $0}' passwd
    • Process control

      • if judgment statement

        # awk '{
        if ($1>5)
        print $0
        print $1*2 
        }' file_name 
        -When this statement,When written in one line,Need to add;As separator 
        # awk '{if ($1<5)print $1*2;else print $1/2}' file_name
      • for loop statement

        # awk '{sum=0;for (i=1;i<5;i++){sum+=$i}print sum}' file_name
        # awk '{
        for (i=0;i<4;i++){
        print sum
        }' file_name
      • while loop statement

        awk '{
        	while (i<4){
        	print sum
        }' file_name
      • do...while statement

        awk '{
        	do {
        	}while (sum<150)
        	print sum
        }' file_name
      • Cycle control

        break Jump out of loop,Continue execution of subsequent statements
  • awk tips

    - Print test Number of lines of text
    # awk 'END{print NR}' file_name
    - Print file_name Last line of text
    # awk 'END{print $0}' file_name
    - Print test Number of text columns
    # awk 'END{print NF}' test

Tags: shell

Posted by robogenus on Fri, 13 May 2022 11:42:26 +0300