OpenWrt skips tools compilation

foreword

It is still time-consuming for OpenWrt to compile tools, so in order to shorten the compilation time, we try to use the compiled tools in the SDK provided by OpenWrt.

Analyze the code

When analyzing the OpenWrt source code, the more you analyze, the more code you see, and the more knowledge you need to understand. It makes my head big. I will divide it into several modules to find out how to judge whether tools should be rebuilt. There is too much nonsense in the front. If you don't want to see the nonsense, just fast forward to Hard change the source code

SDK Analysis

In the SDK provided by OpenWrt, this article mainly uses

• build_dir
• staging_dir

These two folders contain the tools and toolchain s that the text needs to use

Top level Makefile analysis

Let me talk about two important variables first, in case someone is unclear

• TOPDIR This is the top-level path, which is the directory of the source code
• CURDIR This is the current path, which is the directory where the current file is located

export PATH:=$(TOPDIR)/staging_dir/host/bin:$(PATH)

A PATH environment variable $(TOPDIR)/staging_dir/host/bin is imported below, which is the tool directory of tools

$(toolchain/stamp-compile): $(tools/stamp-compile) $(if $(CONFIG_BUILDBOT),toolchain_rebuild_check)
$(target/stamp-compile): $(toolchain/stamp-compile) $(tools/stamp-compile) $(BUILD_DIR)/.prepared

At the end of the world, the toolchain is called here, which is related to the generation of the time stamp file of the tools

tools/Makefile analysis

tools_enabled = $(foreach tool,$(sort $(tools-y) $(tools-)),$(if $(filter $(tool),$(tools-y)),y,n))
$(eval $(call stampfile,$(curdir),tools,compile,,_$(subst $(space),,$(tools_enabled)),$(STAGING_DIR_HOST)))
$(eval $(call stampfile,$(curdir),tools,check,$(TMP_DIR)/.build,,$(STAGING_DIR_HOST)))

Here is also called subdir.mk to generate stamp

subdir.mk analysis

ifndef DUMP_TARGET_DB
# Parameters: <subdir> <name> <target> <depends> <config options> <stampfile location>
# 1: subdir
# 2: target
# 3: build type
# 4: build variant
# 5: all variants
define stampfile
  $(1)/stamp-$(3):=$(if $(6),$(6),$(STAGING_DIR))/stamp/.$(2)_$(3)$(5)
  $$($(1)/stamp-$(3)): $(TMP_DIR)/.build $(4)
	@+$(SCRIPT_DIR)/timestamp.pl -n $$($(1)/stamp-$(3)) $(1) $(4) || \
		$(MAKE) $(if $(QUIET),--no-print-directory) $$($(1)lags-$(3)) $(1)/$(3)
	@mkdir -p $$$$(dirname $$($(1)/stamp-$(3)))
	@touch $$($(1)/stamp-$(3))

  $$(if $(call debug,$(1),v),,.SILENT: $$($(1)/stamp-$(3)))

  .PRECIOUS: $$($(1)/stamp-$(3)) # work around a make bug

  $(1)//clean:=$(1)/stamp-$(3)/clean
  $(1)/stamp-$(3)/clean: FORCE
	@rm -f $$($(1)/stamp-$(3))

endef
endif

timestamp.pl is also called here to generate stamp, so it is probably understood that OpenWrt relies on judging stamp to decide whether to recompile tools and toolchain

host-build.mk analysis

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS)))
HOST_STAMP_CONFIGURED:=$(HOST_BUILD_DIR)/.configured
HOST_STAMP_BUILT:=$(HOST_BUILD_DIR)/.built
HOST_BUILD_PREFIX?=$(if $(IS_PACKAGE_BUILD),$(STAGING_DIR_HOSTPKG),$(STAGING_DIR_HOST))
HOST_STAMP_INSTALLED:=$(HOST_BUILD_PREFIX)/stamp/.$(PKG_NAME)_installed

ifneq ($(if $(HOST_QUILT),,$(CONFIG_AUTOREBUILD)),)
  define HostHost/Autoclean
    $(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED))
    $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))
  endef
endif

• HOST_BUILD_DIR --> build_dir/host/

• HOST_BUILD_PREFIX --> staging_dir/host/ , this is defined in toolchain-build.mk

A bold guess, it should be through several hidden files to determine whether the tools are compiled and installed

• .prepared${md5}_${confvar}
• .configured
• .built
• .$(PKG_NAME)_installed

But there are findmd5 and confvar functions in HOST_STAMP_PREPARED, as well as PKG_FILE_DEPENDS and HOST_PREPARED_DEPENDS variables, and you have to figure out how to generate them.

• In rules.mk, found the confvar function definition

confvar=$(shell echo '$(foreach v,$(1),$(v)=$(subst ','\'',$($(v))))' | $(MKHASH) md5)

• HOST_PREPARED_DEPENDS This thing doesn't seem to appear anywhere else.
• PKG_FILE_DEPENDS is defined in package/base-files/Makefile

PKG_FILE_DEPENDS:=$(PLATFORM_DIR)/ $(GENERIC_PLATFORM_DIR)/base-files/

• $(PLATFORM_DIR) and GENERIC_PLATFORM_DIR in include/target.mk

PLATFORM_DIR:=$(TOPDIR)/target/linux/$(BOARD)
GENERIC_PLATFORM_DIR := $(TOPDIR)/target/linux/generic

• Parameter summary

So we found_md5 passed a total of three parameters, namely ${CURDIR}, $(PLATFORM_DIR), $(GENERIC_PLATFORM_DIR)

depends.mk analysis

# define a dependency on a subtree
# parameters:
#	1: directories/files
#	2: directory dependency
#	3: tempfile for file listings
#	4: find options

DEP_FINDPARAMS := -x "*/.svn*" -x ".*" -x "*:*" -x "*\!*" -x "* *" -x "*\\\#*" -x "*/.*_check" -x "*/.*.swp" -x "*/.pkgdir*"

find_md5=find $(wildcard $(1)) -type f $(patsubst -x,-and -not -path,$(DEP_FINDPARAMS) $(2)) -printf "%p%T@\n" | sort | $(MKHASH) md5

define rdep
  .PRECIOUS: $(2)
  .SILENT: $(2)_check

  $(2): $(2)_check
  check-depends: $(2)_check

ifneq ($(wildcard $(2)),)
  $(2)_check::
	$(if $(3), \
		$(call find_md5,$(1),$(4)) > $(3).1; \
		{ [ \! -f "$(3)" ] || diff $(3) $(3).1 >/dev/null; } && \
	) \
	{ \
		[ -f "$(2)_check.1" ] && mv "$(2)_check.1"; \
	    $(TOPDIR)/scripts/timestamp.pl $(DEP_FINDPARAMS) $(4) -n $(2) $(1) && { \
			$(call debug_eval,$(SUBDIR),r,echo "No need to rebuild $(2)";) \
			touch -r "$(2)" "$(2)_check"; \
		} \
	} || { \
		$(call debug_eval,$(SUBDIR),r,echo "Need to rebuild $(2)";) \
		touch "$(2)_check"; \
	}
	$(if $(3), mv $(3).1 $(3))
else
  $(2)_check::
	$(if $(3), rm -f $(3) $(3).1)
	$(call debug_eval,$(SUBDIR),r,echo "Target $(2) not built")
endif

endef

• wildcard: extended wildcard, here are three paths directly passed in, no processing

• patsubst: replace wildcards, replace -x in DEP_FINDPARAMS with -and -not -path

• $(1): This passes in the above three paths

• $(2): empty

• Dismantling command parameters

• The main command is find

• The second command is sort and md5sum (instead of scripts/mkhash.c), calculate the md5 value after find classification

find {path1, path2, path3} -type f -and -not -path "*/.svn*" -and -not -path ".*" -and -not -path "*:*" -and -not -path "*\!*" -and -not -path "* *" -and -not -path "*\\\#*" -and -not -path "*/.*_check" -and -not -path "*/.*.swp" -and -not -path "*/.pkgdir" -printf "%p%T@\n" | sort | md5sum | awk '{print $1}'

$(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED))
$(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))

The above statement corresponds to the following result

$(call rdep,tools/flock,build_dir/host/flock-2.18/.prepared${find_md5}_${confvar}
$(call rdep,build_dir/host/flock-2.18,build_dir/host/flock-2.18/.built)

The analysis is too big, and I can't go on. I can only use the source code to achieve the adaptation effect.

Hard change source code

Since he uses md5 detection, I also change it to md5 detection, just take md5 for comparison according to the change time of tools/xxx folder, and decisively change host-build.mk

Source code:

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS)))

After the change:

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(shell stat -c %Y ${CURDIR} | $(MKHASH) md5)

In this way, the verification method becomes what we want

later

We moved the staging_dir and build_dir in the SDK to the source directory

mkdir -p staging_dir/host/stamp
tools_name="$(ls -F ./tools/ | grep "/$")"
for a in ${tools_name}; do
    if [ "${a}" != 'include/' ]; then
    PKG_NAME=$(cat "./tools/${a}Makefile" | grep '^PKG_NAME' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')
    PKG_VERSION=$(cat "./tools/${a}Makefile" | grep '^PKG_VERSION' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')

    touch staging_dir/host/stamp/.${PKG_NAME}_installed

    mkdir -p build_dir/host/${PKG_NAME}-${PKG_VERSION}

    prepared_md5=$(stat -c %Y "./tools/${a}" | md5sum | cut -d ' ' -f 1)

    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.prepared${prepared_md5}
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.configured
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.built
    
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.version
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.tarball-version

    fi
done

In this way, you can skip the compilation of tools. The easiest way is to clear all the tools in the tools/Makefile.

Enjoy it ~~