OpenWrt skips tools compilation
foreword
It is still time-consuming for OpenWrt to compile tools, so in order to shorten the compilation time, we try to use the compiled tools in the SDK provided by OpenWrt.
Analyze the code
When analyzing the OpenWrt source code, the more you analyze, the more code you see, and the more knowledge you need to understand. It makes my head big. I will divide it into several modules to find out how to judge whether tools should be rebuilt. There is too much nonsense in the front. If you don't want to see the nonsense, just fast forward to Hard change the source code
SDK Analysis
In the SDK provided by OpenWrt, this article mainly uses
- build_dir
- staging_dir
These two folders contain the tools and toolchain s that the text needs to use
Top level Makefile analysis
Let me talk about two important variables first, in case someone is unclear
- TOPDIR This is the top-level path, which is the directory of the source code
- CURDIR This is the current path, which is the directory where the current file is located
export PATH:=$(TOPDIR)/staging_dir/host/bin:$(PATH)
A PATH environment variable $(TOPDIR)/staging_dir/host/bin is imported below, which is the tool directory of tools
$(toolchain/stamp-compile): $(tools/stamp-compile) $(if $(CONFIG_BUILDBOT),toolchain_rebuild_check) $(target/stamp-compile): $(toolchain/stamp-compile) $(tools/stamp-compile) $(BUILD_DIR)/.prepared
At the end of the world, the toolchain is called here, which is related to the generation of the time stamp file of the tools
tools/Makefile analysis
tools_enabled = $(foreach tool,$(sort $(tools-y) $(tools-)),$(if $(filter $(tool),$(tools-y)),y,n)) $(eval $(call stampfile,$(curdir),tools,compile,,_$(subst $(space),,$(tools_enabled)),$(STAGING_DIR_HOST))) $(eval $(call stampfile,$(curdir),tools,check,$(TMP_DIR)/.build,,$(STAGING_DIR_HOST)))
Here is also called subdir.mk to generate stamp
subdir.mk analysis
ifndef DUMP_TARGET_DB # Parameters: <subdir> <name> <target> <depends> <config options> <stampfile location> # 1: subdir # 2: target # 3: build type # 4: build variant # 5: all variants define stampfile $(1)/stamp-$(3):=$(if $(6),$(6),$(STAGING_DIR))/stamp/.$(2)_$(3)$(5) $$($(1)/stamp-$(3)): $(TMP_DIR)/.build $(4) @+$(SCRIPT_DIR)/timestamp.pl -n $$($(1)/stamp-$(3)) $(1) $(4) || \ $(MAKE) $(if $(QUIET),--no-print-directory) $$($(1)lags-$(3)) $(1)/$(3) @mkdir -p $$$$(dirname $$($(1)/stamp-$(3))) @touch $$($(1)/stamp-$(3)) $$(if $(call debug,$(1),v),,.SILENT: $$($(1)/stamp-$(3))) .PRECIOUS: $$($(1)/stamp-$(3)) # work around a make bug $(1)//clean:=$(1)/stamp-$(3)/clean $(1)/stamp-$(3)/clean: FORCE @rm -f $$($(1)/stamp-$(3)) endef endif
timestamp.pl is also called here to generate stamp, so it is probably understood that OpenWrt relies on judging stamp to decide whether to recompile tools and toolchain
host-build.mk analysis
HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS))) HOST_STAMP_CONFIGURED:=$(HOST_BUILD_DIR)/.configured HOST_STAMP_BUILT:=$(HOST_BUILD_DIR)/.built HOST_BUILD_PREFIX?=$(if $(IS_PACKAGE_BUILD),$(STAGING_DIR_HOSTPKG),$(STAGING_DIR_HOST)) HOST_STAMP_INSTALLED:=$(HOST_BUILD_PREFIX)/stamp/.$(PKG_NAME)_installed ifneq ($(if $(HOST_QUILT),,$(CONFIG_AUTOREBUILD)),) define HostHost/Autoclean $(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED)) $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT))) endef endif
-
HOST_BUILD_DIR --> build_dir/host/
-
HOST_BUILD_PREFIX --> staging_dir/host/ , this is defined in toolchain-build.mk
A bold guess, it should be through several hidden files to determine whether the tools are compiled and installed
- .prepared${md5}_${confvar}
- .configured
- .built
- .$(PKG_NAME)_installed
But there are findmd5 and confvar functions in HOST_STAMP_PREPARED, as well as PKG_FILE_DEPENDS and HOST_PREPARED_DEPENDS variables, and you have to figure out how to generate them.
- In rules.mk, found the confvar function definition
confvar=$(shell echo '$(foreach v,$(1),$(v)=$(subst ','\'',$($(v))))' | $(MKHASH) md5)
- HOST_PREPARED_DEPENDS This thing doesn't seem to appear anywhere else.
- PKG_FILE_DEPENDS is defined in package/base-files/Makefile
PKG_FILE_DEPENDS:=$(PLATFORM_DIR)/ $(GENERIC_PLATFORM_DIR)/base-files/
- $(PLATFORM_DIR) and GENERIC_PLATFORM_DIR in include/target.mk
PLATFORM_DIR:=$(TOPDIR)/target/linux/$(BOARD) GENERIC_PLATFORM_DIR := $(TOPDIR)/target/linux/generic
- Parameter summary
So we found_md5 passed a total of three parameters, namely ${CURDIR}, $(PLATFORM_DIR), $(GENERIC_PLATFORM_DIR)
depends.mk analysis
# define a dependency on a subtree # parameters: # 1: directories/files # 2: directory dependency # 3: tempfile for file listings # 4: find options DEP_FINDPARAMS := -x "*/.svn*" -x ".*" -x "*:*" -x "*\!*" -x "* *" -x "*\\\#*" -x "*/.*_check" -x "*/.*.swp" -x "*/.pkgdir*" find_md5=find $(wildcard $(1)) -type f $(patsubst -x,-and -not -path,$(DEP_FINDPARAMS) $(2)) -printf "%p%T@\n" | sort | $(MKHASH) md5 define rdep .PRECIOUS: $(2) .SILENT: $(2)_check $(2): $(2)_check check-depends: $(2)_check ifneq ($(wildcard $(2)),) $(2)_check:: $(if $(3), \ $(call find_md5,$(1),$(4)) > $(3).1; \ { [ \! -f "$(3)" ] || diff $(3) $(3).1 >/dev/null; } && \ ) \ { \ [ -f "$(2)_check.1" ] && mv "$(2)_check.1"; \ $(TOPDIR)/scripts/timestamp.pl $(DEP_FINDPARAMS) $(4) -n $(2) $(1) && { \ $(call debug_eval,$(SUBDIR),r,echo "No need to rebuild $(2)";) \ touch -r "$(2)" "$(2)_check"; \ } \ } || { \ $(call debug_eval,$(SUBDIR),r,echo "Need to rebuild $(2)";) \ touch "$(2)_check"; \ } $(if $(3), mv $(3).1 $(3)) else $(2)_check:: $(if $(3), rm -f $(3) $(3).1) $(call debug_eval,$(SUBDIR),r,echo "Target $(2) not built") endif endef
-
wildcard: extended wildcard, here are three paths directly passed in, no processing
-
patsubst: replace wildcards, replace -x in DEP_FINDPARAMS with -and -not -path
-
$(1): This passes in the above three paths
-
$(2): empty
-
Dismantling command parameters
-
The main command is find
-
The second command is sort and md5sum (instead of scripts/mkhash.c), calculate the md5 value after find classification
find {path1, path2, path3} -type f -and -not -path "*/.svn*" -and -not -path ".*" -and -not -path "*:*" -and -not -path "*\!*" -and -not -path "* *" -and -not -path "*\\\#*" -and -not -path "*/.*_check" -and -not -path "*/.*.swp" -and -not -path "*/.pkgdir" -printf "%p%T@\n" | sort | md5sum | awk '{print $1}'
$(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED)) $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))
The above statement corresponds to the following result
$(call rdep,tools/flock,build_dir/host/flock-2.18/.prepared${find_md5}_${confvar} $(call rdep,build_dir/host/flock-2.18,build_dir/host/flock-2.18/.built)
The analysis is too big, and I can't go on. I can only use the source code to achieve the adaptation effect.
Hard change source code
Since he uses md5 detection, I also change it to md5 detection, just take md5 for comparison according to the change time of tools/xxx folder, and decisively change host-build.mk
Source code:
HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(if $(HOST_QUILT)$(DUMP),,$(shell $(call find_md5,${CURDIR} $(PKG_FILE_DEPENDS),))_$(call confvar,CONFIG_AUTOREMOVE $(HOST_PREPARED_DEPENDS)))
After the change:
HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(shell stat -c %Y ${CURDIR} | $(MKHASH) md5)
In this way, the verification method becomes what we want
later
We moved the staging_dir and build_dir in the SDK to the source directory
mkdir -p staging_dir/host/stamp tools_name="$(ls -F ./tools/ | grep "/$")" for a in ${tools_name}; do if [ "${a}" != 'include/' ]; then PKG_NAME=$(cat "./tools/${a}Makefile" | grep '^PKG_NAME' | cut -d '=' -f 2 | sed 's/^[ \t]*//g') PKG_VERSION=$(cat "./tools/${a}Makefile" | grep '^PKG_VERSION' | cut -d '=' -f 2 | sed 's/^[ \t]*//g') touch staging_dir/host/stamp/.${PKG_NAME}_installed mkdir -p build_dir/host/${PKG_NAME}-${PKG_VERSION} prepared_md5=$(stat -c %Y "./tools/${a}" | md5sum | cut -d ' ' -f 1) touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.prepared${prepared_md5} touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.configured touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.built echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.version echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.tarball-version fi done
In this way, you can skip the compilation of tools. The easiest way is to clear all the tools in the tools/Makefile.
Enjoy it ~~