OpenWrt skips tools compilation

OpenWrt skips tools compilation


It is still time-consuming for OpenWrt to compile tools, so in order to shorten the compilation time, we try to use the compiled tools in the SDK provided by OpenWrt.

Analyze the code

When analyzing the OpenWrt source code, the more you analyze, the more code you see, and the more knowledge you need to understand. It makes my head big. I will divide it into several modules to find out how to judge whether tools should be rebuilt. There is too much nonsense in the front. If you don't want to see the nonsense, just fast forward to Hard change the source code

SDK Analysis

In the SDK provided by OpenWrt, this article mainly uses

  • build_dir
  • staging_dir

These two folders contain the tools and toolchain s that the text needs to use

Top level Makefile analysis

Let me talk about two important variables first, in case someone is unclear

  • TOPDIR This is the top-level path, which is the directory of the source code
  • CURDIR This is the current path, which is the directory where the current file is located
export PATH:=$(TOPDIR)/staging_dir/host/bin:$(PATH)

A PATH environment variable $(TOPDIR)/staging_dir/host/bin is imported below, which is the tool directory of tools

$(toolchain/stamp-compile): $(tools/stamp-compile) $(if $(CONFIG_BUILDBOT),toolchain_rebuild_check)
$(target/stamp-compile): $(toolchain/stamp-compile) $(tools/stamp-compile) $(BUILD_DIR)/.prepared

At the end of the world, the toolchain is called here, which is related to the generation of the time stamp file of the tools

tools/Makefile analysis

tools_enabled = $(foreach tool,$(sort $(tools-y) $(tools-)),$(if $(filter $(tool),$(tools-y)),y,n))
$(eval $(call stampfile,$(curdir),tools,compile,,_$(subst $(space),,$(tools_enabled)),$(STAGING_DIR_HOST)))
$(eval $(call stampfile,$(curdir),tools,check,$(TMP_DIR)/.build,,$(STAGING_DIR_HOST)))

Here is also called to generate stamp analysis

# Parameters: <subdir> <name> <target> <depends> <config options> <stampfile location>
# 1: subdir
# 2: target
# 3: build type
# 4: build variant
# 5: all variants
define stampfile
  $(1)/stamp-$(3):=$(if $(6),$(6),$(STAGING_DIR))/stamp/.$(2)_$(3)$(5)
  $$($(1)/stamp-$(3)): $(TMP_DIR)/.build $(4)
	@+$(SCRIPT_DIR)/ -n $$($(1)/stamp-$(3)) $(1) $(4) || \
		$(MAKE) $(if $(QUIET),--no-print-directory) $$($(1)lags-$(3)) $(1)/$(3)
	@mkdir -p $$$$(dirname $$($(1)/stamp-$(3)))
	@touch $$($(1)/stamp-$(3))

  $$(if $(call debug,$(1),v),,.SILENT: $$($(1)/stamp-$(3)))

  .PRECIOUS: $$($(1)/stamp-$(3)) # work around a make bug

  $(1)/stamp-$(3)/clean: FORCE
	@rm -f $$($(1)/stamp-$(3))

endif is also called here to generate stamp, so it is probably understood that OpenWrt relies on judging stamp to decide whether to recompile tools and toolchain analysis


  define HostHost/Autoclean
    $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))
  • HOST_BUILD_DIR --> build_dir/host/

  • HOST_BUILD_PREFIX --> staging_dir/host/ , this is defined in

A bold guess, it should be through several hidden files to determine whether the tools are compiled and installed

  • .prepared${md5}_${confvar}
  • .configured
  • .built
  • .$(PKG_NAME)_installed

But there are findmd5 and confvar functions in HOST_STAMP_PREPARED, as well as PKG_FILE_DEPENDS and HOST_PREPARED_DEPENDS variables, and you have to figure out how to generate them.

  • In, found the confvar function definition
confvar=$(shell echo '$(foreach v,$(1),$(v)=$(subst ','\'',$($(v))))' | $(MKHASH) md5)
  • HOST_PREPARED_DEPENDS This thing doesn't seem to appear anywhere else.
  • PKG_FILE_DEPENDS is defined in package/base-files/Makefile
GENERIC_PLATFORM_DIR := $(TOPDIR)/target/linux/generic
  • Parameter summary

So we found_md5 passed a total of three parameters, namely ${CURDIR}, $(PLATFORM_DIR), $(GENERIC_PLATFORM_DIR) analysis

# define a dependency on a subtree
# parameters:
#	1: directories/files
#	2: directory dependency
#	3: tempfile for file listings
#	4: find options

DEP_FINDPARAMS := -x "*/.svn*" -x ".*" -x "*:*" -x "*\!*" -x "* *" -x "*\\\#*" -x "*/.*_check" -x "*/.*.swp" -x "*/.pkgdir*"

find_md5=find $(wildcard $(1)) -type f $(patsubst -x,-and -not -path,$(DEP_FINDPARAMS) $(2)) -printf "%p%T@\n" | sort | $(MKHASH) md5

define rdep
  .PRECIOUS: $(2)
  .SILENT: $(2)_check

  $(2): $(2)_check
  check-depends: $(2)_check

ifneq ($(wildcard $(2)),)
	$(if $(3), \
		$(call find_md5,$(1),$(4)) > $(3).1; \
		{ [ \! -f "$(3)" ] || diff $(3) $(3).1 >/dev/null; } && \
	) \
	{ \
		[ -f "$(2)_check.1" ] && mv "$(2)_check.1"; \
	    $(TOPDIR)/scripts/ $(DEP_FINDPARAMS) $(4) -n $(2) $(1) && { \
			$(call debug_eval,$(SUBDIR),r,echo "No need to rebuild $(2)";) \
			touch -r "$(2)" "$(2)_check"; \
		} \
	} || { \
		$(call debug_eval,$(SUBDIR),r,echo "Need to rebuild $(2)";) \
		touch "$(2)_check"; \
	$(if $(3), mv $(3).1 $(3))
	$(if $(3), rm -f $(3) $(3).1)
	$(call debug_eval,$(SUBDIR),r,echo "Target $(2) not built")

  • wildcard: extended wildcard, here are three paths directly passed in, no processing

  • patsubst: replace wildcards, replace -x in DEP_FINDPARAMS with -and -not -path

  • $(1): This passes in the above three paths

  • $(2): empty

  • Dismantling command parameters

  • The main command is find

  • The second command is sort and md5sum (instead of scripts/mkhash.c), calculate the md5 value after find classification

find {path1, path2, path3} -type f -and -not -path "*/.svn*" -and -not -path ".*" -and -not -path "*:*" -and -not -path "*\!*" -and -not -path "* *" -and -not -path "*\\\#*" -and -not -path "*/.*_check" -and -not -path "*/.*.swp" -and -not -path "*/.pkgdir" -printf "%p%T@\n" | sort | md5sum | awk '{print $1}'
$(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))

The above statement corresponds to the following result

$(call rdep,tools/flock,build_dir/host/flock-2.18/.prepared${find_md5}_${confvar}
$(call rdep,build_dir/host/flock-2.18,build_dir/host/flock-2.18/.built)

The analysis is too big, and I can't go on. I can only use the source code to achieve the adaptation effect.

Hard change source code

Since he uses md5 detection, I also change it to md5 detection, just take md5 for comparison according to the change time of tools/xxx folder, and decisively change

Source code:


After the change:

HOST_STAMP_PREPARED=$(HOST_BUILD_DIR)/.prepared$(shell stat -c %Y ${CURDIR} | $(MKHASH) md5)

In this way, the verification method becomes what we want


We moved the staging_dir and build_dir in the SDK to the source directory

mkdir -p staging_dir/host/stamp
tools_name="$(ls -F ./tools/ | grep "/$")"
for a in ${tools_name}; do
    if [ "${a}" != 'include/' ]; then
    PKG_NAME=$(cat "./tools/${a}Makefile" | grep '^PKG_NAME' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')
    PKG_VERSION=$(cat "./tools/${a}Makefile" | grep '^PKG_VERSION' | cut -d '=' -f 2 | sed 's/^[ \t]*//g')

    touch staging_dir/host/stamp/.${PKG_NAME}_installed

    mkdir -p build_dir/host/${PKG_NAME}-${PKG_VERSION}

    prepared_md5=$(stat -c %Y "./tools/${a}" | md5sum | cut -d ' ' -f 1)

    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.prepared${prepared_md5}
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.configured
    touch build_dir/host/${PKG_NAME}-${PKG_VERSION}/.built
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.version
    echo ${PKG_VERSION} >build_dir/host/${PKG_NAME}-${PKG_VERSION}/.tarball-version


In this way, you can skip the compilation of tools. The easiest way is to clear all the tools in the tools/Makefile.

Enjoy it ~~

Tags: openwrt

Posted by netdog on Sat, 17 Sep 2022 21:56:47 +0300