Explanation of implementation principle of Mini Pack

The article started on my blog https://github.com/mcuking/bl...

Implementation source code, please refer to https://github.com/mcuking/bl...

This paper mainly describes how to implement a front-end application packer similar to webpack step by step.

The essence of webpack

In essence, webpack is a static module bundler for modern JavaScript applications. When webpack processes an application, it recursively builds a dependency graph that contains each module required by the application, and then packages all these modules into one or more bundles.

Webpack is like a production line. It needs a series of processing processes before it can convert the source file into the output result. The responsibility of each processing process in this production line is single. There are dependencies between multiple processes. Only after the current processing is completed can it be handed over to the next process. A plug-in is like a function inserted into the production line. It processes the resources on the production line at a specific time. Webpack organizes this complex production line through Tapable. Webpack will broadcast events during operation. The plug-in only needs to listen to the events it cares about, and then it can join the production line to change the operation of the production line. The event flow mechanism of webpack ensures the order of plug-ins and makes the whole system extensible.

--Easy to understand webpack Wu Haolin

Webpack operation mechanism

The whole operation mechanism is serial, and the following processes will be executed from start to end:

  1. Initialization parameters: read and merge parameters from configuration files and Shell statements to get the final parameters;
  2. Start compilation: initialize the Compiler object with the parameters obtained in the previous step, load all configured plug-ins, and execute the run method of the object to start compilation;
  3. Determine entry: find all entry files according to the entry in the configuration;
  4. Compiling module: starting from the entry file, call all configured loaders to translate the module, find out the module that the module depends on, and then recurse this step until all the entry dependent files have been processed in this step;
  5. Complete module compilation: after translating all modules with Loader in step 4, the final translated content of each module and the dependencies between them are obtained;
  6. Output resources: according to the dependency between entries and modules, assemble chunks containing multiple modules, and then convert each Chunk into a separate file to be added to the output list. This step is the last chance to modify the output content;
  7. Output completion: after determining the output content, determine the output path and file name according to the configuration, and write the file content to the file system.

In the above process, Webpack will broadcast specific events at specific time points. The plug-in will execute specific logic after listening to the events of interest, and the plug-in can call the API provided by Webpack to change the running results of Webpack

Implementation process of Mini Pack

First of all, we need to clarify the objectives of Mini Pack:

Compile the js code in src into es5 version and package it into a bundle js (Note: only focus on js).

Next, we will gradually implement Mini Pack according to the description of webpack operation mechanism just now:

1. First, support the definition of webpack config. JS file, which can be named minipack config. JS, which defines parameters such as output and entry in the file. As shown below:

const path = require('path');

module.exports = {
  entry: path.join(__dirname, './src/index.js'),
  output: {
    path: path.join(__dirname, './dist'),
    filename: 'main.js'
  }
};

2. Then enter the compilation phase: according to minipack config. JS, initialize a Compiler parameter, and execute the run method.

index.js code

const Compiler = require('./compiler');
const options = require('../minipack.config');

// According to minipack config. JS, initialize the Compiler object, and start compilation
new Compiler(options).run();

compiler.js code

const {getAST, getDependencies, transform} = require('./utils');
const path = require('path');

module.exports = class Compiler {
  constructor(options) {
    const {entry, output} = options;
    // Packaging entrance
    this.entry = entry;
    // Export
    this.output = output;
    // Module set
    this.modules = [];
  }

  // Start build
  run() {
    const entryModule = this.buildModule(this.entry, true);

    this.modules.push(entryModule);
  }

  // Compile a single module
  buildModule(filename, isEntry) {
    let ast;

    ast = getAST(filename);

    return {
      filename,
      source: transform(ast),
      dependencies: getDependencies(ast)
    };
  }

  // Output the compiled js module to the specified directory
  emitFiles() {}
};

This step is to compile the entry js file into a module object. The format is as follows:

{
  filename  // file name
  source  // code
  dependencies  // Dependency file, that is, other modules introduced by this module
}

The compilation method getAST, the method transform from ast to code, and the method getDependencies for obtaining module dependencies are separately encapsulated in a utils file.

const fs = require('fs');
const path = require('path');
const {parse} = require('@babel/parser');
const traverse = require('@babel/traverse').default;
const {transformFromAst} = require('@babel/core');

module.exports = {
  // Compile the file js code corresponding to the path into ast
  getAST(path) {
    const content = fs.readFileSync(path, 'utf-8');
    return parse(content, {
      sourceType: 'module'
    });
  },

  // Traverse all nodes through Babel traverse
  // And collect the dependencies of a module according to the ImportDeclaration node
  getDependencies(ast) {
    const dependencies = [];
    traverse(ast, {
      ImportDeclaration({node}) {
        dependencies.push(node.source.value);
      }
    });
    return dependencies;
  },

  // Convert the converted ast code into code again
  // And compile it into es5 by configuring @ Babel / preset env preset plug-in
  transform(ast) {
    const {code} = transformFromAst(ast, null, {
      presets: ['@babel/preset-env']
    });
    return code;
  }
};

3. Determine the entry and find all the entry files according to the entry in the configuration. The above has implemented the compilation of the entry file.

4. Start from the entry file, compile the module (it is not intended to support running loader here), find out the module that the module depends on, and then recurse this step until all the entry dependent files have been processed in this step. That is, use the Babel traverse tool to traverse the ImportDeclaration node on the ast of this module (import in the corresponding code), find all other imported modules of this module, and then compile other modules in a recursive way, repeating the operation just now. The new code is as follows:

compiler.js

module.exports = class Compiler {
  constructor(options) {
    const {entry, output} = options;
    // Packaging entrance
    this.entry = entry;
    // Export
    this.output = output;
    // Module set
    this.modules = [];
  }

  // Start build
  run() {
    this.buildModule(this.entry, true);

    this.emitFiles();
  }

  // Call recursively until all referenced modules are compiled
  buildModule(filename, isEntry) {
    const _module = this.build(filename, isEntry);

    this.modules.push(_module);

    _module.dependencies.forEach(dependency => {
      this.buildModule(dependency, false);
    });
  }

  // Compile a single module
  build(filename, isEntry) {
    let ast;

    if (isEntry) {
      ast = getAST(filename);
    } else {
      const absolutePath = path.join(process.cwd(), './src', filename);
      ast = getAST(absolutePath);
    }

    return {
      filename,
      source: transform(ast),
      dependencies: getDependencies(ast)
    };
  }

  // Output the compiled js module to the specified directory
  emitFiles() {}
};

5. Complete module compilation. The above code has realized recursive compilation of all referenced modules.

6. Output resources. Here, Mini Pack is going to package all modules into one file, instead of assembling chunks containing multiple modules like webpack, and then convert each Chunk into a separate file to be added to the output list.

Since the code of all modules should be packaged into one file, it is bound to lead to naming conflict. In order to ensure that each module does not affect each other, different modules can be wrapped with a function (using the scope of js function). Then there will be another problem - the reference between modules. In this regard, we can customize the require function to refer to variables or methods of other modules, and then pass the customized require method into the package function in the form of parameters for the code in the module to call. The specific modes are as follows:

(function(modules) {
  function require(filename) {
    var fn = modules[filename];
    var module = {exports: {} };

    fn(require, module, module.exports);
    return module.exports;
  }

  return require('./entry');
})({
  './entry': function(require, module, exports) {
      var addModule = require("./add");
      console.log(addModule.add(1, 1));
  },
  './add': function(require, module, exports) {
      module.exports = {
        add: function(x, y) {
            return x + y;
        }
      }
  }
});

Therefore, the Compiler implementation code can be further improved as follows:

module.exports = class Compiler {
  constructor(options) {
    const { entry, output } = options;
    // Packaging entrance
    this.entry = entry;
    // Export
    this.output = output;
    // Module set
    this.modules = [];
  }

  // Start build
  run() {
    this.buildModule(this.entry, true);

    this.emitFiles();
  }

  // Call recursively until all referenced modules are compiled
  buildModule(filename, isEntry) {
    // ditto
  }

  // Compile a single module
  build(filename, isEntry) {
    // ditto
  }

  // Output the compiled js module to the specified directory
  emitFiles() {
    // Put all module codes into one function respectively (use function scope to realize scope isolation and avoid variable conflict)
    // At the same time, a require method has been implemented to introduce the required variables or methods from other modules
    let modules = '';

    this.modules.forEach(_module => {
      modules += `'${_module.filename}': function(require, module, exports) {${_module.source}},`;
    });

    const bundle = `(function(modules) {
      function require(filename) {
        var fn = modules[filename];
        var module = {exports: {}};

        fn(require, module, module.exports);
        return module.exports;
      }

      return require('${this.entry}')
    })({${modules}})`;
  }
};

7. Output completion: after determining the output content, determine the output path and file name according to the configuration, and write the file content to the file system. That is, the fs module outputs the compiled large code to the specified directory. The code is as follows:

module.exports = class Compiler {
  constructor(options) {
    const {entry, output} = options;
    // Packaging entrance
    this.entry = entry;
    // Export
    this.output = output;
    // Module set
    this.modules = [];
  }

  // Start build
  run() {
    this.buildModule(this.entry, true);

    this.emitFiles();
  }

  // Call recursively until all referenced modules are compiled
  buildModule(filename, isEntry) {
    // ditto
  }

  // Compile a single module
  build(filename, isEntry) {
    // ditto
  }

  // Output the compiled js module to the specified directory
  emitFiles() {
    // Put all module codes into one function respectively (use function scope to realize scope isolation and avoid variable conflict)
    // At the same time, a require method has been implemented to introduce the required variables or methods from other modules
    let modules = '';

    this.modules.forEach(_module => {
      modules += `'${_module.filename}': function(require, module, exports) {${_module.source}},`;
    });

    const bundle = `(function(modules) {
      function require(filename) {
        var fn = modules[filename];
        var module = {exports: {}};

        fn(require, module, module.exports);
        return module.exports;
      }

      return require('${this.entry}')
    })({${modules}})`;

    // Write the compiled code to the directory specified by output
    const distPath = path.join(process.cwd(), './dist');
    if (fs.existsSync(distPath)) {
      removeDir(distPath);
    }

    fs.mkdirSync(distPath);

    const outputPath = path.join(this.output.path, this.output.filename);
    fs.writeFileSync(outputPath, bundle, 'utf-8');

    // Insert the compiled js into html and write it to the directory specified by output
    this.emitHtml();
  }

  // Insert html into the script tag (import the packaged bundle js) and output it to the specified directory
  emitHtml() {
    const publicHtmlPath = path.join(process.cwd(), './public/index.html');
    let html = fs.readFileSync(publicHtmlPath, 'utf-8');
    html = html.replace(
      /<\/body>/,
      `  <script type="text/javascript" src="./main.js"></script>
  </body>`
    );

    const distHtmlPath = path.join(process.cwd(), './dist/index.html');
    fs.writeFileSync(distHtmlPath, html, 'utf-8');
  }
};

During this process, Webpack will broadcast specific events at specific time points to notify the corresponding plug-ins to perform the specified tasks and change the packaging results. In this regard, it is not in the initial setting function orientation of Mini Pack, so so so far, the packaging has been completed. Here is the complete code of Compiler:

const path = require('path');
const fs = require('fs');
const {getAST, getDependencies, transform, removeDir} = require('./utils');

module.exports = class Compiler {
  constructor(options) {
    const {entry, output} = options;
    // Packaging entrance
    this.entry = entry;
    // Export
    this.output = output;
    // Module set
    this.modules = [];
  }

  // Start build
  run() {
    this.buildModule(this.entry, true);

    this.emitFiles();
  }

  // Call recursively until all referenced modules are compiled
  buildModule(filename, isEntry) {
    const _module = this.build(filename, isEntry);

    this.modules.push(_module);

    _module.dependencies.forEach(dependency => {
      this.buildModule(dependency, false);
    });
  }

  // Compile a single module
  build(filename, isEntry) {
    let ast;

    if (isEntry) {
      ast = getAST(filename);
    } else {
      const absolutePath = path.join(process.cwd(), './src', filename);
      ast = getAST(absolutePath);
    }

    return {
      filename,
      source: transform(ast),
      dependencies: getDependencies(ast)
    };
  }

  // Output the compiled js module to the specified directory
  emitFiles() {
    // Put all module codes into one function respectively (use function scope to realize scope isolation and avoid variable conflict)
    // At the same time, a require method has been implemented to introduce the required variables or methods from other modules
    let modules = '';

    this.modules.forEach(_module => {
      modules += `'${_module.filename}': function(require, module, exports) {${_module.source}},`;
    });

    const bundle = `(function(modules) {
      function require(filename) {
        var fn = modules[filename];
        var module = {exports: {}};

        fn(require, module, module.exports);
        return module.exports;
      }

      return require('${this.entry}')
    })({${modules}})`;

    // Write the compiled code to the directory specified by output
    const distPath = path.join(process.cwd(), './dist');
    if (fs.existsSync(distPath)) {
      removeDir(distPath);
    }

    fs.mkdirSync(distPath);

    const outputPath = path.join(this.output.path, this.output.filename);
    fs.writeFileSync(outputPath, bundle, 'utf-8');

    // Insert the compiled js into html and write it to the directory specified by output
    this.emitHtml();
  }

  // Insert html into the script tag (import the packaged bundle js) and output it to the specified directory
  emitHtml() {
    const publicHtmlPath = path.join(process.cwd(), './public/index.html');
    let html = fs.readFileSync(publicHtmlPath, 'utf-8');
    html = html.replace(
      /<\/body>/,
      `  <script type="text/javascript" src="./main.js"></script>
  </body>`
    );

    const distHtmlPath = path.join(process.cwd(), './dist/index.html');
    fs.writeFileSync(distHtmlPath, html, 'utf-8');
  }
};

Concluding remarks

Here, a simple front-end project packer has been implemented. Please refer to the complete implementation code mini-pack . After the whole process, I believe readers will have a deeper understanding of the front-end project packaging process.

Tags: Javascript Front-end Webpack

Posted by ryanlwh on Sat, 14 May 2022 04:32:07 +0300