API Injection Mechanism and Plugin Startup Process_VSCode Plugin Development Notes 2

Preface

// The module 'vscode' contains the VS Code extensibility API
import * as vscode from 'vscode';

var disposable = vscode.commands.registerCommand('extension.sayHello', () => {
    // Display a message box to the user
    vscode.window.showInformationMessage('Hello World!');
});

In the plugin process environment, can import vscode module to access APIs available to plugins. If a bit curious, can discover that there's no vscode module under node_modules, and vscode module name hasn't been define()d either. Looks like we required a non-existent module, so, where does this thing come from?

P.S. For more information about define(), please see [VS Code Source Code Brief Analysis | Renderer Process Initialization](/articles/vs-code 源码简析/#articleHeader9)

I. require

Following the clues, first look at what happens when importing a Node module?

Node loads modules through require(name) function, passes in module name name, returns Module instance, rough process as follows:

name parameter is mapped to complete file path through Module._resolveFilename() method
If cache[fullName] exists, return cache[fullName].exports (prioritize cache), a module is only loaded once, thereby improving module loading speed. If don't want to use cache, can delete cache[fullName] before require(name), for example delete require.cache[require.resolve('./my-module.js')]
Otherwise, load source code from corresponding file, and do preprocessing (module-level variable injection), see Module.prototype.load
Finally, compile (execute) transformed source code, return value of module.exports, see Module.prototype._compile

P.S. For more information about module cache, please see node.js require() cache - possible to invalidate?

Look at a simple scenario, suppose there are two source files:

  // my-modue.js
module.exports = 'my-modue';

// index.js
const m = require('./my-module.js');

Executing entry file's first line require('./my-modue.js') rough process is:

// module.js
function require(path) {
  return mod.require(path);
}
Module.prototype.require = function(path) {
  return Module._load(path, this, /* isMain */ false);
}
Module._load = function(request, parent, isMain) {
  var filename = Module._resolveFilename(request, parent, isMain);
  var module = new Module(filename, parent);
  Module._cache[filename] = module;
  tryModuleLoad(module, filename);
  return module.exports;
}

Among them tryModuleLoad() specifically as follows:

function tryModuleLoad(module, filename) {
  module.load(filename);
}
Module.prototype.load = function(filename) {
  // Search all accessible node_modules directories upward
  this.paths = Module._nodeModulePaths(path.dirname(filename));
  // Load module by file extension
  Module._extensions[extension](this, filename);
}
Module._extensions['.js'] = function(module, filename) {
  // Read source code
  var content = fs.readFileSync(filename, 'utf8');
  // Compile (execute)
  module._compile(internalModule.stripBOM(content), filename);
}
Module.prototype._compile = function(content, filename) {
  // Wrap module source code with IIFE, inject module-level variables, see NativeModule.wrap()
  var wrapper = Module.wrap(content);
  // Equivalent to safer eval(), compile wrapped function source code, get executable Function instance
  var compiledWrapper = vm.runInThisContext(wrapper, {
    filename: filename,
    lineOffset: 0,
    displayErrors: true
  });
  var dirname = path.dirname(filename);
  // Module-level require() method to inject
  var require = internalModule.makeRequireFunction(this);
  // Inject module parameters, execute
  result = compiledWrapper.call(this.exports, this.exports, require, this, filename, dirname);
  // This return value is discarded, useless, module content is brought out by this.exports
  return result;
}

IIFE wrapped outside module source code is like this:

NativeModule.wrap = function(script) {
  // NativeModule.wrapper[0] = "(function (exports, require, module, __filename, __dirname) { "
  // NativeModule.wrapper[1] = "\n});"
  return NativeModule.wrapper[0] + script + NativeModule.wrapper[1];
};

Simply organize, actually the core work of entire process is equivalent to:

// 1.Read file
const moduleScript = fs.readFileSync(fullFilename, 'utf8');
// 2.Construct module (isolate module scope, declare module-level variables)
const wrapped = `(function (exports, require, module, __filename, __dirname) {
  ${moduleScript}
});`;
// 2.5.Compile to get executable module
const moduleFunction = eval(wrapped);
// 3.Execute (inject module-level variable values)
let exportsHost = {};
moduleFunction.call(exportsHost, exportsHost);
const m = exportsHost;

So, since require is a (module-level) local variable, not convenient to do tricks (hijack/tamper), then must have done something to Module, to be able to support loading non-existent virtual modules

P.S. Don't think about hijacking require('internal/module').makeRequireFunction factory method to tamper require, because not allowed to access internal module:

NativeModule.nonInternalExists = function(id) {
  return NativeModule.exists(id) && !NativeModule.isInternal(id);
};
NativeModule.isInternal = function(id) {
  return id.startsWith('internal/');
};

At Module._resolveFilename will be treated as outsider, search from outside, cannot access the instance we want

II. extension API Injection

Debug the process of require('vscode'), easy to discover where tricks were done:

// ref: src/vs/workbench/api/node/extHost.api.impl.ts
function defineAPI(factory: IExtensionApiFactory, extensionPaths: TernarySearchTree<IExtensionDescription>): void {

  // each extension is meant to get its own api implementation
  const extApiImpl = new Map<string, typeof vscode>();
  let defaultApiImpl: typeof vscode;

  const node_module = <any>require.__$__nodeRequire('module');
  const original = node_module._load;
  node_module._load = function load(request, parent, isMain) {
    if (request !== 'vscode') {
      return original.apply(this, arguments);
    }

    // get extension id from filename and api for extension
    const ext = extensionPaths.findSubstr(parent.filename);
    if (ext) {
      let apiImpl = extApiImpl.get(ext.id);
      if (!apiImpl) {
        apiImpl = factory(ext);
        extApiImpl.set(ext.id, apiImpl);
      }
      return apiImpl;
    }

    // fall back to a default implementation
    if (!defaultApiImpl) {
      defaultApiImpl = factory(nullExtensionDescription);
    }
    return defaultApiImpl;
  };
}

Module._load() method is hijacked, when encountering vscode returns a virtual module, called apiImpl. Note, APIs obtained by each plugin are independent (possibly for plugin security isolation considerations, avoid hijacking API affecting other plugins)

P.S. Note, reason why require.__$__nodeRequire('module'), is because global.require has already been hijacked (see [VS Code Source Code Brief Analysis | Renderer Process Initialization](/articles/vs-code 源码简析/#articleHeader9) loader part)... VS Code team's approach is very wild

III. Plugin Mechanism Initialization Process

Previously in [VS Code Startup Process](/articles/vs-code 源码简析/#articleHeader9) UI layout part mentioned:

UI 入口
src/vs/workbench/electron-browser/bootstrap/index.html
  src/vs/workbench/electron-browser/bootstrap/index.js
    src/vs/workbench/workbench.main js index 文件
      src/vs/workbench/electron-browser/main.ts
        src/vs/workbench/electron-browser/shell.ts 界面与功能服务的接入点
          src/vs/workbench/electron-browser/workbench.ts 创建界面
            src/vs/workbench/browser/layout.ts 布局计算，绝对定位

Starting from creating WorkbenchShell formally enters function area UI layout, UI is called Shell, considered as container for carrying functions ("shell")

That is starting from src/vs/workbench/electron-browser/shell.ts begin working on interface creation, and interface docking with function services. Last time only focused on parts related to main startup process, this time look at plugin mechanism initialization process

Plugin mechanism initialization file progressive relationship:

src/vs/workbench/electron-browser/shell.ts 界面与功能服务的接入点
  src/vs/workbench/services/extensions/electron-browser/extensionService.ts
    src/vs/workbench/services/extensions/electron-browser/extensionHost.ts
      src/vs/workbench/node/extensionHostProcess.ts
        src/vs/workbench/node/extensionHostMain.ts

Create ExtensionService

src/vs/workbench/electron-browser/shell.ts's createContents() method is related to ExtensionService, main content as follows:

private createContents(parent: Builder): Builder {
  // Instantiation service with services
  const [instantiationService, serviceCollection] = this.initServiceCollection(parent.getHTMLElement());
}
private initServiceCollection(container: HTMLElement): [IInstantiationService, ServiceCollection] {
  this.extensionService = instantiationService.createInstance(ExtensionService);
  serviceCollection.set(IExtensionService, this.extensionService);
}

ExtensionService comes from src/vs/workbench/services/extensions/electron-browser/extensionService.ts, key parts as follows:

lifecycleService.when(LifecyclePhase.Running).then(() => {
  // delay extension host creation and extension scanning
  // until after workbench is running
  // 1.初始化 extensionHost
  this._startExtensionHostProcess([]);
  // 2.扫描已安装的插件
  this._scanAndHandleExtensions();
});

private _startExtensionHostProcess(initialActivationEvents: string[]): void {
  // Kill already existing ExtensionHost process
	  this._stopExtensionHostProcess();
  // Create and start ExtensionHostProcessWorker
  this._extensionHostProcessWorker = this._instantiationService.createInstance(ExtensionHostProcessWorker, this);
  this._extensionHostProcessProxy = this._extensionHostProcessWorker.start().then(
    //...
  );
  // Register events that trigger activation by scenario (such as activating plugin only when opening specific file)
  this._extensionHostProcessProxy.then(() => {
    initialActivationEvents.forEach((activationEvent) => this.activateByEvent(activationEvent));
  });
}

First start extensionHost process through ExtensionHostProcessWorker, simultaneously scan installed plugins, after extensionHost process creation completes register on-demand activated plugins (plugins with activationEvents not ["*"])

Start extensionHost Process

ExtensionHostProcessWorker comes from src/vs/workbench/services/extensions/electron-browser/extensionHost.ts, key parts as follows:

public start(): TPromise<IMessagePassingProtocol> {
  const opts = {
    env: objects.mixin(objects.deepClone(process.env), {
      AMD_ENTRYPOINT: 'vs/workbench/node/extensionHostProcess'
    })
  };

  // Run Extension Host as fork of current process
  this._extensionHostProcess = fork(URI.parse(require.toUrl('bootstrap')).fsPath, ['--type=extensionHost'], opts);
}

This fork() seems unrelated to AMD_ENTRYPOINT, actually, entry of child process obtained by fork is:

// URI.parse(require.toUrl('bootstrap')).fsPath
// After toUrl conversion corresponds to
// out/bootstrap

That is src/bootstrap.js, key parts as follows:

require('./bootstrap-amd').bootstrap(process.env['AMD_ENTRYPOINT']);

Go around and come back, is to walk through loader to execute entry file:

var loader = require('./vs/loader');
exports.bootstrap = function (entrypoint) {
  loader([entrypoint], function () { }, function (err) { console.error(err); });
};

So now, step into entry src/vs/workbench/node/extensionHostProcess.ts:

// setup things
const extensionHostMain = new ExtensionHostMain(renderer.rpcProtocol, renderer.initData);
onTerminate = () => extensionHostMain.terminate();
return extensionHostMain.start();

Turned to ExtensionHostMain again, corresponding source file is src/vs/workbench/node/extensionHostMain.ts:

public start(): TPromise<void> {
  return this._extensionService.onExtensionAPIReady()
    // Start the most eager batch of plugins
    .then(() => this.handleEagerExtensions())
    .then(() => this.handleExtensionTests())
    .then(() => {
      this._logService.info(`eager extensions activated`);
    });
}
// Handle "eager" activation extensions
private handleEagerExtensions(): TPromise<void> {
  this._extensionService.activateByEvent('*', true).then(null, (err) => {
    console.error(err);
  });
  return this.handleWorkspaceContainsEagerExtensions();
}

At this point, unconditionally started plugins are also activated, plugin mechanism initialization completes

Activate Plugins

Specific plugin activation process is quite tedious, because supports Extension Pack type plugins (allows plugins to depend on other plugins), so activating plugins also needs to handle plugin dependency tree, only activate current plugin after all depended plugins successfully activate

P.S. If want to understand specific process, can look at these two files:

src/vs/workbench/api/node/extHostExtensionService.ts
src/vs/workbench/api/node/extHostExtensionActivator.ts

Due to space limitations, we skip tedious dependency handling part, directly look at part loading plugin pkg.main entry file:

private _doActivateExtension() {
  // require load plugin entry file
  loadCommonJSModule(this._logService, extensionDescription.main, activationTimesBuilder),
		this._loadExtensionContext(extensionDescription).then(values => {
    // Execute its activate() method
    return ExtHostExtensionService._callActivate(this._logService, extensionDescription.id, <IExtensionModule>values[0], <IExtensionContext>values[1], activationTimesBuilder);
  });
}

// Load entry file
function loadCommonJSModule() {
  r = require.__$__nodeRequire<T>(modulePath);
  return TPromise.as(r);
}
// Execute agreed activate() method
private static _callActivateOptional() {
  if (typeof extensionModule.activate === 'function') {
    const activateResult: TPromise<IExtensionAPI> = extensionModule.activate.apply(global, [context]);
  }
}

Directly node require execute plugin entry file to get module instance, then apply call its activate method, plugin runs up

IV. Process Model

At this point, we understand there are at least 3 processes in VS Code:

Electron Main Process: App main process
Electron Renderer Process: UI process
Extension Host Process: Plugin host process, provides execution environment for plugins

Among them Extension Host Process (each VS Code window) only exists one, all plugins execute in this process, not one independent process per plugin

Note, plugin host process is an ordinary Node process (forked out by childProcess.fork()), not Electron process, and restricted from using electron:

// Environment variable
ELECTRON_RUN_AS_NODE: '1'

So cannot use require('electron').BrowserWindow.getAllWindows() in plugin runtime environment to curve modify UI

P.S. For discussion about plugin customizing UI capabilities, see access electron API from vscode extension

Inter-Process Communication Method

      <Electron IPC>
Main ---------------- Renderer
 |
 |
 | <Child Process IPC>
 |
 |
Extension Host

Among them, communication between Extension Host and Main is completed through IPC built into fork(), specifically as follows:

// Support logging from extension host
this._extensionHostProcess.on('message', msg => {
  if (msg && (<IRemoteConsoleLog>msg).type === '__$console') {
    this._logExtensionHostMessage(<IRemoteConsoleLog>msg);
  }
});

Here is only one-way communication (plugin -> Main), actually can complete the other half (Main -> plugin) through this._extensionHostProcess.send({msg})

P.S. For more information about inter-process communication, please see [Nodejs Inter-Process Communication](/articles/nodejs 进程间通信/)