Small tool to print a mapping between module file "primary source" and dart files that are part of that module

module_explorer.dart

import 'dart:convert';
import 'dart:core';
import 'dart:io';

main() {
  final packageName = Directory.current.path.split('/').last;
  final generated = new Directory('.dart_tool/build/generated/$packageName');

  // The build system's documentation reads "[e]mit .module assets which contain
  // a filtered view of the package level meta-module specific to a single
  // module. These are emitted for the 'primary source' of each module, as well
  // as each library which is an entrypoint." which does not appear to be
  // completely correct. It does seem that these are emitted for every 'primary
  // source' i.e. root of the dependency tree but these do not appear to be
  // emitted for every entry point.
  // https://github.com/dart-lang/build/blob/master/build_modules/README.md#module-creation
  final moduleFilesInLib = new RegExp(r'.*lib.*\.module$');
  final moduleFiles = generated
      .listSync(recursive: true)
      .where((e) => moduleFilesInLib.hasMatch(e.path))
      .fold(new List<File>(), addFiles);

  final moduleToFiles =
      <String /* module name */, Set<String> /* filenames in module */ >{};
  for (final file in moduleFiles) {
    final module = json.decode(file.readAsStringSync());
    // 'p' is the json key for primarySource
    // https://github.com/dart-lang/build/blob/68b3d29a624754959d65aefcc509a918cca11746/build_modules/lib/src/modules.dart#L64
    final primarySource = primarySourceToKey(module['p']);

    Set<String> files = new Set<String>.from([primarySource]);

    // According to the docs sources should not necessarily contain all the dart
    // files that are compiled as part of this module, but it always seems to,
    // including cases where there does not appear to be import cycles.
    // https://github.com/dart-lang/build/blob/68b3d29a624754959d65aefcc509a918cca11746/build_modules/lib/src/modules.dart#L88
    for (String source in module['s'].map(primarySourceToKey)) {
      files.add(source);
    }

    moduleToFiles[primarySource] = files;
  }

  // Invariant check: no two files should be compiled as part of the same module
  for (final i in moduleToFiles.keys) {
    for (final j in moduleToFiles.keys) {
      if (i != j) {
        final intersect = moduleToFiles[i]
            .intersection(moduleToFiles[j])
            .where((String s) => s.startsWith('text'))
            .toList();

        if (intersect.isNotEmpty) {
          print('invariant violation, two modules contain the same dart file');
          exit(1);
        }
      }
    }
  }

  final dartFileInLib = new RegExp(r'^\.\/lib.*dart$');
  final expectedDartFiles = new Directory('.')
      .listSync(recursive: true)
      .fold(new List<String>(), (List<String> value, FileSystemEntity entity) {
    if (entity is File && dartFileInLib.hasMatch(entity.path)) {
      value.add(entity.path.replaceFirst('./', ''));
    }
    return value;
  });

  final notFound = <String>[];
  for (final file in expectedDartFiles) {
    final containingModule = search(moduleToFiles, file);
    if (containingModule == null) {
      notFound.add(file);
    }
  }

  if (notFound.isNotEmpty) {
    print(
        'The following ${notFound.length} dart files were not found $notFound to be part of a module file');
  }

  for (final k in moduleToFiles.keys) {
    print('module $k contains ${moduleToFiles[k].length} files');
  }
}

List<File> addFiles(List<File> files, FileSystemEntity entity) {
  if (entity is File) {
    files.add(entity);
  }
  return files;
}

/// Find which module file is compiled in.
String search(Map<String, Set<String>> moduleToFile, String fileName) {
  for (final k in moduleToFile.keys) {
    if (moduleToFile[k]
        .map((s) => s.split('|')[1])
        .toList()
        .contains(fileName)) {
      return k;
    }
  }
  return null;
}

String primarySourceToKey(p) {
  return '${p[0]}|${p[1]}';
}

these do not appear to be emitted for every entry point.

@jakemac53 - do you recall if we changed something here?

We haven't changed anything I am aware of regarding that - we should be emitting module files for all application entry points. We may not emit modules for "entry points" under lib if they are in a cycle though? I can't remember for sure.

We may not emit modules for "entry points" under lib if they are in a cycle though

This might be what I was seeing. My process was to grab all the modules that had the "is" key == true from the .ddc.meta_module.clean and compare against the .module files found. The latter was a subset of the former.

In general does this seem like this will do what we think it does (print a mapping between module file "primary source" and the number of dart files that are compiled as part of that module)?

In general does this seem like this will do what we think it does

After a fresh build, yes I believe it will. One potential problem is that if things change and a .module file becomes unnecessary, we won't necessarily delete it from the cache directory. So if you have run multiple builds and changed imports around and added or removed files you might see modules that no longer exist.

Good to know! Thanks for taking a look!