Shadowed submodule error

error.md

Shadowed submodule error

Problem

I am adding Swift Package Manager support to the camera_avfoundation package. See: flutter/packages#6710

However, the Mac_x64 ios_build_all_packages master check fails:

Warning: Building for device with codesigning disabled. You will have to manually codesign before deploying to device.
Building com.example.allPackages for device (ios)...
Running pod install...                                              3.6s
Running Xcode build...                                          

Xcode build done.                                           16.7s
Failed to build iOS app
Parse Issue (Xcode): Build a shadowed submodule 'camera_avfoundation.messages_g'

Encountered error while building for device.

Building all_packages using Cocoapods failed.

Reproduction

Simple reproduction

You can reproduce this locally using Xcode 15.2 or older; Xcode 15.3 and newer do not reproduce this error:

1. Checkout my branch: https://github.com/loic-sharma/flutter-packages/tree/spm_camera

   git checkout -b loic-sharma-spm_camera main
   git pull [email protected]:loic-sharma/flutter-packages.git spm_camera
   

2. Add use_frameworks! to packages/camera/camera_avfoundation/example/ios/Podfile:

   diff --git a/packages/camera/camera_avfoundation/example/ios/Podfile b/packages/camera/camera_avfoundation/example/ios/Podfile
   index bcdae34190..61fe86ef41 100644
   --- a/packages/camera/camera_avfoundation/example/ios/Podfile
   +++ b/packages/camera/camera_avfoundation/example/ios/Podfile
   @@ -28,6 +28,8 @@ require File.expand_path(File.join('packages', 'flutter_tools', 'bin', 'podhelpe
    flutter_ios_podfile_setup
   
    target 'Runner' do
   +  use_frameworks!
   +
      flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
   
      target 'RunnerTests' do

3. Now build the camera_avfoundation example app using Cocoapods:

   sudo xcode-select --switch /path/to/xcode/15.2/or/older
   flutter config --no-enable-swift-package-manager
   cd packages/camera/camera_avfoundation/example
   flutter clean
   flutter build ios
   

   This will error:

   Running Xcode build...
   Xcode build done.                                           16.1s
   Failed to build iOS app
   Parse Issue (Xcode): Build a shadowed submodule 'camera_avfoundation.messages_g'
   
   
   Encountered error while building for device.
   

Minimal reproduction

1. Follow the simple reproduction steps above

2. Open the example app in Xcode 15.2 or older

3. In Project navigator, open Runner > Flutter > Debug. Append VERBOSE_SCRIPT_LOGGING=true.

4. Clean your build folder & do a build

5. Open the Report navigator and expand the failing step:

   

6. Copy the clang command.

7. In your terminal, and run the clang command. This should reproduce the build error:

   fatal error: build a shadowed submodule 'camera_avfoundation.FLTCamMediaSettingsAVWrapper'
   /Users/loicsharma/Library/Developer/Xcode/DerivedData/Runner-ajljfhewikpbjigrasflzcnbhont/Build/Intermediates.noindex/Pods.build/Debug-iphonesimulator/camera_avfoundation.build/module.modulemap:1:18: note: previous definition is here
   framework module camera_avfoundation {
                    ^
   1 error generated.
   

The build error is caused a clang invocation that compiles a .m file. The intermediate build folder included in this build invocation contains only the correct Cocoapods module map (it does not contain the Swift Package Manager module map).

Known workarounds

Use Xcode 15.3 or newer

The app builds fine using Cocoapods if you use Xcode 15.3.

If you edit the clang compile command from the minimal repro above to use Xcode 15.3's clang executable, the file compiles successfully.

Use Swift Package Manager

This build error only reproduces if you use Flutter's Cocoapods integration. This build error does not reproduce if you enable Swift Package Manager.

Remove use_frameworks! from your Podfile

This build error only reproduces if your app's Podfile has use_frameworks!.

The camera_avfoundation example app builds fine if this line is removed.

Rename the Swift Package Manager module map

Flutter plugins that support Swift Package Manager and have custom module maps will have two .modulemap files.

For camera_avfoundation:

1. Cocoapods module map: packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/include/cocoapods_camera_avfoundation.modulemap
2. Swift package manager module map: packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/include/module.modulemap

If you rename or delete the Swift package manager module map, the build error no longer reproduces.

This indicates that clang is unexpectedly using the Swift Package Manager module map when Swift Package Manager is disabled.

Potential solution

It appears that the Swift package manager module map is used unexpectedly when Flutter's Swift package manager feature is disabled. This hypothesis would also explain bug flutter#148307.

I've created an experiment that separates the public header file directories for Swift package manager and Cocoapods: flutter/packages#6732

The camera_avfoundation plugin now has two directories containing public header files:

1. packages/camera/camera_avfoundation/Sources/camera_avfoundation/include - This is the Cocoapods public header directory
2. packages/camera/camera_avfoundation/Sources/camera_avfoundation/public - This is the SPM public header directory

This experiment appears to solve this issue.

However, this experiment has a drawback: it requires significant code duplication. I plan to do some follow-up experiments:

1. Reduce the number of public header files by converting them to private header files. This would be a breaking change but it does reduce the code duplication significantly
2. Use symlinks instead of duplicating header files. This would make it easier to keep header files synced across SPM and Cocoapods.

So I played around with this too, and I'm also not sure why it sometimes seems to use the module.modulemap and sometimes doesn't. Perhaps because module.modulemap is the implicit modulemap file.

I did discover that if you change the imports to be not-relative, except for messages.g.h, it works fine:

- #import "./include/camera_avfoundation/CameraPlugin.h"
+ #import "CameraPlugin.h"
#import "./include/camera_avfoundation/messages.g.h"

I don't understand why, though...

Since have 2 modulemaps seems to be an issue, we could also consider solving with tooling. An idea could be to update documentation to name the SwiftPM modulemap something else (not module.modulemap), perhaps plugin_name_swift_package.modulemap and then via Flutter tooling when using Swift Package Manager, create a symlink for module.modulemap to plugin_name_swift_package.modulemap. We'd need to figure out how to do that without altering pub-cache, though, since currently SwiftPM is using the plugin directly from pub-cache.

---

I think the solution of using separate directories for CocoaPods vs SwiftPM (flutter/packages#6732) could also work and may be better so it's independent of the Flutter CLI version, but I would probably do something like the structure below. Since SwiftPM is going to eventually become the new standard and we'll eventually deprecate CocoaPods, we want it to be the more simple and the CocoaPods can be more difficult.

Structure:
-> in snippet below represents a symlink

camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation-umbrella.h
camera_avfoundation/Sources/camera_avfoundation/include/module.modulemap
camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation/CameraPlugin.h
camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation/messages.g.h
other headers...

camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/camera_avfoundation-umbrella.h -> camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation-umbrella.h
camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/camera_avfoundation -> camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation
camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/cocoapods_camera_avfoundation.modulemap

Package.swift

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import PackageDescription

let package = Package(
  name: "camera_avfoundation",
  platforms: [
    .iOS("12.0")
  ],
  products: [
    .library(name: "camera-avfoundation", targets: ["camera_avfoundation"])
  ],
  dependencies: [],
  targets: [
    .target(
      name: "camera_avfoundation",
      dependencies: [],
      exclude: ["cocoapods_headers"],
      resources: [
        .process("Resources")
      ]
    )
  ]
)

camera_avfoundation.podspec

#
# To learn more about a Podspec see http://guides.cocoapods.org/syntax/podspec.html
#
Pod::Spec.new do |s|
  s.name             = 'camera_avfoundation'
  s.version          = '0.0.1'
  s.summary          = 'Flutter Camera'
  s.description      = <<-DESC
A Flutter plugin to use the camera from your Flutter app.
                       DESC
  s.homepage         = 'https://github.com/flutter/packages'
  s.license          = { :type => 'BSD', :file => '../LICENSE' }
  s.author           = { 'Flutter Dev Team' => '[email protected]' }
  s.source           = { :http => 'https://github.com/flutter/packages/tree/main/packages/camera_avfoundation' }
  s.documentation_url = 'https://pub.dev/packages/camera_avfoundation'
  s.source_files = 'camera_avfoundation/Sources/camera_avfoundation/**/*.{h,m}'
  s.exclude_files = 'camera_avfoundation/Sources/camera_avfoundation/include/**/*.h'
  s.public_header_files = 'camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/**/*.h'
  s.module_map = 'camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/cocoapods_camera_avfoundation.modulemap'
  s.dependency 'Flutter'

  s.platform = :ios, '12.0'
  s.pod_target_xcconfig = { 'DEFINES_MODULE' => 'YES' }
  s.resource_bundles = {'camera_avfoundation_privacy' => ['camera_avfoundation/Sources/camera_avfoundation/Resources/PrivacyInfo.xcprivacy']}
end

Pub doesn't support symlinks. Example error: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8747737440527014529/+/u/Run_package_tests/publishability/stdout

ERROR: Pub does not support publishing packages with directory symlinks: `/b/s/w/ir/x/w/packages/packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/cocoapods_headers/camera_avfoundation`.

Tracking issue: dart-lang/pub#3143
Pull request that attempted to add symlink support: dart-lang/pub#3298