Include only parts of others' repository as submodule in your project and arrange them to desired locations.

01_Overview.md

This gist provides an example of how to only include parts of others' repository as submodule in your project.

What is more, it will also show how to "put" the folders of submodule to desired location (e.g. under your project's root), without breaking the submodule.

Consider we have a repository

Simulation

• README.md

There is one another repository we want to use, but only the algorithm folder is needed.

Robot

• examples/
• tests/
• algorithms/
• README.md

Additionally, we would like the algorithms folder be put under the root of Simulation, not under the submodule folder Robot.

So the desired goal will be having a repository as this

Simulation

• README.md
• algorithms/ @xxx

02_Submodule.md

We start by adding the other repository as submodule as usual. For this example, we will put all the submodules under external folder.

cd Simulation
git submodule add "Robot_Git_URL" external/Robot

We then use sparse-checkout to include only part of the submodule folder. Note that sparse-checkout init has to be done inside the submodule folder

cd external/Robot
git sparse-checkout init --cone

After this step, there should be a new file named sparse-checkout in your project's .git/modules/external/Robot/info/

You may use sparse-checkout set to update which folder to keep, but this will also keep all the root files.

git sparse-checkout set "algorithm"

Since only need the folder is needed, we can turn to a more direct approach by setting the file itself.

cd ../..
echo "/algorithm/" >> .git/modules/<NameOfSubmodule>/info/sparse-checkout

That's it!

Just to test if it is properly set up

rm -rf external/Robot
git submodule update --init --recursive

03_Arrange.md

Although the submodule is added and tracked properly. It's usually cumbersome to go into many levels to find the scripts we need.

Let's assume we are using some python scripts in algorithm folder. We will need to do

from external.Robot.algorithm.ik import ik

while the ideal case should be

from algorithm.ik import ik

To this aim, it's desired to "move" the algorithm folder out to the root. But this will inevitably break the submodule connection. One elegant alternative is to use symbolic_links to create a virtual link in the desired location that links back to the actual folder.

In the end I will provide an example setup.py file to set up the symbolic_links, so that the folder could be directly access in the root level.

04_Conclusion.md

In conclusion, we use sparse-checkout to maintain only the folders we need from submodules. We then can use symbolic_links to "place" the folder to desired positions to be used.

setup.py

import os
import sys


# Define the paths
# Path to the external folder
EXTERNAL_DIR = os.path.join(os.getcwd(), "external", "robot")
# Root of your package
ROOT_DIR = os.getcwd()
LINKS = [("algorithm", "algorithm")]

def create_symbolic_links():
    for src, dest in LINKS:
        src_path = os.path.join(EXTERNAL_DIR, src)
        dest_path = os.path.join(ROOT_DIR, dest)

        # Remove existing link or file if it exists
        if os.path.exists(dest_path) or os.path.islink(dest_path):
            print(f"Removing existing path: {dest_path}")
            os.remove(dest_path)

        # Ensure the source exists
        if not os.path.exists(src_path):
            print(f"Source path does not exist: {src_path}")
            sys.exit(1)

        # Create symbolic link
        try:
            print(f"Linking {src_path} to {dest_path}")
            os.symlink(src_path, dest_path)
        except OSError as e:
            print(f"Error creating symlink: {e}")
            sys.exit(1)


if __name__ == "__main__":
    print("Setting up package by creating symbolic links...")
    create_symbolic_links()
    print("Setup complete.")

Great write-up.

One note, if the submodule you're attempting to checkout has submodules of its own (that you don't want/need), don't include --recursive in the git submodule update --init command.