Spack singularity package debugging session

I've fetched/merged your branch and am at commit ec296438fd41b18ea310ea83acf534e8f295a25a.

Here is the definition of source_path for a stage:

Here's the interesting bit, it's returning the first thing that it finds in the output of os.listdir that is a directory (order of things returned by listdir is unspecified):

for p in [os.path.join(self.path, f) for f in os.listdir(self.path)]:
    if os.path.isdir(p):
        return p

So, set a breakpoint at the very first line of "our" do_stage:

--- a/var/spack/repos/builtin/packages/singularity/package.py
+++ b/var/spack/repos/builtin/packages/singularity/package.py
@@ -52,6 +52,7 @@ class Singularity(MakefilePackage):
     # Unpack the tarball as usual, then move the src dir into
     # its home within GOPATH.
     def do_stage(self, mirror_only=False):
+        import pdb; pdb.set_trace()
         super(Singularity, self).do_stage(mirror_only)
         source_path = self.stage.source_path
         if not os.path.exists(self.singularity_gopath_dir):

Then do

spack clean -a
spack uninstall -a -y singularity
spack install [email protected]

I expected to find myself sitting at the call to super(...), but oddly enough I'm sitting at the assignment to source_path. Surprising, but lets keep on going.

Look at source_path:

==> No binary for singularity found: installing from source
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(57)do_stage()
-> source_path = self.stage.source_path
(Pdb) p self.stage.source_path
None
(Pdb)

Print out the stage dir:

(Pdb) p self.stage.path
'/home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr'
(Pdb)

In another window, see what's in that directory.

Hmmm, I see nothing (perhaps the call to super hasn't actually run yet and python's fibbing...):

[ghartzell@ft spack]$ ls /home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr
[ghartzell@ft spack]$

Take one step on the debugger, with n:

(Pdb) n
==> Fetching https://github.com/sylabs/singularity/releases/download/v3.1.1/singularity-3.1.1.tar.gz
###################################################################################################################################################################### 100.0%
==> Staging archive: /home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr/singularity-3.1.1.tar.gz
==> Created stage in /home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(58)do_stage()
-> if not os.path.exists(self.singularity_gopath_dir):
(Pdb)

Yep, there the Super() ran.

Now, in the other window, check the contest of the staging directory:

[ghartzell@ft spack]$ ls /home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr
singularity  singularity-3.1.1.tar.gz
[ghartzell@ft spack]$

Good, we fetched the tarball and unpacked it. Now check the value of source_path:

-> if not os.path.exists(self.singularity_gopath_dir):
(Pdb) p self.stage.source_path
'/home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr/singularity'
(Pdb)

That makes sense, it's the first/only subdir of the staging dir.

NOTE: I wonder what you see for this value and what's in the staging directory. I'm kind of betting that you see a directory named src and/or that source_dir is src If that's so, then we have a place to start digging.

Use n to step down to the mkdirp line, and check the value of source_path:

(Pdb) n
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(61)do_stage()
-> mkdirp(self.sylabs_gopath_dir)
(Pdb) p self.stage.source_path
'/home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr/singularity'
(Pdb)

Now take several steps (n, a bunch of times), until you're past the shutil.move and at the function's return:

(Pdb) n
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(62)do_stage()
-> shutil.move(source_path,
(Pdb) n
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(63)do_stage()
-> self.singularity_gopath_dir)
(Pdb) n
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(64)do_stage()
->
(Pdb) n
--Return--
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(64)do_stage()->None
->
(Pdb)

Now, in the other window, check the contents of the staging directory:

[ghartzell@ft spack]$ ls /home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr
singularity-3.1.1.tar.gz  src
[ghartzell@ft spack]$

Yay, it looks just like we'd want.

Unfortunately, I've stepped too far to be able to check source_path, but we can step through into the parent and check it there:

--Return--
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(64)do_stage()->None
->
(Pdb) p self.stage.source_dir
*** AttributeError: 'StageComposite' object has no attribute 'source_dir'
(Pdb) l
 59  	            tty.debug("Moving {0} to {1}".format(
 60  	                self.stage.source_path, self.singularity_gopath_dir))
 61  	            mkdirp(self.sylabs_gopath_dir)
 62  	            shutil.move(source_path,
 63  	                        self.singularity_gopath_dir)
 64  ->
 65  	    # MakefilePackage's stages use this via working_dir()
 66  	    @property
 67  	    def build_directory(self):
 68  	        return self.singularity_gopath_dir
 69
(Pdb) n
> /home/ghartzell/spack/lib/spack/spack/package.py(997)do_patch()
-> has_patch_fun = hasattr(self, 'patch') and callable(self.patch)
(Pdb) l
992
993  	        # Kick off the stage first.  This creates the stage.
994  	        self.do_stage()
995
996  	        # Package can add its own patch function.
997  ->	        has_patch_fun = hasattr(self, 'patch') and callable(self.patch)
998
999  	        # Get the patches from the spec (this is a shortcut for the MV-variant)
1000 	        patches = self.spec.patches
1001
1002 	        # If there are no patches, note it.
(Pdb) p self
<spack.pkg.builtin.singularity.Singularity object at 0x7f909b671e48>
(Pdb) p self.stage.source_path
'/home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr/src'
(Pdb)

YAY: also what I'd expect.

Note that I see the same thing if I unwind your "caching the stage.source_path value" line. If you've made it this far and everything's worked (it probably did, with your workaround in place), perhaps you can remove the workaround and step through it again?

--- a/var/spack/repos/builtin/packages/singularity/package.py
+++ b/var/spack/repos/builtin/packages/singularity/package.py
@@ -52,13 +52,13 @@ class Singularity(MakefilePackage):
     # Unpack the tarball as usual, then move the src dir into
     # its home within GOPATH.
     def do_stage(self, mirror_only=False):
+        import pdb; pdb.set_trace()
         super(Singularity, self).do_stage(mirror_only)
-        source_path = self.stage.source_path
         if not os.path.exists(self.singularity_gopath_dir):
             tty.debug("Moving {0} to {1}".format(
                 self.stage.source_path, self.singularity_gopath_dir))
             mkdirp(self.sylabs_gopath_dir)
-            shutil.move(source_path,
+            shutil.move(self.stage.source_path,
                         self.singularity_gopath_dir)

     # MakefilePackage's stages use this via working_dir()

How does your run through compare to the above?

hartzell/step-by-step.md