I've found that you can manually control which .dlls to reference and which .dlls to copy to output from a NuGet package if you turn off all assets on the PackageReference:
<PackageReference Include="Foo.MyPackage" Version="1.0.0" ExcludeAssets="all" GeneratePathProperty="true" />
and then manually reference the .dlls you need: