Generating and testing precompile statements for COSMO + Convex + MOI

README.md

Generating and testing precompile statements for COSMO + Convex + MOI

1. I generated precompile statements via generate_precompiles.jl.
2. Then I dev'd each of Convex, MOI, and COSMO, and added the precompile statements. (See the SnoopCompile docs).
3. I removed the ProblemDepot specific precompiles from Convex, and added the SparseArray, Base (but minus the printing ones), and LinearAlgebra (minus isapprox) precompiles to Convex too
4. Then I ran test_precompiles.jl, resulting in output.txt
5. I ran it a second time, and saved the file as output2.txt.

E.g.,

include("generate_precompiles.jl")

# Now `dev` Convex, MOI, and COSMO, and add the precompile statements as above

include("test_precompiles.jl")

Effect on "first-solve" time

See output.txt and output2.txt for two runs of the timing test (which just solves one problem via readme_test). It seems just adding precompile statements to COSMO reduces the first-solve time by 15-25 seconds. Adding precompile statements to MOI and Convex reduces it by another 4-8 seconds.

Effect on precompilation time

Adding precompile statements will naturally increase precompilation time.

With the precompile statements, I get the following timings:

ephwork:COSMO eh540$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using COSMO
[ Info: Precompiling COSMO [1e616198-aa4e-51ec-90a2-23f7fbd31d8d]
 46.775588 seconds (18.79 M allocations: 1.028 GiB, 0.63% gc time)

julia> @time using COSMO
  0.520894 seconds (1.04 M allocations: 49.060 MiB, 2.87% gc time)

julia> @time using COSMO
  0.000238 seconds (359 allocations: 19.828 KiB)

ephwork:COSMO eh540$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using COSMO
 11.267857 seconds (17.62 M allocations: 995.843 MiB, 2.89% gc time)

julia> @time using COSMO
  0.744716 seconds (1.04 M allocations: 49.060 MiB, 2.06% gc time)

julia> @time using COSMO
  0.000240 seconds (359 allocations: 19.828 KiB)

Without precompile statements, I get the following times:

ephwork:COSMO eh540$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using COSMO
[ Info: Precompiling COSMO [1e616198-aa4e-51ec-90a2-23f7fbd31d8d]
 21.610615 seconds (17.64 M allocations: 971.520 MiB, 0.84% gc time)

julia> @time using COSMO
  0.537329 seconds (1.04 M allocations: 49.061 MiB, 2.90% gc time)

julia> @time using COSMO
  0.000260 seconds (359 allocations: 19.828 KiB)

julia> 
ephwork:COSMO eh540$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using COSMO
  8.708156 seconds (16.46 M allocations: 914.094 MiB, 1.77% gc time)

julia> @time using COSMO
  0.503618 seconds (1.04 M allocations: 49.061 MiB)

julia> @time using COSMO
  0.000420 seconds (359 allocations: 19.828 KiB)

readme_test.jl

using Convex, COSMO

# Generate random problem data
m = 4;  n = 5
A = randn(m, n); b = randn(m, 1)

# Create a (column vector) variable of size n x 1.
x = Variable(n)

# The problem is to minimize ||Ax - b||^2 subject to x >= 0
# This can be done by: minimize(objective, constraints)
problem = minimize(sumsquares(A * x - b), [x >= 0])

# Solve the problem by calling solve!
@time solve!(problem, COSMO.Optimizer(verbose=false))

# Check the status of the problem
problem.status # :Optimal, :Infeasible, :Unbounded etc.

# Get the optimal value
problem.optval

generate_precompiles.jl

using SnoopCompile
inf_timing = @snoopi tmin=0.01 include("precompile_script.jl")
pc = SnoopCompile.parcel(inf_timing)
SnoopCompile.write("precompiles", pc)

Manifest.toml

# This file is machine-generated - editing it directly is not advised

[[AMD]]
deps = ["Libdl", "LinearAlgebra", "SparseArrays", "Test"]
git-tree-sha1 = "7b8e22e91af4ccbfbcae87c36d141441defe6f0f"
uuid = "14f7f29c-3bd6-536c-9a0b-7339e30b5a3e"
version = "0.3.1"

[[AbstractTrees]]
deps = ["Markdown"]
git-tree-sha1 = "86d092c2599f1f7bb01668bf8eb3412f98d61e47"
uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
version = "0.3.2"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[BenchmarkTools]]
deps = ["JSON", "Logging", "Printf", "Statistics", "UUIDs"]
git-tree-sha1 = "9e62e66db34540a0c919d72172cc2f642ac71260"
uuid = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
version = "0.5.0"

[[BinaryProvider]]
deps = ["Libdl", "SHA"]
git-tree-sha1 = "5b08ed6036d9d3f0ee6369410b830f8873d4024c"
uuid = "b99e7846-7c00-51b0-8f62-c81ae34c0232"
version = "0.5.8"

[[COSMO]]
deps = ["DataStructures", "LinearAlgebra", "MathOptInterface", "Pkg", "Printf", "QDLDL", "Random", "SparseArrays", "Statistics", "SuiteSparse", "Test", "UnsafeArrays"]
git-tree-sha1 = "bbde44383ff077d4796f3819436a50f787b69d00"
uuid = "1e616198-aa4e-51ec-90a2-23f7fbd31d8d"
version = "0.7.0"

[[CodecBzip2]]
deps = ["BinaryProvider", "Libdl", "TranscodingStreams"]
git-tree-sha1 = "5db086e510c11b4c87d05067627eadb2dc079995"
uuid = "523fee87-0ab8-5b00-afb7-3ecf72e48cfd"
version = "0.6.0"

[[CodecZlib]]
deps = ["BinaryProvider", "Libdl", "TranscodingStreams"]
git-tree-sha1 = "05916673a2627dd91b4969ff8ba6941bc85a960e"
uuid = "944b1d66-785c-5afd-91f1-9de20f533193"
version = "0.6.0"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "ed2c4abadf84c53d9e58510b5fc48912c2336fbb"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "2.2.0"

[[Convex]]
deps = ["AbstractTrees", "BenchmarkTools", "LinearAlgebra", "MathOptInterface", "OrderedCollections", "SparseArrays", "Test"]
git-tree-sha1 = "f75e9538a9f91a7edb05913a5f18bac74601d51a"
uuid = "f65535da-76fb-5f13-bab9-19810c17039a"
version = "0.13.1"

[[DataStructures]]
deps = ["InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "5a431d46abf2ef2a4d5d00bd0ae61f651cf854c8"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.17.10"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[HTTP]]
deps = ["Base64", "Dates", "IniFile", "MbedTLS", "Sockets"]
git-tree-sha1 = "8d9bdd55c9d0d6ddf08f8b5229f90b7f274b6777"
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
version = "0.8.12"

[[IniFile]]
deps = ["Test"]
git-tree-sha1 = "098e4d2c533924c921f9f9847274f2ad89e018b8"
uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f"
version = "0.5.0"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "b34d7cef7b337321e97d22242c3c2b91f476748e"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.0"

[[JSONSchema]]
deps = ["BinaryProvider", "HTTP", "JSON"]
git-tree-sha1 = "b0a7f9328967df5213691d318a03cf70ea8c76b1"
uuid = "7d188eb4-7ad8-530c-ae41-71a32a6d4692"
version = "0.2.0"

[[LibGit2]]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MathOptInterface]]
deps = ["BenchmarkTools", "CodecBzip2", "CodecZlib", "JSON", "JSONSchema", "LinearAlgebra", "MutableArithmetics", "OrderedCollections", "SparseArrays", "Test", "Unicode"]
git-tree-sha1 = "f0d60e42d8b64dd1b511e2dc13e0b72ba1dfc9cf"
uuid = "b8f27783-ece8-5eb3-8dc8-9495eed66fee"
version = "0.9.12"

[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "a9e2221f06b42f56052f43ad7edecb01d0ef5ab4"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.1"

[[MbedTLS_jll]]
deps = ["Libdl", "Pkg"]
git-tree-sha1 = "066a4467008745eed36dad973ceb66405785a621"
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.16.0+1"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MutableArithmetics]]
deps = ["LinearAlgebra", "SparseArrays", "Test"]
git-tree-sha1 = "020d4f22e1151e0613edf91a56535379564c1ce8"
uuid = "d8a4904e-b15c-11e9-3269-09a3773c0cb0"
version = "0.2.7"

[[OrderedCollections]]
deps = ["Random", "Serialization", "Test"]
git-tree-sha1 = "c4c13474d23c60d20a67b217f1d7f22a40edf8f1"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.1.0"

[[Parsers]]
deps = ["Dates", "Test"]
git-tree-sha1 = "0c16b3179190d3046c073440d94172cfc3bb0553"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "0.3.12"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Test", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[QDLDL]]
deps = ["AMD", "LinearAlgebra", "Random", "SparseArrays", "Test"]
git-tree-sha1 = "bdd19e6df448f46d994c7db6c6e7d0dae8b587e8"
uuid = "bfc457fd-c171-5ab7-bd9e-d5dbfc242d63"
version = "0.1.3"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[SnoopCompile]]
deps = ["OrderedCollections", "Pkg", "Serialization"]
git-tree-sha1 = "ba4d5e2fbf54ca0068149301442df93abcfbb4c0"
uuid = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"
version = "1.2.3"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[SuiteSparse]]
deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"]
uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"

[[Test]]
deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TranscodingStreams]]
deps = ["Random", "Test"]
git-tree-sha1 = "7c53c35547de1c5b9d46a4797cf6d8253807108c"
uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
version = "0.9.5"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[UnsafeArrays]]
deps = ["Compat"]
git-tree-sha1 = "460f4f7bab69dd47512ac8ec4b2c14b28c6cb4a0"
uuid = "c4a57d5a-5b31-53a6-b365-19f8c011fbd6"
version = "0.3.0"

output.txt

("add Convex", "add COSMO", "add MathOptInterface")
 80.454257 seconds (113.68 M allocations: 5.571 GiB, 5.17% gc time)

("develop Convex", "add COSMO", "add MathOptInterface")
 61.071458 seconds (107.14 M allocations: 5.260 GiB, 5.18% gc time)

("add Convex", "develop COSMO", "add MathOptInterface")
 54.125889 seconds (92.13 M allocations: 4.549 GiB, 4.80% gc time)

("develop Convex", "develop COSMO", "add MathOptInterface")
 51.917905 seconds (85.67 M allocations: 4.240 GiB, 5.48% gc time)

("add Convex", "add COSMO", "develop MathOptInterface")
 77.087202 seconds (112.14 M allocations: 5.492 GiB, 4.71% gc time)

("develop Convex", "add COSMO", "develop MathOptInterface")
 66.320530 seconds (105.68 M allocations: 5.180 GiB, 5.18% gc time)

("add Convex", "develop COSMO", "develop MathOptInterface")
 63.775371 seconds (90.57 M allocations: 4.468 GiB, 4.55% gc time)

("develop Convex", "develop COSMO", "develop MathOptInterface")
 47.120227 seconds (84.28 M allocations: 4.164 GiB, 5.22% gc time)

output2.txt

("add Convex", "add COSMO", "add MathOptInterface")
 65.151737 seconds (113.55 M allocations: 5.566 GiB, 4.65% gc time)

("develop Convex", "add COSMO", "add MathOptInterface")
 60.208842 seconds (107.16 M allocations: 5.257 GiB, 5.01% gc time)

("add Convex", "develop COSMO", "add MathOptInterface")
 50.760969 seconds (92.08 M allocations: 4.547 GiB, 5.22% gc time)

("develop Convex", "develop COSMO", "add MathOptInterface")
 49.766633 seconds (85.63 M allocations: 4.237 GiB, 5.49% gc time)

("add Convex", "add COSMO", "develop MathOptInterface")
 64.258489 seconds (112.16 M allocations: 5.490 GiB, 4.64% gc time)

("develop Convex", "add COSMO", "develop MathOptInterface")
 61.297292 seconds (105.73 M allocations: 5.183 GiB, 5.89% gc time)

("add Convex", "develop COSMO", "develop MathOptInterface")
 57.630073 seconds (90.68 M allocations: 4.474 GiB, 5.17% gc time)

("develop Convex", "develop COSMO", "develop MathOptInterface")
 46.187760 seconds (84.28 M allocations: 4.164 GiB, 5.30% gc time)

precompile_script.jl

using Convex, COSMO

include("readme_test.jl")

# copied from ConvexTests.jl
Convex.ProblemDepot.run_tests(;exclude=[r"dual", r"mip"]) do p
    solve!(p, COSMO.Optimizer(verbose = false, decompose = false, max_iter = 40000, eps_abs = 5e-7, eps_rel = 5e-7))
end

Project.toml

[deps]
COSMO = "1e616198-aa4e-51ec-90a2-23f7fbd31d8d"
Convex = "f65535da-76fb-5f13-bab9-19810c17039a"
MathOptInterface = "b8f27783-ece8-5eb3-8dc8-9495eed66fee"
SnoopCompile = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"

test_precompiles.jl

using Pkg, Logging

_julia_cmd() = `$(Base.julia_cmd()) --color=yes --startup-file=no` # taken from @tkf's Run.jl

pkgs = ("Convex", "COSMO", "MathOptInterface")
script = abspath("readme_test.jl")


function run_timings(pkgs, script)
    fs = (Pkg.add, Pkg.develop)
    combinations = Iterators.product(Iterators.repeated(fs, length(pkgs))...) |> collect |> vec

    results = []
    for c in combinations
        mktempdir() do dir
            with_logger(NullLogger()) do # suppress logs
                mktemp() do path, io # temp file to send Pkg output
                    redirect_stdout(io) do
                        Pkg.activate(dir)
                        for (i, f) in enumerate(c)
                            f(pkgs[i]) # dev or add the package
                        end
                    end
                end
            end
            io = IOBuffer()
            run(pipeline(`$(_julia_cmd()) --project=$(dir) $script`; stdout = io))
            setup = string.(c) .* " " .* pkgs
            result = String(take!(io))
            push!(results, (setup, result))
            println(setup)
            println(result)
        end
    end
    return results
end

results = run_timings(pkgs, script)

open("output.txt", "w") do io
    foreach( r -> println.((io,), r), results)
end