foreman-install w/katello and Puppet

var_log_foreman-installer_katello.log

foreman-installer --scenario katello --enable-foreman-plugin-puppet --enable-foreman-cli-puppet --foreman-proxy-puppet true --foreman-proxy-puppetca true --foreman-proxy-content-puppet true --enable-puppet --puppet-server true --puppet-server-foreman-ssl-ca /etc/pki/katello/puppet/puppet_client_ca.crt --puppet-server-foreman-ssl-cert /etc/pki/katello/puppet/puppet_client.crt --puppet-server-foreman-ssl-key /etc/pki/katello/puppet/puppet_client.key
2022-10-21 00:31:14 [NOTICE] [root] Loading installer configuration. This will take some time.                                                                                                                                             
2022-10-21 00:31:18 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.                                                                                                                                      
2022-10-21 00:31:18 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.                                                                                   
2022-10-21 00:34:09 [NOTICE] [configure] Starting system configuration.                                                                                                                                                                    
2022-10-21 00:36:14 [NOTICE] [configure] 250 configuration steps out of 1807 steps complete.                                                                                                                                               
2022-10-21 00:36:56 [NOTICE] [configure] 500 configuration steps out of 1809 steps complete.                                                                                                                                               
2022-10-21 00:38:03 [NOTICE] [configure] 750 configuration steps out of 1815 steps complete.                                                                                                                                               
2022-10-21 00:38:33 [NOTICE] [configure] 1000 configuration steps out of 1838 steps complete.                                                                                                                                              
2022-10-21 00:39:51 [NOTICE] [configure] 1250 configuration steps out of 1838 steps complete.                                                                                                                                              
2022-10-21 00:40:18 [ERROR ] [configure] Systemd start for puppetserver failed!                                                                                                                                                            
2022-10-21 00:40:18 [ERROR ] [configure] journalctl log for puppetserver:                                                                                                                                                                  
2022-10-21 00:40:18 [ERROR ] [configure] -- Logs begin at Fri 2022-10-21 00:30:33 UTC, end at Fri 2022-10-21 00:40:18 UTC. --                                                                                                              
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:09 foreman.fm.libvirt.vm systemd[1]: Starting puppetserver Service...                                                                                                                
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:12 foreman.fm.libvirt.vm puppetserver[13920]: WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs                     
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:17 foreman.fm.libvirt.vm puppetserver[13920]: Failed to load feature test for posix: can't find user for 0                                                                           
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:17 foreman.fm.libvirt.vm puppetserver[13920]: Execution error (RuntimeError) at RUBY/<main> (/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb:19).                 
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:17 foreman.fm.libvirt.vm puppetserver[13920]: (Error) Cannot determine basic system flavour                                                                                          
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:17 foreman.fm.libvirt.vm puppetserver[13920]: Full report at:                                                                                                                        
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:17 foreman.fm.libvirt.vm puppetserver[13920]: /tmp/clojure-7259191360659662110.edn                                                                                                   
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:18 foreman.fm.libvirt.vm systemd[1]: puppetserver.service: Control process exited, code=exited status=1                                                                              
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:18 foreman.fm.libvirt.vm systemd[1]: puppetserver.service: Failed with result 'exit-code'.                                                                                           
2022-10-21 00:40:18 [ERROR ] [configure] Oct 21 00:40:18 foreman.fm.libvirt.vm systemd[1]: Failed to start puppetserver Service.                                                                                                           
2022-10-21 00:40:18 [ERROR ] [configure] /Stage[main]/Puppet::Server::Service/Service[puppetserver]/ensure: change from 'stopped' to 'running' failed: Systemd start for puppetserver failed!

Red Hat Enterprise Linux 8.6 (FIPS 140-2)
Foreman 3.4
Katello 4.6
Puppet 7

I have tried the same process on a non-FIPS RHEL 8 install and an Alma Linux 8 install, all with the above error.

foreman-installer --scenario katello \
--enable-foreman-plugin-puppet \
--enable-foreman-cli-puppet \
--foreman-proxy-puppet true \
--foreman-proxy-puppetca true \
--foreman-proxy-content-puppet true \
--enable-puppet \
--puppet-server true \
--puppet-server-foreman-ssl-ca /etc/pki/katello/puppet/puppet_client_ca.crt \
--puppet-server-foreman-ssl-cert /etc/pki/katello/puppet/puppet_client.crt \
--puppet-server-foreman-ssl-key /etc/pki/katello/puppet/puppet_client.key

/tmp/clojure edn

{:clojure.main/message
 "Execution error (RuntimeError) at RUBY/<main> (/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb:19).\n(Error) Cannot determine basic system flavour\n",
 :clojure.main/triage
 {:clojure.error/class org.jruby.exceptions.RuntimeError,
  :clojure.error/line 19,
  :clojure.error/cause "(Error) Cannot determine basic system flavour",
  :clojure.error/symbol RUBY/<main>,
  :clojure.error/source
  "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb",
  :clojure.error/phase :execution},
 :clojure.main/trace
 {:via
  [{:type java.lang.IllegalStateException,
    :message "There was a problem adding a JRubyInstance to the pool.",
    :at
    [puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__34563$add_instance__34568$fn__34572
     invoke
     "jruby_agents.clj"
     58]}
   {:type org.jruby.embed.EvalFailedException,
    :message "(Error) Cannot determine basic system flavour",
    :at
    [org.jruby.embed.internal.EmbedEvalUnitImpl
     run
     "EmbedEvalUnitImpl.java"
     131]}
   {:type org.jruby.exceptions.RuntimeError,
    :message "(Error) Cannot determine basic system flavour",
    :at
    [RUBY
     <main>
     "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb"
     19]}],
  :trace
  [[RUBY
    <main>
    "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/feature/base.rb"
    19]
   [org.jruby.RubyKernel require "org/jruby/RubyKernel.java" 1017]
   [org.jruby.RubyKernel
    require_relative
    "org/jruby/RubyKernel.java"
    1045]
   [RUBY
    <module:Puppet>
    "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb"
    94]
   [RUBY
    <main>
    "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb"
    41]
   [org.jruby.RubyKernel require "org/jruby/RubyKernel.java" 1017]
   [RUBY
    require
    "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb"
    85]
   [RUBY <main> "uri:classloader:/puppetserver-lib/puppet/server.rb" 1]
   [org.jruby.RubyKernel require "org/jruby/RubyKernel.java" 1017]
   [RUBY
    require
    "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb"
    85]
   [RUBY
    <main>
    "uri:classloader:/puppetserver-lib/puppet/server/master.rb"
    1]
   [org.jruby.RubyKernel require "org/jruby/RubyKernel.java" 1017]
   [RUBY
    require
    "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb"
    85]
   [RUBY <main> "<script>" 1]],
  :cause "(Error) Cannot determine basic system flavour"}}

Root cause: /tmp was mounted with noexec (due to CIS compliance).

Solution:

• mkdir /opt/puppetlabs/server/apps/puppetserver/tmp
• Add -Djava.io.tmpdir=/opt/puppetlabs/server/apps/puppetserver/tmp to JAVA_ARGS in /etc/sysconfig/puppetserver
• chcon system_u:object_r:usr_t:s0 /opt/puppetlabs/server/apps/puppetserver/tmp
• chown puppet: /opt/puppetlabs/server/apps/puppetserver/tmp
• systemctl restart puppetserver

I also added the following options to foreman-installer (persist JAVA_ARGS changes and use JDK 11 with ParallelGC):

--puppet-server-jvm-extra-args "-Djava.io.tmpdir=/opt/puppetlabs/server/apps/puppetserver/tmp -XX:+UseParallelGC" \
--puppet-server-jvm-java-bin "/usr/lib/jvm/jre-11/bin/java"