zone_prov_issue

I have an image that fails to boot on the latest PI, but only when provisioned on the latest PI.
If I provision the same image on an older platform, (20150622T171240Z), and then reboot to the
latest PI the zone comes up fine.


The crux of the issue appears to be when provisioning on the new PI the SMF service
system/boot-archive is missing.

For refernce base64 14.2.0 and base64 13.2.0 provision fine. The issue is with a custom image.
Which is base64 14.2.0 essentially with puppet installed ready to go.


As below:

[root@broken-dev ~]# svccfg list | grep boot
[root@broken-dev ~]#


[root@broken-dev ~]# svcs -xv
svc:/system/filesystem/usr:default (read/write root file systems mounts)
 State: offline since Thu Nov  5 13:10:39 2015
Reason: Dependency svc:/system/boot-archive is absent.
   See: http://illumos.org/msg/SMF-8000-E2
Impact: 37 dependent services are not running:
        svc:/network/ipsec/ipsecalgs:default
        svc:/milestone/network:default
        svc:/network/initial:default
        svc:/network/service:default
        svc:/network/dns/client:default
        svc:/milestone/single-user:default
        svc:/milestone/sysconfig:default
        svc:/milestone/multi-user:default
        svc:/system/zoneinit:default
        svc:/smartdc/mdata:execute
        svc:/milestone/multi-user-server:default
        svc:/network/inetd:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/network/ssh:default
        svc:/system/filesystem/local:default
        svc:/system/cron:default
        svc:/pkgsrc/rsyslog:default
        svc:/network/ipsec/policy:default
        svc:/network/iptun:default
        svc:/system/device/local:default
        svc:/system/filesystem/minimal:default
        svc:/system/manifest-import:default
        svc:/network/routing-setup:default
        svc:/system/coreadm:default
        svc:/system/rbac:default
        svc:/system/name-service-cache:default
        svc:/system/rmtmpfiles:default
        svc:/system/identity:domain
        svc:/system/hostid:default
        svc:/system/logadm-upgrade:default
        svc:/milestone/devices:default
        svc:/system/filesystem/smartdc:default
        svc:/network/physical:default
        svc:/smartdc/mdata:fetch
        svc:/system/keymap:default
        svc:/system/pfexec:default

svc:/network/physical:nwam (?)
 State: disabled since Thu Nov  5 13:10:39 2015
Reason: Disabled by an administrator.
   See: http://illumos.org/msg/SMF-8000-05
Impact: 19 dependent services are not running:
        svc:/milestone/network:default
        svc:/network/initial:default
        svc:/network/service:default
        svc:/network/dns/client:default
        svc:/milestone/single-user:default
        svc:/milestone/sysconfig:default
        svc:/milestone/multi-user:default
        svc:/system/zoneinit:default
        svc:/smartdc/mdata:execute
        svc:/milestone/multi-user-server:default
        svc:/network/inetd:default
        svc:/system/utmp:default
        svc:/system/console-login:default
        svc:/network/ssh:default
        svc:/system/filesystem/local:default
        svc:/system/cron:default
        svc:/pkgsrc/rsyslog:default
        svc:/network/routing-setup:default
        svc:/network/iptun:default
[root@broken-dev ~]#


If i look at an image that was provisiond on the older platform release,
now running on the new release with out issue the service is present:

[root@builder ~]# svcs | grep boot
online         12:10:40 svc:/system/boot-archive:default



I can't see how/what has changed that would affect this, my first
thought was that the brand definition had changed. But looking at
smartos-live/overlay/generic/usr/lib/brand/joyent the manifests
file has had system/boot-archive.xmlcommented out at least as far
back as 2013.


So how does this service make it into the SMF repository on older
platforms at provision time, but not newer ?


Looking at a broken zone in /etc/svc it looks like something snapshots the SMF
repository every boot. This is handy, i can see the repository.db file as
it was when the image was created (boot-archive) present.

[root@hyp2 (dev) /zones/ed3c477c-83cf-11e5-a491-23f5529e6654/root/etc/svc]# ls
profile                          repository-boot-20141002_104544  repository-boot-20151105_151439  repository.db
repository-boot                  repository-boot-20141002_104606  repository-boot-20151105_151500  volatile

And the repository.db as it was when the zone was created at first boot prior to provisioning

In this case file repository-boot-20151105_151439, again boot-archive is present.

And then finally after the provision has happened and the VM reboots,
repository-boot-20151105_151500 boot-archive SMF service had gone AWOL


Runing diff -r against the broken provisioned zone, and the image directory the below was spotted in one of the SMF logs

diff -r root//var/svc/log/system-manifest-import:default.log /zones/a545a234-4a22-11e4-8f84-879a091260f2/root//var/svc/log/system-manifest-import:default.log
1,17c1,2
< [ Nov  5 15:14:40 Enabled. ]
< [ Nov  5 15:14:41 Executing start method ("/lib/svc/method/manifest-import"). ]
< [ Nov  5 15:14:41 Timeout override by svc.startd.  Using infinite timeout. ]
< svccfg: Loaded 1 smf(5) service descriptions
< /etc/svc system profiles not found: upgrade system profiles
< svccfg: Multiple definitions for property value_F5SGK5RPMNXW443PNRSQ----_description_C in property group tm_proppat_nt_ttymon_device.
< Delete service svc:/network/ldap/client as there are no supporting manifests
< Delete service svc:/network/nfs/mapid as there are no supporting manifests
< Delete service svc:/network/nfs/status as there are no supporting manifests
< Delete service svc:/network/nis/client as there are no supporting manifests
< Delete service svc:/network/nfs/client as there are no supporting manifests
< Delete service svc:/network/nfs/nlockmgr as there are no supporting manifests
< Delete service svc:/system/boot-archive as there are no supporting manifests
< Delete service svc:/system/sac as there are no supporting manifests
< Delete service svc:/system/sysidtool as there are no supporting manifests
< [ Nov  5 15:14:42 Method "start" exited with status 0. ]
< [ Nov  5 15:15:00 Enabled. ]