Apple EFI Package Grabber

Apple_EFI_Grab.md

Apple EFI Package Grabber

Description

Parses user-provided (DB) list of Apple Software Update CatalogURL .sucatalog links and saves all newer (since last run) EFI firmware package links into a text file. It removes any xml formatting, ignores false positives, removes duplicate links and sorts them in alphabetical order for easy comparison afterwards.

Usage

First, you need to familiarize a bit with the DB (i.e. Apple_EFI_Grab.txt file). It consists of 3 sections: Last run DateTime (YYYY-MM-DD HH:MM:SS), Sucatalog links to check and EFI Package links which have been gathered so far across all runs. Before running the utility for the fist time, you need to insert the Sucatalog links into the DB, below the 1st line (DateTime). The Sucatalog links in the DB are stored in partial form, starting from "index" string. For example: "https://swscan.apple.com/content/catalogs/others/index-12.merged-1.sucatalog" must be stored as "index-12.merged-1.sucatalog" in the DB. If the Sucatalog links are not pre-included in the DB or are outdated, you can find them by searching online (e.g. https://github.com/zhangyoufu/swscan.apple.com/blob/master/url.txt).

Compatibility

Should work at all Windows, Linux or macOS operating systems which have at least Python 3.7 support.

Prerequisites

To run the python script, you need to have the following 3rd party Python module installed:

• urllib3

pip3 install urllib3

Apple_EFI_Grab.py

#!/usr/bin/env python3
#coding=utf-8

"""
Apple EFI Grab
Apple EFI Package Grabber
Copyright (C) 2018-2024 Plato Mavropoulos
"""

title = 'Apple EFI Package Grabber v2.6'

print('\n' + title)

import sys

sys_ver = sys.version_info
if sys_ver < (3,7):
    sys.stdout.write('\n\nError: Python >= 3.7 required, not %d.%d!\n' % (sys_ver[0], sys_ver[1]))
    (raw_input if sys_ver[0] <= 2 else input)('\nPress enter to exit') # pylint: disable=E0602
    sys.exit(1)

import traceback

def show_exception_and_exit(exc_type, exc_value, tb):
    if exc_type is KeyboardInterrupt:
        print('\nNote: Keyboard Interrupt!')
    else:
        print('\nError: %s crashed, please report the following:\n' % title)
        traceback.print_exception(exc_type, exc_value, tb)
        input('\nPress enter to exit')
    
    sys.exit(1)

sys.excepthook = show_exception_and_exit

import urllib3
import datetime

def fetch_cat_info(name):
    mod = None
    
    url = cat_url[:-len('others/')] + name if name in ['index.sucatalog','index-1.sucatalog'] else cat_url + name
    
    for _ in range(retry_count):
        try:
            mod = urllib3.PoolManager().request('HEAD', url, retries=retry_count).headers['Last-Modified']
        except:
            continue
        break
    else:
        print(f'\nSkipped {name}, could not get info!')
    
    return name, url, mod

def fetch_cat_links(cat_file):
    cat_links = []
    cat_data = b''
    
    for _ in range(retry_count):
        try:
            cat_data = urllib3.PoolManager().request('GET', cat_file[1], retries=retry_count).data
        except:
            continue
        break
    else:
        print(f'\nSkipped {cat_file[0]}, could not get data!')
    
    cat_text = cat_data.decode('utf-8','ignore')
    
    if cat_data:
        for line in [l.strip() for l in cat_text.split('\n')]:
            if 'http' in line and ('.pkg' in line or '.tar' in line) and ('FirmwareUpd' in line or '/BridgeOSUpdateCustomer' in line or 'EFIUpd' in line or 'InstallAssistant' in line) \
            and 'Bluetooth' not in line and 'DPVGA' not in line and 'Thunderbolt' not in line and 'PMG5' not in line and 'HardDrive' not in line and 'AssistantAuto' not in line:
                down_link = line[line.find('http'):(line.find('.pkg') if '.pkg' in line else line.find('.tar')) + 4]
                
                down_link_hash_val = '0' * 40
                
                down_link_off = cat_text.find(down_link)
                
                down_link_dict_bgn = cat_text[:down_link_off].rfind('<dict>')
                down_link_dict_end_rel = cat_text[down_link_off:].find('</dict>')
                
                if down_link_dict_bgn != -1 and down_link_dict_end_rel != -1:
                    down_link_dict_end = down_link_off + down_link_dict_end_rel
                    
                    down_link_hash_key_rel = cat_text[down_link_dict_bgn:down_link_dict_end].find('<key>Digest</key>')
                    
                    if down_link_hash_key_rel != -1:
                        down_link_hash_key_off = down_link_dict_bgn + down_link_hash_key_rel
                        
                        down_link_hash_val_rel = cat_text[down_link_hash_key_off:].find('<string>')
                        
                        if down_link_hash_val_rel != -1:
                            down_link_hash_val_off = down_link_hash_key_off + down_link_hash_val_rel
                            
                            down_link_hash_val = cat_text[down_link_hash_val_off:].split('<string>')[1].split('</string>')[0]
                
                down_link = down_link.replace('http:','https:')
                
                cat_links.append([down_link, down_link_hash_val])
    
    return cat_links

dat_db = 'Apple_EFI_Grab.txt'
cat_url = 'https://swscan.apple.com/content/catalogs/others/'
apple_cat = []
down_links = []
server_date = None
retry_count = 20

with open(dat_db, 'r', encoding='utf-8') as dat: db_lines = dat.readlines()
db_lines = [line.strip('\n') for line in db_lines]

db_date = datetime.datetime.strptime(db_lines[0], '%Y-%m-%d %H:%M:%S')
db_links = {line for line in db_lines if line.startswith('https')}
db_hashes = {line.split('digest_')[1] for line in db_lines if line.startswith('digest_')}
db_sucat = [line for line in db_lines if line.startswith('index')]

print('\nGetting Catalog Listing...')

if not db_sucat:
    input('\nError: Failed to retrieve Catalogs from DB!\n\nDone!')
    sys.exit(1)

apple_mod = [fetch_cat_info(sucat) for sucat in db_sucat]

for name,url,mod in apple_mod:
    if mod is not None:
        dt = datetime.datetime.strptime(mod, '%a, %d %b %Y %H:%M:%S %Z')
        if not server_date or dt > server_date:
            server_date = dt
        
        apple_cat.append((name, url, dt))

if not server_date:
    input('\nError: Failed to retrieve Current Catalog Datetime!\n\nDone!')
    sys.exit(1)

print('\n    Previous Catalog Datetime :', db_date)
print('    Current Catalog Datetime  :', server_date)

if server_date <= db_date:
    input('\nNothing new since %s!\n\nDone!' % db_date)
    sys.exit()

print('\nGetting Catalog Links...')

down_links = [fetch_cat_links(cat) for cat in apple_cat]

down_items = [item for sublist in down_links for item in sublist]

down_links = [value[0] for value in down_items]

down_hashes = [value[1] for value in down_items]

if not down_links:
    input('\nError: Failed to retrieve Catalog Links!\n\nDone!')
    sys.exit(1)

new_links = set()

for down_item in down_items:
    down_item_link, down_item_hash = down_item

    if down_item_hash == '0' * 40:
        if down_item_link not in db_links:
            new_links.add(down_item_link)
    else:
        if down_item_hash not in db_hashes:
            new_links.add(down_item_link)

if new_links:
    print('\nFound %d new link(s) between %s and %s!' % (len(new_links), db_date, server_date))
    
    cur_date = datetime.datetime.now().isoformat(timespec='seconds').replace('-','').replace('T','').replace(':','') # Local UTC Unix
    
    with open('Apple_%s.txt' % cur_date, 'w', encoding='utf-8') as lout: lout.write('\n'.join(map(str, new_links)))
else:
    print('\nThere are no new links between %s and %s!' % (db_date, server_date))

new_db_sucat = '\n'.join(map(str, db_sucat))

new_db_links = '\n'.join(map(str, sorted(list(dict.fromkeys(down_links + list(db_links))))))

new_db_hashes = ''
for new_db_hash in sorted(set(down_hashes + list(db_hashes))):
    new_db_hashes += f'digest_{new_db_hash}\n'

new_db_lines = '%s\n\n%s\n\n%s\n\n%s' % (server_date, new_db_sucat, new_db_links, new_db_hashes)

with open(dat_db, 'w', encoding='utf-8') as dbout: dbout.write(new_db_lines)

input('\nDone!')

Apple_EFI_Grab.txt

2000-01-01 00:00:00

index.sucatalog
index-1.sucatalog
index-leopard.merged-1.sucatalog
index-snowleopard.merged-1.sucatalog
index-leopard-snowleopard.merged-1.sucatalog
index-lion.merged-1.sucatalog
index-lion-snowleopard-leopard.merged-1.sucatalog
index-mountainlion.merged-1.sucatalog
index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-mountainlionseed.merged-1.sucatalog
index-mountainlionseed-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.9.merged-1.sucatalog
index-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.9publicseed.merged-1.sucatalog
index-10.9publicseed-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.9seed.merged-1.sucatalog
index-10.9seed-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.10.merged-1.sucatalog
index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.10beta.merged-1.sucatalog
index-10.10beta-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.10seed.merged-1.sucatalog
index-10.10seed-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.10customerseed.merged-1.sucatalog
index-10.11.merged-1.sucatalog
index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.11beta.merged-1.sucatalog
index-10.11beta-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.11seed.merged-1.sucatalog
index-10.11seed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.11customerseed.merged-1.sucatalog
index-10.11customerseed-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.12.merged-1.sucatalog
index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.12beta.merged-1.sucatalog
index-10.12beta-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.12seed.merged-1.sucatalog
index-10.12seed-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.12customerseed.merged-1.sucatalog
index-10.12customerseed-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.13.merged-1.sucatalog
index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.13beta.merged-1.sucatalog
index-10.13beta-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.13seed.merged-1.sucatalog
index-10.13seed-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.13customerseed.merged-1.sucatalog
index-10.13customerseed-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.14.merged-1.sucatalog
index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.14beta.merged-1.sucatalog
index-10.14beta-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.14seed.merged-1.sucatalog
index-10.14seed-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.14customerseed.merged-1.sucatalog
index-10.14customerseed-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.15.merged-1.sucatalog
index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.15beta.merged-1.sucatalog
index-10.15beta-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.15seed.merged-1.sucatalog
index-10.15seed-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.15customerseed.merged-1.sucatalog
index-10.15customerseed-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.16.merged-1.sucatalog
index-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.16beta.merged-1.sucatalog
index-10.16beta-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.16seed.merged-1.sucatalog
index-10.16seed-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-10.16customerseed.merged-1.sucatalog
index-10.16customerseed-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-11.merged-1.sucatalog
index-11-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-12.merged-1.sucatalog
index-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-12beta.merged-1.sucatalog
index-12beta-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-12seed.merged-1.sucatalog
index-12seed-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-12customerseed.merged-1.sucatalog
index-12customerseed-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-13.merged-1.sucatalog
index-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-13beta.merged-1.sucatalog
index-13beta-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-13seed.merged-1.sucatalog
index-13seed-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-13customerseed.merged-1.sucatalog
index-13customerseed-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-14.merged-1.sucatalog
index-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-14beta.merged-1.sucatalog
index-14beta-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-14seed.merged-1.sucatalog
index-14seed-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-14customerseed.merged-1.sucatalog
index-14customerseed-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-15.merged-1.sucatalog
index-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-15beta.merged-1.sucatalog
index-15beta-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-15seed.merged-1.sucatalog
index-15seed-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog
index-15customerseed.merged-1.sucatalog
index-15customerseed-15-14-13-12-10.16-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog

LICENSE

Copyright (c) 2019-2024 Plato Mavropoulos

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Subject to the terms and conditions of this license, each copyright holder and contributor hereby grants to those receiving rights under this license a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except for failure to satisfy the conditions of this license) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer this software, where such license applies only to those patent claims, already acquired or hereafter acquired, licensable by such copyright holder or contributor that are necessarily infringed by:

(a) their Contribution(s) (the licensed copyrights of copyright holders and non-copyrightable additions of contributors, in source or binary form) alone; or

(b) combination of their Contribution(s) with the work of authorship to which such Contribution(s) was added by such copyright holder or contributor, if, at the time the Contribution is added, such addition causes such combination to be necessarily infringed. The patent license shall not apply to any other combinations which include the Contribution.

Except as expressly stated above, no rights or licenses from any copyright holder or contributor is granted under this license, whether expressly, by implication, estoppel or otherwise.

DISCLAIMER

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.