Skip to content

Instantly share code, notes, and snippets.

@kentux
Last active May 3, 2021 06:22
Show Gist options
  • Select an option

  • Save kentux/e6b2eca8633821202445baf0eea7e147 to your computer and use it in GitHub Desktop.

Select an option

Save kentux/e6b2eca8633821202445baf0eea7e147 to your computer and use it in GitHub Desktop.
Extract rpm list from rocky and centos repos to populate an sqlite database
#!/usr/bin/env bash
# Quentin Fourel <quentin@fourel.eu>
# Extract rpm list from rocky and centos repos to populate an sqlite database
# Script executed from a rocky server with a .repo file that contain all centos repos (suffixed by "Centos-")
# Database is created where you launch the script
set -euo pipefail
DB_FILE=rocky_centos_rpms.sqlite
# All repos
REPOSLIST="appstream appstream-source baseos baseos-source cr debuginfo devel extras extras-source fasttrack ha media-appstream media-baseos plus plus-source powertools"
# All Centos repos available
REPOSLIST="appstream appstream-source baseos baseos-source cr debuginfo devel extras extras-source fasttrack ha plus plus-source powertools"
# All Rocky repos available
REPOSLIST="appstream baseos extras ha powertools"
# Temp dir
WORKDIR="$(mktemp -d)"
function cleanup {
# Clean temp dir used in trap
rm -rf "${WORKDIR}"
}
# Force temp cleanup when script end
trap cleanup EXIT
function rpmslist2csv {
# Get rpm list for specified repo and write it in csv file formated
# Columns : Name, Arch, Version, Release1, Release2, Repo
local repo=${1}
echo "Create csv that contain rpms available in ${repo}"
dnf repository-packages "${repo}" list --all > "${WORKDIR}/${repo}.list"
perl -ne "s/^(\S+)\.(\S+)\s+(.+)\-(\d+)\.(\S+)\s+@?(\S+)\s*$/\1,\2,\3,\4,\5,\6\n/g&&print" "${WORKDIR}/${repo}.list" > "${WORKDIR}/${repo}.csv"
}
function sql_command {
# Launch sqlite3 command
echo -e "$1" | sqlite3 "${DB_FILE}"
}
function init_db {
# Init db (delete old db and create an empy one with "rpms" table
echo "Init DB"
if [[ -f "${DB_FILE}" ]]; then
rm "${DB_FILE}"
fi
sql_command "CREATE TABLE rpms(name,arch,version,release1,release2,repo);"
}
function csv2sqlite {
# Import csv file in sqlite db
local repo="${1}"
echo "Insert csv data in sqlite DB"
sql_command ".mode csv\n.import ${WORKDIR}/${repo}.csv rpms"
}
function rpmslist2sqlite {
# Combine rpmslist2csv & csv2sqlite functions
local repo=$1
rpmslist2csv "${repo}"
csv2sqlite "${repo}"
}
function main {
init_db
# Clean repos cache
dnf clean all 1>/dev/null
for repo in ${REPOSLIST}
do
rpmslist2sqlite "${repo}"
rpmslist2sqlite "Centos-${repo}"
done
echo "Done"
}
main
@kentux
Copy link
Author

kentux commented May 2, 2021

Before launching it (on rocky server), you just have to create /etc/yum.repos.d/Centos.repo that contain :

[Centos-appstream]
name=CentOS Linux $releasever - AppStream
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=AppStream&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-baseos]
name=CentOS Linux $releasever - BaseOS
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=BaseOS&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-cr]
name=CentOS Linux $releasever - ContinuousRelease
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=cr&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-debuginfo]
name=CentOS Linux $releasever - Debuginfo
baseurl=http://debuginfo.centos.org/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-devel]
name=CentOS Linux $releasever - Devel WARNING! FOR BUILDROOT USE ONLY!
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=Devel&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-extras]
name=CentOS Linux $releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-fasttrack]
name=CentOS Linux $releasever - FastTrack
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=fasttrack&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-ha]
name=CentOS Linux $releasever - HighAvailability
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=HighAvailability&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-media-baseos]
name=CentOS Linux $releasever - Media - BaseOS
baseurl=file:///media/CentOS/BaseOS
        file:///media/cdrom/BaseOS
        file:///media/cdrecorder/BaseOS
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-media-appstream]
name=CentOS Linux $releasever - Media - AppStream
baseurl=file:///media/CentOS/AppStream
        file:///media/cdrom/AppStream
        file:///media/cdrecorder/AppStream
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-plus]
name=CentOS Linux $releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-powertools]
name=CentOS Linux $releasever - PowerTools
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=PowerTools&infra=$infra
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial


[Centos-baseos-source]
name=CentOS Linux $releasever - BaseOS - Source
baseurl=http://vault.centos.org/$contentdir/$releasever/BaseOS/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-appstream-source]
name=CentOS Linux $releasever - AppStream - Source
baseurl=http://vault.centos.org/$contentdir/$releasever/AppStream/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-extras-source]
name=CentOS Linux $releasever - Extras - Source
baseurl=http://vault.centos.org/$contentdir/$releasever/extras/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[Centos-plus-source]
name=CentOS Linux $releasever - Plus - Source
baseurl=http://vault.centos.org/$contentdir/$releasever/centosplus/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

In the sqlite database created you can request all packages without "twin" : SELECT count(*) AS nb, name, arch, version, repo from rpms GROUP BY name, arch HAVING nb = 1;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment