Skip to content

Instantly share code, notes, and snippets.

@milnak
Created September 4, 2025 16:28
Show Gist options
  • Save milnak/95afc6199082789cfd5780baaa2f5fda to your computer and use it in GitHub Desktop.
Save milnak/95afc6199082789cfd5780baaa2f5fda to your computer and use it in GitHub Desktop.
Mirror a website using wget
param (
[Parameter(Mandatory)][string]$Domain,
[string]$WgetCommand = 'wget.exe'
)
Get-Command -Name $WgetCommand -CommandType Application -ErrorAction Stop | Out-Null
$TargetPath = '{0}_{1}' -f $Domain, (Get-Date -Format 'yyMMdd')
if (Test-Path -LiteralPath $TargetPath) {
Write-Warning "Path '$TargetPath' already exists. Can't continue."
exit
}
mkdir $TargetPath -ErrorAction Stop
Push-Location $TargetPath
# --exclude-domains=LIST comma-separated list of rejected domains
# -E, --adjust-extension save HTML/CSS documents with proper extensions
# -H, --span-hosts go to foreign hosts when recursive
# -k, --convert-links make links in downloaded HTML or CSS point to local files
# -l, --level=NUMBER maximum recursion depth (inf or 0 for infinite)
# -nv, --no-verbose turn off verboseness, without being quiet
# -p, --page-requisites get all images, etc. needed to display HTML page
# -r, --recursive specify recursive download
# -U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION
# -X, --exclude-directories=LIST list of excluded directories
$wgetArgs = `
'--exclude-domains=googleapis.com,youtube.com', `
'--span-hosts', `
'--convert-links', `
'--level=inf', `
'--no-verbose', `
'--page-requisites', `
'--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"', `
"http://$Domain"
$process = Start-Process -FilePath $WgetCommand -ArgumentList $wgetArgs -NoNewWindow -Wait -PassThru
$exitCode = $process.ExitCode
if ($exitCode -ne 0) {
Write-Warning "'$WgetCommand' failed, error $exitCode"
exit
}
Pop-Location
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment