Skip to content

Instantly share code, notes, and snippets.

@mnalis
Created August 25, 2022 00:48
Show Gist options
  • Save mnalis/f77bbf938230616960ed3947fbcaa212 to your computer and use it in GitHub Desktop.
Save mnalis/f77bbf938230616960ed3947fbcaa212 to your computer and use it in GitHub Desktop.
StreetComplete 45.0-alpha1 invalid shoulder=* tag detection (OSM changeset parsing in shell)
### Detecting StreetComplete AddShoulder corruption from SC/Zazolc 45.0-alpha1
### or: using hacky shell commands for (rapid, but noncompliant XML) fun and profit
### See https://github.com/streetcomplete/StreetComplete/issues/4170
% time pbzip2 -dc changesets-220815.osm.bz2 | env -i LC_ALL=C grep -B1 -A2 'StreetComplete:quest_type.*AddShoulder' > shoulder_quest.txt
pbzip2 -dc changesets-220815.osm.bz2 1579.50s user 14.95s system 379% cpu 7:00.41 total
env -i LC_ALL=C grep -B1 -A2 'StreetComplete:quest_type.*AddShoulder' > 47.80s user 10.19s system 13% cpu 7:00.40 total
% fgrep -c changeset shoulder_quest.txt
23333
% fgrep -c quest_type shoulder_quest.txt
23333
% fgrep -c created_by shoulder_quest.txt
23333
% fgrep -B3 '45.0-alpha1' shoulder_quest.txt | fgrep '<changeset' | wc -l
32
% fgrep -B3 '45.0-alpha1' shoulder_quest.txt | sed -nEe 's/^.*user="([^"]*)".*$/\1/p' | sort | uniq -c | sort -nr
5 TS-R
5 David Dean
4 mcliquid
4 fghj753
2 mnalis ALTernative
2 Mateusz Konieczny
2 legofahrrad
2 hiierundda
1 westnordost
1 ldonetc
1 Katzenstreu
1 Jean-Baptiste CHARRON
1 endim8
1 confusedbuffalo
% fgrep -B3 '45.0-alpha1' shoulder_quest.txt | sed -nEe 's/^.*changeset id="([^"]*)".*$/\1/p' | tee shoulder_chanesets.txt
122408830
122497877
122529357
122532873
122538293
122544257
122555287
122555406
122556260
122557403
122558693
122564903
122570719
122576976
122578908
122600321
122607032
122643404
122661053
122683508
122689085
122714515
122779581
122794907
122804550
122814611
122817709
123116676
123127612
123195865
123242374
123482596
% xargs -i wget -q https://www.openstreetmap.org/api/0.6/changeset/{}/download -O shoulder_changeset_{}.txt < shoulder_chanesets.txt
% fgrep 'k="shoulder"' shoulder_changeset_* | egrep -v 'v="no|both"'
shoulder_changeset_122570719.txt: <tag k="shoulder" v="right"/>
shoulder_changeset_122570719.txt: <tag k="shoulder" v="right"/>
shoulder_changeset_122804550.txt: <tag k="shoulder" v="right"/>
# This ways need deleting the shoulder (values not in "both|no"):
% xmlstarlet sel -T -t -m "//tag[@k='shoulder' and @v!='both' and @v!='no']" -v '../@id' -o " " shoulder_changeset_*
93167152 424095545 4692786
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment