Notes on how to query all pubspecs on `pub.dev`.

query-all-pubspecs.md

Query pubspec.yaml for all packages

When considering introduction of a new field in pubspec.yaml, or wondering how many packages use a certain feature, it can be useful to query all pubspecs from pub.dev. The following notes have some scripts to make this easy.

File yaml2json.sh

Helper script to convert YAML to JSON.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

python2 -c 'import sys, yaml, json; y=yaml.load(sys.stdin.read(),Loader=yaml.SafeLoader); print json.dumps(y)'

File fetch_pubspec.sh

Helper script to fetch the pubspec for a package into the all-pubspecs-json/ folder.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

if [ ! -f "all-pubspecs/$1.yaml" ]; then
  echo "$1";
  URL=`curl -s "https://pub.dartlang.org/api/packages/$1/" | jq -r '.latest.archive_url'`
  curl -s -L "$URL" | tar -zxO pubspec.yaml | ./yaml2json.sh | jq -r . | tee > "all-pubspecs-json/$1.yaml"
fi

File fetch_all_pubspecs.sh

Create all-pubspecs-json/ folder and download all pubspecs into this folder as JSON files.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

mkdir -p all-pubspecs-json
curl 'https://pub.dartlang.org/api/packages?compact=1' | jq -r '.packages[]' | parallel -j 200 ./fetch_pubspec.sh {}

File query_pubspecs.sh

Query over all pubspecs from the all-pubspecs-json/ folder in parallel using jq queries.

#!/bin/bash -e

# Copyright 2020 Google LLC.
# SPDX-License-Identifier: Apache-2.0

if [ "$1" == "" ]; then
  echo 'Usage: query_pubspecs.sh <jq-query>';
  exit 1;
fi;


export QUERY="$1"
process() {
  VAL=`cat "all-pubspecs-json/$1" | jq -r "$QUERY"`;
  if [ "$VAL" != "null" ]; then
    echo "$1: $VAL";
  fi;
}
export -f process

ls all-pubspecs-json | parallel -j 20 process {};