govuk scrape

$ paster govuk_publications --config=/var/ckan/ckan.ini scrape
...
After 2387/2387 pages:

Publications:
        Created: 94880 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consulta...
        Unchanged: 484 ['publications/the-ombudsmans-annual-report-and-accounts-2015-16', 'publications/rg1-8nh-kingfisher-colours-limited-environmental-permit-applicati...
        Updated: 83 ['publications/oil-and-gas-public-statements-relating-to-2014-operations', 'statistics/tabulation-tool-questionnaire-statistical-notice', 'publicat...
        Error - Incomplete publication - title: 7 ['statistics/womens-smoking-status-at-time-of-delivery-in-england-october-2014-to-december-2014', 'statistics/summary-hospital-level-mortality-indic...
        Error - Publication redirect: 1 ['publications/preventing-illegal-working-guidance-for-employers-october-2013']
        Time taken (h:m:s): 9:15:00

Fields:
        Format found (method 1): 183079 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publications/tra...
        Publish date found: 95447 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publica...
        Type found: 95447 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publica...
        Title found: 95447 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publica...
        Gov.uk ID found: 95447 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publica...
        Details found: 95411 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publica...
        Organization found: 95369 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/pet-travel-approved-air-sea-rail-and-charter -routes-for-the-movement-of-pets', 'publica...
        Summary found: 91994 ['publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publications/travel-with-assistance-dogs-transport-companies-and-routes', 'publications/technical-guidance-for-body-worn-video-bwv-devices-cast-2016', 'publications/international-child-abduction-and-...
        Attachments (embedded) found: 87736 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publications/travel-with-assistance-dogs-transport-companies-and-routes', 'publications/technical-guidance-for...
        Updated not found - check: 79837 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/technical-guidance-for-body-worn-video-bwv-devices-cast-2016', 'publications/so20-6jf-w...
        Ignoring "Part of" type /government/policies: 36517 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor /government/policies/radioactive-and-nuclear-substances-and-waste', 'publications/ofqual-programme-of-events /government/policies/school-and-college-qualifications-and-curriculum', 'publications/community-p...
        Updated found: 15610 ['publications/pet-travel-approved-air-sea-rail-and-charter-routes-for-the-movement-of-pets', 'publications/travel-with-assistance-dogs-transport-companies-and-routes', 'publications/international-child-abduction-and-contact-unit-application-form', 'publications/accreditation-of-gcses-as-a-lev...
        Format found (method 2): 12629 ['publications/current-catch-limits-over-10-metre-non-sector-pool', 'publications/current-catch-limits-over-10-metre-non-sector-pool', 'publications/current-catch-limits-over-10-metre-non-sector-pool', 'publications/current-catch-limits-over-10-metre-non-sector-pool', 'publications/current-cat...
        Publication external (method 1) so no attachments: 6487 ['consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'publications/ofqual-programme-of-events', 'statistics/magistrates-court-bulletin-july-to-september-2016', 'statistics/crown-court-bulletin-july-to-september-2016', 'statistics/families-and-h...
        Ignoring "Part of" type /government/world: 3670 ['publications/ofqual-programme-of-events /government/world/united-kingdom', 'publications/mozambique-consular-fees /government/world/mozambique', 'publications/belarus-consular-fees /government/world/belarus', 'publications/bereavement-information-for-macao /government/world/macao', 'publicati...
        Summary found (method 2): 3453 ['consultations/gda-of-hitachi-ge-nuclear-energy-ltds-uk-advanced-boiling-water-reactor', 'consultations/postgraduate-doctoral-loans', 'consultations/part-time-undergraduate-maintenance-loan', 'consultations/official-statistics-proposed-changes-to-defra-statistics', 'consultations/basement-deve...
        Collection summary found: 3367 ['assessing-new-nuclear-power-station-designs', 'centre-for-applied-science-and-technology-information', 'reform-of-as-and-a-level-qualifications-by-ofqual', 'reform-of-gcse-qualifications-by-ofqual', 'protected-food-name-scheme-uk-registered-products', 'tax-treaties', 'dwp-provider-guidance', ...
        Collection title found: 3367 ['assessing-new-nuclear-power-station-designs', 'centre-for-applied-science-and-technology-information', 'reform-of-as-and-a-level-qualifications-by-ofqual', 'reform-of-gcse-qualifications-by-ofqual', 'protected-food-name-scheme-uk-registered-products', 'tax-treaties', 'dwp-provider-guidance', ...
        Collection organization found: 3367 ['assessing-new-nuclear-power-station-designs', 'centre-for-applied-science-and-technology-information', 'reform-of-as-and-a-level-qualifications-by-ofqual', 'reform-of-gcse-qualifications-by-ofqual', 'protected-food-name-scheme-uk-registered-products', 'tax-treaties', 'dwp-provider-guidance', ...
        Ignoring "Part of" type /government/statistical-data-sets: 1447 ['statistics/vehicle-licensing-statistics-2015 /government/statistical-data-sets/veh04-licensed-light-goods-vehicles', 'statistics/vehicle-licensing-statistics-2015 /government/statistical-data-sets/veh05-licensed-heavy-goods-vehicles', 'statistics/vehicle-licensing-statistics-2015 /government/...
        Attachments not found - check: 1166 ['statistical-data-sets/unclaimed-estates-list', 'statistical-data-sets/veh02-licensed-cars', 'statistical-data-sets/ras45-quarterly-statistics', 'statistical-data-sets/outcome-of-unduly-lenient-sentence-referrals', 'statistical-data-sets/other-tb-statistics', 'statistical-data-sets/buses-stati...
        Organization title found: 863 ['environment-agency', 'department-for-education', 'animal-and-plant-health-agency', 'department-for-environment-food-rural-affairs', 'home-office', 'official-solicitor-and-public-trustee', 'ofqual', 'hm-passport-office', 'office-of-the-public-guardian', 'national-college-for-teaching-and-leader...
        Organization description found (external org): 618 ['monitor', 'the-parliamentary-and-health-service-ombudsman', 'forestry-commission', 'northern-ireland-court-service', 'department-of-justice-northern-ireland', 'office-for-national-statistics', 'office-of-the-chief-electoral-officer-for-northern-ireland', 'joint-nature-conservation-committee', ...
        Ignoring "Part of" type /government/topical-events: 608 ['publications/factsheet-the-uks-humanitarian-aid-response-to-the-syria-crisis /government/topical-events/supporting-syria-conference-2016', 'publications/action-plan-for-anti-money-laundering-and-counter-terrorist-finance /government/topical-events/anti-corruption-summit-london-2016', 'consulta...
        Organization description found: 226 ['environment-agency', 'department-for-education', 'animal-and-plant-health-agency', 'department-for-environment-food-rural-affairs', 'home-office', 'official-solicitor-and-public-trustee', 'ofqual', 'hm-passport-office', 'office-of-the-public-guardian', 'national-college-for-teaching-and-leader...
        Organization not found - error: 78 ['publications/english-conventional-names', 'publications/toponymic-guidelines', 'publications/united-nations-and-geographical-names', 'publications/antarctic-place-names', 'publications/place-names-of-the-united-kingdom', 'publications/the-importance-of-geographical-names', 'publications/geograp...
        Publication page redirected to known publication type: 73 ['statistics/oil-and-gas-public-statements-relating-to-2014-operations', 'statistics/tabulation-tool-questionnaire-statistical-notice', 'publications/norovirus-national-update', 'statistics/benefit-expenditure-and-caseload-tables-2016', 'statistics/benefit-expenditure-and-caseload-tables-2015', '...
        Format not found - check: 56 ['publications/official-development-assistance-oda-international-subscriptions-january-to-september-2013', 'statistical-data-sets/oil-and-petroleum-products-weekly-statistics', 'publications/stamp-duty-land-tax-technical-specifications', 'publications/self-assessment-technical-specifications-2015...
        Detail not found - check: 36 ['statistical-data-sets/road-freight-statistical-tables-index', 'statistical-data-sets/free-flow-speeds-statistical-tables-index', 'statistical-data-sets/vehicles-statistical-tables-index', 'statistical-data-sets/table-1-total-gross-public-expenditure-on-development-2007-08-2011-12', 'statistical...
        Attachments (unmarked) found: 35 ['consultations/closing-the-public-sector-pay-gap', 'statistical-data-sets/tsgb01-modal-comparisons', 'statistical-data-sets/transport-expenditure-tsgb13', 'publications/update-on-the-autumn-statement-2013-ndr-appeals-commitment', 'statistics/renewables-obligation-certificates-and-generation-dece...
        Attachment title (unmarked) not found: 35 ['consultations/closing-the-public-sector-pay-gap', 'publications/update-on-the-autumn-statement-2013-ndr-appeals-commitment', 'publications/update-on-the-autumn-statement-2013-ndr-appeals-commitment', 'statistics/renewables-obligation-certificates-and-generation-december-2014-data', 'statistics/...
        Publication is a consultation outcome so no attachments: 23 ['consultations/farne-deeps-vessel-eligibility', 'consultations/decision-making-and-mandatory-reconsideration-ssac-consultation', 'consultations/allowing-vessels-targeting-plaice-in-the-north-sea-to-use-tr1-gears', 'consultations/office-of-tax-simplification-itnics-closer-alignment-project', 'con...
        Organization description not found - error: 19 ['highways-agency', 'prime-ministers-office-10-downing-street', 'department-of-agriculture-environment-and-rural-affairs-northern-ireland', 'department-of-health-northern-ireland', 'department-of-finance-northern-ireland', 'independent-anti-slavery-commissioner', 'department-for-communities-north...
        Organization page redirected - error: 12 ['publications/professional-deputy-costs https://www.gov.uk/courts-tribunals/senior-courts-costs-office', 'publications/ips-annual-report-and-accounts-2012-to-2013 https://www.gov.uk/government/organisations/identity-and-passport-service', 'publications/ips-annual-report-and-accounts-2011-to-2012...
        Title not found - error: 7 ['statistics/womens-smoking-status-at-time-of-delivery-in-england-october-2014-to-december-2014', 'statistics/summary-hospital-level-mortality-indicator-shmi-deaths-associated-with-hospitalisation-in-england-july-2013-to-june-2014', 'publications/brazil-notarial-services-guide', 'publications/how-...
        Publication page redirected to publications/failure-to-make-an-immigration-right-to-rent-check-penalty-guidance - check: 1 ['publications/failure-to-make-an-immigration-right-to-rent-check-penalty-guidance']
        Publication page redirected to publications/right-to-rent-immigration-checks-guidance-on-who-is-affected - check: 1 ['publications/right-to-rent-immigration-checks-guidance-on-who-is-affected']
        Publication page redirected - error: 1 ['publications/preventing-illegal-working-guidance-for-employers-october-2013']
        Publication page redirected to publications/living-in-brazil - check: 1 ['publications/living-in-brazil']
        Publication page redirected to publications/how-to-make-right-to-rent-checks - check: 1 ['publications/how-to-make-right-to-rent-checks']
        Publication page redirected to publications/brazil-notarial-services-guide - check: 1 ['publications/brazil-notarial-services-guide']
        Time taken (h:m:s): 9:15:01

Page 2388/2387: Time remaining: -1:47 https://www.gov.uk/government/publications?page=2388