davidread

#!/bin/bash
#
# For the git repo in the current directory, this script will re-initialize
# git-crypt with a new secret and re-add all the gpg keys.
#
# Purpose: by running this script, users who used to have their GPG keys in this
# git-crypt'd repo will not be able to view future changes.
#
# Notes:
# 1. Before running this you should have already removed GPG keys of old users.

davidread

git-crypt guidance

Some repositories have git-crypt enabled, which encrypts certain files. This doc describes how to add and remove users, using their GPG keys, and has associated scripts. (You can also just share the repo's symmetric key, which is simpler, but not covered here.)

Understanding git-crypt

A repo that has been git-crypt'd should have in its repo:

• .gitattributes - defines which files should be encrypted
• .git-crypt/keys/default/0/*.gpg - .gpg file for every user (Each .gpg file is the repo's symmetric encryption key, which has been encrypted for a particular user with their individual public key. The filename is the user key's fingerprint.)

davidread

(default) vagrant@ubuntu-xenial:/vagrant/src/ckan$ nosetests --nologcapture --with-pylons=test-core.ini -v ckan ckanext
/usr/lib/ckan/default/local/lib/python2.7/site-packages/webassets/loaders.py:162: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  obj = self.yaml.load(f) or {}
No handlers could be found for logger "ckan.lib.uploader"
ckan.tests.config.test_environment.TestDisplayTimezone.test_missing_timezone ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_missing_siteurl ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_missing_schema ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_removes_backslash ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_wrong_schema ... ok
CKAN_SQLALCHEMY_URL in the env takes precedence over CKAN_DB ... ok

davidread

#! /bin/bash

# Resets a CKAN install e.g. after you switch branches, particularly downgrading.
# Destructive! Designed for developers, not production installs.
# Run this script from the ckan repo directory.
# Make sure you have $CKAN_INI set.

set -ex

CKAN_MAJOR_VERSION=$(grep '__version__' ckan/__init__.py | awk '{print $3}' |sed "s/'\([0-9]\+\)\.[0-9]\+\..*/\1/")

davidread

I ran the migrate_package_activity.py (ckan/ckan#3972) on a real existing CKAN database to get a sense of the time taken and the change in database size

Timing: Migration took 49s for 236 datasets. If you scale that linearly to 900k datasets (europeandataportal.eu) then you're talking 1 hour. There's likely to be more activities per dataset though, so it would be more like several hours. Seems reasonable to want to do the migration while the site is up.

Size: size of the activity table went up about 20%, because every revision we now store resources, extras and tags on each package. However the overall size came down when we deleted the content of the activity_detail table (which is no longer needed), because that had a quirk that when you add a tag, it stored the dictized package as well in that table - this seems like a mistake.

Test Details

Test database properties:

davidread

"working tree" = "repository"
"untracked files" - diff
"changes"

---- "add" --->
<--- "rm" ----

"Staged" = "Changes to be committed" = "index"
Similar: "indexed" "tracked files" "snapshot"

davidread

$ nosetests --ckan --with-pylons=test-core.ini ckan/tests/lib/test_helpers.py:TestBuildNavMain -x
No handlers could be found for logger "ckan.lib.uploader"
E
======================================================================
ERROR: ckan.tests.lib.test_helpers.TestBuildNavMain.test_dataset_navigation_legacy_routes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/vagrant/src/ckan/ckan/tests/lib/test_helpers.py", line 606, in test_dataset_navigation_legacy_routes

davidread

diff --git a/ckan/lib/navl/validators.py b/ckan/lib/navl/validators.py
index 172da1531..b041d478e 100644
--- a/ckan/lib/navl/validators.py
+++ b/ckan/lib/navl/validators.py
@@ -4,7 +4,7 @@ from six import text_type

 import ckan.lib.navl.dictization_functions as df

-from ckan.common import _, json
+from ckan.common import _, json, config

davidread

Lots of stack traces and error messages for CKAN, followed by a comment about how I fixed it (that time).
NB don't email me privately for help - ask on StackOverflow #ckan instead.
David

File "/vagrant/src/ckan/ckan/templates/home/snippets/search.html", line 1, in top-level template code
    {% set tags = h.get_facet_items_dict('tags', limit=3) %}
  File "/vagrant/src/ckan/ckan/lib/helpers.py", line 949, in get_facet_items_dict
    if not c.search_facets or \
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)

davidread

auth:
  type: custom
  custom:
    className: "oauthenticator.auth0.Auth0OAuthenticator"
    config:
      client_id: "3maZCyDXXXXXXXXXIDbz0W8A"
      client_secret: "aMSobNMr0OkPjXXXXXXXXX7MdZkAcpL"
      oauth_callback_url: "http://a172XXXX77d46cd-18XXXXX48.eu-west-1.elb.amazonaws.com/hub/oauth_callback"
      scope:
        - "openid"