Leo Lapworth ranguard

Tue May 30 19:24:04 BST 2017

cat /var/log/nginx/metacpan-api/access.log | awk '{print $13}' | sort | uniq -c

    340 "-"
      9 "alfred-metacpan/0.01"
    100 "analysis.cpantesters.org/1.1.1"
     37 "AppEngine-Google;

Trying to create the S3 repo...

curl -X POST 'http://localhost:9200/_snapshot/backups?pretty=1&error_trace=true' -d '
{"settings":{"storage_class":"standard","region":"us-east-1","canned_acl":"private","protocol":"https","access_key":"XXXXXXXXX","bucket":"AAAAAAA","server_side_encryption":1,"secret_key":"YYYYYYY"},"type":"s3"}'

{
  "error" : {
    "root_cause" : [ {
      "type" : "no_class_def_found_error",

Goals / ideas for the MetaCPAN Hackathon

Switch to new ES index

Tasks:

Review ES configuration is optomal and does not need a re-index (PRE MC::Hack)
Look at snapshots/backups

MetaCPAN has been using Elasticsearch for a long time... a really long time, we last 'upgraded' to version 0.20.2! Our current setup relies on a single box and this very old version, and it works great, that said...

We have spend the last couple of years (this is a voluneer based project), doing major work in reoganising how we index and have update all our code, such that we now have a 3 node cluster (and long term we will have 2 datacenters with a cluster in each) running Elasticsearch version 2.4.0 this will give us reliability and access to a vast number of extra features, from autocomplete to aggrgations.

	### Mapping

	```json
	"suggest": {
	"type" : "completion",
	"analyzer" : "simple",
	"search_analyzer" : "simple",
	"payloads" : true
	},
	```

	package Foo::A;

	use Moose::Role;

	requires 'bar';

	1;

	package Foo::B;

	package Foo::A;

	use Moose::Role;

	requires 'bar';

	1;

	package Foo::B;

	#!/usr/bin/env perl

	# USAGE: run_test.pl AUTHOR

	use 5.20.0;
	use warnings;

	use MetaCPAN::Client;

	my $opt_v = 2; # Verbosity of debug

	80.100.130.165 23.235.40.38 - - [06/Jul/2016:21:29:37 +0100] "GET /v1/module/Release::Checklist HTTP/1.1" 404 48 "-" "MetaCPAN::Client/1.018000" 500238 0.054 443 fastapi.metacpan.org 0.054
	80.100.130.165 23.235.40.50 - - [06/Jul/2016:21:29:38 +0100] "POST /v1/favorite/_search?scroll=5m&search_type=scan&size=1000 HTTP/1.1" 200 295 "-" "HTTP-Tiny/0.058" 500240 0.142 443 fastapi.metacpan.org 0.059
	80.100.130.165 23.235.40.51 - - [06/Jul/2016:21:29:38 +0100] "POST /v1/rating/_search?scroll=5m&search_type=scan&size=1000 HTTP/1.1" 200 295 "-" "HTTP-Tiny/0.058" 500217 0.190 443 fastapi.metacpan.org 0.055
	80.100.130.165 23.235.40.29 - - [06/Jul/2016:21:29:44 +0100] "GET /v1/module/Spreadsheet::Read HTTP/1.1" 200 33633 "-" "MetaCPAN::Client/1.018000" 500235 0.070 443 fastapi.metacpan.org 0.063
	80.100.130.165 23.235.40.43 - - [06/Jul/2016:21:29:44 +0100] "POST /v1/favorite/_search?scroll=5m&search_type=scan&size=1000 HTTP/1.1" 200 296 "-" "HTTP-Tiny/0.058" 500244 0.128 443 fastapi.metacpan.org 0.053
	80.100.130.165 23.2

	#!/usr/bin/env perl
	# PODNAME: fetch_dist_versions_for_tests.pl
	#
	# takes a distribution (e.g. Data-Page), looks up all versions, downloads and
	# creates configs to copy into the testing data dir
	# 'test-data/fakecpan/configs/from_cpan'
	# which work for CPAN::Faker to /tmp/faked/<DIST>-version.json

	#### USAGE:
	# Use the real content