Skip to content

Instantly share code, notes, and snippets.

View pcn's full-sized avatar

Peter N pcn

28 followers · 4 following
  • New York, NY
View GitHub Profile
@pcn
pcn / grover_backtrace
Created July 6, 2012 07:21
Trying to use nodetool refresh with Grover data
pn@ip-10-30-138-35:/mnt/data/db/cassandra$ nodetool -h 0 -p 7199 refresh Grover Commits
Exception in thread "main" java.io.IOError: java.io.IOException: Invalid argument
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:225)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:202)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:412)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:186)
at org.apache.cassandra.db.ColumnFamilyStore.loadNewSSTables(ColumnFamilyStore.java:573)
at org.apache.cassandra.db.ColumnFamilyStore.loadNewSSTables(ColumnFamilyStore.java:522)
at org.apache.cassandra.service.StorageService.loadNewSSTables(StorageService.java:2884)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
@pcn
pcn / Gertrude.template.json
Created July 15, 2012 03:53
gertrude launch template
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Creates a Gertrude cluster",
"Parameters" : {
"Environment" : {
"Description" : "The environment where this stack belongs. 'Production' and 'Staging' are reserved words. Do NOT use them for testing.",
"Type" : "String"
},
@pcn
pcn / tcpdump_from_loggly
Created August 9, 2012 23:44
packet traces goign to loggly with an API key that seems to fail for reasons that are not clear.
23:31:34.098028 IP ip-10-141-133-76.ec2.internal.34031 > ec2-107-23-3-233.compute-1.amazonaws.com.www: Flags [.], ack 1, win 115, options [nop,nop,TS val 8930817 ecr 413342485], length 0
E..4n @.@...
..Lk......P..(
a4.E...s.......
..F.....
23:31:34.098872 IP ip-10-141-133-76.ec2.internal.34031 > ec2-107-23-3-233.compute-1.amazonaws.com.www: Flags [P.], seq 1:206, ack 1, win 115, options [nop,nop,TS val 8930817 ecr 413342485], length 205
E...n
@.@...
..Lk......P..(
a4.E...s.......
@pcn
pcn / sstablemessage
Created August 11, 2012 02:05
Corrupt sstable I guess
(CassandraHaag 1/12) pn@domU-12-31-39-05-46-11:/mnt/tmp$ sudo sstablekeys /mnt/tmp/Haag/StudentEvents-hd-103-Data.db
log4j:WARN No appenders could be found for logger (org.apache.cassandra.config.DatabaseDescriptor).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.io.IOError: java.io.EOFException
at org.apache.cassandra.io.sstable.KeyIterator.computeNext(KeyIterator.java:66)
at org.apache.cassandra.io.sstable.KeyIterator.computeNext(KeyIterator.java:36)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.tools.SSTableExport.enumeratekeys(SSTableExport.java:217)
@pcn
pcn / cassandra_log
Created August 11, 2012 15:13
Nodetool cleanup errors on a fresh restore
2012-08-11T14:18:59.88547 INFO 14:18:59,885 Cleaning up SSTableReader(path='/mnt/data/db/cassandra/Haag/StudentEvents-hd-1003-Data.db')
2012-08-11T14:18:59.94836 ERROR 14:18:59,946 Fatal exception in thread Thread[CompactionExecutor:9,1,RMI Runtime]
2012-08-11T14:18:59.94840 java.io.IOError: java.io.IOException: dataSize of 189633 starting at 82070879 would be larger than file /mnt/data/db/cassandra/Haag/StudentEvents-hd-1003-Data.db length 82083840
2012-08-11T14:18:59.94840 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:154)
2012-08-11T14:18:59.94841 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:86)
2012-08-11T14:18:59.94842 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70)
2012-08-11T14:18:59.94842 at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177)
2012-08-11T14:18:59.94843 at org.apac
@pcn
pcn / majority_node_grep_of_minority_events
Created August 30, 2012 15:53
Cassandra gossip has abandoned a node
@4000000050234dda249f5fb4.s:2012-07-31T20:16:55.00512 INFO 20:16:55,004 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@4000000050234dda249f5fb4.s:2012-07-31T20:17:07.32295 INFO 20:17:07,322 Finished hinted handoff of 231 rows to endpoint /10.192.178.255
@4000000050234dda249f5fb4.s:2012-08-01T07:47:17.87990 INFO 07:47:17,878 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@4000000050234dda249f5fb4.s:2012-08-01T07:47:18.59174 INFO 07:47:18,591 Finished hinted handoff of 13 rows to endpoint /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-28T23:09:48.58580 INFO 23:09:48,585 InetAddress /10.192.178.255 is now dead.
@40000000503e41c804822eb4.s:2012-08-28T23:09:49.58861 INFO 23:09:49,588 Node /10.192.178.255 has restarted, now UP
@40000000503e41c804822eb4.s:2012-08-28T23:09:49.59137 INFO 23:09:49,590 InetAddress /10.192.178.255 is now UP@40000000503e41c804822eb4.s:2012-08-28T23:09:49.59140 INFO 23:09:49,
@pcn
pcn / log_excerpt
Created August 31, 2012 19:22
cassandra for grover fails with no meaningful message
2012-08-31T15:45:40.98224 INFO 15:45:40,980 Compacted to [/mnt/data/db/cassandra/Grover/Branches.Branches_subgraph_id_idx-hd-134-Data.db,]. 12,097,232 to 11,540,283 (~95% of original) bytes for 96 keys at 4.795500MB/s. Time: 2,295ms.2012-08-31T15:46:14.51641 INFO 15:46:14,516 GC for PS Scavenge: 236 ms for 1 collections, 1430079048 used; max is 33994833922012-08-31T15:47:03.55341 INFO 15:47:03,553 GC for PS Scavenge: 235 ms for 1 collections, 1395463120 used; max is 34038743042012-08-31T15:47:54.59539 INFO 15:47:54,595 GC for PS Scavenge: 225 ms for 1 collections, 1400291552 used; max is 34026291202012-08-31T15:48:51.84810 INFO 15:48:51,847 GC for PS Scavenge: 305 ms for 1 collections, 1383144256 used; max is 34078064642012-08-31T15:49:48.87291 INFO 15:49:48,872 GC for PS Scavenge: 261 ms for 1 collections, 1401799744 used; max is 34064302082012-08-31T15:50:34.89669 INFO 15:50:34,896 GC for PS Scavenge: 227 ms for 1 collections, 1421621232 used; max is 3412983808
2012-08-31T15:51:27.92456 INFO 15:51
@pcn
pcn / permission_amis.py.patch
Created October 28, 2012 15:34
patch to use permission_amis.py in the west
~/tmp/permission_amis.py
--- bin/permission_amis.py 2012-03-26 14:00:05.000000000 -0400
+++ /Users/pn/tmp/permission_amis.py 2012-10-27 16:24:24.000000000 -0400
@@ -14,10 +14,12 @@
import sys
import boto
from optparse import OptionParser
+import boto.ec2
+import boto.exception
@pcn
pcn / gist:4587793
Created January 21, 2013 17:46
loggly OOM freeze when there is an output 503
2012-10-05T03:21:11.60448 W, [2012-10-05T03:21:11.604000 #2204] WARN -- : Output thread exception {"timestamp":"2012-10-05T03:21:11.604000 +0000",
"plugin":"LogStash::Outputs::Loggly: {\"host\"=>\"logs01.loggly.net\", \"key\"=>\"SANITIZED\", \"type\"=>\"\", \"tags\"=>[], \"proto\"=>\"http\"}","exception":"503 \"Server down\"","backtrace":["file:/opt/knewton/logstash/logstash-monolithic.jar!/META-INF/jruby.home/
lib/ruby/1.9/net/http.rb:2303:in `error!'","file:/opt/knewton/logstash/logstash-monolithic.jar!/logstash/outputs/loggly.rb:77:in `receive'","file:/
opt/knewton/logstash/logstash-monolithic.jar!/logstash/outputs/base.rb:48:in `handle'","file:/opt/knewton/logstash/logstash-monolithic.jar!/logstas
h/agent.rb:674:in `run_output'","file:/opt/knewton/logstash/logstash-monolithic.jar!/logstash/agent.rb:333:in `start_output'"],"message":"Output th
read exception","level":"warn"}
2012-10-05T09:28:03.30075 Exception in thread "filter|worker|0" java.lang.OutOfMemoryError: Java heap space
2012-10-05T09:28:03.3011
@pcn
pcn / chef-solo running the apt cookbook
Created January 22, 2013 20:06
pgp.mit.edu is timing out?
What happened here?
STDOUT: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver pgp.mit.edu --recv B999A372
STDERR: gpg: requesting key B999A372 from hkp server pgp.mit.edu
gpg: keyserver timed out
gpg: keyserver receive failed: keyserver error