Pchelolo

#!/bin/bash
kafkacat -b localhost:9092 -t 'eqiad.change-prop.error' -p 0 -o beginning -e | jq 'select((.message == "500") or (.message == "504")) | .meta.uri' > errors
cat errors | sort | uniq -c | awk '$1>100'
rm errors

Pchelolo

var array = [ { b: 'c' } ];
function test() {
  var value = array[0];
  if (!value) return;
  setTimeout(() => {
    console.log(value.b); // Throws can't read property b of undefined.
  }, 1000);
}

Pchelolo

kafkacat -b localhost:9092 -t 'eqiad.mediawiki.revision-create' -p 0 -o -1300000 -c 1300000 | jq -c 'select(.meta.domain = "en.wikipedia.org") | . ' > edit_dump

vagrant ssh
cat edit_dump | kafkacat -b localhost:9092 -t 'datacenter1.mediawiki.revision-create' -p 0

Set it to Integer.MAX_VALUE in here: https://github.com/wikimedia/mediawiki-services-trending-edits/blob/master/lib/edit-stream.js#L22
Change this to 'smallest' in here: https://github.com/wikimedia/mediawiki-services-trending-edits/blob/master/lib/edit-stream.js#L88

Pchelolo

cd TILERATOR_SOURCE_REPO
docker build -t tolerator-deploy .
docker run -it --name tolerator-builder --rm -v PATH_TO_DEPLOY_REPO:/opt/service -w /opt/service tilerator-deploy /bin/bash

Pchelolo

              page_edit:
                topic: mediawiki.revision-create
                retry_on:
                  status:
                    - '5xx'
                    - 404 # Sometimes occasional 404s happen because of the mysql replication lag, so retry
                # Test-only. We use undefined rev_parent_id to test backlinks so we
                # don't want transclusions to interfere with backlinks test
                match_not:
                  - rev_parent_id: undefined

Pchelolo

oh... there's actually another much bigger problem...
So, when we rerender the HTML for transclusions, we rerender the latest version. If what we've just rendered is the same as what's stored, we don't store
When normal traffic is happening it's OK, because the template was actually edited and the page might've changed
When replaying historic events, the template is not changed, it's on it's latest revision for all rerenders of a particular page. So nothing will get stored
That's another reason to split the life traffic into the dev cluster

Pchelolo

MERGED + role::changeprop https://gerrit.wikimedia.org/r/#/c/333110/
MERGED + role::citoid  https://gerrit.wikimedia.org/r/#/c/333102/
MERGED + role::cxserver https://gerrit.wikimedia.org/r/#/c/333105/
MERGED + role::eventstreams https://gerrit.wikimedia.org/r/#/c/333107/
MERGED + role::graphoid https://gerrit.wikimedia.org/r/#/c/333111/
MERGED + role::mathoid https://gerrit.wikimedia.org/r/#/c/333113/
MERGED + role::mobileapps https://gerrit.wikimedia.org/r/#/c/333108/
MERGED + role::pdfrender https://gerrit.wikimedia.org/r/333114
MERGED + role::trendingedits https://gerrit.wikimedia.org/r/#/c/333112/

Pchelolo

let x = 1;
console.log({
    [x === 1] : "X is One",
    [x === 2] : "X is Two"
}.true);

Pchelolo

const TimeUUID = require('cassandra-uuid').TimeUuid;
require('../test/utils/common').factory.createProducer(()=>{})
.then((producer) => {
    const produce = () => {
        return P.try(() => producer.produce('test_dc.mediawiki.revision-create', 0,
            Buffer.from(TimeUUID.now().toString()),
            TimeUUID.now().toString()))
        .then(() => {
            global.gc();
            return produce();

Pchelolo

OFFSET=`kafka run-class  kafka.admin.ConsumerGroupCommand --describe --group change-prop-on_transclusion_update --bootstrap-server kafka1001.eqiad.wmnet:9092 --new-consumer | sed -n 3p | tr ',' '\n' | sed -n 6p` /home/otto/kafkacat -b kafka1001.eqiad.wmnet:9092 -t eqiad.change-prop.transcludes.resource-change -p 0 -c $OFFSET -o -$OFFSET | jq 'select(.meta.schema_uri == "continue/1") | .original_event.page_title'