Pchelolo

{
  "$schema": "/mediawiki/job/1.0.0",
  "meta": {
    "uri": "https://placeholder.invalid/wiki/Special:Badtitle",
    "request_id": "ef97f889-8e3c-46e2-b6f5-2c0859ed78fa",
    "id": "f4c01e9b-2846-418a-834b-5cfa1bab7c3e",
    "dt": "2020-09-27T22:46:48Z",
    "domain": "en.wikipedia.org",
    "stream": "mediawiki.job.MassMessageJob"
  },

Pchelolo

Master:
-------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations
-------------------------------------------------------------------------
BM_AccessLogFormatter                6512 ns         6454 ns       107716
BM_JsonAccessLogFormatter          184852 ns       183447 ns         3785
BM_TypedJsonAccessLogFormatter     180106 ns       178997 ns         3922

-------------------------------------------------------------------------
Benchmark                               Time             CPU   Iterations

Pchelolo

{"name":"eventgate-wikimedia","hostname":"eventgate-analytics-production-58db4b564c-lwsnl","pid":299,"level":60,"err":{"message":"","name":"TypeError","stack":"TypeError: Cannot read property 'toString' of null\n    at EventGate.mapToErrorEvent (/srv/service/eventgate-wikimedia.js:192:67)\n    at _.map (/srv/service/node_modules/eventgate/lib/eventgate.js:247:41)\n    at arrayMap (/srv/service/node_modules/lodash/lodash.js:639:23)\n    at Function.map (/srv/service/node_modules/lodash/lodash.js:9580:14)\n    at Timeout.setTimeout [as _onTimeout] (/srv/service/node_modules/eventgate/lib/eventgate.js:246:47)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)","levelPath":"fatal/service-runner/unhandled"},"msg":"Cannot read property 'toString' of null","time":"2020-08-19T14:41:21.265Z","v":0}

Pchelolo

--- envoy.yaml
                  # Generate two rate limiting descriptors if we can:

                  # - {value: ("user_class_per_ip", "<class>")
                  #
                  rate_limits:
                    # If JWT is present in the request, is valid and contains ratelimit override,
                    # this will generate the following descriptor:
                    # - {value: ("user_class","<class>"), override:(value:1000,unit:MINUTE)}
                    # The ratelimit override will be applied by the service.

Pchelolo

Step1: Envoy: DynamicMetadataAction
	- { dynamic: { key: ‘blabla_key”, value: “blablakey_to_get_value_from_dynamic_metadata” } }


Step2: Ratelimiter service & data-plane: rate limits overrides in a request, descriptor etc
Step3: Envoy, envoy filter: pass dynamic limits from actions into descriptors. Somehow.
	- { dynamic: { key: ‘blabla_key”, value: “blablakey_to_get_value_from_dynamic_metadata”, limit_override: { dynamic: { limit: “key_to_get_limit”, “unit”: “kley_to_get_unit” } } } }

Normal endpoint: 
(client_id: 1234) - global limit

Pchelolo

Step1: Envoy: DynamicMetadataAction
	- { dynamic: { key: ‘blabla_key”, value: “blablakey_to_get_value_from_dynamic_metadata” } }


Step2: Ratelimiter service & data-plane: rate limits overrides in a request, descriptor etc
Step3: Envoy, envoy filter: pass dynamic limits from actions into descriptors. Somehow.
	- { dynamic: { key: ‘blabla_key”, value: “blablakey_to_get_value_from_dynamic_metadata”, limit_override: { dynamic: { limit: “key_to_get_limit”, “unit”: “kley_to_get_unit” } } } }

Normal endpoint: 
(client_id: 1234) - global limit

Pchelolo

Method call
mediawiki
extensions/MathSearch/includes/specials
SpecialMlpEval.php
SpecialMlpEval
printFormulaRef
			$this->selectedMathTag->getWikiText(), $this->revision->getTimestamp() );
extensions/Video/includes/specials
SpecialUndeleteWithVideoSupport.php
SpecialUndeleteWithVideoSupport

Pchelolo

- MW stores the config for user classes -> rate limit + attributes 
- The lua filter parses them out from the JWT: now lua filter has a rate number
- OPTIONAL: we can set the overrides as https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter#metadata
- applies the proper metadata -> we now have the number = my limit is 1000 req/s

now... descriptors...

Overrides as hits_added. Can't be < 0 :(
- hits_addend=100.000 / limit ???

Pchelolo

Introduce UserRecord and UserFactory
UserRecord is a value-object interface implemented by the old User class and a new UserRecordValue (or DatabaseUserRecord?) class. UserRecord extends UserIdentity and in addition defines the following methods:
hasLocalAccount
getRealName
isLocked (?!)
isHidden (?!)
getTouched/getDBTouched
getEmail
getEmailAuthenticationTimestamp
getRegistration

Pchelolo

[2020-02-19T01:54:43.548Z] FATAL: changeprop/17 on changeprop-test-569cbf654d-mqth6: Message not supplied (levelPath=fatal/startup, err.found=", err.expression=message.meta.domain")
    SyntaxError: Expected "(", ".", "[", [ \t\n], [a-z0-9_$-]i or end of input but "\"" found.
        at peg$buildException (/srv/service/node_modules/template-expression-compiler/ExpressionParser.js:332:14)
        at Object.peg$parse [as parse] (/srv/service/node_modules/template-expression-compiler/ExpressionParser.js:1551:13)
        at Object.parse (/srv/service/node_modules/template-expression-compiler/index.js:92:43)
        at compileExpression (/srv/service/node_modules/swagger-router/lib/reqTemplate.js:28:31)
        at splitAndPrepareTAssemblyTemplate (/srv/service/node_modules/swagger-router/lib/reqTemplate.js:220:44)
        at replaceComplexTemplates (/srv/service/node_modules/swagger-router/lib/reqTemplate.js:341:40)
        at Object.keys.forEach (/srv/service/node_modules/swagger-router/lib/reqTemplate.js:323:24)