Jon Bartels jonbartels

Source queues in MC are Evil

I was called out on Slack this week for my long standing assertion that "source queues are evil". I will cede that this is said somewhat jokingly, but it is based in fact. I have never been able to quickly explain my position so I wrote this gist to organize my thoughts.

Brief Listing

source queues are hard to manage compared to destination queues
Destination queues will retry if the downstream channel is down for some reason.
Destination queues allow the engineer to add or reduce threads to process the message volume at the desired rate
source queues trigger message recovery which is slow

How we use it (I may be overly devops-ey, and I’m also not the closest to this implementation right now) : We break up specific integrations (like for instance a specific health system) into different repositories.

Each integration is in it’s own channel group. Dev Workflow:

a developer works on the dev environment, building his channels and testing it in that environment When he is ready to push it to the higher environments, he runs a ‘mirthsync’ command to pull down the channels from dev, and place them in the local git repo

they make a PR against the dev branch of the repo

From - nextgenhealthcare/connect#5193 (comment)

PS C:\Program Files\Mirth Connect\server-lib> java -jar .\rhino-1.7.13.jar Rhino 1.7.13 2020 09 02
js> x = <A/>

js> x += <B/>
<A/>
<B/>

	var dump = {};

	var factory = new org.quartz.impl.StdSchedulerFactory();
	var allSchedulers = factory.getAllSchedulers();

	dump.schedulerCount = allSchedulers.size();

	dump.schedulers = [];

	var iter = allSchedulers.iterator();

	WITH channel_query AS (
	SELECT dc.local_channel_id,
	c.name,
	format('SELECT ' \|\|
	'%L::TEXT as channel_name, ' \|\|
	'date_trunc(''day'', received_date)::TEXT as day, ' \|\|
	'count(*)::BIGINT as cnt FROM d_m%s' \|\|
	' WHERE received_date >= now() - INTERVAL ''%s DAYS'' GROUP BY 1, 2', c.name,
	dc.local_channel_id, 90) as query
	FROM d_channels dc

	--see https://github.com/nextgenhealthcare/connect/issues/4320
	-- this generates create index statements for the metadata columns in MC excluding the default columns
	-- the user SHOULD NOT BLINDLY CREATE ALL INDEXES
	-- MC is update/insert heavy and having indexes can slow that performance.
	-- These indexes should be used only on metadata colums which are searched frequently, on large tables, and where constraining by other indexed columns is not practical
	-- Further, if you're searching on large tables you should run the damn pruner. see https://gist.github.com/MichaelLeeHobbs/40b4b7cf70ecbe30b73eed763367e626
	SELECT t.table_name, c.column_name, format('CREATE INDEX CONCURRENTLY metadata_hax_%1$s_%2$s ON %1$s("%2$s");', t.table_name, c.column_name) as create_stmt
	FROM information_schema.tables t
	INNER JOIN information_schema.columns c ON c.table_name = t.table_name AND c.table_schema = c.table_schema
	WHERE t.table_schema='public'

	/*
	I did a thing.
	I wrote a channel that reads the current config map and then writes it to the Mirth DB so that you can switch from file backed config maps to DB backed config maps.
	Just let the code run in a JS Reader and then flip the mirth.properties entry whenever you’re ready to change over.
	*/
	// read mirth.properties and check the configurationmap.location property
	var classpathResource = com.mirth.connect.server.tools.ClassPathResource.getResourceURI('mirth.properties');
	var propsInputStream;
	var configmapLocation = 'Could not read!';
	try {

	with channel_xml as (
	select
	name,
	xmlparse(document channel) as channel_xml
	from channel c
	)
	, destination_connector_xml as (
	SELECT
	name as channel_name,
	unnest(xpath('//destinationConnectors/connector/name/text()', channel_xml))::TEXT as connector_name,

	do $$
	DECLARE r record;
	declare stmt text;
	begin
	drop table if exists temp_last_message;
	create table temp_last_message (chan text, cnt INTEGER);
	FOR r in select c.name::TEXT as chan, dc.local_channel_id as local_channel_id from d_channels dc inner join channel c on c.id = dc.channel_id
	loop
	stmt := format('INSERT INTO temp_last_message select ''%s'' AS name, count(*) as cnt from d_m%s as messages_last_2_weeks where received_date >= now() - interval ''2 weeks''', r.chan, r.local_channel_id);
	--raise notice 'generated %', stmt;

	jonb 11:31
	I would do many dirty deeds if globalMap and globalChannelMap and configMap had a lastUpdated flag. I have a use case with a large-ish object in the map and don’t want to re-load or re-parse it if it hasn’t changed. (edited)

	pacmano 11:39
	We manually put that in there. That would be really nice.

	chris 11:51
	lastUpdated flag, or timestamp-of-last-update?
	11:51
	And do you want that per-key, or per-map?