Why...why are we here good sir? Well...I don't want to be here right now. I really don't. It's Saturday night, the Warriors beat the Clippers by the hair on the chinnies and dodged a late fourth quarter comeback, and I really want to eat. But...I have to blog. I have to do it now. If I don't, this story will not be the same. Why are we here? Because I'm on production support, that's why. I got an email from pagerduty that was like:
Unhandled rejection TypeError: next is not a function
I'm like huh? Wtf. What broke now? Why now? It's the weekend, I don't want to be here. I want to do me!!!!! But...I'm on production support. I wonder what this could be. Will it go away in time. Is it transient? This needs to resolve itself. My sanity is relying upon it.
Login into pagerduty... Resolve... Prey...
Hey, it's me again...I'm still broken You need to take action
But why?!?! Why are you still broken? Why won't you resolve yourself so I can just make it to Monday and like...just live some Login into pagerduty... Resolve... Prey...
Hi, I'm still no resolved You need to actually like....fix me
You should know that I had a similar issue with one of our worker processes the last time I was on call. I submitted a pullr request at like 3 in the morning. So I really don't want to be here right now. Ok...we have a real issue here that I actually need to look at. WTF. Ok...let's see here. Something about next. Is this express middleware related? Express uses next. Well...it's not a web process. This is way a worker process. Ok...bluebird is in the stacktrace...also bookshelf. Gah...this must be some kinda change event. It's broken. Somebody is gonna say the site is down or that their favorite feature that I've never worked on is currently broken. Don't do that. Don't post that on Intercom and trigger the shit show. No!!!! I need time. Maybe this isn't all that serious. Maybe no one has noticed yet. Ok kool. Relax. You're going to figure this out. Hmm...ampq you said? Yes it's in the strack trace. What's this? addMessageType? That looks core to this amqp stuff and not actually app code. This is third party. This npm package is so broken. This isn't my fault. Nothing to do here. Wait...it has to be something we did. Something we're using. Something we told it to do deep within its core. Wait...we haven't changed anything in this file. Why is it broken? This makes no sense. What changed?!?!
Hmm...that sync process was updated recently, and it involes amqp, doesn't it. Blame game? It was those guys...yes!!..wait...no. That's on a completely separate dyno in a completely different heroku app. WTF. I need to figure this out. amqp...WTF...which ampq? Hmm...node-lanetix-amqp-easy doesn't appear to be the culprit. X. what about this here servicebus thing. It has some middleware. It looks express like. It probably hasn't chagned for a while. Maybe it's not that. Google...Google now!!!
What?! September 4, 2013? That was like going on 3 years ago! Damn! Fuck!!! It's not that?! But that thing is middleware based just like express is. In fact...we wrote a custom middleware just like in the README for servicebus on github instructed us to do. I didn't write that middleware. It hasn't changed!!! Why!? What is this? Nothing has changed. Wait...are you sure? Check the amqp lib stuff. How?
npm ls
Yea..do that...you've seen people track down react related version issues through this trick. Use this...you have the force. Go now!! Do this!!
Ok...if anything has changed, you have an older version of the app with older npm packages still disc. npm ls that. Ok..keep that in your terminal. Clone a new version of the app. npm install. npm ls that. What do you see?
2 changes
- patch for servicebus (old = 1.0.18, new = 1.0.21)
- patch for amqplib (old = 0.4.0, new = 0.4.1)
Forget ampqlib. It can't be that. Wait...servicebus was changed? But Google says it hasn't been udpated since September 4, 2013, and that's coming directly from the npm website since it's ususally the number 1 hit when you Google a package name. Ok...false..clearly false Antwan. A new package has been published. Click on that link from Google that points to npm. What do you see? Last published a week ago?!?! Whaaaaaaa... something changed!!! It did...oh snap. Oh?! I see. What changed? Was it middleware related? Check pull request #64. Yes!!! Yes it was!! They changed some middleware related stuff. We're using a "I want to accept all patches" versioning scheme in package.json. heroku could straight up deploy a new version right under our noses while we're asleep. But we trust the library authors. They won't sale us out, leave us in the cold. They'll be responsible and publish a major version if there's breakage. But they're human!!! Humans make mistakes. This change isn't backward compatible...it broke stuff for us without us changing any code. Wait...back up a second...the README...when would someone update their README.
- they first publish the package (initial release)
- they break something (major release)
- they add a new feature (minor realease)
- patches? No...no...the interface is still the same. Just an internal detail was changed...nothing new or old here...just maintenance 4a. tests pass...REAME stays the same...new package is deployed containing new patch...all is well in paradise 4b. clients get new update...we're bakcward compatible...we promise...nothing broke
I kept going...I dug deeper...nothing seemingly changed but I knew this to not be true. I almost gave up...went down a different rabbit hole. I was persistent. I didn't trust the first thought that came to mind. I knew better. I had to know better. I've seen this sort of thing before. I knew it had to be this sort of thing. No code changed, but something did. It had to be an npm package. Check the stacktrace ...what do you see...amqp related...but in what way? https://www.npmjs.com/package/servicebus...yes...in that way...you found it man, you're done!!! You can rest easy. Lock into version 1.0.18 and you can keep Saturdaying it up. You can live. mateodelnorte/servicebus#65 is born. What will happen there? Not sure...I don't know...we'll find out. Lessons learned Antwan!!! Lessons learned!!! Always check the last published date on npm. Always check for the versioning scheme specified in package.json. Is it ^ based? Then yes...you can auto receive patches. Always use npm ls for before an after comparison. It is your friend. Remember to keep the science pure. Have a control...have a variable.. hone in and go for the kill. Be a https://github.com/github/scientist. Ok Antwan...the people believe you..they get the point...you're done here...so done here...way done here...you've never blogged this way before..this is so informal..what's with all the ellipses n stuffez.. this is kray kray...way to kray kray...oh well...it was fun...something different...thoughts flowing...just going...trendsetting...doing and being something different...they will appreciate this...this style...this perspective...I sure hope so...it's so different...man is it different...
- ESPN Sportscenter
- I'm hungry
- Must eat
- Must go for pizza in East Bay near UC Berkeley
- Must go now...
- Check email now
- Any response...TBC there
- Signing off
.....Goodbye