Erlang/OTP libraries/modules replaced by third party projects

e.md

I'm looking to collect information on why certain functionality found in Erlang/OTP is usually handled instead by a third party library in people's projects.

This could be bugs, missing functionality, poor interface, performance, etc.

Examples off the top of my head that need expanding on are:

• httpc
• httpd
• http_uri
• reltool

Please comment on this gist with libs and reasons, even if you aren't sure they are accurate or still accurate today.

base64url instead of base64 because whenever I need to base64 encode something, it's usually to put in URL. Erlang makes throwing around raw binary easy so base64 doesn't really have a place.

Its a right of passage writing your own proplists replacement module using lists:key* functions.

There doesn't appear to be any clear benefit when attempting to push for relx instead of reltool in the sense that automatic configuration can still feed a config through reltool. Why bother to rewrite reltool, when releases are important and need to be dependable? Doesn't that just fragment development effort?

Would be good to know where Emakefiles get used on more than toy projects, or where the make module gets used, and why they don't have any features... and why isn't rebar in Erlang/OTP?

A couple of items and gotchas:

• pg2 and global currently makes assumptions about the guarantees you want in a distributed system. This makes newcomers use them in situations where the premises are broken. This is especially true around netsplit situations.
• crypto:strong_rand_bytes/1 is icky. Modern advice is either to use a dedicated getrandom() call (arc4random_buf() on OpenBSD) or use /dev/urandom (with the caveat that you must wait until /dev/random exists as a device). It is not a priori clear strong_rand_bytes/1 gives these guarantees.
• The rpc module doesn't scale. People learn this the hard way.
• The xmerl module have several problems. Most importantly it leaks atom() values so an enemy can overfill your atom table using it. For backwards compatibility it also uses string() data which often gives memory pressure problems when processing large XML documents.
• The array module implements persistent arrays. People often want mutable arrays, which they can't get and thus people resort to NIF implementations.
• All of mnesia assumes certain things which is in stark contrast to the environment in which most Erlang systems are running. I've seen people latch onto it in situations where a simple eleveldb plug is enough to handle persistence.

Inverse problems:

• People don't know how powerful sofs is and then they skip it, even though it can simplify lots of code.

I use Lager in every project because I want some basic logging (beyond io:format) and because I prefer lager's error printing (OTP default is massive and being multi-line makes log aggregators handle it poorly).

The 3 functions: unicode:to_lower/1 unicode:to_upper/1 unicode:to_title/1 are very basic functionality and are missing from Erlang/OTP. They should be like the implementation in https://github.com/rambocoder/unistring but automatically generated from unicode data to make it maintainable.

Is cowboy considered a replacement for httpd? If it is, the main thing I like from cowboy is cowboy_rest. I build REST-like APIs all the time and cowboy is simple to use and very reliable for that purpose.

A gofmt for Erlang, not a solution but a beginning (and links) here

We use gproc all over the show because of pretty much all the features here:
https://github.com/uwiger/gproc#introduction

Looks like gproc like functionality should be part of OTP (or maybe vm?).

The biggest part of the projects have some kind of process worker pools.

Ordered data structures like ordsets etc. I have a few projects with add hook handwritten "reverse" modules aka rordsets.

cowboy > httpd
httpc usable from time to time, but not for any hard work.

Roll your own protocol for moderately big messages instead of disterl.

http decode is overall annoying in erlang. The decode_packet parser returns a mix of atoms and strings. So now that http2 is all lowercase, you need to normalize all your headers if you want to share code. Also decode_packet does not take things into consideration like max size. A much simpler decode_packet for active receive should be created. Also decode_packet could perhaps start to cover http2 and hpack, with the end user being responsible for managing the streams.

Something like accept(Socket, [{packet, {http, 1024, 8192}}, {active, once}]) perhaps.   1024 = max prefix_size, 8192 max size of headers.
It procs:

{http, {error, header_header_size_exceeded}
{http, {error, header_sze_exceeded}
{http, {...Type...}, {...Headers...}, BodySize}

I have a bunch of things I include often because OTP does not have that functionality, but that is beyond the scope of the question.

@jlouis, How do you get around the limitations of RPC module for scaling?

@vans163 missing functionality could be within the scope. The unicode example from @okeuday is a good one. Stuff that really should be in OTP but is missing.

eldap support is poor, the app isn't even proper OTP

JSON's prevalent enough that it feels a surprising not to have a JSON-parsing library built in (like how xmerl is). I use jsx because compiling jiffy hasn't always been a smooth experience.

Unicode is a good example. And we should look at Go for the solution:

• https://godoc.org/golang.org/x/text
• https://blog.golang.org/normalization

Just providing to_upper and to_lower is 99% of the time going to be a mistake. It is a complex area where you need to take care about the details.

@vans163 (RPC) Most applications use the rpc module sparingly in which case it is fine. The problem occurs as soon as you use it for every cross-call you have in the system (which you don't have to). You can build your own more efficient variant, or you can avoid even using the module in the first place. The problem is the gotcha: people see rpc and go "I know RPC! I'll use it!". But if you have a pid() in Erlang you can send it a message, even if it is on a another host.

As for @waisbrot's comment: xmerl should never have been part of the stdlib. Likewise, jsx shall not be part of the stdlib. I'd much rather have a small set of "blessed", "maintained" packages on hex.pm which we use for these kinds of things. Once these things goes into a stdlib, they are set in stone and will move very slowly. The world around them changes, and then you have the same problem as always. jsx is an excellent example because it makes the trade-off of being written in Erlang, and thus about 10 times slower than jiffy which uses a NIF. But using a NIF has its own set of complexities, so not everyone wants that library. You must be able to pick among several poisons.

It also goes for my view on library design: include the things in the stdlib which are impossible to implement as a library (ETS, ...), or are so tightly coupled to the ERTS system that you can't evolve them separately on their own lifecycle.

When it comes to HTTP it has never been a top priority to Ericsson. The httpc absolutely still suffers from the problems I described regarding eldap. E.i something half done was contributed and we have to try making it better. We do have some plans for further improvements but alas I can not make any promises, as we also have a problem prioritizing it.

When it comes to httpd our vision is to try to simplify it and make it a small simplistic embedded web server. We do not have the people to make it otherwise. Also there are Ericsson projects that use it this way and are happy that it solves their needs.

Some better calendar module would be great. Current calendar have only basic primitives, but some more high-order library often required to perform date arithmetics, timezone calculations, formatting and parsing. (And I guess everybody hates to type this datetime_to_gregorian_seconds, gregorian_seconds_to_datetime each time).

Humm... I got this dupplicated so I thought I removed one copy but it seems it went away altogether! So here we go again!
@aerosol
This is a symptom of that the OTP team have not had a good enough process for including contributions. I think it is just lately that PR-process has started to work good enough.
In the past it was quite common that prototypes that work well for some use case where included without proper tests, documentation or code reviews. We do have it in our backlog to
make eldap into a proper OTP application, it just has not been prioritized high enough yet. PR are welcome.

I also guess that not everybody are happy with the way how we do configuration. sys.config files and application:get_env sometimes don't fit our needs: config files should be written in erlang syntax; sometimes configs should have some other logical structure, not just <application>.<key>.

Yes, calendar module is a disaster in the user experience sense. It should really be replaced.