October 2019 Archives

Wed Oct 16 01:00:01 IRST 2019

ejabberd vs. Prosody

Like many people, I recently have been re-discovering the XMPP messaging protocol and ended up running my own server. In fact, this blog is essentially a result of that effort, though it functions independently from the XMPP service.

You see, I had used XMPP back in my college days when we used to call it Jabber and before today's most popular messaging platforms based their protocols on it. To me and many others, XMPP just meant jabber.org. (Sure, there were other servers, but one would expect the original implementation to be the leader and cutting edge.)

I kept using the protocol as much as was practical, though its practicality diminished greatly over the years. Over a decade passed where logging into jabber.org with Psi or maybe Pidgin was a pointless affair. The protocol used to interoperate with Google Talk, but that eventually came to an end when Google gradually became more and more evil. (And this is a true story: They literally removed the "don't be evil" clause from their official corporate policy.)

So, what was the point of using XMPP anymore? Nobody used it, and messaging had revolutionized and gone mobile; nobody needed desktop messengers, but mobile phone ones. Even an app like Skype could not compete with services like WhatsApp and Telegram. XMPP would be a joke by comparison, with a handful of Android clients of abysmal quality and not suited for mobile use and mobile connections.

However, over the years XMPP silently evolved and grew powerful. Someone randomly mentioned to me online that they not only still use it, but that it's awesome thanks to modern clients like Conversations. Eventually I too decided to use Conversations, but for that I needed a server.

As it turned out, jabber.org had dropped the ball. My account on their server still works even, but their server is not modern in the sense that it doesn't even support half the features Conversations needs to provide a great mobile experience on par with WhatsApp. (And yes, it really does this.)

I figured OK, I have what I need to run my own server. I just need to install the free software and let it do its thing. I think I had even played around with ejabberd once many years ago.

And that's where I fell into the trap that you might fall into as well. I knew that ejabberd existed and was available, but I thought, "Gosh, ejabberd is ancient! It's like the original XMPP server. And it's written in that probably obsolete language, Erlang." If jabber.org is obsolete, what would ejabberd be? jabber.org even stopped using ejabberd long ago in favor of a newer server.

That's why I did a quick search around, and Prosody caught my attention. I had heard of it once before, and at the time it had a pretty good reputation, and now it seemed to me like it was the most cutting edge and modern server. It certainly seemed to be the most currently popular, so people who know their stuff must all be using Prosody, right?

Well, Prosody got installed on my server the day I downloaded Conversations, and setting it up was indeed painless at first. It was simple enough to install and get up and running, and the base configuration was a breeze. I was able to log in right away and things were looking alright.

However, it didn't take long for me to realize that something was amiss. First of all, Conversations supports all these fantastic features that Prosody also supports, but they aren't ready out of the box. Some of the advanced features have to be enabled manually by loading "community" modules. That is, these are modules that implement some tricky features in the protocol that are not officially part of Prosody or supported by its devs, but are provided as a separate thing you can install made by community members who contribute them. Not to say that community modules are bad or anything like that, but the fact that they're separated is not totally desirable.

Now, Conversations is nice enough to tell you for the most part what is working on your server and what isn't. So the first step of enabling the modules I needed in order to have all the features I wanted was not too difficult. However, then problems started creeping in.

Some modules had dependencies to other modules that were not entirely clear. Some modules have zero documentation. Modules have their dependencies automatically loaded if you forget one, so that's not such a big deal, but some modules can conflict with each other, and you have to make choices as to which module is right for you. (For example, stream management for mobile devices.) But how do you make that choice when the models that you apparently have to choose between may not be sufficiently documented, or one is beta quality and may not work?

I spent hours and days tweaking my Prosody configuration to make Conversations happy. I talked to the Prosody people, Googled as many tutorials and configuration examples as I could find, and there was never any clear known-good setup. It was always just, "Well, maybe if you load this or that module it'll work this way." Every example config I looked at recommended different modules apparently without anyone really being sure which ones are needed and which ones aren't.

My hard work did yield some progress, and I got to the point where I had a config or two that should be perfect for my client, but there was just one show-stopper:

Without getting too technical, a mobile phone can't maintain a solid connection to the server all the time because mobile phones are very touch-and-go. The app gets backgrounded and put to sleep, the Internet comes and goes depending on your signal, and so on.

What Conversations is designed to do is work together with the server to understand that these things happen and not terminate your session and break your communications at the slightest hiccup. That means the client can lose connectivity and regain the original connection by letting the server know it's still there.

This is exactly what wasn't working with my Prosody server, and when Conversations tried to resume a preestablished connection it would just hang and never connect. This meant that it would frustratingly stop working after maybe half an hour from the time of connecting until I manually reset the connection. I could fix it by disabling that feature on the server, but then Conversations would always start a new connection to the server every time it needed to do something. That kind of inefficiency is bad for the phone's battery life, as it's working too hard on something it shouldn't have to be doing.

In the mean time, some of the connectivy problems manifested as message delivery errors, where Prosody would tell my client that messages had failed to deliver because of timeouts, but the other person actually was getting them. This is also a known issue with mobile configurations that a certain module can address, allowing Prosody to ignore apparent delivery failtures.

As it stood, my server wasn't cutting it, and after talking to the nice Prosody guys, the eventual conclusion that we were able to come to was that my modules were outdated, and I was encountering a bug that had already been fixed. The frustrating part there was that when I installed Prosody in the first place I made sure to check its version, as the whole point was to use the latest tech. My Prosody version was shown as the last minor release, so the package was fairly recent.

So in the end, I could not use my distro's Prosody packages if I wanted it to work, and I didn't really feel like manually installing things outside of my package manager, possibly experimenting with nightly builds or something like that. In the mean time I had heard a curious thing, which was that actually ejabberd was not a dead project, but in fact had lept ahead and was one of the most advanced servers out there.

I was a little skeptical, still clinging to the idea that ejabberd was outdated or at best inefficient, as Prosody claims to be faster or lighter on resources. However, after some discussion with others, I also learned that the Conversations project openly prefers and recommends ejabberd. But why?

Well, at this point I had to try it. It seems like a scary program that's difficult to set up, but at least I could tell my package manager to install it, so at worst I might waste a couple hours attempting to run it, let alone configure it. In a single afternoon I had Prosody fully replaced with a working ejabberd install, with virtually all features working out of the box.

You see, ejabberd is a very sophisticated piece of software. What it does is actually quite amazing (thanks in no small part to features built into Erlang), and the effectiveness of its design has not diminished with age. All the advanced features that a client like Conversations needs are built into the server and available out of the box. They're still modular by design, but they are included in the base package, and the integration is tight. Its cohesive design saves you a lot of pain.

Was it as easy to set up as Prosody? Not exactly, but I'd say it's only marginally more difficult. After installing the ejabberd package on my system, I had to resolve a slight misconfiguration in the server's config file that wasn't obvious at first. (It was a bind problem that may have been an oversight by the package maintainer.) The log output was helpful to that end, and once the server was able to load and run, adding user accounts was simple.

Over time, what I learned to appreciate about ejabberd is how easy it is to work with. There might be a bit of a learning curve at first, and its many features and endlessly configurable behavior is truly daunting, but it mainly just works as-is and needs only a light touch to fit your needs perfectly.

Contrary to what I was led to believe, ejabberd is definitely fast and efficient. It is also stable and reliable, and seemingly handles whatever gets thrown at it. It can do some things that no other server can, and it's flexible and straightforward. My personal opinion is that it's the opposite of Prosody in the sense that Prosody starts out easy but gets harder when you need to tweak it, but ejabberd is a little harder to start out with but makes your life much easier down the line.

ejabberd can also do incredible things like hot-reload the server's configuration while the server is running, and there's no break in service. You can literally reconfigure almost all of the server's essential functionality with no downtime. (Want to reconfigure ports? Change modules? No problem!) Less impressive but greatly appreciated is the server's web interface, which makes user management and status monitoring as simple as possible.

On top of all this, my version of ejabberd was not even the latest one at the time I installed it. (It has since been updated.) Some major XMPP servers are using an even older version of it and still do everything Conversations needs. I also have since customized the server to provide the exact functionality I want, and I'm completely satisfied. There's maybe only one or two features left that I could enable, and I technically don't need them. (For example, XMPP over HTTP if I wanted to offer a web client.)

Any downside? Well, my little server has severely limited storage, and ejabberd likes to use storage space. For one thing, message history never expires, so that'll grow indefinitely unless you do something about it. A simple command to the server in a cron job handles that nicely. The other thing is that the server by default logs basically all network activity, which will get out of hand even with normal activity levels. Knocking the log level down a notch solves this easily.

Is Prosody bad? Not at all. Just make sure you're using the latest version possible, and keep it up to date as well. ejabberd and Prosody are probably the two best XMPP servers out there, though to be fair I have not tried the others to see how they truly compare. However, if you're seriously thinking of running an XMPP server of your own, just use ejabberd. I really can't recommend it highly enough, and I'm no stranger to running network servers. It's really not every day that you get a chance to discover software as great as this.


Posted by admin | Permanent link | Translate | File under: english

Sat Oct 12 10:45:41 IRST 2019

Duolingo Incubator

I've been using Duolingo off and on for the past several years, and over that time I've ended up sometimes waiting a long time for a course to come out. I probably started some time back in 2014, and at that time there were only a handful of language courses available to choose from. To be quite frank, most of them didn't even interest me all that much, but I remember casually messing around with Irish and some Germanic languages.

As it happened, really interesting courses were coming up in the incubator, with some of them (such as Polish) having already been there for such a long time. (Years, I suppose.) The only thing I could really do was wait, but fortunately after a year or so interesting languages started going into beta. That is actually how I learned Esperanto, though I hadn't been waiting for it.

After all these years, many courses have been introduced to the platform—some quickly, some slowly. I used to check the incubator very often just to see what new language I could look forward to trying in the near-ish future (say, in the following months). Now I've grown used to not checking it, but today I decided to give incubator.duolingo.com a look because I remembered how fun it was.

Well, a few things were to be expected. Still no end in sight for the Yiddish course, but I was pleased to see that Scottish Gaelic is now upcoming, and Finnish is coming along. I have been particularly wanting to try Finnish, so I'll be sure to check that out. Additionally, I happened to notice that the Hungarian course, which came out years ago, is still in beta. Taking a look at their page, it appears that the reason is they are now expanding the course into a second, extended version, as many courses have done.

Most of the other languages in beta (Latin, Navajo, Klingon) I find interesting but have not really made the time to do anything more with them than a few lessons here and there. When the Navajo course came out it was particularly bad, but maybe they've managed to improve it since then. (Hopefully they've added audio.) The Klingon course is surprisingly good if you like bizarre alien languages. (Of course, if you like that then try Hungarian as well.)

Also of note is that many of the courses in incubation now are for speakers of languages other than English, a trend that has been increasing and is sure to continue doing so now that there are so many courses for English speakers. Courses like "Arabic for Spanish speakers" sound nice.

Since lately I've been getting into Nordic langauges again, a course I'd like to see would be Icelandic or Old Norse, and maybe even a Nynorsk version of Norwegian, though the Bokmål course is already fantastic. (It is, in fact, one of the longest and best-quality courses on Duolingo.)

I could write more about how great it is for English speakers to study Norwegian, but I can save that for another time. As for "X for Y" language combos on Duolingo, it seems quite impossible for them to ever cross-connect all languages, but they've already laid some foundational connections by having most languages available through English and, conversely, English available through most languages. It's my suggestion, however, that they do this with Esperanto instead. That is, they should make Esperanto available to speakers of all languages, and then make all languages available to speakers of Esperanto. The simple reason is that people learn Esperanto an order of magnitude faster and more effectively than English, so making knowledge of English a condition for accessing other knowledge is at worst a grave injustice and foolish at best.

I would love to respond to responses to my posts, so I encourage anyone reading this to send me an e-mail with their own thoughts or questions. And, of course, if there's languages that you'd like to see on Duolingo, you should let Duolingo know! (And wait five years!)


Posted by admin | Permanent link | Translate | File under: english

Sat Oct 12 00:49:10 IRST 2019

Image Test


Posted by admin | Permanent link | Translate | File under: english

Fri Oct 11 22:39:40 IRST 2019

La Blogo Funkcias

Jen la unua afiŝo en Esperanto.

Posted by admin | Permanent link | Translate | File under: esperanto

Fri Oct 11 21:27:13 IRST 2019

Blog Up and Running

Everything appears to be working.

Posted by admin | Permanent link | Translate | File under: english