Experiments with xmpp

Monday, November 22, 2010

"Re-scoping" changes to XMPP draft

Looks like my timing was really lucky - I raised the "re-scoping" issue before the bis spec was submitted to IESG : so after a slightly protracted discussion on the xmpp ietf mailing list, the new spec addresses the concerns in a reasonably good way - adds constraints which should, hopefully, eliminate the issues.

Please let me know in case you see any issues !

Monday, November 15, 2010

Giving up on tigase

Wanted to run the same experiments on tigase which I did against ejabberd and openfire : and gave up. Too many installation (configuration rather) issues. It keeps complaining about db setup issues, wont allow for overriding ports, etc.

If I was more interested, I might try to hunt down the issues : but somehow, not that driven unfortunately.

This is actually an interesting takeaway actually ... administration of the server should be as easy as possible. I am actually very pleased with the way openfire folks have handled this aspect !

Particularly when you want to target developers, it really helps if you can bring up server and manage it without needing to muck around unnecessarily with internals of the deployment, config db, etc. For powerusers, this is required ofcourse (so no binary blobs for config please !) but for a casual user, a simple and elegant UI goes a long way.

Using NC to simulate prefix-free canonicalization

The rough sample of clients I looked at did not allow me to hack around with how is handled namespacing, and I did not want to play too much with the guts of an api to run this simple test.

So I simulated it with nc - to hack up a "client" to do the same.

Essentially took a socket dump of a 'valid' session between psi and wildfire server, and modified it simulate prefix-free canonicalization.

The actual flow was :

$ nc

This will open a socket to hostname:port, and allow us to read/write off the socket, neat.

Now we essentially just copy each line over, wait for response (if required) and on to next line.

Whitespace/newlines can be liberally sprinkled between lines for legibility.

--- start --


<stream:stream xmlns:stream="http://etherx.jabber.org/streams" version="1.0" to="openfire.daigoro.net" xml:lang="en" xmlns:xml="http://www.w3.org/XML/1998/namespace" xmlns='jabber:client'>


<auth xmlns="urn:ietf:params:xml:ns:xmpp-sasl" mechanism="PLAIN">AG1yaWR1bABhYmMxMjM=</auth>


<stream:stream xmlns:stream="http://etherx.jabber.org/streams" version="1.0" to="openfire.daigoro.net" xml:lang="en" xmlns:xml="http://www.w3.org/XML/1998/namespace" >


<iq type="set" id="bind_1"  xmlns='jabber:client'>
<bind xmlns="urn:ietf:params:xml:ns:xmpp-bind">
<resource>direct_conn</resource>
</bind>
</iq>


<iq type="set" id="aad8a"  xmlns='jabber:client'>
<session xmlns="urn:ietf:params:xml:ns:xmpp-session"/>
</iq>


<iq type="get" id="aad9a"  xmlns='jabber:client'>
<query xmlns="jabber:iq:roster"/>
</iq>


<presence  xmlns='jabber:client'>
<priority>5</priority>
<c xmlns="http://jabber.org/protocol/caps" node="http://psi-im.org/caps" ver="0.13-dev-rev2" ext="cs ep-notify html"/>
</presence>

--- end ---

Sunday, November 14, 2010

Re-scoping experiment 2 - openfire to ejabberd

Update: I re-ran the exact same experiment using nc to simulate a client which does prefix free canonicalization and got exact same result.

To do so, initial stream is opened with xmlns="jabber:client", but subsequent (post-auth) stream does not have xmlns set to jabber:client. Here is an example of how to do it.

In continuation of the re-scoping experiment series, the next attempt was to send stanza's from openfire to ejabberd and look at the behavior of how they are handled.

For the experiment, the versions used are :

ejabberd, Version: 2.1.5

Openfire, Version: 3.6.4

The control stanza.

As with previous experiment, the control stanza is a simple message with a body from a user connected to openfire server to a recipient in ejabberd server.

<message  to="mridul@ejabberd.daigoro.net" xml:lang="en">
<body>Test 0</body>
</message>

This was received as expected by the user connected to ejabberd.

So our basic setup is working fine !

Top level stanza with explicit namespace.

<ns:message  xml:lang="en" xmlns:ns="jabber:client" to="mridul@ejabberd.daigoro.net" >
<ns:body>Test 1</ns:body>
</ns:message>

If you recall, ejabberd DOES NOT allow this to be transmitted, but openfire does !

On the S2S connection, openfire sends following to ejabberd.


<message xmlns:ns="jabber:client" xml:lang="en" to="mridul@ejabberd.daigoro.net" from="mridul@openfire.daigoro.net/daigoro">
<body>Test 1</body>
</message>

Interestingly, openfire re-scopes not just the top-level message stanza, but also the children !

There is a redundent namespace declaration - which is not used, we can ignore that for now.

At the client connected to ejabberd, we recieve

<message from="mridul@openfire.daigoro.net/daigoro" xml:lang="en" to="mridul@ejabberd.daigoro.net">
<body>Test 1</body>
</message>

which is essentially what was sent across to begin with.

So final received stanza is same as initially transmitted stanza : but the re-scoping done over s2s if quite interesting.

To further investigate this, we try our second stanza : explicit namespaces all the way to the children.

Explicit namespace all over.

<ns:message  xml:lang="en" xmlns:ns="jabber:client" to="mridul@ejabberd.daigoro.net" >
<ns1:body xmlns:ns1="jabber:client">Test 2</ns1:body>
</ns:message>

Over S2S, we have the following sent from openfire to ejabberd :

<message xmlns:ns="jabber:client" xml:lang="en" to="mridul@ejabberd.daigoro.net" from="mridul@openfire.daigoro.net/daigoro">
<body xmlns:ns1="jabber:client">Test 2</body>
</message>

We observe that openfire tries to rescope all elements corresponding to "jabber:client" to "jabber:server".

Explicit namespace all over, with custom attributes and child element.

This is an additional test corresponding to previous experiment - to see what happens to attributes and 'unknown' child elements.

<ns:message  xml:lang="en" xmlns:ns="jabber:client" to="mridul@ejabberd.daigoro.net" ns:test_attr_1="test_val1" >
<ns1:body xmlns:ns1="jabber:client" ns1:test_attr2="test_val2">Test 2.1</ns1:body>
<ns1:custom_elem xmlns:ns1="jabber:client" ns1:test_attr2="test_val2">Test 2.2</ns1:custom_elem>
</ns:message>

So here we have two additional attributes : one at the stanza level and one at the child level qualified by ns1 and ns2 for the "jabber:client" namespace.

We also have a additional child element qualified to "jabber:client" namespace.

What gets transmitted from openfire to ejabberd is :

<message xmlns:ns="jabber:client" xml:lang="en" to="mridul@ejabberd.daigoro.net" ns:test_attr_1="test_val1" from="mridul@openfire.daigoro.net/daigoro">
<body xmlns:ns1="jabber:client" ns1:test_attr2="test_val2">Test 2.1</body>
<custom_elem xmlns:ns1="jabber:client" ns1:test_attr2="test_val2">Test 2.2</custom_elem>
</message>

What is received by client in the end is :

<message from="mridul@openfire.daigoro.net/daigoro" ns:test_attr_1="test_val1" xmlns:ns="jabber:client" xml:lang="en" to="mridul@ejabberd.daigoro.net">
<body ns1:test_attr2="test_val2" xmlns:ns1="jabber:client">Test 2.1</body>
<custom_elem ns1:test_attr2="test_val2" xmlns:ns1="jabber:client">Test 2.2</custom_elem>
</message>

So essentially, while re-scoping, openfire seems to be attempting to re-scope all elements while leaving attributes as-is.

This is definitely DIFFERENT from what is done by ejabberd, so we have a definite incompatibility in terms of how re-scoping is handled.

Explicit namespace only for child elements.

<message  xml:lang="en" to="mridul@ejabberd.daigoro.net" >
<ns1:body xmlns:ns1="jabber:client" >Test 3</ns1:body>
</message>

What gets sent over S2S from openfire to ejabberd is :

<message xml:lang="en" to="mridul@ejabberd.daigoro.net" from="mridul@openfire.daigoro.net/daigoro">
<body xmlns:ns1="jabber:client">Test 3</body>
</message>

So re-scoping is attempted for not just the stanza but also for all child elements.

So essentially, re-scoping rule for openfire seems to be : for any element in "jabber:client" namespace, change namespace to "jabber:server" namespace. This is applicable ONLY to elements, for attributes, openfire does not do any rescoping.

Variant of the above - with explicit "jabber:server" namespace.

<message  xml:lang="en" to="mridul@ejabberd.daigoro.net" >
<ns1:body xmlns:ns1="jabber:server" >Test 4</ns1:body>
</message>

What gets sent over S2S is:

<message xml:lang="en" to="mridul@ejabberd.daigoro.net" from="mridul@openfire.daigoro.net/daigoro">
<body xmlns:ns1="jabber:server">Test 4</body>
</message>

And what is received by the client is :

<message from="mridul@openfire.daigoro.net/daigoro" xml:lang="en" to="mridul@ejabberd.daigoro.net">
<body>Test 4</body>
</message>

So again, we have an incompatibility - what is sent is NOT what is received.

From these two experiments, we can clearly see that there is incompatibility in terms of how re-scoping is handled by ejabberd and openfire : and so the expectation of what gets received by clients/components compared to what gets sent.

Please note that this was just a simple set of examples targeted at how re-scoping is to be handled, and not contrived cases ! I am sure there are a lot of usecases and variants I am missing !

Hopefully, this issue might get addressed in this or next version of the bis XMPP spec.

Re-scoping experiment 1 - ejabberd to openfire

Update: I re-ran the exact same experiment using nc to simulate a client which does prefix free canonicalization and got exact same result.

To do so, initial stream is opened with xmlns="jabber:client", but subsequent (post-auth) stream does not have xmlns set to jabber:client. Here is an example of how to do it.

As mentioned in a previous post, I was wondering how rescoping was supposed to be handled in a general case. That is, server gets an xml stanza from client meant for a (user on) remote server and has to, as robustly possible, re-scope it to ensure successful delivery.

ASCII magic :-P

user1 -- c2s --> server1 --- s2s ---> server2 -- c2s --> user2

Re-scoping is required since the stanza namespace expected over s2s is "jabber:server" while over c2s is "jabber:client".

Please note: server1 and server2 here will be on different domains, most probably different implementations : ditto for the clients. So we are touching atleast 4 different XMPP codebases.

Now that intro is done, let us move on.

As a first experiment I tried a simple one - I set up ejabberd and openfile servers on my box, tapped their s2s communication, used psi as both transmitting and receiving client and then started sending stanza's of interest to see the behavior.

I will, time permitting ofcourse, try other servers later on !

My assumption was, these are fairly popular servers, and so would be instructive to see how they currently handle this.

From ejabberd to openfire.

Openfire, Version: 3.6.4

ejabberd, Version: 2.1.5

The control stanza.

<message  to="mridul@openfire.daigoro.net" xml:lang="en">
<body>Test 0</body>
</message>

This made it all the way through, the only (expected) change was the addition of a "from" attribute by ejabberd server before sending stanza to openfire.

So our basic setup is sound and works fine.

Top level stanza with explicit namespace.

<ns:message xml:lang="en" xmlns:ns="jabber:client" to="mridul@openfire.daigoro.net" ><ns:body>Test 1</ns:body>
</ns:message>

This DOES NOT make it through.

It reaches the ejabberd server, but nothing over the S2S connection to openfire.

On the other hand, the bis spec does discourage this practice - I will need to hack up my own xmpp client (suggestions anyone ?) to try out prefix-free canonicalization to see if it works.

I hope this did not work 'cos of ejabberd conforming to this restriction from the spec !

Explicit namespace all over.

<ns:message  xml:lang="en" xmlns:ns="jabber:client" to="mridul@openfire.daigoro.net" >
<ns1:body xmlns:ns1="jabber:client" >Test 2</ns1:body>
</ns:message>

Since the previous stanza had issues, I did not really expect this one to go through, and it did not.

Explicit namespace only for child elements.

This was interesting case - the stanza did not have any explicit namespace, while the child (body) was explicitly qualified to "jabber:client".

The observations were :

ejabberd DID NOT re-scope the child element. So, the body continued to be in the explicitly declared "jabber:client" namespace.
openfire stripped the explicit "jabber:client" namespace before forwarding to user2.

Interesting result this.

Variant of above - with explicit "jabber:server" namespace.

The body sub-element is explicitly qualified with "jabber:server" namespace by the client.

<ns1:body xmlns:ns1="jabber:server" >Test 4</ns1:body>
</message>

And there-in lay the surprise, ejabberd exhibited a behavior consistent with the previous case : it seems to check only namespace at the stanza level and pretty much treats everything else inside as 'opaque'.

So it transmitted the body with the explicit "jabber:server" namespace to openfire.

<message from='mridul@ejabberd.daigoro.net/daigoro' to='mridul@openfire.daigoro.net' xml:lang='en'>
<ns1:body xmlns:ns1='jabber:server'>Test 4</ns1:body>
</message>

But openfire's behavior was unexpected (atleast compared to how ejabberd worked).

It rescoped the body to "jabber:client" namespace !

So user 2 received :

<message from="mridul@ejabberd.daigoro.net/daigoro" xml:lang="en" to="mridul@openfire.daigoro.net">
<body>Test 4</body>
</message>

which is definitely NOT what was sent !

Ah, so we have our first 'breakage'.

What was sent was NOT what was received.

Variant of above - with explicit custom namespace.

The breakage above could be either due to

re-scoping error by openfire
a general namespace handling error.

In order to validate which one it was, I sent another stanza : this time, body was explicitly namespaced with a custom namespace.

<message  xml:lang="en" to="mridul@openfire.daigoro.net" >
<ns1:body xmlns:ns1="dummy:custom:ns" >Test 5</ns1:body>
</message>

We received this as-is at the recipient client : so what we observed in the previous case was definitely a re-scoping error : and NOT a general namespace handling issue with openfire.

So we have our first case of re-scoping incompatibility.

Sure, it can be argued that this is contrived since client's dont use "jabber:server" namespace. My point would essentially be, I would rather prevent this from happening to begin with than to see breakages in the wild.

In next post, we will try to see how the openfire to ejabberd communication works.

Rescoping XMPP stanza's

An interesting aspect of XMPP is 're-scoping' of XML stanza's between client and server namespace. To elaborate briefly : xmpp clients send xml stanza's namespace'd by 'jabber:client', while stanza's are namespace'd by 'jabber:server' between servers.

So if user1 on server1 needs to send a xml stanza to user2 on server2, server1 will need to 're-scope' the stanza from "jabber:client" namespace to "jabber:server" namespace before transmitting it to server2, and server2 will need to re-scope it back to "jabber:client".

The interesting thing here is, re-scoping xml snippets in a general case, is quite tricky unless it is fully specified : and while going over the xmpp-bis spec for 3920 rfc, I found the description insufficiently robust. Dont get me wrong, it does describe what is required : but there seems to be a lot of loose ends.

As part of hacking up a new xml streaming parser, I was struck by the ambiguity regarding this aspect in XMPP spec.

To give a few examples, consider the following xml stanza's, how are they to be 're-scoped'.

<message  to="user2@server2" xml:lang="en"><body>Test 0</body></message>

<ns:message  to="user2@server2" xml:lang="en" xmlns:ns="jabber:client"><body>Test 1</body></ns:message>

<ns:message  to="user2@server2" xml:lang="en" xmlns:ns="jabber:client"><ns1:body xmlns:ns1="jabber:client" >Test 2</ns1: body></ns:message>

Actually, all three of them are exactly equivalent from client's xml parser point of view. There are, definitely, many more variants of the above : I am just restricting to some simple examples.

If you think this is an elaborate stretch, let me assure you it is not : a simple example of xmpp client which is aggregating xml snippets from various producers and sending it out : so the content (like body here) is not too much in its control.

Not to mention, the XMPP spec is defining xml streams, so definitely needs to be as robust as possible in terms of defining what this means.

There are, ofcourse, other variations of this - like explicitly namespaced attributes, etc. I will post some experimental results later on.

Introduction

This blog attempts to document my experiments with and thoughts on XMPP.

My association with XMPP has been sporadic over the years and yet intense when I could devote time to it :-)

I used to work for the Instant Messaging team at Sun Microsystems : where I worked on the server, http binding component and client api. There might be some interesting XMPP/Sun IM related info in the blog there.

Since joining Yahoo! Labs, I have been unable to do anything serious on/with XMPP ...

This blog is another attempt to jump start and attempt to get involved with XMPP again :-)

Standard disclaimer applies - this blog expresses my personal thoughts and opinions and past or present employers are not associated with it or liable in any way.