[elm-discuss] Implementing websocket-based protocol - lessons learned

[Christophe de Vienne]

Hi everyone,

Last week I coded a NATS [1] client, which is a PUB/SUB messaging
system, in Elm [2]. It was very instructive and the result is pretty
satisfying so far.

Along the way I learned a few things that might be of use for others
wanting to do this kind of client.

* The WebSocket API is a little short for this use-case:
- we need an API for being informed when the connection is
opened/closed.
- it lacks an API for sending ordered messages. It could be
Task-based, allowing to interrupt it one of the send fails,
or it could be a "sendAll" Cmd that takes a list of messages
to send.

Both this issues were fortunately worked-around in my case, but it
is only because the protocol allowed it. Others may not be so lucky.

* Using an effect module is not mandatory...

* ... but I had to mimick Cmd and Sub, which adds a lot of boilerplate
for the user (see the project README)

* not being an effect module has clear pros :
- we can rely on other effect modules (in my case: WebSocket and
Random)
- Implementation is somehow easier, and more friendly to maintain
- The module can be published on package.elm-lang.org

* Being an effect module would allow a much simpler and natural API.
But then we could not use WebSocket or Random


Finally, I would like to discuss a notion of middleware (which I think
is what I did with elm-nats):

We could simplify the API as much as with an effect module if we could
manipulate the Cmd and Sub.

I suspect that being able to filter a Cmd to extract all Cmd of a given
type in it would permit to implement some kind of middleware component.
Such a component would be explicitely inserted by the user between its
program and its TEA entry points.

It could define some custom Cmd and Sub, and would transform them to
other effect modules commands and subs just before returning them to the
program.

Has anything like this been attempted already?

Am I making any sense?


Best regards,

Christophe

[1] https://nats.io
[2] https://github.com/orus-io/elm-nats

['Rupert Smith' via Elm Discuss]

Post by Christophe de Vienne
- we can rely on other effect modules (in my case: WebSocket and
Random)
* Being an effect module would allow a much simpler and natural API.
But then we could not use WebSocket or Random

If you had used elmq to solve the issue of passing the messages around
within the Elm application you would not have run into this limitation.
That is to say that the dependency structure of your Nats module and the
effects module it uses would have looked like this:

Nats
-> WebSocket
-> Random
-> elmq

Where -> means depends on. But Nats itself would not be an effects module.

Post by Christophe de Vienne
Finally, I would like to discuss a notion of middleware (which I think
We could simplify the API as much as with an effect module if we could
manipulate the Cmd and Sub.
I suspect that being able to filter a Cmd to extract all Cmd of a given
type in it would permit to implement some kind of middleware component.
Such a component would be explicitely inserted by the user between its
program and its TEA entry points.
It could define some custom Cmd and Sub, and would transform them to
other effect modules commands and subs just before returning them to the
program.
Has anything like this been attempted already?

This is basically what elmq is. When you send a message the result is a Cmd:

send : String -> Value -> Cmd msg

And when you listen for messages the result is a Sub:

listen : String -> (Value -> msg) -> Sub msg

The only 'filtering' currently available is to be able to name a channel
(the String parameter in 'send' and 'listen'). More sophisticated
filtering, mapping, folding / whatever could be added to it.

[Christophe de Vienne]

I should have had a better look at what elmq allowed indeed.

That said until elmq (or something equivalent) is published on
package.elm-lang.org, we get a non-publishable package.

I will make some tests with now that the plain version is working properly.

Thanks!

Christophe

Post by Christophe de Vienne
- we can rely on other effect modules (in my case: WebSocket and
Random)
* Being an effect module would allow a much simpler and natural API.
But then we could not use WebSocket or Random
If you had used elmq to solve the issue of passing the messages around
within the Elm application you would not have run into this limitation.
That is to say that the dependency structure of your Nats module and the
Nats
-> WebSocket
-> Random
-> elmq
Where -> means depends on. But Nats itself would not be an effects module.
Finally, I would like to discuss a notion of middleware (which I think
We could simplify the API as much as with an effect module if we could
manipulate the Cmd and Sub.
I suspect that being able to filter a Cmd to extract all Cmd of a given
type in it would permit to implement some kind of middleware component.
Such a component would be explicitely inserted by the user between its
program and its TEA entry points.
It could define some custom Cmd and Sub, and would transform them to
other effect modules commands and subs just before returning them to the
program.
Has anything like this been attempted already?
send : String -> Value -> Cmd msg
listen : String -> (Value -> msg) -> Sub msg
The only 'filtering' currently available is to be able to name a channel
(the String parameter in 'send' and 'listen'). More sophisticated
filtering, mapping, folding / whatever could be added to it.
--
You received this message because you are subscribed to the Google
Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

['Rupert Smith' via Elm Discuss]

Post by Christophe de Vienne
I should have had a better look at what elmq allowed indeed.
That said until elmq (or something equivalent) is published on
package.elm-lang.org, we get a non-publishable package.

Yes, it would make your package non-publishable too.

Post by Christophe de Vienne
I will make some tests with now that the plain version is working properly.

Would be interesting to see if elmq has enough features to support your use
case, or if it needs some extension. I think gathering use cases, reviewing
what other language do and so on is the work that needs to be done prior to
proposing adding it as a supported effects module. Also, nice that you did
it without an effects module, as that is one more example to compare the
code with/without elmq, if you were to redo it on top of elmq.

I'm going to the upcoming Elm Europe conference, so I can talk to the core
devs there and see what they think of it. As I said before, perhaps its not
an 'architectural' direction that they like; on the other hand it does seem
both simple and useful.

['Rupert Smith' via Elm Discuss]

Post by 'Rupert Smith' via Elm Discuss
I'm going to the upcoming Elm Europe conference, so I can talk to the core
devs there and see what they think of it. As I said before, perhaps its not
an 'architectural' direction that they like; on the other hand it does seem
both simple and useful.

There was someone else who wrote a 'middleware' component for Elm, but did
it with 'out messages' instead of as an effect module. There was a routing
part that you put in you top-level update function. Let me see if I can
find it...

One criticism of elmq might be that it is not so transparent. So if a
module sends a message, the Cmd for that is passed into the Elm kernel, and
from then on it sort of disappears from the compilers view - the messages
are untyped in so far as that they are Json.Encode.Values. There is no way
for the compiler to check that whatever receives them has a matching type,
or indeed that there is any receiver listening at all. At least with 'out
messages' the type checking will flow across the modules. Of course there
could still be a bug in code where the 'out messages' are ignored and not
processed. It may be possible to improve the elmq API to mitigate these
issues - for example, an option to discard messages when there is no
receiver and so on.

['Rupert Smith' via Elm Discuss]

Post by 'Rupert Smith' via Elm Discuss
There was someone else who wrote a 'middleware' component for Elm, but did
it with 'out messages' instead of as an effect module. There was a routing
part that you put in you top-level update function. Let me see if I can
find it...

This:

https://groups.google.com/forum/#!searchin/elm-discuss/messaging%7Csort:relevance/elm-discuss/gSkvCgTj4q0/TJJCtcX9BgAJ

https://github.com/RJWoodhead/elm-memo

[Christophe de Vienne]

Hi

Post by Christophe de Vienne
I will make some tests with now that the plain version is working properly.
Would be interesting to see if elmq has enough features to support your
use case, or if it needs some extension. I think gathering use cases,
reviewing what other language do and so on is the work that needs to be
done prior to proposing adding it as a supported effects module. Also,
nice that you did it without an effects module, as that is one more
example to compare the code with/without elmq, if you were to redo it on
top of elmq.

I hope I am wrong but I do not think I can achieve a natural
subscription API on top of elmq. I cannot work on it right now but will.

Post by Christophe de Vienne
I'm going to the upcoming Elm Europe conference, so I can talk to the
core devs there and see what they think of it. As I said before, perhaps
its not an 'architectural' direction that they like; on the other hand
it does seem both simple and useful.

A discussion around a concept of "middleware", able to define its own
Cmd and Sub that it could rewrite along with updating its own state
(which can be a part of the app model), would be interesting.

Cmd and Sub are pretty opaque beasts from where I stand, may be we can
already do such a thing.

[Christophe de Vienne]

Post by Christophe de Vienne
A discussion around a concept of "middleware", able to define its own
Cmd and Sub that it could rewrite along with updating its own state
(which can be a part of the app model), would be interesting.
Cmd and Sub are pretty opaque beasts from where I stand, may be we can
already do such a thing.
http://package.elm-lang.org/packages/mdgriffith/elm-style-animation/3.5.5/Animation
provides a function to build a subscription. This is just a convenience
function; under the covers the animations are being iterated as a timer
ticks, so the real Sub being created is a timer one.
So I think if you can define what flexible 'command' and 'subscription'
constructor functions you think the middleware needs, and we can try and
map those down to the simplest Cmd and Sub implementations that cover
all the needed use cases, then that gives us what we would need to
implement in an effects module. Essentially, it would be a messaging
kernel that is flexible enough that any reasonable messaging application
you like can be built on top of it.

It is what I did at first, but it has serious drawbacks. The biggest was
that a subscription implies an update of the Nats state... so I had to
push the state to all the update functions and they had to sent it
back... Very messy.

The final API I have, although it has some boilerplate, is a lot nicer.

['Rupert Smith' via Elm Discuss]

Post by Christophe de Vienne
It is what I did at first, but it has serious drawbacks. The biggest was
that a subscription implies an update of the Nats state... so I had to
push the state to all the update functions and they had to sent it
back... Very messy.

This use case can be catered for by using a correlation id, I think. Also,
perhaps the word 'subscription' is getting overloaded here, you have the
Elm Sub(scription), and the NATS protocol subscription, and the middleware
subscription (listen), making this conversation a little difficult to nail
down...

== Module Consuming the NATS Protocol

Elmq.sendString "subscribe" "nats0"
Elmq.listen "nats0"

== The module implementing the NATS protocol

type Msg =
NewNatsSubscription String
| ...

Elmq.listenString "subscribe" (\channel -> NewNatsSubscription)
Elm.send channelName ...

==

So I sent a subscription request to Nats, and when Nats processed it, it
sends back messages on the named channel. The channel name is used as a
correlation id to link together the act of creating the subscription and
the messages that then flow over that subscription.

I don't know the specific of NATS and whether it has named channels etc.
but hopefully you get what I am on about?