[elm-discuss] Batching HTTP requests and then performing something after they all finish

[Rafał Cieślak]

I have a dict which acts as an album cover cache:

1. I download a list of albums from Last.fm.
2. I send a separate request for each album to get its cover image URL
(right now i use Cmd.batch).
2.1 After I receive an image URL from Last.fm, I add it to the cache, so
that I have a mapping from album ID to album image URL.
3. Once all the album image URL requests finish (no matter if any of them
fails), I'd like to send the cache through a port so that JS can save it in
localStorage.

I don't know how to approach this. The Cmd API allows for batching, but not
sequencing and the Task API allows for sequencing, but not batching. I can
turn a Task into Cmd, but not the other way around.

So if I have a bunch of batched HTTP requests, I'm not really able to
sequence something after they happen, or at least I don't see how I can do
this without adding some dirty workarounds to the update function.

Theoretically I could wrap the album image URLs into RemoteData and then
after receiving any of them, see if the rest of the album image URLs is
finished and if so, send the cache to JS.

[Brian Hicks]

It sounds like you might actually want to do this in update! You're going
to have to write an Msg for getting new album covers anyway, right? And,
conceptually, it sounds like you want to save the results you get but cache
them for future use through a port.

Your logic branches there, so it makes sense to say, in response to your
Msg, "save these to display to the user. Also, store them so I don't have
to make this request again."

If my assumption there is wrong, please correct me. The mechanical way to
do *exactly *what you're asking is putting your tasks in Task.sequence and
sending the result through Task.andThen. But I'm not sure you want to do
that, for the reasons above. Your task handling logic would get really
hairy, when it has a natural fit in the Elm Architecture.

Post by Rafał Cieślak
1. I download a list of albums from Last.fm.
2. I send a separate request for each album to get its cover image URL
(right now i use Cmd.batch).
2.1 After I receive an image URL from Last.fm, I add it to the cache, so
that I have a mapping from album ID to album image URL.
3. Once all the album image URL requests finish (no matter if any of them
fails), I'd like to send the cache through a port so that JS can save it in
localStorage.
I don't know how to approach this. The Cmd API allows for batching, but
not sequencing and the Task API allows for sequencing, but not batching. I
can turn a Task into Cmd, but not the other way around.
So if I have a bunch of batched HTTP requests, I'm not really able to
sequence something after they happen, or at least I don't see how I can do
this without adding some dirty workarounds to the update function.
Theoretically I could wrap the album image URLs into RemoteData and then
after receiving any of them, see if the rest of the album image URLs is
finished and if so, send the cache to JS.

[Eric G]

If you model the dict values (image URLs) with something like

type AlbumImageState
= NotLoaded
| Cached Url
| Loaded Url
| Error Http.Error

Would that help? When you get back the list in the first request,
initialize the values to NotLoaded. Then you could then write a function
like

allLoadedOrError : List ImageUrlState -> Bool

to check if all the album-image requests have returned, before writing the
successes to localstorage.

[Eric G]

If you model the dict values (image URLs) with something like

type AlbumImageState
= NotLoaded
| Cached Url
| Loaded Url
| Error Http.Error

Would that help? When you get back the list in the first request,
initialize the values to NotLoaded. Then you could then write a function
like

allLoadedOrError : List AlbumImageState -> Bool

to check if all the album-image requests have returned, before writing the
successes to localstorage.

[Rafał Cieślak]

Eric: This is what I meant when I suggested wrapping the URLs with
RemoteData
<http://package.elm-lang.org/packages/krisajenkins/remotedata/latest>. ;)
It does pretty much exactly what you described and I'm thinking about using
it anyway, as it seems that it's going to be helpful in other parts of the
app as well.

Brian: As far as I understand your approach, your way of solving that
involves sending each new album cover through a port separately, which is
okay. I was thinking about sending the whole dict (cache) to JS once I know
that I finished downloading all the images.

it sounds like you want to save the results you get but cache them for

Post by Brian Hicks
future use through a port

More-or-less yes, with the difference that right now the cache lives in Elm
and the only thing I want to do outside of Elm is to save it to
localStorage so that I can rebuild the cache the next time I reload the app.

So your approach could certainly work, but I think I'm going to use the
second solution and wrap the values in the cache with RemoteData. I think
I'd rather have as few interactions with JS as possible – in the second
solution I'd pass the cache from JS to Elm through flags and then pass it
back from Elm to JS once I finish downloading all images.

---

Obviously we could come up with even more ideas. I think my main gripe was
that you can either sequence then batch or just batch, but you can't batch
and then sequence, because the API doesn't allow you to do that.

I haven't given much thought to it, but I feel it's because Task and Cmd
have fundamentally different approach to error handling. Adding Task.batch
would require either imposing certain semantics of errors (like
Task.sequence does) or offering an API that would allow the user to
implement error semantics themselves. And I'm not sure how easy of a task
the latter thing is.

Even in JS with promises such sequencing would require some
not-quite-elegant solutions.

[Brian Hicks]

I think there's been something lost here. I'm suggesting you do:

Task.sequence [ allYourThings ] |> Task.perform Cmd

And then in the update,

{ model | covers = covers } ! [ storeInJS covers ]

Eric: This is what I meant when I suggested wrapping the URLs with RemoteData <http://package.elm-lang.org/packages/krisajenkins/remotedata/latest>. ;) It does pretty much exactly what you described and I'm thinking about using it anyway, as it seems that it's going to be helpful in other parts of the app as well.
Brian: As far as I understand your approach, your way of solving that involves sending each new album cover through a port separately, which is okay. I was thinking about sending the whole dict (cache) to JS once I know that I finished downloading all the images.
it sounds like you want to save the results you get but cache them for future use through a port
More-or-less yes, with the difference that right now the cache lives in Elm and the only thing I want to do outside of Elm is to save it to localStorage so that I can rebuild the cache the next time I reload the app.
So your approach could certainly work, but I think I'm going to use the second solution and wrap the values in the cache with RemoteData. I think I'd rather have as few interactions with JS as possible – in the second solution I'd pass the cache from JS to Elm through flags and then pass it back from Elm to JS once I finish downloading all images.
---
Obviously we could come up with even more ideas. I think my main gripe was that you can either sequence then batch or just batch, but you can't batch and then sequence, because the API doesn't allow you to do that.
I haven't given much thought to it, but I feel it's because Task and Cmd have fundamentally different approach to error handling. Adding Task.batch would require either imposing certain semantics of errors (like Task.sequence does) or offering an API that would allow the user to implement error semantics themselves. And I'm not sure how easy of a task the latter thing is.
Even in JS with promises such sequencing would require some not-quite-elegant solutions.
--
You received this message because you are subscribed to a topic in the Google Groups "Elm Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elm-discuss/22Tt_p2OpMY/unsubscribe <https://groups.google.com/d/topic/elm-discuss/22Tt_p2OpMY/unsubscribe>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

[Rafał Cieślak]

Oh okay, sorry. :D

But this won't batch the requests, right? By "batching" I mean "run all in
parallel" as Cmd.batch seems to do. As far as I understand Task.sequence
docs
<http://package.elm-lang.org/packages/elm-lang/core/5.1.1/Task#sequence>,
it's going to make the requests one-by-one. I'd rather load them in
parallel.

What's more important though is that Task.sequence is going to fail if any
task fails – I'm okay with some of the requests failing and this behavior
would make this API a bit harder to work with in my case.

[Eric G]

Post by Rafał Cieślak
Eric: This is what I meant when I suggested wrapping the URLs with
RemoteData
<http://package.elm-lang.org/packages/krisajenkins/remotedata/latest>. ;)
It does pretty much exactly what you described and I'm thinking about using
it anyway, as it seems that it's going to be helpful in other parts of the
app as well.

I had assumed you were caching at the individual album-image level, which
is why I had that additional `Cached Url` case. But maybe you don't need it.