beatrichartz / parallel_stream Goto Github PK

View Code? Open in Web Editor NEW

99.0 4.0 22.0 81 KB

A parallelized stream implementation for Elixir

License: MIT License

Elixir 99.06% Shell 0.94%

stream elixir parallelize parallelism

parallel_stream's Introduction

ParallelStream

Parallelized stream implementation for elixir

What does it do?

Parallelize some stream operations in Elixir whilst keeping your stream in order. Operates with a worker pool.

How do I get it?

Add

{:parallel_stream, "~> 1.1.0"}

to your deps in mix.exs like so:

defp deps do
  [
    {:parallel_stream, "~> 1.1.0"}
  ]
end

Note: Elixir 1.5.0 is required

How to use

Do this to parallelize a map:

stream = 1..10 |> ParallelStream.map(fn i -> i * 2 end)
stream |> Enum.into([])
[2,4,6,8,10,12,14,16,18,20]

The generated stream is sorted the same as the input stream.

More supported functions are each (to produce side-effects):

1..100 |> ParallelStream.each(&IO.inspect/1)

filter:

stream = 1..20 |> ParallelStream.filter(fn i -> i |> rem(2) == 0 end)
stream |> Enum.into([])
[2,4,6,8,10,12,14,16,18,20]

and filter's counterpart, reject:

stream = 1..20 |> ParallelStream.reject(fn i -> i |> rem(2) == 0 end)
stream |> Enum.into([])
[1,3,5,7,9,11,13,15,17,19]

License

MIT

Contributions & Bugfixes are most welcome!

parallel_stream's People

Contributors

Stargazers

Watchers

parallel_stream's Issues

Deprecation warnings: "coveralls" and Stream.chunk/4 -> Stream.chunk_every/4

warning: found quoted keyword "coveralls" but the quotes are not required. Note that keywords are always atoms, even when quoted. Similar to atoms, keywords made exclusively of ASCII letters, numbers, and underscores do not require quotes
/Users/ernest/Developer/igwet/deps/parallel_stream/mix.exs:17:30

==> parallel_stream
Compiling 11 files (.ex)
warning: Stream.chunk/4 is deprecated. Use Stream.chunk_every/4 instead
lib/parallel_stream/producer.ex:16: ParallelStream.Producer.build!/4

Better documentation on `num_workers`, `worker_work_ratio`

Hi!

Thanks for the library!

Could you write up some documentation on how those options work for ParallelStream.map?

My intuition says that as long as num_workers: 1, it's effectively not parallel, and worker_work_ratio is for managing the batch size passed to each worker for fine tuning acceptable parallel overhead?

Lower overhead of parallelism if possible

Currently, the cutoff for getting performance gains is at about 20 microseconds (see #1) - although that's quite good, there are possibly ways to further lower the overhead by doing some performance optimisations.

Possible to version bump and release?

I only ask because I'm getting a bunch of compiler warnings about the zero arity functions.

I was planning on fixing them and sending a pull request before I noticed that they were already fixed.

If you were waiting on adding some features before releasing a new version, I understand. Just thought I'd ask. Thanks!

Remove warnings

Not a big deal but I have those warnings which rise when I compile my app:

warning: found quoted keyword "coveralls" but the quotes are not required. Note that keywords are always atoms, even when quoted. Similar to atoms, keywords made exclusively of Unicode letters, numbers, underscore, and @ do not require quotes
  /app/deps/parallel_stream/mix.exs:17
==> parallel_stream
warning: Stream.chunk/4 is deprecated. Use Stream.chunk_every/4 instead
  lib/parallel_stream/producer.ex:16
Generated parallel_stream app

Trying to get rid of it.

Infinite Stream does run forever

I have an infinite stream comming in and want to map over it, then filter
and as a last step I have a simple Enum.take(1).

Using stdlib Stream functions, my tests finish after a couple of seconds,
using ParallelStream my testsuite does time out.

When I'm back home this evening, I'll try to add some example to reproduce the
behaviour.

What git version is hex version 1.0.6 based upon?

I see there's a 1.0.6 in hex but the corresponding github tag is missing. Maybe this tag needs to be created ?

Bump version

The latest version on hex.pm is outdated, from 2017. Since there are some new changes, including silencing compiler warnings, could you please bump the version so those changes will be accessible through mix?