If all issues for alpha are done, the last thing to do is some extensive real life testing to see if it all works as intended. Also a durability test might be good, running it for about an hour.

Currently a lot of new threads are started for asynchronous tasks. We should use a thread pool for better use of resources.

To minimize the impact of stop-the-world GC, we should produce as little garbage as possible. The main idea is to reuse as many data structures as possible used in a step calculation. This can be achieved in various ways, some ideas:

• Use a StepData object that is part of an ActiveScenario and holds all data structures needed (mainly arrays, maybe also maps?) that can be reused every step
• These StepData objects then need to be object pooled to prevent littering old gen with these. Apache Commons Pool is a good library for this
• Another option is to pool the data structures directly, but that introduces more complexity in acquiring and releasing.

To keep in mind:

• All arrays should be the max possible size, when amountOfFrames needed covers the full buffer millis
• You need an actual size as extra parameter all over the code handling arrays, cause the real array size will be that maximum
• calculateAmountOfFramesNeeded should never return a higher value then needed for the full buffer millis (implementation should be checked on this)
• byte[]'s for the sound card streams should be polled separately, because they are the only one 'escaping from the step' and written to the stream in a separate thread.
• Measuring is knowing!

https://github.com/ewjmulder/tinylog/blob/v1.3/tinylog/src/org/pmw/tinylog/Tokenizer.java
new inner class MaxSizeToken that just cuts off first or last part that is too long
new inner class FixedSizeToken that combines behavior of min-size and max-size

Starting the first scenario will give hickups because of JVM warmup. Currently that is not solved by letting the AudioMixer start early, cause there is no scenario and that code is not hit. So create a fake warmup scenario that has an audio input stream of silence or random but just have fixed speaker volumes of 0 or so. At least let the scenario code run without producing sound for a few seconds so it is ready to start the next one at full JVM speed.
Note: also let this warmup scenario restart a few times, cause that code part also needs some warmup!

Test Immerse to see where a possible limitation may be in the amount of sound cards supported / smooth playback, etc.

This is the same idea as fixed speaker ratios, but can be implemented even easier when taken into account explicitly.

When there are no active scenario's, do not actually provide all 0's to the hardware
So do not 'keep the engine running' when there is no music to play.
Need to refill buffers first after new scenario pops up.

Is this a useful feature? What does it solve?

Currently, when calculating a dynamic volume, only the angle to the speaker is taken into account. But it makes sense that the same angle, but much difference in distance should also have it's effect on the volume. This might be tricky in the current setup, cause normalizing will remove this effect. Maybe it should be some kind of extra feature build in the system that the algorithms should be aware of. Needs some further thinking...
See also notes about this in #24.

First go low profile for the alpha release, but when reaching a beta release, use Github Pages to get some more attention and a flashy website to point to.

See AudioSystem.isConversionSupported
See also: https://www.javaworld.com/article/2076227/java-se/add-mp3-capabilities-to-java-sound-with-spi.html for using the SPI implementation system. But actually no default conversion seems to be present, so maybe better use your own taste.
Actually the isConversionSupported returns true for a lot of possibilities, so do use that if it indeed actually works.

Store TBD: Postgres / MongoDB / ??

What to persist:

• Speaker Matrix
• Sound card data
• Sound card to speaker mapping
• Room and Scenario data

Even more dynamic would be to use audio input streams that change based on what is happening. Dynamic sound waves based on movement, etc. But the downside is that you must keep the read buffer very low for that.

Not a feature for the short term!

Use thread (pool) to read from AudioInputStream to BufferedInputStream or so and control the buffer size explicitly. Initially this can be set high to always have bytes available quickly when needing to read.

• Total JVM memory (min=max)
• Eden space memory (min=max)
• max tenure limit?
• young gen fraction?
• anything else?

First only for Linux, use the /dev/snd/by-path mapping

Consider using the extra USB information from LibUSB:

       Context context = new Context();
        int result = LibUsb.init(context);
        if (result != LibUsb.SUCCESS)
            throw new LibUsbException("Unable to initialize libusb.", result);
        // Read the USB device list
        DeviceList list = new DeviceList();
        int result2 = LibUsb.getDeviceList(null, list);
        if (result2 < 0)
            throw new LibUsbException("Unable to get device list", result);

        try {
            // Iterate over all devices and scan for the right one
            for (Device device : list) {
                DeviceDescriptor descriptor = new DeviceDescriptor();
                result = LibUsb.getDeviceDescriptor(device, descriptor);
                if (result != LibUsb.SUCCESS)
                    throw new LibUsbException("Unable to read device descriptor", result);

                System.out.println("Bus number: " + LibUsb.getBusNumber(device));
                System.out.println("Port number: " + LibUsb.getPortNumber(device));
                System.out.println(descriptor.dump());
                System.out.println("");
            }
        } finally {
            // Ensure the allocated device list is freed
            LibUsb.freeDeviceList(list, true);
        }

Given the nature of Immerse, a stereo input stream has no added value, but it would be silly to fail on it. Possible handling: give channel as extra parameter or merge channels 'manually' before volume processing. Merging probably means either taking the highest amplitude or the average. Research this somewhat and maybe find a library that can do it. But in general providing a stereo input stream with quite different sound in the channels is not useful. Also add a warning in the logs about providing a stereo input stream, cause it should be prevented if possible.

More CPU in the background instead of stop the world from time to time is preferred

Using this info: http://soundfile.sapp.org/doc/WaveFormat/
It should be pretty simple to generate your own wave streams and use those for testing purposes.
Both for unit tests that need an input stream as for a generator that creates a certain sound, although there are better libs for that, since sine wave generation is not that trivial.

Seems like a audio stream generated by SineWaveAudioInputStreamGenerator is not restarting properly.

Scenario:
Scenario scenario = scenario(room, settings(fixed(generate(format, 500, 10_000)), fixed(5, 10, 10), fixed(5, 5, 5),
fixed(fixedSpeakerVolumeRatios), fractional(), forever()));

Output:
2018-04-23 14:41:47 [main] ImmerseMixer.initializeSoundCardStreams() - INFO: Exception for mixer info: 'PCH [plughw:0,0]'. Known Java Sound API issue, falling back to default audio device.
2018-04-23 14:41:47 [main] ImmerseMixer.lambda$32() - INFO: Main mixer changed from state NEW to WARMUP
2018-04-23 14:41:49 [Warmup Mixer Executor] ImmerseMixer.lambda$32() - INFO: Main mixer changed from state WARMUP to INITIALIZED
2018-04-23 14:41:49 [Warmup Mixer Executor] ImmerseMixer.warmup() - INFO: Warmup completed in 1.471 seconds
2018-04-23 14:41:49 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer changed from state INITIALIZED to STARTED
2018-04-23 14:41:49 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer restarted scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer restarted scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer restarted scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer restarted scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer restarted scenario Scenario
2018-04-23 14:41:59 [Main Mixer Worker] ImmerseMixer.lambda$32() - INFO: Main mixer started scenario Scenario

Because of the new optimization of the merge method, warmup should have overlapping scenarios to cover the full code base again. Also good for reduced warmup runtime. Maybe calculate scenario runtime in advanced and starting the next one after 30% runtime, so max 3/4 scenario's at the same time?

Implement some kind of StepStatistics that will fill during the execution of a step and saved/cached/logged/sent somewhere. NOTE: if cached locally, this will influence memory management!

Possible data to include:

• amount of frames needed
• step millis
• nr of scenario's (re)started/stopped
• ???

Can be quite bumpy (even 20ms steps) depending on the hardware.
The bigger the steps, the more risk of hickups in playback.
So smoothen by always writing some frames, even when current frame position has not changed in the sound card streams. We can do this by for example:

• Taking some kind of linear interpolation for the frame position based on past measurements
• Using a real time clock to calculate the approximate frame position based on the previous (calculated) frame position

Some readme docs are already in place, but it needs:

• more structure
• a good general intro
• a getting started section
• some explanation about the audio-streaming internals

NOTE: what is the relation between the README's and the wiki?
Maybe general description in README and forward to wiki for more details?

For example:

• build server (what's its name)
• quality check (10 points company)
• public kind of QA server thingy (pdm, checkstyle, findbugs, etc)
• security report?
• auto deploy to maven central upon release

Does Immerse work on OSX?

We have 2 measurements:

1. System.out.println("Frames needed diff: " + (maxFramesNeeded - minFramesNeeded));

2. We can also try to use current micro position to see the difference between streams

3. Will fluctuate but should be close to 0, the diff for 2 should also be close to 0, but not tested yet and unknown how reliable this is, maybe just based on frame position.

NB: Actually so far no hearable out of sync, so maybe no high prio.

Does Immerse work on Windows?

Since we want this to be a general purpose library, better remove the Spring dependency and any big lib that is just used for a small piece. Optionally include a few classes from other open source projects directly to lower the footprint.

Only for internal code quality: upgrade the Frame concept to a first class citizen by having some kind of FrameStream in and out as wrappers for the underlying byte streams. That will greatly simplify code in SoundCardStream. Also for Format we can have a better abstraction then the current Java AudioFormat. See for instance the PyhAudioFormat from the Program Your Home project.

Previous testing showed that all usable sound cards will have plughw:x,0 in their name, with x being a number from 0 till n. One of these will likely be the default. But it also seemed to suggest that the one that is in use as default is not accessible by it's plughw:x,0 index. But testing without any USB sound card that was not the case. So this should be further tested and the solution should be put into code here:
ScenarioPlayer:initializeStreams

Since the input can be any odd audio file, it would be nice to support as many formats as possible. Currently only PCM signed is supported. But it should be fairly easy to include PCM unsigned and float, since it takes just a slightly different calculation to process the amplitude (volume) alteration.

See: SoundCardStream.calculateFrameBytes

PS: Output will always use PCM signed for simplicity.

Send stats to stats gatherer, that will async send stats to a server / process and display for instance avg step time last sec/min/since start.
Use existing tools for gathering/saving and visualiztion, but maybe some internal class as proxy that can have different backends: sysout, GUI, network service

INFO: Main mixer changed from state NEW to WARMUP
Exception in thread "Warmup Mixer Executor" java.util.ConcurrentModificationException
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1558)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at one.util.streamex.AbstractStreamEx.rawCollect(AbstractStreamEx.java:68)
at one.util.streamex.AbstractStreamEx.toSet(AbstractStreamEx.java:1219)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.getScenariosInPlayback(ImmerseMixer.java:156)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.hasScenariosInPlayback(ImmerseMixer.java:147)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.lambda$warmup$2(ImmerseMixer.java:246)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.waitFor(ImmerseMixer.java:472)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.warmup(ImmerseMixer.java:246)
at com.programyourhome.immerse.audiostreaming.mixer.ImmerseMixer.lambda$initialize$0(ImmerseMixer.java:189)
at java.lang.Thread.run(Thread.java:748)

The audio-streaming module can run inside a JVM and controller directly from there. It would also be nice to be able to run immerse as a network service that can be controller from the network.

The performance is currently good enough to run on a Pine64, but not by a lot. So might be useful to do some actual profiling to find out what the current bottleneck is. And maybe in case of a big percentage to a small function, we can fix that to improve performance.

For Windows and OSX support, we might consider using Port Audio:
http://portaudio.com/

Create some kind of mechanism (maybe just plain .properties file) to configure some important parameters from the outside, like:

• buffer millis
• step sleep time
• warmup config
• etc

Continuous looping:

• first and last location must be the same
• millis entry is modulo'ed with the total path time and calcuted subsequently

Everything already seems dynamically calculated, but it makes sense to have separate control over the volume of the audio resource. This can simulate things getting closer or further away or something turning up / down volume, talking softer / louder, etc.

Related to this is issue #25, that deals with dynamic volume based on distance. The same problems apply here, namely that in the current setup the normalizing process might be in the way. Or both this and #25 should be realized by some post-processing. That could work for this one, but for #25 we need the distance which might be lost by the time of post processing. Although recalculating is not that hard of course.