Comments (13)
Hi @PRamoneda, thanks for reporting the issue.
The Resample
algorithm uses the external dependency of libsamplerate
library. But the current Essentia WASM builds were not linked with any of these third-party dependencies to make it lightweight. Hence it won't work with the current builds. Somehow we made a mistake by including it with the current JS API. We are working on it and will include support for this algorithm in the next version release.
Meanwhile, for downsampling, you could do that using native JS by directly downsampling the JS typed array from the getChannelData method of the Web Audio API. An example of this can be found here.
Hope it helps
from essentia.js.
It is working yeah!!
But without nnls symbolic transcription. I am raising to use CQT chroma :S. I am testing CQT, HPCP and NNLS.
Thank you other time!!
from essentia.js.
Thank you so much!!!.
I have another question about how can I create a vectorFloatFloat to feed ChomaNNLS.
If I call ArrayToVector with an array of VectorFloat. Should it return a vectorFloatFloat??
I get later Expected null or instance of VectorVectorFloat, got an instance of VectorFloat
because logFreqSpectrum is VectorFloat
This code is inspired in https://github.com/MTG/essentia/blob/9bca80eb331efa550975d00a353e9928815a2b3f/test/src/unittests/tonal/test_nnlschroma.py
let audioURL = document.getElementById("audio").currentSrc;
console.log(audioURL);
// load audio file from an url
let audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
// let audioData2 = essentia.arrayToVector(audioData);
audioData = downsample(audioData, 44100, 8000); // sample rate
if (isComputed) { plotChroma.destroy(); };
let frames = essentia.FrameGenerator(audioData, 1024, 512);
// compute for overlapping frames
let logFreqSpectrum = new Array(frames.length);
let meanTuning = 0;
let localTuning = 0;
for (var i=0; i<frames.size(); i++) {
let log_spectrum = essentia.LogSpectrum(essentia.Spectrum(frames.get(i)).spectrum,
3, // bins per semitone
1024, //frameSize
0, // rollon
8000);// sample rate
// let c = essentia.vectorToArray(log_spectrum.logFreqSpectrum);
console.log(log_spectrum.logFreqSpectrum);
logFreqSpectrum.push(log_spectrum.logFreqSpectrum);
meanTuning = log_spectrum.meanTuning;
localTuning = log_spectrum.meanTuning;
}
logFreqSpectrum = essentia.arrayToVector(logFreqSpectrum);
console.log(logFreqSpectrum)
from essentia.js.
Removed the redundant comment:)
You can create VectorVectorFloat
type on the JS side using
let vecvecFloat = new essentia.module.VectorVectorFloat();
arrayToVector
and vectorToArray
methods only works for 1D arrays/vector. For 2D vector/arrays, you need to manually convert by iterating through it.
The below example should work. Haven't tested it though!
for (var i=0; i<frames.size(); i++) {
let log_spectrum = essentia.LogSpectrum(essentia.Spectrum(frames.get(i)).spectrum,
3, // bins per semitone
1024, //frameSize
0, // rollon
8000);// sample rate
vecvecFloat.push_back(log_spectrum.logFreqSpectrum);
meanTuning = log_spectrum.meanTuning;
localTuning = log_spectrum.meanTuning;
let nnlsChroma = essentia. NNLSChroma(vecvecFloat, meanTuning, localTuning);
// you need manually resize the 2D vector after its use
vecvecFloat.resize(0, 1);
console.log(nnlsChroma);
}
from essentia.js.
Why I need to manually resize the 2D vector after using it??? Is It due to memory allocation, isnt it??
This,
let nnlsChroma = essentia. NNLSChroma(vecvecFloat, meanTuning, localTuning);
// you need manually resize the 2D vector after its use
vecvecFloat.resize(0, 1);
console.log(nnlsChroma);
have to be called after the loop. I understand. As https://github.com/MTG/essentia/blob/ba79be6515f2fd0cde75ee3f6fa98706a66f4c36/src/examples/standard_nnls.cpp#L125
However, I have imitated the cpp version and NNLSchroma compute all the pitch classes to zero.
example.html
<!DOCTYPE html>
<html lang="en">
<head>
<title>essentia.js examples</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link
rel="stylesheet"
type="text/css"
href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css"
/>
</head>
<center>
<body style="background-color: #000000!important;">
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia-wasm.web.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia.js-core.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia.js-plot.js"></script>
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
<script src="script.js" defer></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div>
<a>
<h1> HARMONIC CHANGE DETECTION FUNCTION </h1>
</a>
</div>
<h2 style="color: azure;">
HPCP chroma example
</h2>
<div class="ui divider" style="height: 2px; width: 2px;"></div>
<input type="file" id="upload" />
<audio id="audio" controls>
<source src="https://freesound.org/data/previews/328/328857_230356-lq.mp3" id="src" />
</audio>
<div id="logDiv" style="color: azure;"><br /></div>
<div class="ui divider" style="width: 2px; height: 5px;"></div>
<div id="plotDiv"></div>
<br />
<br />
</body>
</center>
</html>
script.js
function handleFiles(event) {
var files = event.target.files;
$("#src").attr("src", URL.createObjectURL(files[0]));
document.getElementById("audio").load();
}
document.getElementById("upload").addEventListener("change", handleFiles, false);
let essentia
/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */
let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";
let isComputed = false;
function downsample(buffer, old_sr, new_sr) {
if (new_sr == old_sr) {
return buffer;
}
if (new_sr > old_sr) {
throw "downsampling rate show be smaller than original sample rate";
}
var sampleRateRatio = old_sr / new_sr;
var newLength = Math.round(buffer.length / sampleRateRatio);
var result = new Float32Array(newLength);
var offsetResult = 0;
var offsetBuffer = 0;
while (offsetResult < result.length) {
var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
var accum = 0, count = 0;
for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
accum += buffer[i];
count++;
}
result[offsetResult] = accum / count;
offsetResult++;
offsetBuffer = nextOffsetBuffer;
}
return result;
}
// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
let audioURL = document.getElementById("audio").currentSrc;
console.log(audioURL);
// load audio file from an url
let audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
// let audioData2 = essentia.arrayToVector(audioData);
audioData = downsample(audioData, 44100, 8000); // sample rate
if (isComputed) { plotChroma.destroy(); };
let frames = essentia.FrameGenerator(audioData, //
1024, //
512);
// compute for overlapping frames
let logFreqSpectrum = new essentia.module.VectorVectorFloat();
let meanTuning = new essentia.module.VectorFloat();
let localTuning = new essentia.module.VectorFloat();
for (var i=0; i<frames.size(); i++) {
let log_spectrum = essentia.LogSpectrum(essentia.Spectrum(frames.get(i), 1024).spectrum,
3, // bins per semitone
1024, //frameSize
0, // rollon
8000);// sample rate
console.log(essentia.vectorToArray(log_spectrum.logFreqSpectrum));
logFreqSpectrum.push_back(log_spectrum.logFreqSpectrum);
localTuning.push_back(log_spectrum.localTuning);
//as in python script https://github.com/MTG/essentia/blob/9bca80eb331efa550975d00a353e9928815a2b3f/test/src/unittests/tonal/test_nnlschroma.py
meanTuning = log_spectrum.meanTuning;
}
console.log(typeof logFreqSpectrum, typeof meanTuning, typeof localTuning);
console.log(logFreqSpectrum, meanTuning, localTuning);
// Running NNLSchroma algorithm on an input audio signal vector
// check https://essentia.upf.edu/reference/std_NNLSChroma.html
// NNLSChroma(logSpectrogram: any, meanTuning: any, localTuning: any, chromaNormalization: string='none', frameSize: number=1025, sampleRate: number=44100, spectralShape: number=0.7, spectralWhitening: number=1, tuningMode: string='global', useNNLS: boolean=true)
let chroma = essentia.NNLSChroma( logFreqSpectrum, // input
meanTuning,
localTuning,
'none', //chromaNormalization
1024, //frameSize
8000, //sampleRate
0.7, //spectralShape
1, //spectralWhitening
'global', //tuningMode
true).chromagram; //useNNLS
debugger;
let chromagram = Array(chroma.size());
for (var i = 0; i < chroma.size(); i++) {
console.log(essentia.vectorToArray(chroma.get(i)));
chromagram[i] = essentia.vectorToArray(chroma.get(i));
}
console.log(typeof chromagram);
// plot the feature
plotChroma.create(
chromagram, // input feature array
"NNLS Chroma", // plot title
audioData.length, // length of audio in samples
8000 // audio sample rate
);
isComputed = true;
}
$(document).ready(function() {
// create EssentaPlot instance
plotChroma = new EssentiaPlot.PlotHeatmap(
Plotly, // Plotly.js global
plotContainerId, // HTML container id
"chroma", // type of plot
EssentiaPlot.LayoutChromaPlot // layout settings
);
// Now let's load the essentia wasm back-end, if so create UI elements for computing features
EssentiaModule().then(async function(WasmModule) {
essentia = new Essentia(WasmModule);
// essentia version log to html div
$("#logDiv").html(
"<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
);
$("#logDiv").append(
'<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
);
var button = document.getElementById("btn");
// add onclick event handler to comoute button
button.addEventListener("click", () => onClickFeatureExtractor(), false);
});
});
Thank you!!
from essentia.js.
Moreover, log spectrum always throw this warning. Even with default parameters.
[0;32m[ INFO ] LogSpectrum: input spectrum size does not match '_frameSize' parameter. Reconfiguring the algorithm.
Thank you so much!!!!
from essentia.js.
Yes, NNLSChroma, accepts frames of log spectrum as input. Also, you may need to apply windowing to each frame before computing the log spectrum. The documentation suggests the following:
This code is ported from NNLS Chroma [1, 2]. To achieve similar results follow this processing chain: frame slicing with sample rate = 44100, frame size = 16384, hop size = 2048 -> Windowing with Hann and no normalization -> Spectrum -> LogSpectrum.
References:
[1] Mauch, M., & Dixon, S. (2010, August). Approximate Note Transcription for the Improved Identification of Difficult Chords. In ISMIR (pp. 135-140).
[2] Chordino and NNLS Chroma, http://www.isophonics.net/nnls-chroma
For example, try this out
const frameSize = 16384;
const hopSize = 2048;
let frames = essentia.FrameGenerator(audioData,
frameSize,
hopSize)
let logSpectFrames = new essentia.module.VectorVectorFloat();
for (var i=0; i<frames.size(); i++) {
// default hanning window (you can change it according to your need)
let windowing = essentia.Windowing(frame.get(i), false, 1024, 'hann');
let spect = essentia.Spectrum(windowing.frame, frameSize); // frameSize
let logSpectrum = essentia.LogSpectrum(spect.spectrum,
3, // bins per semitone
frameSize
0, // rollon
8000);// sample rate
logSpectFrames.push_back(logSpectrum.logFreqSpectrum);
meanTuning = logSpectrum.meanTuning;
localTuning = logSpectrum.localTuning;
}
let nnlsChroma = essentia. NNLSChroma(logSpectFrames, meanTuning, localTuning);
delete windowing:
delete spect;
delete logSpectrum;
Regarding memory allocation, you may need to manually delete any JS objects created from Essentia algorithms as Emscripten documentation suggests. Check here for more details.
Another tip, it might be good for the web app to run your audio feature extraction process inside Web Workers to achieve better performance.
from essentia.js.
Thank you so much!!!
But it is not working :(. NNLSchroma doesnt compute anything. Results of NNLS are 0 too.
Here a web editor with everything. https://jsfiddle.net/PRamoneda/zc1bnhxk/2/
HTML file, JS file and console output screenshot
script.js
function handleFiles(event) {
var files = event.target.files;
$("#src").attr("src", URL.createObjectURL(files[0]));
document.getElementById("audio").load();
}
document.getElementById("upload").addEventListener("change", handleFiles, false);
let essentia
/* "https://freesound.org/data/previews/328/328857_230356-lq.mp3"; */
let audioData;
// fallback for cross-browser Web Audio API BaseAudioContext
const AudioContext = window.AudioContext || window.webkitAudioContext;
let audioCtx = new AudioContext();
let plotChroma;
let plotContainerId = "plotDiv";
let isComputed = false;
function downsample(buffer, old_sr, new_sr) {
if (new_sr == old_sr) {
return buffer;
}
if (new_sr > old_sr) {
throw "downsampling rate show be smaller than original sample rate";
}
var sampleRateRatio = old_sr / new_sr;
var newLength = Math.round(buffer.length / sampleRateRatio);
var result = new Float32Array(newLength);
var offsetResult = 0;
var offsetBuffer = 0;
while (offsetResult < result.length) {
var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
var accum = 0, count = 0;
for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
accum += buffer[i];
count++;
}
result[offsetResult] = accum / count;
offsetResult++;
offsetBuffer = nextOffsetBuffer;
}
return result;
}
// callback function which compute the frame-wise HPCP chroma of input audioURL on a button.onclick event
async function onClickFeatureExtractor() {
let audioURL = document.getElementById("audio").currentSrc;
console.log(audioURL);
// load audio file from an url
let audioData = await essentia.getAudioChannelDataFromURL(audioURL, audioCtx, 0);
if (isComputed) { plotChroma.destroy(); };
const frameSize = 16384;
const hopSize = 2048;
const sampleRate = 8000;
console.log("audio antes downsampling", audioData);
audioData = downsample(audioData, 44100, sampleRate);
console.log("audio despues downsampling", audioData);
let frames = essentia.FrameGenerator(audioData,
frameSize,
hopSize)
let logSpectFrames = new essentia.module.VectorVectorFloat();
for (var i=0; i<frames.size(); i++) {
// default hanning window (you can change it according to your need)
let windowing = essentia.Windowing(frames.get(i), false, hopSize, 'hann');
let spect = essentia.Spectrum(windowing.frame, frameSize); // frameSize
let logSpectrum = essentia.LogSpectrum(spect.spectrum,
3, // bins per semitone
frameSize,
0, // rollon
sampleRate);// sample rate
logSpectFrames.push_back(logSpectrum.logFreqSpectrum);
meanTuning = logSpectrum.meanTuning;
localTuning = logSpectrum.meanTuning;
}
let nnlsChroma = essentia.NNLSChroma(logSpectFrames, meanTuning, localTuning).chromagram;
delete windowing;
delete spect;
delete logSpectrum;
for (var i = 0; i < nnlsChroma.size(); i++)
console.log(essentia.vectorToArray(nnlsChroma.get(i)));
// plot the feature
plotChroma.create(
nnlsChroma, // input feature array
"NNLS Chroma", // plot title
audioData.length, // length of audio in samples
sampleRate // audio sample rate
);
isComputed = true;
delete nnlsChroma;
}
$(document).ready(function() {
// create EssentaPlot instance
plotChroma = new EssentiaPlot.PlotHeatmap(
Plotly, // Plotly.js global
plotContainerId, // HTML container id
"chroma", // type of plot
EssentiaPlot.LayoutChromaPlot // layout settings
);
// Now let's load the essentia wasm back-end, if so create UI elements for computing features
EssentiaModule().then(async function(WasmModule) {
essentia = new Essentia(WasmModule);
// essentia version log to html div
$("#logDiv").html(
"<h5> essentia-" + essentia.version + " wasm backend loaded ... </h5>"
);
$("#logDiv").append(
'<button id="btn" class="ui white inverted button">Compute HPCP Chroma </button>'
);
var button = document.getElementById("btn");
// add onclick event handler to comoute button
button.addEventListener("click", () => onClickFeatureExtractor(), false);
});
});
Example.html
<!DOCTYPE html>
<html lang="en">
<head>
<title>essentia.js examples</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link
rel="stylesheet"
type="text/css"
href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css"
/>
</head>
<center>
<body style="background-color: #000000!important;">
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia-wasm.web.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia.js-core.js"></script>
<script src="https://unpkg.com/[email protected]/dist/essentia.js-plot.js"></script>
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
<script src="script.js" defer></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div>
<a>
<h1> HARMONIC CHANGE DETECTION FUNCTION </h1>
</a>
</div>
<h2 style="color: azure;">
HPCP chroma example
</h2>
<div class="ui divider" style="height: 2px; width: 2px;"></div>
<input type="file" id="upload" />
<audio id="audio" controls>
<source src="https://freesound.org/data/previews/328/328857_230356-lq.mp3" id="src" />
</audio>
<div id="logDiv" style="color: azure;"><br /></div>
<div class="ui divider" style="width: 2px; height: 5px;"></div>
<div id="plotDiv"></div>
<br />
<br />
</body>
</center>
</html>
Here the console output:
from essentia.js.
Okay, I just saw that this is a known issue with the NNLS chroma algorithm. See issue MTG/essentia#951 and MTG/essentia#948.
So you need to change the parameter useNNLS=False
.
Btw, please only post the necessary code snippet in the comments. No need to share all of your web app code in the comments unless it is related to the issue. Sharing a link to a web editor is enough. In that way, it would be easier for others to find the information in these threads :)
Thanks for reporting the issue.
Hope this helps, cheers!
from essentia.js.
from essentia.js.
According to the comments in the above-mentioned issue threads, the NNLS symbolic transcription approach is not fully tested and guaranteed to work in every use-case.
Let me know if this setting works for you in the web application.
from essentia.js.
Does Essentia WASM builds includes Resample method for now?
from essentia.js.
@floydback It is included in the build, but it does not work yet, sorry. You can try writing the resampling function yourself as suggested above, using a reference like this one or the downsample function implemented by the OP. You can also use an OfflineAudioContext to do the resampling for you (see this StackOverflow answer)
from essentia.js.
Related Issues (20)
- Using deep learning models with Essentia.js and Tensorflow.js HOT 1
- Tensor must have a shape comprised of positive integers but got shape [,96]
- Real-time mood classification - "most" vs "least" danceable part of a track HOT 1
- Error 404 when accessing the source code from the demos HOT 1
- Build errors trying to build onsets demo
- Real-time analysis.
- JS bindings for SBic
- GapsDetector not returning timestamp of anomaly in NodeJS
- RuntimeError: abort(TypeError: Failed to parse URL from asturiasfine.mp3). Build with -s ASSERTIONS=1 for more info. at process.abort (C:\Users\Rosanna\Desktop\node_modules\essentia.js\dist\essentia-wasm.umd.js:19:12157)
- Is this up-to-date? HOT 3
- Can't import Essentia
- Get sections/markers info (intro, verse, chorus etc) from an audio HOT 1
- FrameGenerator should provide access to other FrameCutter parameters
- I would like to request the implementation of SBic.
- Beginner documentation improvements HOT 1
- How to import ML models ad use for autotagging HOT 1
- VectorFloat params incorrectly converted on Typescript before passed to WASM
- Float params accept JS string containing numeric characters
- Webpack: error for missing modules in WASM version HOT 1
- Can essentia js be used in typescript for the web?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from essentia.js.