LokiDB : build process and cross platform about lokidb HOT 19 OPEN

Hmm, strange I will take a deeper look at it ...

• running npm run test:node on Windows does nothing o.O
• npm run build does not terminate
• and integration tests does also not work

from lokidb.

Thanks. I am going to try to focus on making sure my build process is both expedient for me as well as not requiring changes to configuration when i eventually merge into main branch. I can either work around by manually selecting files to check in or applying gitignore if acceptable.

I am also wondering if doing an npm run build on a clean clone should produce pending changes in the dist/packages folder.

I had not yet noticed the build does not terminate, I recently re-baselined and it must have just started doing that within last few months... worked before. Maybe try node inspect build.js later if I have time.

Maybe later this week I will address my findings with btree indexing, solicit feedback, and suggest short term integration main branch, and longer term refactoring of indexes.

from lokidb.

npm run build yields only the only js and map artifacts similar to above picture... and it now sucessfully completes. I would assume they show up even though they are in gitignore because they actually in the repo (under version control) and changed.

You are right. Git only would ignore them by using:

git update-index --assume-unchanged

described here. But this cannot be committed with the repository...

do get an error within instanbul which I or you can look into later but the karma tests completed. Since instanbul errors out i can not tell if i have the html issue.

Please check if cleaning the coverage directory solves the issue.

So, it would seem i do not need dist/packages nor dist/packages-dist in order to perform an "npm run test". I know karma is configured with preprocessors and eventually dumps to chrome headless, i am just not sure where its own internal compilation artifacts are/were put or if it was able to do so within memory?

Yes, karma and node tests don't need dist/packages. They compile typescript to javascript at runtime. I think they hold it in memory or inside temporary directories.

As a test, if I temporarily wipe out the dist/packages and dist/packages-dist folder and do an "npm run build", both folders are generated, the only difference i can see is there are minified versions added to the dist/packages folder(s).

The dist/packages-dist folder should be generated for deployment (to npm) and integration testing (it contains the package.json and the README.md). Minified versions don't get published to npm.

Integration tests does not work on windows yet. The package installation seems to be a bit different than on linux...

So correct me if I'm wrong but I thing the major use cases are :

• Developers make changes they want to test before checking in
• Developers want to package and check in packages to github so others do not need toolchain
• You are publishing a new build to npm

Why can't we just have one 'dist' folder that covers both subdirectories? Or if we want to allow developers to publish 'personal' copies of the package it would only update one non-version controlled folder. If they want to build and checkin packages we might run a different build script to update only the version controlled directory. And for npm, possibly use that latter folder within publish script?

If I understand you correctly, this is a good idea. ;)

So to summarize the idea:

• the dist folder will be created over a special script/parameter
• the prebuild packages dist/packages will be committed during deployment at a release build (over travis) to github
• the dist/packages-dist folder will not be under version control and will be published to npm during deployment at a release build (over travis)
• the normal build command (npm run build), used by developers for tests and benchmarking, will be not under version control and inside the folder build/

Do you have some remarks with this summarized idea?

Sorry if any of that should have seemed obvious, it can be difficult reverse engineering the build process because of window not behaving similarly to original intent.

Please ask as much as you like. ;)

from lokidb.

You already have 'test', 'build', and 'deploy' npm scripts. So the extra script/parameter would be in addition or those or a parameter to build? If so that makes sense.

The script npm run deploy will do the release build inside dist folder and do the actually deploy. Publishing to npm does only works with travis.

npm run build will build the packages to build/packages/ (for benchmarking, not under version control)

npm run build -- --dist will build and bundle the packages to dist/ (for integration tests and deploying)

My benchmarks may change later, but for now they also represent a use case of using prebuilt js modules without having a toolchain of its own. I guess at some point i will rewrite in typescript and would probably need to set up webpack on it and at that point would it be better to move into its own (package?)

I think using prebuild js modules for benchmarking is good enough yet. We can refactor this after a final release and check if the results with pure typescript are nearly the same.

I will also check again if I can finally fix the html problem.

from lokidb.

Wow, thank you... you have done more than I expected. That makes a lot of sense and i can live with 3rd party lib issues for coverage. I may need to catch up on a lot of this over the weekend but I feel I understand this process and will hopefully make it easier to collaborate.

After all your hard work maybe i need to wait a week or so before soliciting your feedback on comparator organization/registration and index serialization :P

I will summarize a suggested organization in #95 in a few days to see what your thoughts are.

from lokidb.

Hey @Viatorus, I am trying to determine the best way to avoid artifact overload on Windows with a configuration that is acceptable to main branch.

On Windows (only it would seem), when I run (manually, or as part of npm run test) :

npm run coverage

The output of the individual html files (about 46 of them) go right next to the source file they are reporting about. I can only assume these are supposed to have been generated into the coverage folder at the root. The output of the final summary report goes into correct coverage folder. I have tried configuring an .istanbul.yml to turn on verbose output and it shows this.

Unless you have any ideas, I think i am just going to turn on .gitignore for html files in packages folder.

from lokidb.

The dist/packages are currently only updated during a release. Otherwise nearly every pull request would have to commit the dist/packages...

I also recommend rebasing instead of merging. Sometimes rebasing is a little bit nasty but it covers all the issues if you just merge bigger changes.

from lokidb.

Thanks for changes, I am still figuring out what's normal expected behavior... seems I don't need to rebuild before a test, just before generating package files which I can reference in my js benchmark.

I just tried the current fix_build branch and did a npm run build and its much cleaner, only js and map files... not sure if that is what you intended but its better. (still have to forcibly exit)

from lokidb.

The forcible exit was fixed in the branch fix_coverage and is now merged into the master.

Inside the gitignore file dist/ is ignored. Strange, that changes are still tracked.

from lokidb.

As I see, the HTML issue is still present. o.O !!!

from lokidb.

npm run build yields only the only js and map artifacts similar to above picture... and it now sucessfully completes. I would assume they show up even though they are in gitignore because they actually in the repo (under version control) and changed.

I do get an error within instanbul which I or you can look into later but the karma tests completed. Since instanbul errors out i can not tell if i have the html issue.

So, it would seem i do not need dist/packages nor dist/packages-dist in order to perform an "npm run test". I know karma is configured with preprocessors and eventually dumps to chrome headless, i am just not sure where its own internal compilation artifacts are/were put or if it was able to do so within memory?

As a test, if I temporarily wipe out the dist/packages and dist/packages-dist folder and do an "npm run build", both folders are generated, the only difference i can see is there are minified versions added to the dist/packages folder(s).

So correct me if I'm wrong but I thing the major use cases are :

• Developers make changes they want to test before checking in
• Developers want to package and check in packages to github so others do not need toolchain
• You are publishing a new build to npm

Why can't we just have one 'dist' folder that covers both subdirectories? Or if we want to allow developers to publish 'personal' copies of the package it would only update one non-version controlled folder. If they want to build and checkin packages we might run a different build script to update only the version controlled directory. And for npm, possibly use that latter folder within publish script?

Sorry if any of that should have seemed obvious, it can be difficult reverse engineering the build process because of window not behaving similarly to original intent.

from lokidb.

Thanks, clearing the coverage folder did allow npm run test to complete correctly :)

I have not summarized the idea and tbh i have not implemented ts build/toolchain so i can only speculate about ideal use cases which myself or others might attempt to perform. As such, feel free to push back if I ask for anything impractical.

You already have 'test', 'build', and 'deploy' npm scripts. So the extra script/parameter would be in addition or those or a parameter to build? If so that makes sense.

My benchmarks may change later, but for now they also represent a use case of using prebuilt js modules without having a toolchain of its own. I guess at some point i will rewrite in typescript and would probably need to set up webpack on it and at that point would it be better to move into its own (package?)

Thanks again for your work setting all of this up, I hope to help out working more in main modules now that I am getting closer to a point where i can pr these changes into main branch.

from lokidb.

So the problem with the coverage HTML output under windows seems to be the path separator.

The test:node coverage has the correct path separator \\ (Windows style). The test:web coverage has the path separator // (Unix style, bad for Windows).

I opened a issue here:
https://github.com/mattlewis92/karma-coverage-istanbul-reporter/issues/47

from lokidb.

Great!

from lokidb.

I've got maybe a stupid question to ask, why the webpack is needed for this library-like project to compile ts code? Is it because of ts compiling to a single js file using webpack declarations files are currently messed up?

Is the release version going to contain in the dist folder the same files list as in source folder, but files would have js extension + d.ts file for each js file (file to file ts=>js conversion)? I mean a dist structure that would be convenient for use in typescript environment with automatically wired up declarations, but also totally usable in js environment.

What is the recommended way to use 2.0.0-beta.7 with wired up declarations (in typescript environment)?

from lokidb.

Using npm install @loki/... should provide a package.json with the correct path to the typings.

I am not really sure why I use webpack. I think I saw it on other projects. I didn't find many monorepository Typescript library projects so I put it together what I found.

You are correct, all the d.ts files are not what I want but utilities like dts-bundle don't work here (for there known limitations).

If you have a good idea, how to manage the ts compiling, I am open for PRs or ideas. :)

from lokidb.

Using npm install @loki/... should provide a package.json with the correct path to the typings.

It doesn't work out of the box in typescript environment, at least v2.0.0-beta.7.

I'd suggest writing at least minimal tests set in typescript and run them on the compiled code (on the js files + declarations). Such testing approach allows detecting obvious issues of running code in typescript environment.

The idea is to compile using just typescript compiler, running tsc command. Ideally, but not necessary, that would be a composite project structure, introduced with recently released typescript v3. The idea of the composite projects is explained in TSConf 2018 - Opening Remarks by Anders Hejlsberg (starting from 54:45).

Webpack then could step into if there will be a need to provide bundled dist version for browsers. But generally I treat bundling as a burden of the end library users, ie end users would have to setup bundling on their side if they need it.

I'm not ready to work on the PR at the moment.

from lokidb.

@Viatorus, minor request to see if you foresee any issues with it :

I was trying to do an offline build the other day and it was failing due to the call to fetchBranchesAndTags() within getBuildInformation() in common.js.

Could we either have a try catch around that and just use the required package.json version as fallback?
Or possibly add boolean flag param to getBuildInformation() indicating whether we want to issue git refresh requests (and have build.js pass false while deploy passes true?)

No rush since I have a workaround, just curious if we could support that as some point.

@vladimiry : 3.0 project references definitely look interesting for my offline/experimental personal projects... beyond my capability integrate that approach into LokiDB :) If you find a nice 'reference' project implementing this approach with unit testing, please mention it for research.

from lokidb.

@vladimiry
Thank you for your links and hints.

I found the issue why @lokidb/loki did not work in TypeScript (see #154).

from lokidb.