Comments (20)
Need more information in order to reproduce this.
- What are you trying to count?
- Is there some code you can put somewhere that I can use to replicate.
- Did you download the binary or compile from source?
- Is it possible that some other background process is modifying what you are trying to count?
- Can you create a checksum of the directory you are trying to count, then run scc then run the checksum and then scc again to prove that nothing is changing and that scc is the culprit
from scc.
All I did was run scc ~/go/src
several times in a row. For comparison I also ran gocloc and tokei the exact same way and they gave me consistent numbers each time.
I built all of them from source: tokei via Brew and the others via go get
.
I'm not doing anything else in the background that would be altering these files and the fact that the other 2 tools get consistent results suggests that the issue is only found in scc.
My guess is that whatever the issue is must be particular to OS X though, since you haven't seen it on either Linux or WIndows.
from scc.
That is very odd. Can you try running on a single directory with one file in it and see if its reproducible there? Then add the to option --trace
scc --trace singledirectory
That should allow me to at least guess what might be going on.
from scc.
OK, so when I try it on a single dir it's consistent. Even on a dir that contains other dirs that contain code it is fine. But I think I have a reproducible test case for you:
go get github.com/davidrjenni/reftools
Then run scc on the davidrjenni dir a few times and the results are not consistent.
Strangely, if I run it on the only subdir in there (i.e. reftools) then I get consistent results!
from scc.
Are you seeing different counts on the totals?
You probably will see flipping between the following
HTML 15 998 898 41 59 0
JavaScript 15 63202 54101 8816 285 11205
That's because the sorting us done by Files rather than the name which is what Tokei does. Since they are the same value of 15 its down to whatever the runtime decides to do when the sort is invoked. You can control it to be like tokei by doing the following scc -s name .
which will make the sorting the same.
I might move the default over to sort by lines, however it will have the same issue if the line counts match. I made it based on files because that's what I wanted to see.
from scc.
I'm seeing the file count for Go flip between 519 (the correct value) and something between 235 and 240 (it varies each time).
from scc.
Must be something to do with the Go runtime on macOS. That is frustrating. I don't have access to an macOS machine.
My guess at the moment is that its something to do with the parallel walking of the file tree. If you drop down a single directory and run it does it work? That is drop down so that there is only a single directory where you run scc
which will force it to run the walk in a single GoRoutine. I know you did that with the reftools
directory but this should really make things sure.
If that turns out to be the case I will turn off the parallel file walk for Darwin arch until I can determine what the root cause issue is.
from scc.
I believe that's what I already tried with the parent dir of reftools, which only contains reftools. Here's the console output:
amdsouza-macbookair:davidrjenni amdsouza$ ls
reftools
amdsouza-macbookair:davidrjenni amdsouza$ scc .
Language Files Lines Code Comments Blanks Complexity
Go 217 31285 21179 6720 3386 4849
Markdown 7 308 232 0 76 0
JavaScript 6 832 629 78 125 119
YAML 6 110 99 0 11 4
CSS 4 904 785 4 115 0
JSON 3 91 83 0 8 0
BASH 3 166 110 27 29 10
Dockerfile 2 152 112 19 21 36
ASP.NET 2 2 2 0 0 0
TypeScript 2 215 149 34 32 2
TOML 1 7 6 0 1 0
Happy 1 202 175 0 27 0
Plain Text 1 2 2 0 0 0
Makefile 1 19 12 3 4 0
Total 256 34295 23575 6885 3835 5020
Estimated Cost to Develop $745,881
Estimated Schedule Effort 13.724775 months
Estimated People Required 6.437527
amdsouza-macbookair:davidrjenni amdsouza$ scc .
Language Files Lines Code Comments Blanks Complexity
Go 240 33478 22693 7121 3664 5242
Markdown 7 308 232 0 76 0
JavaScript 6 832 629 78 125 119
YAML 6 110 99 0 11 4
CSS 4 904 785 4 115 0
BASH 3 166 110 27 29 10
JSON 3 91 83 0 8 0
TypeScript 2 215 149 34 32 2
Dockerfile 2 152 112 19 21 36
ASP.NET 2 2 2 0 0 0
Makefile 1 19 12 3 4 0
TOML 1 7 6 0 1 0
Happy 1 202 175 0 27 0
Plain Text 1 2 2 0 0 0
Total 279 36488 25089 7286 4113 5413
Estimated Cost to Develop $796,256
Estimated Schedule Effort 14.069895 months
Estimated People Required 6.703732
from scc.
Odd that it only affects the Go counts.
Can you please try checking out and installing from this branch https://github.com/boyter/scc/tree/Issue1
Then see if the new option scc --npw .
resolves the issue for you. It disables the parallel walk. If it does I will have a look at what impact this has in general and either disable it just for macOS or possibly for everything depending on the results.
from scc.
Even with the --npw flag I still get inconsistent results for the number of Go files in the davidrjenni dir.
from scc.
Have been trying this out on a 2013 Retina Macbook Pro. I am totally unable to replicate the issue. I have included the results I get. Tried running it a dozen times and the results are always 100% the same.
# boyter @ Bens-MacBook-Pro in ~/go/src/github.com/davidrjenni/reftools on git:master o [15:50:33]
$ scc .
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks Complexity
-------------------------------------------------------------------------------
Go 522 115819 83117 20749 11953 15415
JavaScript 15 63202 54101 8816 285 11205
HTML 15 998 898 41 59 0
Markdown 10 376 277 0 99 0
YAML 7 136 121 0 15 2
CSS 6 1773 1572 5 196 0
JSON 3 91 83 0 8 0
Plain Text 3 172 113 0 59 0
BASH 3 166 110 27 29 10
TypeScript 2 215 149 34 32 2
Dockerfile 2 152 112 19 21 36
Happy 1 202 175 0 27 0
XML 1 11 11 0 0 0
C 1 19 10 4 5 0
Assembly 1 30 26 0 4 0
Makefile 1 19 12 3 4 0
TOML 1 7 6 0 1 0
-------------------------------------------------------------------------------
Total 594 183388 140893 29698 12797 26670
-------------------------------------------------------------------------------
Estimated Cost to Develop $4874490
Estimated Schedule Effort 28.009405 months
Estimated People Required 20.614841
-------------------------------------------------------------------------------
However this is running macOS Sierra 10.12.6 and not High Sierra which may explain the difference. Are you using High Sierra?
from scc.
I am using High Sierra but also you are doing one key thing differently.
Can you try it from the parent dir (i.e. inside davidrjenni not reftools)?
from scc.
I copied the above from the last run when I was trying to replicate. No matter where I ran it I had the same result. For the record I tried the following all inside ~/go/src/github.com/davidrjenni/
and then tried again inside reftools
scc .
scc -wl go .
I am starting to suspect that it may be something to do with High Sierra. The only thing I can think of is that somehow when it traverses the file-tree it does so in a way that nothing else does and possibly causes the files to be excluded.
When you run this with the --files
option you should be seeing that some files are missing, however if you add --verbose
or -v
you should see output at the top similar to the below
WARN 2018-04-22T21:58:55Z: skipping directory due to being in blacklist: .git
WARN 2018-04-22T21:58:55Z: skipping file unknown extension: Jenkinsfile
WARN 2018-04-22T21:58:55Z: skipping file unknown extension: quartz.properties
The order of the above is not deterministic but the count should be 100% the same and they should match if you for instance sorted the output. I have a feeling that in your case they would not be. You can verify this using scc -v . | grep "WARN" | sort
and capturing the output.
I will see if I can get the person whom I borrowed the Macbook from to upgrade to High Sierra but I suspect that may be unlikely.
from scc.
Ah, using -v & looking at the warning provided a very useful clue; many lines like this:
error reading: reftools/vendor/golang.org/x/tools/go/buildutil/allpackages.go open reftools/vendor/golang.org/x/tools/go/buildutil/allpackages.go: too many open files
That probably explains what's happening: scc is bumping up against some limit on the number of files that can be open concurrently.
from scc.
Ah yes. It is designed to pull as much from the disk as possible. Sounds like you are hitting the ulimit issues.
https://superuser.com/questions/261023/how-to-change-default-ulimit-values-in-mac-os-x-10-6#306555
https://unix.stackexchange.com/questions/108174/how-to-persistently-control-maximum-system-resource-consumption-on-mac/221988#221988
I will update the documentation to mention this I think. Interesting that tokei does not hit the limit. I might have to look into how to control or at least change the limit when the application runs.
https://stackoverflow.com/questions/17817204/how-to-set-ulimit-n-from-a-golang-program#17818022
Looks promising.
from scc.
Dang, that warning was a red herring. I fixed the limits an now I no longer see those warnings but I still get inconsistent numbers :(
from scc.
That is rather annoying. Are the verbose responses different at all when the numbers are different?
from scc.
N/m: I had make a wee mistake earlier; raising the open file limit to 4096 seems to have done the trick after all.
from scc.
Ah great. It seemed such a perfect fit. Ill have a look at implementing the above then.
Oh annoyingly its one of those ones that requires sudo apparently. I might just add it to the documentation. Most dev's raise the ulimit anyway. Might be possible to add it as better reporting output.
from scc.
Going to close this down now. Thanks so much for working through it with me.
from scc.
Related Issues (20)
- "Text" rather than "Plain Text"? HOT 2
- Suggested primary branch name HOT 1
- Could you please release a new version? HOT 3
- Multiple directories do not sum correctly HOT 7
- Error in 'm'ain.ts' File Prevents Installation in Go HOT 1
- Recommended install method fails due to malformed file name HOT 5
- Flag --not-match/-M not working, got ignored entirely HOT 2
- Feature: Add custom languages.json support
- Feature: Add option to count files not supported by scc
- BUG: Multicharacter support HOT 4
- Docker: aquasec/trivy reports CRITICAL and HIGH vulnerabilities HOT 1
- Malware Alerts in SCC x86 Windows Release HOT 3
- REGRESSION: Performance HOT 2
- Consider disabling comoco and size by default HOT 1
- Github action removed HOT 1
- Feature - support language Clipper HOT 10
- Migrated previous GitLab CI code to a GitLab CI/CD Component HOT 1
- Proposal: Upgrade to Go 1.22 HOT 2
- Feature: add support for shebang for R HOT 2
- ULOC missing in JSON output format HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scc.