Comments (10)
Hmm, first I wanted to share this admittedly not great repo to help you set up a dev machine in windows. I'm pretty sure I've used py-in-toto to generate metadata on windows and consume it on linux and vice versa.
Somehow generateKeyID calculates a different checksum on Windows. I have no explanation for this.
I thought I had explained this on slack, but I think this has to do to the fact that windows defaults to utf-16, so you may be hashing a bunch of additional zeros. Have you printed the buffers themselves before they are being passed to the hashing function? (i.e., dump the whole memory of the hashable buffer and then compare them)
from in-toto-golang.
Note. I fixed the issue in RecordArtifact in this commit here: 04b13d5
from in-toto-golang.
Hi @SantiagoTorres
I dig a little bit deeper. Quickfix for most of the test data is using LF for the files. Maybe we should add a .gitattributes
file.
Another Problem I have spotted is this one here:
// Read key bytes
pemBytes, err := ioutil.ReadAll(pemFile)
if err != nil {
return err
}
key, err := decodeAndParse(pemBytes)
if err != nil {
return err
}
// Use type switch to identify the key format
switch key.(type) {
case *rsa.PublicKey:
if err := k.setKeyComponents(pemBytes, []byte{}, rsaKeyType, scheme, keyIdHashAlgorithms); err != nil {
return err
}
As shown in the snippet above, we are reading the pemBytes and using them as input for setKeyComponents. This means, that we are operating on the bytes from the file, hence it works as long as the line separator is the same (LF-LF). But when we try to import a CLRF formatted file, we will end up with a different checksum, because we will load the file bytes directly as public key string in our key object.
from in-toto-golang.
Update:
I wonder if .gitattributes
:
* eol=lf
would make sense for the whole project, although I think that fixing the problem above in the source code, might fix the problem in general. I will need to have a look on the other test output.
from in-toto-golang.
Did we ever execute tests via appveyor on Windows, besides go fmt?
Judging from the Appveyor build log it does look like it:
...
go test ./...
go: downloading golang.org/x/crypto v0.0.0-20191029031824-8986dd9e96cf
go: extracting golang.org/x/crypto v0.0.0-20191029031824-8986dd9e96cf
go: finding golang.org/x/crypto v0.0.0-20191029031824-8986dd9e96cf
ok github.com/in-toto/in-toto-golang/in_toto 0.837s
Build success
You could use rdp and see what's going on on the appveyor workers
from in-toto-golang.
Judging from the Appveyor build log it does look like it:
Has appveyor some different git settings? Maybe autocrlf = false? It definitely does not work in my Windows VM with CRLF files.
from in-toto-golang.
I have build a patch for the issue, I have explained here: #75 (comment)
and indeed, this patch fixes our Windows compatibility problems, that I have introduced in my in-toto run PR.
Have a look on my Github Actions report for Windows: https://github.com/shibumi/in-toto-golang/runs/1029349929?check_suite_focus=true
This works now (please ignore, the send coverage error).
You can find the patch here: #76
from in-toto-golang.
@lukpueh This can be closed now :)
from in-toto-golang.
@lukpueh Ok, I think we are not done here.. I played around with the github actions branch I have and it looks like it only worked, because I have accidentally pushed a gitattributes file to it. The bugs around key loading are fixed through the PR we have merged, but we still have issues:
shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
env:
GOROOT: C:\hostedtoolcache\windows\go\1.14.7\x64
go: downloading golang.org/x/crypto v0.0.0-20191029031824-8986dd9e96cf
json: unsupported type: func(*testing.T)
open bad/path: The system cannot find the path specified.
--- FAIL: TestRecordArtifacts (0.00s)
runlib_test.go:216: RecordArtifacts returned '(map[demo.layout.template:map[sha256:991b055e7001d6f1481849cecb55292e99058ca3d45be0ccaf3ba9a5b34ca6fa] foo.tar.gz:map[sha256:52947cb78b91ad01fe81cd6aef42d1f6817e92b9e6936c1e5aabb7c98514f355] tmpdir/tmpfile:map[sha256:ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad]], %!s(<nil>))', expected '(map[demo.layout.template:map[sha256:019e121a1e0a34aecde0aebb642162b11db4248c781cb8119f81f592723a0424] foo.tar.gz:map[sha256:52947cb78b91ad01fe81cd6aef42d1f6817e92b9e6936c1e5aabb7c98514f355] tmpdir/tmpfile:map[sha256:ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad]], nil)'
--- FAIL: TestInTotoRun (0.15s)
runlib_test.go:331: InTotoRun returned '({{link Name map[demo.layout.template:map[sha256:991b055e7001d6f1481849cecb55292e99058ca3d45be0ccaf3ba9a5b34ca6fa]] map[foo.tar.gz:map[sha256:52947cb78b91ad01fe81cd6aef42d1f6817e92b9e6936c1e5aabb7c98514f355]] map[return-value:%!s(float64=0) stderr:err stdout:out] [sh -c printf out; printf err >&2] map[]} [{be6371bc627318218191ce0780fd3183cce6c36da02938a477d2e4dfae1804a6 c0d67be0689b1f9470929c023b6733124833aea33c345d737f1a5047cac75a46a3d88b9b9c81610dbc2c18d9fdb08ad40170cdc6ce3e45609c8fdcd5bff82a08}]}, %!s(<nil>))', expected '({{link Name map[demo.layout.template:map[sha256:019e121a1e0a34aecde0aebb642162b11db4248c781cb8119f81f592723a0424]] map[foo.tar.gz:map[sha256:52947cb78b91ad01fe81cd6aef42d1f6817e92b9e6936c1e5aabb7c98514f355]] map[return-value:%!s(float64=0) stderr:err stdout:out] [sh -c printf out; printf err >&2] map[]} [{be6371bc627318218191ce0780fd3183cce6c36da02938a477d2e4dfae1804a6 08a6c42b8433502f2869bb3dc73f8348f6b6f89e42bbc63f91a33e7171d762e138ed5d695fb83cebec958203e17b2285f95b198d758bc62cf30e1f7408d6c10c}]}, nil)'
WARNING: syntax error in pattern, pattern was '['
[begin test warning output]
WARNING: Expected command for step 'foo' (rm -rf .) and command reported by 'foo.a.link' (rm -rf /) differ.
[end test warning output]
FAIL
coverage: 95.6% of statements
FAIL github.com/in-toto/in-toto-golang/in_toto 1.284s
FAIL
##[error]Process completed with exit code 1.
This time it's the demo.layout.template
file
from in-toto-golang.
The problem seems to be the RecordArtifact function:
func RecordArtifact(path string) (map[string]interface{}, error) {
hashObjectMap := createMap()
// Read file from passed path
contents, err := ioutil.ReadFile(path)
hashedContentsMap := make(map[string]interface{})
if err != nil {
return nil, err
}
// Create a map of all the hashes present in the hash_func list
hashFunc := []string{"sha256"}
for _, element := range hashFunc {
result := hashObjectMap[element].Compute(contents)
hashedContentsMap[element] = result
}
// Return it in a format that is conformant with link metadata artifacts
return hashedContentsMap, nil
}
We are reading the raw input of a file path here. It's only failing for demo.layout.template, because foo.tar.gz is a binary archive (so no line separators) and the test file tmpfile
is just a string without any line separator.
I don't know how we are going to fix this now.
For the Github Actions PR I plan to ship the attributes file for now, this fixes this temporary, but we really need to think about reading layout files and link files and everything file related. What we need is a function, that normalizes line separators. @trishankatdatadog pointed me to a securesystemslib function, that does this.
from in-toto-golang.
Related Issues (20)
- Add version/SHA to provevanceBuilder HOT 3
- Support older SLSA version and v1.0 HOT 8
- CycloneDX predicate is wrong HOT 1
- non unique dictionary key error HOT 9
- Support v1.0 SLSA provenance format
- in-toto-golang does not provide defaults for excluding artifacts HOT 7
- Inconsistent path match behaviour compared to python implementation HOT 4
- Option to substitute signing algorithm to a custom version HOT 3
- Verification using ED25519 keys for functionaries returns error HOT 9
- Typo in provenance structure HOT 1
- Revisit type assertions
- in-toto-golang record stop doesn't removes unfinished link. HOT 2
- Add OpenSSF Scorecards HOT 3
- Use in-toto/attestation Go bindings for Statement and SLSA provenance predicate layers
- ALLOW pattern doesn't work as expected on windows HOT 1
- Oversight in the SLAS v1 field `invocationID`? HOT 7
- Define intoto v1.0 statement type HOT 6
- Improve Scorecard score for in-toto-golang HOT 1
- CII OpenSSF best-practice badge HOT 1
- Source code missing for test/data/helloworld binary HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from in-toto-golang.