unnoted / fileb0x Goto Github PK

View Code? Open in Web Editor NEW

633.0 18.0 53.0 603 KB

a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

License: MIT License

Batchfile 0.47% Go 94.14% HTML 5.09% Shell 0.18% CSS 0.12%

go embed files go-bindata virtual-file-system embed-files embedded-files update http http-filesystem

fileb0x's People

Contributors

Stargazers

Watchers

fileb0x's Issues

How to disable directory browsing?

i run _example/gin example,

open http://127.0.0.1:8080/css/ can show all file list,

how to disable directory browsing?

run go fmt on output by default

Is there any reason to not perform go fmt on the code that is generated? (by default)

file birthtime

Wondering why you didn't implement file birthtime?

stat -c '%w'

Thanks for the laugh!

I just got a kick out of this:

https://github.com/UnnoTed/fileb0x/blob/master/_example/simple/public/secrets.txt

Thanks!

Some errors from vbauerster/mpb

+ make generate
# github.com/UnnoTed/fileb0x/updater
/go/src/github.com/UnnoTed/fileb0x/updater/updater.go:231:9: p.Stop undefined (type *mpb.Progress has no field or method Stop)
/go/src/github.com/UnnoTed/fileb0x/updater/updater.go:268:5: undefined: decor.Name
/go/src/github.com/UnnoTed/fileb0x/updater/updater.go:279:7: bar.Incr undefined (type *mpb.Bar has no field or method Incr)

See vbauerster/mpb@23769ee

go get Fail

go get -u github.com/UnnoTed/fileb0x

# github.com/ungerik/go-dry
src/github.com/ungerik/go-dry/debug.go:46:8: undefined: strings.Builder
src/github.com/ungerik/go-dry/debug.go:58:8: undefined: strings.Builder
src/github.com/ungerik/go-dry/debug.go:64:55: undefined: strings.Builder
src/github.com/ungerik/go-dry/errors.go:115:8: undefined: strings.Builder
src/github.com/ungerik/go-dry/string.go:148:8: undefined: strings.Builder
src/github.com/ungerik/go-dry/string.go:257:8: undefined: strings.Builder
src/github.com/ungerik/go-dry/string.go:274:8: undefined: strings.Builder

go version
go version go1.9.2 linux/amd64

Generated code needs gofix

Generated code imports golang.org/x/net/context but since go 1.7 it became a part of standard library and gofix replaces import with context.

Before gofix:

import (
	"bytes"
	"compress/gzip"
	"io"
	"net/http"
	"os"
	"path"

	"golang.org/x/net/context"
	"golang.org/x/net/webdav"
)

After gofix:

import (
	"bytes"
	"compress/gzip"
	"io"
	"net/http"
	"os"
	"path"

	"context"
	"golang.org/x/net/webdav"
)

webdav file system requires context to open files

After this commit golang/net@4bb47a1 it's going to be required to send context. At the very minimum opening files needs to be sent context.Background().

bufio.Scanner limitations

Hey.
When trying to build a big project came across a problem:

panic: bufio.Scanner: token too long

goroutine 1 [running]:
main.main()
        Q:/Soft/Go/GOPATH/src/github.com/UnnoTed/fileb0x/main.go:127 +0x24f1
main.go:1: running "fileb0x": exit status 2

As a solution I would suggest using bufio.Reader (PR on the way).

How to prefer local file overrides?

Let's say that I want to use the in-memory filesystem only when a local file doesn't exist (i.e. the in-memory is the fallback).

Is there some option for fileb0x to handle this?

Or what might be a good way to wrap it to do so?

An attempt is made to create the root directory

My directory structure looks something like this:

.
├── b0x.yaml
└── server
    ├── ab0x.go
    └── static
        └── css
            └── styles.min.css

b0x.yaml contains the following:

pkg: server
dest: server
compression:
  compress: true
custom:
  - files:
    - server/static
    base: server

However, when I generate server/ab0x.go, the following is added to init():

err = FS.Mkdir(CTX, "/", 0777)
if err != nil {
  log.Fatal(err)
}

This is a problem because attempting to create the root directory results in an error.

"Base" option stopped working

Hi,

Seems that 5494112 changes the behavior of base.

I have something like this:

custom:
  - files:
    - "public/js"
    - "public/css"
    base: "public/"

Before that commit, I could serve files as /js/... and /css/... instead of /public/js/. But now public/ is not being removed from the path anymore.

Thanks for this awesome lib.

panic: invalid argument

error info:

panic: invalid argument

goroutine 1 [running]:
github.com/NiceLabs/DriveIndex/listing/embedded.init.0()
	/Users/septs/Projects/DriveIndex/listing/embedded/aembedded.go:58 +0xdda
exit status 2

generated code:

	err := CTX.Err()
	if err != nil {
		panic(err)
	}

	err = FS.Mkdir(CTX, "../", 0777)
	if err != nil && err != os.ErrExist {
		panic(err)
	}

	err = FS.Mkdir(CTX, "../frontend/", 0777)
	if err != nil && err != os.ErrExist {
		panic(err)
	}

configuration:

pkg: embedded
dest: "./embedded"

fmt: true
clean: true

output: "embedded.go"

custom:
  - files:
      - "../frontend/dist/*"

compression:
  compress: true
  method: BestCompression

go get fail: bad checksum

On Windows 10 with Go 1.11 I ran go get github.com/UnnoTed/fileb0x, which has this as a dependency, and I get this error:

go: verifying github.com/karrick/[email protected]: checksum mismatch
        downloaded: h1:e5iv87oxunQtG7S9MB10jrINLmF7HecFSjiTYKO7P2c=
        go.sum:     h1:UP4CfXf1LfNwXrX6vqWf1DOhuiFRn2hXsqtRAQlQOUQ=

This also happens on CircleCI, but it doesn't happen on OS X.

I'm wondering if there isn't some CRLF or filename case sensitivity issues. Any thoughts?

github.com/UnnoTed/fileb0x/custom

custom\custom.go:58:31: cannot use customFile (type string) as type fs.FS in argument to doublestar.Glob:
string does not implement fs.FS (missing Open method)

debug mode issues

Hello.

I have found several problems, in attempts to use 'debug' mode.

If try to enable 'debug' without changing my configuration file (which include compression options), I get:
./a_fileb0x.go:8: imported and not used: "compress/gzip".

When comment/remove the 'compression' section, get quite expected ressult:
template: files:8:37: executing "files" at <$Compression.Compres...>: can't evaluate field Compress in type *compression.Options

But if set compress: false and debug: true - then I cannot access any files via http.FileServer.

Example: test_fileb0x.zip

can't install on windows: can't find doublestar

I'm not clear as to why yet, but when I try to install on windows, it can't find the doublestar module:

C:\Users\aj\github.com\UnnoTed\fileb0x>go install .
custom\custom.go:17:2: unknown import path "github.com/bmatcuk/doublestar": cannot find module providing package github.com/bmatcuk/doublestar

Also won't build on windows:

C:\Users\aj\github.com\UnnoTed\fileb0x> go mod vendor
C:\Users\aj\github.com\UnnoTed\fileb0x>go build -mod=vendor
custom\custom.go:17:2: cannot find package "." in:
        C:\Users\aj\github.com\UnnoTed\fileb0x\vendor\github.com\bmatcuk\doublestar

And go get fails within fileb0x:

C:\Users\aj\github.com\UnnoTed\fileb0x>go get
go build github.com/bmatcuk/doublestar: no Go files in

Yet this doesn't seem to be a direct problem with doublestar because go get github.com/bmatcuk/doublestar works just fine. Very confused...

labstack/echo dependency is vulnerable

❯ go mod why -m github.com/labstack/echo
# github.com/labstack/echo
...
github.com/UnnoTed/fileb0x
github.com/labstack/echo

Warning:(178, 2)  Dependency go:github.com/labstack/echo:v3.2.1+incompatible is vulnerable, safe version v3.3.6+incompatible CVE-2022-40083 9.6 URL Redirection to Untrusted Site ('Open Redirect') vulnerability with high severity found   Results powered by Checkmarx(c)

GHSA-crxj-hrmp-4rwf

Update timestamp only if there are other changes

Hi!

Is there a way to make the asset generation idempotent so that it won't update the timestamp in the assets file if there are no other changes within the data?

My configuration looks like this:

pkg: assets
dest: "."
fmt: true
output: assets.go
updater:
  enabled: false
custom:
  - files:
    - "./data/*.jpeg"
    base: "./data/"

mpb.New().SetWidth and bar.Completed are undefined

I am receiving compilation errors with the latest revision:

src/github.com/UnnoTed/fileb0x/updater/updater.go:225: mpb.New().SetWidth undefined (type *mpb.Progress has no field or method SetWidth)
src/github.com/UnnoTed/fileb0x/updater/updater.go:333: bar.Completed undefined (type *mpb.Bar has no field or method Completed)

go get -u github.com/UnnoTed/fileb0x error

github.com/airking05/termui

C:\Users\administrator\go\pkg\mod\github.com\airking05\[email protected]+incompatible\render.go:107:30: not enough arguments in call to stack.ParseDump
have (*bytes.Reader, *os.File)
want (io.Reader, io.Writer, bool)
C:\Users\administrator\go\pkg\mod\github.com\airking05\[email protected]+incompatible\render.go:112:10: undefined: stack.Palette
C:\Users\administrator\go\pkg\mod\github.com\airking05\[email protected]+incompatible\render.go:113:15: undefined: stack.SortBuckets
C:\Users\administrator\go\pkg\mod\github.com\airking05\[email protected]+incompatible\render.go:113:33: undefined: stack.Bucketize
C:\Users\administrator\go\pkg\mod\github.com\airking05\[email protected]+incompatible\render.go:114:22: undefined: stack.CalcLengths

Please tag as v1.0.0 (or whatever)

For the sake of being able to easily track likely breaking changes it would be great to have some tags.

git tag v1.0.0
git push --tags

Since this is already much used and well-tested in the wild, starting at v1.0.0 makes sense to me - but you do what you like :)

debug prefixes issue

Hey.
When I tried to use debug mode, while some files had a prefix in the configuration file, I did't access them by "old" link. Access to these files can be obtained only if you request it by path, which corresponds to its real and actual location in the host file system.

Test files: test_fileb0x.zip
While debug: true try to get a file by url http://127.0.0.1:10000/test_prefix/static/test.html.

Generated code does not check all errors

The code that fileb0x generates does not have all its errors checked, causing it to fail checkers like errcheck and staticcheck. While this can be worked around, it would be nice if the code generated was free of lint.

simple example errors

So I tried the example as per the README on my MBP (Mojave) system and got to...

$ go generate
go: finding module for package example.com/foo/simple/static
go: downloading golang.org/x/net v0.0.0-20180921000356-2f5d2388922f
go: downloading github.com/labstack/echo v3.2.1+incompatible
go: downloading github.com/karrick/godirwalk v1.7.3
go: downloading github.com/airking05/termui v2.2.0+incompatible
verifying github.com/karrick/[email protected]: checksum mismatch
        downloaded: h1:e5iv87oxunQtG7S9MB10jrINLmF7HecFSjiTYKO7P2c=
        go.sum:     h1:UP4CfXf1LfNwXrX6vqWf1DOhuiFRn2hXsqtRAQlQOUQ=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
main.go:1: running "go": exit status 1

$ go build
go: finding module for package example.com/foo/simple/static
main.go:10:2: cannot find module providing package example.com/foo/simple/static: unrecognized import path "example.com/foo/simple/static": reading https://example.com/foo/simple/static?go-get=1: 404 Not Found

$ go version go1.15.7 darwin/amd64

I don't know if the security error is also causing the module finding logic to fail as well. Any thoughts?

Hash in file name?

Is there a way to serve the same files with hash in their name for aggressive caching?

looks like the last couple of commits broke code generation

here is what we are getting in our CI:

09:41:52 # github.com/Aptomi/aptomi/pkg/server/ui
09:41:52 pkg/server/ui/ab0x.go:40:2: undefined: err
09:41:52 pkg/server/ui/ab0x.go:41:5: undefined: err
09:41:52 pkg/server/ui/ab0x.go:42:9: undefined: err
09:41:52 pkg/server/ui/ab0x.go:45:2: undefined: err
09:41:52 pkg/server/ui/ab0x.go:46:5: undefined: err
09:41:52 pkg/server/ui/ab0x.go:47:9: undefined: err
09:41:52 pkg/server/ui/ab0x.go:50:2: undefined: err
09:41:52 pkg/server/ui/ab0x.go:51:5: undefined: err
09:41:52 pkg/server/ui/ab0x.go:52:9: undefined: err
09:41:52 pkg/server/ui/ab0x.go:55:2: undefined: err
09:41:52 pkg/server/ui/ab0x.go:55:2: too many errors
09:41:58 Makefile:85: recipe for target 'build' failed

Manual initialization

Initialization is automatically done in init() function. This means that even if the binary knows it is not going to use the files (e.g. through command line options) the data will still be unpacked into memory. This wastes memory and slows down start up time.

It would be really good if an option could be added to export an init function which must be called manually to build the webdav FS object for the first time.

too many arguments in call to {FS.Mkdir, FS.OpenFile}

This error is happening while compiling the generated source.

Rolling back to 89859f8 fixed for me.

Go complains about unused variables

When compiling my main file, which imports a file generated with fileb0x, go finds err f rb and r to be "declared but not used". The only way I found to fix this was to manually delete the variables from inside the generated file.

Can you guys make a stable release?

Our CI has been failing recently several times because we point to fileb0x master.

First time it was failing because of
2e8a2eb

Then lately:

09:40:29 ../../UnnoTed/fileb0x/updater/updater.go:317:6: bar.Complete undefined (type *mpb.Bar has no field or method Complete)

A stable release would be great. We could point to specific commit too, but that's not ideal.

repository discoverability

From a previous use i knew this repository (and really liked it!) by its name ("filebox") and tried to google it ("golang filebox") but didn't find anything. I had to look it up on awesome golang. I'd suggest to add some tags like "filebox" to make this repo easiert to discover.

"prefix" bugs?

Hey.

Has faced with several problems.

First

In attempt to use such configuration - I receive strange behavior. First dot in filenames will be replaced with a prefix var.

custom:
  - files:
    - "./main.go"
    - "./fileb0x.yaml"
    base: "./"
    prefix: "_bug?_"

Second

The prefix option doesn't work for me :(

  - files:
    - "./static"
    prefix: "test_prefix/"

Result:

go get -u -v github.com/UnnoTed/fileb0x fails

▶ go get -u -v github.com/UnnoTed/fileb0x
github.com/UnnoTed/fileb0x (download)
github.com/BurntSushi/toml (download)
github.com/vbauerster/mpb (download)
github.com/VividCortex/ewma (download)
github.com/bmatcuk/doublestar (download)
github.com/karrick/godirwalk (download)
github.com/pkg/errors (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/labstack/echo (download)
github.com/labstack/gommon (download)
github.com/mattn/go-colorable (download)
github.com/mattn/go-isatty (download)
github.com/valyala/fasttemplate (download)
Fetching https://golang.org/x/crypto/acme/autocert?go-get=1
Parsing meta tags from https://golang.org/x/crypto/acme/autocert?go-get=1 (status code 200)
get "golang.org/x/crypto/acme/autocert": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/acme/autocert?go-get=1
get "golang.org/x/crypto/acme/autocert": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/crypto/acme?go-get=1
Parsing meta tags from https://golang.org/x/crypto/acme?go-get=1 (status code 200)
get "golang.org/x/crypto/acme": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/acme?go-get=1
get "golang.org/x/crypto/acme": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/webdav?go-get=1
Parsing meta tags from https://golang.org/x/net/webdav?go-get=1 (status code 200)
get "golang.org/x/net/webdav": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/webdav?go-get=1
get "golang.org/x/net/webdav": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
Fetching https://golang.org/x/net/context?go-get=1
Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
get "golang.org/x/net/context": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
get "golang.org/x/net/context": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/webdav/internal/xml?go-get=1
Parsing meta tags from https://golang.org/x/net/webdav/internal/xml?go-get=1 (status code 200)
get "golang.org/x/net/webdav/internal/xml": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/webdav/internal/xml?go-get=1
get "golang.org/x/net/webdav/internal/xml": verifying non-authoritative meta tag
github.com/UnnoTed/fileb0x/updater
# github.com/UnnoTed/fileb0x/updater
../../UnnoTed/fileb0x/updater/updater.go:253:31: cannot use 0 (type int) as type decor.WC in argument to decor.StaticName
../../UnnoTed/fileb0x/updater/updater.go:254:32: cannot use 0 (type int) as type decor.WC in argument to decor.StaticName
../../UnnoTed/fileb0x/updater/updater.go:254:32: cannot use decor.DSyncSpace (type int) as type decor.WC in argument to decor.StaticName
../../UnnoTed/fileb0x/updater/updater.go:255:45: cannot use 18 (type int) as type decor.WC in argument to decor.CountersKibiByte
../../UnnoTed/fileb0x/updater/updater.go:255:45: cannot use decor.DSyncSpace (type int) as type decor.WC in argument to decor.CountersKibiByte
../../UnnoTed/fileb0x/updater/updater.go:257:34: not enough arguments in call to decor.ETA
../../UnnoTed/fileb0x/updater/updater.go:257:38: undefined: decor.DwidthSync

Possible to pack SQLite file?

I have a program that read data from local sqlite db, and show data to users, I wanna to pack all assets include sqlite file to one binary file, this program only execute read operation from sqlite, is it possible to pack sqlite file?

Undefined: err

I'm getting an error that looks very similar to #21 but I've double-checked and I have the fixed version of the code with the commit mentioned in that PR:

# github.com/akerl/github-auth-lambda/static
static/ab0x.go:45:5: undefined: err
static/ab0x.go:46:5: undefined: err
static/ab0x.go:47:9: undefined: err
static/ab0x.go:50:5: undefined: err
static/ab0x.go:51:5: undefined: err
static/ab0x.go:52:9: undefined: err
static/ab0x.go:55:2: undefined: err
static/ab0x.go:56:5: undefined: err
static/ab0x.go:57:9: undefined: err

Here's my box.yaml: https://github.com/akerl/github-auth-lambda/blob/9f2de9c3a6fe8bde4484aada59ad872ab57ef638/box.yaml

unescaped paths on windows

When trying to build a big project came across a second problem:

.\a_fileb0x.go:66:14: unknown escape sequence
.\a_fileb0x.go:69:12: unknown escape sequence
.\a_fileb0x.go:72:17: unknown escape sequence
.\a_fileb0x.go:75:12: unknown escape sequence
.\a_fileb0x.go:78:12: unknown escape sequence
.\a_fileb0x.go:81:12: unknown escape sequence
.\a_fileb0x.go:84:12: unknown escape sequence
.\a_fileb0x.go:87:17: unknown escape sequence
.\a_fileb0x.go:90:17: unknown escape sequence
.\a_fileb0x.go:93:12: unknown escape sequence
.\a_fileb0x.go:93:12: too many errors

There was something similar to (we could receive unescaped \n,\t, etc in string):

var remap = map[string]map[string]string{
  "fileb0x.test.yaml": {
		"prefix": "external/",
		"base": "",
	},"static\test.html": {
		"prefix": "test_prefix/",
		"base": "",
	},"static\test.txt": {
		"prefix": "test_prefix/",
		"base": "",
	},"main.go": {
		"prefix": "external/",
		"base": "",
	},
}

PS: PR on the way