maddyblue / esc Goto Github PK
View Code? Open in Web Editor NEWA simple file embedder for Go
Home Page: http://godoc.org/github.com/mjibson/esc
License: MIT License
A simple file embedder for Go
Home Page: http://godoc.org/github.com/mjibson/esc
License: MIT License
Hey,
This is the lowest of priorities I know... I have recently started using a library scanner that detects vulnerabilities in libraries that I use. It seems to have caught on to the usage of the embedded jquery in example/static.go
Would it be possible to update the example to something else?
$ esc -o static.go index.html style.css
http.Handle("/", http.FileServer(FS(Debug)))
I can access /style.css but not /index.html.
[amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/index.html
HTTP/1.1 301 Moved Permanently
Location: ./
Date: Mon, 24 Nov 2014 08:29:07 GMT
Content-Type: text/plain; charset=utf-8
[amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Date: Mon, 24 Nov 2014 08:29:15 GMT
Content-Length: 19
[amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/style.css
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 1378
Content-Type: text/css; charset=utf-8
Date: Mon, 24 Nov 2014 08:31:26 GMT
[amitu@AmitUs-MacBook-Air martd (master ✗)]$
I cannot use FS() directly since I need to do some templating on the index.html. This is what I'm doing:
// static
router.HandleFunc("/", MkIndexHandler(mc))
http.Handle("/js/", http.FileServer(_escDir(false, "/js")))
the index handler works fine:
func MkIndexHandler(mc *MeasurementCache) func(http.ResponseWriter, *http.Request) {
mainTemplate, err := _escFSString(false, "/index.html")
However, any request to /js
ends in http 404.
assets has the respective files and I'm using the head version with Readdir
support:
var _escData = map[string]*_escFile{
"/js/app.js": {
name: "app.js",
...
var _escDirs = map[string][]os.FileInfo{
"assets": {
_escData["/index.html"],
_escData["/js"],
},
"assets/js": {
_escData["/js/app.js"], ...
},
I don't even see a call to _escDirectory.Open()
. Would appreciate a hint if this is a supported scenario and what might be a potential mistake.
Would you accept a PR to fix all golint errors and warnings?
Let me preface my feature request by first stating that I really have enjoyed using esc. It is simple, uses the Go toolchain (generate), and makes me happy.
I have run across a use case where a possible feature enhancement could make this tool a little easier to use for specific use cases. Below is a sample of some code that demonstrates a scenario where I have implemented a renderer for use in the Echo framework. Simply put it provides a Render method that tells the framework how to render requests.
In the LoadTemplates method is where the imaginary code comes in. In here I imagine the ability to do something similar to ioutil.ReadDir() and get a slice of FileInfo structs. This way I can then iterate over the files and compile them as templates.
package ui
import (
"fmt"
"html/template"
"io"
"path/filepath"
"strings"
"myproject/app/compiledassets"
"github.com/labstack/echo"
)
var templates map[string]*template.Template
/*
TemplateRenderer describes a handlers for rendering layouts/pages
*/
type TemplateRenderer struct {
templates *template.Template
}
/*
NewTemplateRenderer creates a new struct
*/
func NewTemplateRenderer(debugMode bool) *TemplateRenderer {
result := &TemplateRenderer{}
result.LoadTemplates(debugMode)
return result
}
/*
Implements render interface expected by Echo
*/
func (t *TemplateRenderer) Render(w io.Writer, name string, data interface{}, ctx echo.Context) error {
var tmpl *template.Template
var ok bool
if tmpl, ok = templates[name]; !ok {
return fmt.Errorf("Cannot find template %s", name)
}
return tmpl.ExecuteTemplate(w, "layout", data)
}
/*
Traverses the asset pages directory and compiles templates into a package level variable
*/
func (t *TemplateRenderer) LoadTemplates(debugMode bool) {
templates = make(map[string]*template.Template)
/*
* This code isn't real. It is what I imagine esc could look like
* to get a directory of assets. Perhaps returns a slice of
* os.FileInfo structs
*/
files := compiledassets.DirContents("/www")
for _, file := range files {
if !file.IsDir() {
basename := file.Name()
trimmedName := strings.TrimSuffix(basename, filepath.Ext(basename))
templates["mainLayout:"+trimmedName], _ = template.Must(
template.New("layout").Parse(www.FSMustString(debugMode, "/www/myproject/layouts/mainLayout.gohtml")),
).Parse(www.FSMustString(debugMode, "/www/myproject/pages/"+basename))
}
}
}
What are your thoughts on this? Thanks for reading!
First of all, thanks for creating this! It works well and the API is clean and simple.
I used it to bundle wasm and js into an example app so that a server and all content is bundled into the app. Some of my content is generated dynamically, so delegating everything to
http.FileServer(FS(false))
didn't seem workable. I ended up writing handler functions like the following to allow setting proper content headers. Seems to work very well, though I don't know if that's the cleanest possible way to code it.
// fsSendStatic serves the requested file from the ESC FS.
func fsSendStatic(w http.ResponseWriter, r *http.Request) {
_, err := w.Write(FSMustByte(false, r.URL.Path))
if err != nil {
log.Fatalf("Writing file %s returned error: %v", r.URL.Path, err)
}
}
// fsSendStaticJS serves the requested javascript file from the ESC FS.
func fsSendStaticJS(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/javascript")
fsSendStatic(w, r)
}
// fsSendStaticWasm serves the requested WebAssembly file from the ESC FS.
func fsSendStaticWasm(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/wasm")
fsSendStatic(w, r)
}
It took a bit of trial and error to get it working. Just wondering if it might save others some time if the README included an example showing how to serve a mixture of static and dynamic content.
Useful when multiple versions of static files are uses - dev, production.
// +build dev
The current tag/release is quite old now. I suggest creating a new release as the current version is much better then the old one.
I am invoking esc like:
esc -o static.go -modtime=1483228800 -ignore=".*server$" static
I have a folder, which contains files I want to ignore (in the above example a binary called static/server
). However, the regex passed into the -ignore
flag only checks whether the foldername static
matches the regex. Is this intended? I was hoping all the files in the folder static
would be matched against the regex
The ModTime feature added in #13 causes our CI builds to fail due to the timestamps changing and causing our checks to think that the static.go and Tsc generated bosun.js file are out of date. I don't believe git syncs the last modified date, so anyone using multiple systems or having multiple contributors is likely to have the same issue.
Initially after we updated to the new version of esc (see bosun-monitor/bosun#1307 ) I though it was just an issue with TypeScript, but after ensuring it only updates the .js file when the content changes (see bosun-monitor/bosun#1308 ) we are still seeing issues with our CI builds.
I don't think we need the ModTime feature, so having a flag to disable it would meet our needs. @captncraig or I can probably create a PR to disable ModTime (or enable it if you want it disabled by default)
I'm using the /text/template
package for html rendering. Now I want to use esc
for including the templates files in my binary. I'm looking for a good example, but I can't find it. Is there a way to access a file from the generated sources as an asset which will be accepted by the template.-functions?
tmpl, parseErr := template.ParseFiles(baseFile, headerFile, footerFile)
if parseErr != nil {
http.Error(w, parseErr.Error(), http.StatusInternalServerError)
return
}
if templErr := tmpl.Execute(w, renderObject); templErr != nil {
http.Error(w, templErr.Error(), http.StatusInternalServerError)
}
It would be useful to have the hashes of each files pre-computed and accessible with according methods so that we can send strong HTTP Etag headers for caching.
Hi there,
I just discovered this project. I have a similar one called vfsgen
, that also generates a .go file that implements http.FileSystem
and only uses standard library.
In its README, I have a list of alternatives because I think it's a good idea, and I'd like to add your project there. Can you help me understand in what ways this project is unique, what does it focus on/optimize for, and what would be a good description?
Thanks!
I've put together a small update with some additional documentation to the README in a branch found in my fork of this project. I would love some feedback before submitting a pull request. It can be found at https://github.com/adampresley/esc/tree/advanced-readme-example.
I recently upgraded to the newest version of this module and I noticed that esc.Run()
now requires an io.Writer
. That's awesome, however old code requires a lot of template (see StackExchange/dnscontrol#515 for an example).
Would it make sense to have (for example) a nil writer result in the file being created automatically? This would help with migrating old code, and also would reduce the amount of template code needed.
Just a though!
Hi,
Thank you so much for this project.
Many of us felt much pain when go-bindata unexpectedly went down and hundreds of projects stopped building. This is to verify that there are no plans for esc to repeat the same trick.
That will really help with dependency management.
The Open() function for the localFS (ie dev mode where original files are exposed) first checks that the file exists in the last esc-generated file list, and errors if it's not there. More serious refactoring & renaming of files would be permitted if this was simply a pass through to Open(). You wouldn't want this in production for security reasons though - perhaps it a new mode, DANGER_MODE.
Would be great if there was FSGzippedByte
method which would return the compressed representation of the data, afterall pretty much every browser sends Accept-Encoding: gzip
.
Awesome project that we use daily! One thing we do with go generate
is to invoke esc as part of our code generation step to bundle the assets and whatnot into a few "static.go"esq files. It would be very useful to be able to query the version of esc to validate its existence before attempting to use it.
Golang has made changes to how the gzip compression scheme works in Golang 1.8. So if you are embedding files into your program and you would like to support users who have earlier versions of Golang, you will need to manage inconsistent compression errors.
It would be nice to toggle compression on and off.
A generated file should have a comment matching this regex: ^// Code generated .* DO NOT EDIT.$
.
$ esc -o static.go static/
Generates:
var data = map[string]*file{
"/static/mart.js": {compressed: "....", size: 519, local: "static/static/mart.js"},
}
Which is wrong as I do not have static/static/mart.js, and hence it fails.
I hit this data race in my CI:
==================
WARNING: DATA RACE
Read at 0x00c0000d2a68 by goroutine 7:
github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escFile).Name()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:130 +0x3e
github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).Init()
/home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:48 +0x34c
github.com/golang-migrate/migrate/v4/source/httpfs.New()
/home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/driver.go:21 +0xaf
github.com/hortbot/hortbot/internal/db/migrations.newMigrate()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:51 +0x77
github.com/hortbot/hortbot/internal/db/migrations.Up()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:15 +0x8d
github.com/hortbot/hortbot/internal/db/migrations_test.withDatabase()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:67 +0x11e
github.com/hortbot/hortbot/internal/db/migrations_test.TestUp()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:33 +0x52
testing.tRunner()
/home/travis/.gimme/versions/go/src/testing/testing.go:993 +0x1eb
Previous write at 0x00c0000d2a68 by goroutine 77:
github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare.func1()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:59 +0xad
sync.(*Once).doSlow()
/home/travis/.gimme/versions/go/src/sync/once.go:66 +0x103
sync.(*Once).Do()
/home/travis/.gimme/versions/go/src/sync/once.go:57 +0x68
github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:58 +0x160
github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.Open()
/home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:78 +0x46
github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escStaticFS).Open()
<autogenerated>:1 +0x5f
github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).ReadUp()
/home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:111 +0x1b9
github.com/golang-migrate/migrate/v4/source/httpfs.(*driver).ReadUp()
<autogenerated>:1 +0x5b
github.com/golang-migrate/migrate/v4.(*Migrate).newMigration()
/home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:837 +0x96
github.com/golang-migrate/migrate/v4.(*Migrate).readUp()
/home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:612 +0x22b
It appears as though there's a path where the static file is being returned before it has been prepared, but I can't see how at first glance.
I'm using the prefix flag and I noticed that when I run this
esc -o="static.go" -pkg="nictl" -prefix="docs/public" docs/public
on the command line the prefix is stripped but if I use go generate:
//go:generate esc -o="static.go" -pkg="nictl" -prefix="docs/public" docs/public
it isn't. The rest is working absolutely fine...
With prefix specified, make sure that if both prefix/file.name
and file.name
are present, an error is produced.
Add a comment on top of the autogenerated files—something like
// This file was autogenerated by esc (https://github.com/mjibson/esc), do not edit.
In some cases it is desirable to add files to the local fs and test without a need to rebuild an executable. Previously _ascLocalFS.Open() implementation needed the path to exist in the generated map of files in the fs or would fail. This change removes the dependency for the path to already exist in static FS generation.
Eventually It would be nice to change FS to take a string argument that would direct which file system path to use instead of defaulting to to predefined local path. so after deployment of binary an argument could be provided to allow user to specify a path to use for content.
Please See PR I have raised to begin discussion of this change.
I am trying to figure out how to handle passing the 'output' file to a cross compiler so I can cross compile my binary while using this library.
I am fine with working with either goxc
or gox
for cross compiling so if you have a solution for either of those cross compilers, I would be interested in how you do it.
Thanks...
static.go:14:6: don't use underscores in Go names; type _esc_localFS should be _escLocalFS
static.go:16:5: don't use underscores in Go names; var _esc_local should be _escLocal
static.go:18:6: don't use underscores in Go names; type _esc_staticFS should be _escStaticFS
static.go:20:5: don't use underscores in Go names; var _esc_static should be _escStatic
static.go:22:6: don't use underscores in Go names; type _esc_file should be _escFile
static.go:166:5: don't use underscores in Go names; var _esc_data should be _escData
I've been toying with having my CI pipeline re-generate the embedded front-end as a way to check I've remembered to check in any changes:
go get github.com/mjibson/esc
go generate ./frontend/...
git diff --exit-code ./frontend # --exit-code => exits w. 1 on changes, 0 otherwise
But I am getting lots of small changes like this
".": {
- _escData["/icon.png"],
- _escData["/index.html"],
_escData["/icon.svg"],
+ _escData["/index.html"],
+ _escData["/icon.png"],
},
}
Yes, it is a map -- and the spec says they are un-ordered, and I guess that's kind of hard to change. But it would be nice if one could test this regularly, to make sure nothing is forgotten.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.