ncw / directio Goto Github PK
View Code? Open in Web Editor NEWThis is library for the Go language to enable use of Direct IO under all OSes
License: MIT License
This is library for the Go language to enable use of Direct IO under all OSes
License: MIT License
from the knowledge of page size alignment, I presume that AlignedBlock should return the multiple size of AlignSize, eg: given AlignedSize 4096, when blocksize 3000 is given, it should align to 4096, but AlignedBlock return 3000 instead, do I misunderstood something?
Hey Nick,
If I call AlignedBlock
with 0
, I get a panic! The following source code:
package main
import (
"github.com/ncw/directio"
)
func main() {
directio.AlignedBlock(0)
}
produced the following panic:
panic: runtime error: index out of range
goroutine 1 [running]:
runtime.panic(0x49c2a0, 0x56ad17)
/usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/ncw/directio.AlignedBlock(0x0, 0xc21001f140, 0x4160c0, 0x400c6d)
/home/vagrant/go/src/github.com/ncw/directio/direct_io.go:43 +0x1c2
main.main()
/home/vagrant/go/src/github.com/10gen/mongo-zoo/mem/foo.go:8 +0x26
exit status 2
This is a really cool repository by the way!
Best,
Michael
Is there any better way to use directio with scanner
type directioReader struct {
r io.Reader
buf, rest []byte
}
// read through buf without allocating more to allow usage in bufio.Scanner
func (r *directioReader) Read(b []byte) (n int, err error) {
if r.rest != nil {
n = copy(b, r.rest)
if n < len(r.rest) {
r.rest = r.rest[n:]
return
} else {
r.rest = nil
}
}
rx, err := r.r.Read(r.buf)
if err != nil {
return n + rx, err
}
nx := copy(b[n:], r.buf[:rx])
if (rx - nx) > 0 {
r.rest = r.buf[nx:rx]
}
return nx + n, nil
}
func NewDirectIOReader(r io.Reader) io.Reader {
return &directioReader{
r: r,
buf: directio.AlignedBlock(directio.BlockSize),
}
}
f, err := directio.OpenFile("filename.txt", os.O_RDONLY|os.O_SYNC, 0)
if err != nil {
panic(err)
}
s := bufio.NewScanner(NewDirectIOReader(f))
/// s scanner usage for s.Scan { , etc...
Hello!
I'm not sure about problem in ncw/directio but it's possible.
Version with directio.OpenFile:
echo 3 > /proc/sys/vm/drop_caches
/root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
539MB 0:00:25 [21,1MB/s]
Version with os.OpenFile:
echo 3 > /proc/sys/vm/drop_caches
/root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
539MB 0:00:03 [ 148MB/s]
As u can see O_DIRECT version is much times slower then same code without O_DIRECT. But it's very strange because I cleaned page cache before tests.
Full code you can find here: https://gist.github.com/pavel-odintsov/f303309f5284e09bc359
Thank you!
Hey there,
thanks for this awesome library! I was playing around with this today a bit.
I do have a somewhat silly requirement in that some writes may not be aligned with the block size (forced flushes/fsyncs out of my control). On Linux it's fairly easy to toggle O_DIRECT on and off through fcntl, I assume on MAC it works the same using F_NOCACHE.
Now I'm currently browsing the win APIs, but I don't find anything similar. Do you have a clue?
Would you be ok if I'd send you a PR for such a feature?
Reference Commit: 17018d9
TestDirectIO
fails on my Linux system because of the mode syscall.O_DIRECT
, which is set by directio.OpenFile
.
As proof, I noted that changing line 29 to use os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0666)
eliminates the error. The only difference between these two calls is the use of syscall.O_DIRECT
.
Has this mode been deprecated? Is this project still active? Thanks!
go version
= go1.7 linux/amd64
uname -a
= Linux localhost.localdomain 4.13.16-202.fc26.x86_64 #1 SMP Thu Nov 30 15:39:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[jess@localhost directio]$ go test
--- FAIL: TestDirectIo (0.00s)
direct_io_test.go:32: Failed to directio.OpenFile for read open /tmp/direct_io_test087816965: invalid argument
FAIL
exit status 1
FAIL directio 0.001s
Hello.
I have tried to access to a physical drive (flash drive on windows machine) by using this code:
drive, err := directio.OpenFile(`\\.\PhysicalDrive9`, os.O_RDWR, 0666)
But without root I can't get access to the device (open \\.\PhysicalDrive9: Access is denied
).
Tried to play with different os.FileMode's, but the result remained the same. Please can you tell me if i fail with mode's/os.FileMode's, or direct access to the device can be got only with the admin rights only?
I would be very grateful for any help.
Perhaps it should be: http://godoc.org/github.com/ncw/directio?
when buf size less than blockSize, it reports error when write after calling AlignedBlock(). It should be noted down if not support this case.
You hard code alignment in windows to 4096 you should either use http://msdn.microsoft.com/en-us/library/aa364935%28VS.85%29.aspx which gives the sector size or io control to get it instead http://msdn.microsoft.com/en-us/library/windows/desktop/aa363147(v=vs.85).aspx
Also some of the comments are wrong re disabling drive level caches directio does nothing with this the controller can still cache etc (and most do even if they are not safe!)
Cheers,
Greg
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.