ncw / directio Goto Github PK

from the knowledge of page size alignment, I presume that AlignedBlock should return the multiple size of AlignSize, eg: given AlignedSize 4096, when blocksize 3000 is given, it should align to 4096, but AlignedBlock return 3000 instead, do I misunderstood something?

Hey Nick,

If I call AlignedBlock with 0, I get a panic! The following source code:

package main

import (
    "github.com/ncw/directio"
)

func main() {
    directio.AlignedBlock(0)
}

produced the following panic:

panic: runtime error: index out of range

goroutine 1 [running]:
runtime.panic(0x49c2a0, 0x56ad17)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/ncw/directio.AlignedBlock(0x0, 0xc21001f140, 0x4160c0, 0x400c6d)
    /home/vagrant/go/src/github.com/ncw/directio/direct_io.go:43 +0x1c2
main.main()
    /home/vagrant/go/src/github.com/10gen/mongo-zoo/mem/foo.go:8 +0x26
exit status 2

This is a really cool repository by the way!

Best,
Michael

Is there any better way to use directio with scanner

type directioReader struct {
	r io.Reader
	buf, rest []byte
}
// read through buf without allocating more to allow usage in bufio.Scanner
func (r *directioReader) Read(b []byte) (n int, err error) {
	if r.rest != nil {
		n = copy(b, r.rest)
		if n < len(r.rest) {
			r.rest = r.rest[n:]
			return
		} else {
			r.rest = nil
		}
	}
	rx, err := r.r.Read(r.buf)
	if err != nil {
		return n + rx, err
	}
	nx := copy(b[n:], r.buf[:rx])
	if (rx - nx) > 0 {
		r.rest = r.buf[nx:rx]
	}
	return nx + n, nil
}
func NewDirectIOReader(r io.Reader) io.Reader {
	return &directioReader{
		r: r,
		buf: directio.AlignedBlock(directio.BlockSize),
	}
}

f, err := directio.OpenFile("filename.txt", os.O_RDONLY|os.O_SYNC, 0)
if err != nil {
   panic(err)
}
s := bufio.NewScanner(NewDirectIOReader(f))
/// s scanner usage for s.Scan { , etc...

Hello!

I'm not sure about problem in ncw/directio but it's possible.

Version with directio.OpenFile:

echo 3 > /proc/sys/vm/drop_caches
 /root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
 539MB 0:00:25 [21,1MB/s] 

Version with os.OpenFile:

echo 3 > /proc/sys/vm/drop_caches
/root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
 539MB 0:00:03 [ 148MB/s] 

As u can see O_DIRECT version is much times slower then same code without O_DIRECT. But it's very strange because I cleaned page cache before tests.

Full code you can find here: https://gist.github.com/pavel-odintsov/f303309f5284e09bc359

Thank you!

Hey there,

thanks for this awesome library! I was playing around with this today a bit.

I do have a somewhat silly requirement in that some writes may not be aligned with the block size (forced flushes/fsyncs out of my control). On Linux it's fairly easy to toggle O_DIRECT on and off through fcntl, I assume on MAC it works the same using F_NOCACHE.

Now I'm currently browsing the win APIs, but I don't find anything similar. Do you have a clue?

Would you be ok if I'd send you a PR for such a feature?

Reference Commit: 17018d9

TestDirectIO fails on my Linux system because of the mode syscall.O_DIRECT, which is set by directio.OpenFile.

As proof, I noted that changing line 29 to use os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0666) eliminates the error. The only difference between these two calls is the use of syscall.O_DIRECT.

Has this mode been deprecated? Is this project still active? Thanks!

System information

• go version = go1.7 linux/amd64
• uname -a = Linux localhost.localdomain 4.13.16-202.fc26.x86_64 #1 SMP Thu Nov 30 15:39:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Test Output

[jess@localhost directio]$ go test
--- FAIL: TestDirectIo (0.00s)
	direct_io_test.go:32: Failed to directio.OpenFile for read open /tmp/direct_io_test087816965: invalid argument
FAIL
exit status 1
FAIL	directio	0.001s

Hello.
I have tried to access to a physical drive (flash drive on windows machine) by using this code:
drive, err := directio.OpenFile(`\\.\PhysicalDrive9`, os.O_RDWR, 0666)
But without root I can't get access to the device (open \\.\PhysicalDrive9: Access is denied).
Tried to play with different os.FileMode's, but the result remained the same. Please can you tell me if i fail with mode's/os.FileMode's, or direct access to the device can be got only with the admin rights only?
I would be very grateful for any help.

Perhaps it should be: http://godoc.org/github.com/ncw/directio?

when buf size less than blockSize, it reports error when write after calling AlignedBlock(). It should be noted down if not support this case.

You hard code alignment in windows to 4096 you should either use http://msdn.microsoft.com/en-us/library/aa364935%28VS.85%29.aspx which gives the sector size or io control to get it instead http://msdn.microsoft.com/en-us/library/windows/desktop/aa363147(v=vs.85).aspx

Also some of the comments are wrong re disabling drive level caches directio does nothing with this the controller can still cache etc (and most do even if they are not safe!)

Cheers,

Greg