velocidex / go-ntfs Goto Github PK

Hi!

There's good support here for LZNT1 compression (created from Windows Compress contents to save space option or compact.exe /c).

In Windows 10 / Windows Server 2016, some extra compression algorithms XPRESS4K, XPRESS8K, XPRESS16K, and LZX were added (created from compact.exe /c /EXE LZX). These have higher compression ratios but do not support in-place modification.

On disk these files show up as reparse points, with the compressed data stored in a WofCompressedData ADS stream. In real Windows decompression is performed by a filesystem filter driver above ntfs. So it's arguable if it belongs in this project or not. But to correspond to the existing lznt1 support, there should be support for the other compression algorithms too.

Some links:

• In other libraries like ntfs-3g, it's available as an add-on plugin: https://github.com/ebiggers/ntfs-3g-system-compression (GPLv2/LGPLv3)
  • The ntfs-3g version was the basis for the Linux kernel ntfs3 implementation: https://github.com/torvalds/linux/blob/master/fs/ntfs3/lib/xpress_decompress.c
  • And the ntfs-3g version is based on wimlib's version: https://github.com/matchaxnb/wimlib/blob/master/src/xpress-decompress.c
• Sleuthkit doesn't have an implementation
• https://github.com/comaeio/rust-lzxpress Some rust code in MIT
• https://www.swiftforensics.com/2016/10/wofcompressed-streams-in-windows-10.html (2016) discussion about NTFS forensics tools

i wrote the following code for file-log search on my windows,
but it get "Invalid magic" error, how can i fix up this demo code ? thank you~

`
package main

import (
"context"
"fmt"
"io"
"log"
"os"
"strings"
"github.com/Velocidex/go-ntfs/parser"
)

const (
record_directory = "out.txt"
)

func getReader(reader io.ReaderAt) io.ReaderAt {
if record_directory == "" {
return reader
}
return parser.NewRecorder(record_directory, reader)
}

func main() {
rd, er := os.Open(\\.\D:)
if er != nil {
log.Fatalln(er)
}
defer rd.Close()
reader, err := parser.NewPagedReader(getReader(rd), 1024, 10000)
if err != nil {
log.Fatalln(err)
}
ntfs_ctx, err := parser.GetNTFSContext(reader, 0)
if err != nil {
log.Fatalln(err)
}
for record := range parser.ParseUSN(context.Background(), ntfs_ctx, 0) {
fmt.Println(record.Usn())
fmt.Println(record.Offset)
fmt.Println(record.Filename())
fmt.Println(record.FullPath())
fmt.Println(record.TimeStamp())
fmt.Println(strings.Join(record.Reason(), ", "))
fmt.Println(strings.Join(record.FileAttributes(), ", "))
fmt.Println(strings.Join(record.SourceInfo(), ", "))
}
}

`

The redirect at https://www.velocidex.com/golang/go-ntfs/parser is not working, so e.g. https://godoc.org/www.velocidex.com/golang/go-ntfs/parser will not work.

Hi,
I'm looking for a way to add more precise timestamp for SI and FN fields.
If I'm right, the actual resolution of the SI and FN timestamps in NTFS is 100 nanoseconds counting from Jan 1, 1601.
Could we get such an output in MFT module output?
Thanks in advance

Hi,

I have an NTFS image that contains a compressed (LZNT1) file. When trying to read its content with go-ntfs, i get the Decompression error - shift is too large error message.

7-Zip's ntfs library is successfully able to extract this file from the disk image.

Steps to reproduce:

1. Create a compressible file with size > 10 MiB
   • In this case the file was created by something like: for i in {1..512} ; do dd if=/dev/zero bs=4K count=10 >> combofile.bin ; dd if=/dev/urandom bs=4K count=10 >> combofile.bin
2. Enable LZNT1 (e.g. via compact.exe /c filename)
3. Finalise NTFS image and try to extract files with go-ntfs

Here is an affected filesystem: http://ms11.ivysaur.me/pub/raw.ntfs.xz . Try to extract the interior compresstest/combofile.bin file

e.g. v0.2.0, the current v0.1 does not seem to work

The current paradigm of accessing data streams via their Attribute IDs is incorrect and results in wrong data being fetched in certain scenarios. The library relies on the inode notation (MftEntry-AttributeType-AttributeID) to uniquely identify data streams. Eg: 38-128-3

However in case of complex MFT entries where an attribute for the main MFT entry is stored in another another one, the attribute ID is no longer unique as it is likely going to be zero in the second one (while in the base entry, id 0 would always be STDINFO). If there are more than one streams, there is no way to distinguish them. This scenario is commonly seen in large fragmented files, most commonly $USNJRNL:$J.

This is not easily noticed as it only occurs if the file is very large and highly fragmented like the $MFT and $USNJRNL:$J. The current implementation partially works, but gives incorrect results that are otherwise difficult to validate without a lot of advanced testing (creation of large and fragmented files!).

The attached disk image charlie.zip demonstrates this problem via a handcrafted MFT entry that simulates the same fragmented behaviour by adding an ATTRIBUTE_LIST and storing some $DATA attributes in other MFT entries. The file /Nine.txt has the following streams:

The picture below shows the MFT layout of this file.

It is not possible to access the streams 111 and 333 via the inode notation. To access alternate data streams, we need to address them by name. In NTFS, no other unique identifier exists.

Bug demo

Below is a listing of the contents and file sizes of the streams in Nine.txt.

To demonstrate the issue, we run the compiled exe against this image on the entry Nine.txt.

C:\go-ntfs>ntfs.exe cat "C:\temp\vr\tests\images\charlie_edited.dd” Nine.txt
"111111111111111111..._<snipped>_

Hi, thanks so much for this project, it's incredibly useful.

I have an NTFS disk image where the MFT is sufficiently large and fragmented, such that all the extents are not immediately in the $DATA attribute. The $DATA attribute contains only a single extent (4 long x cluster size 4096 / MFT entry size 1024 = 16 MFT IDs) and the rest of the MFT IDs have overflown into the $ATTRIBUTE_LIST (0x20).

Go-ntfs is only able to read MFT IDs up to 16. However, the NTFS implementation in Sleuthkit handles this edge case and is able to see all the files inside this disk image. It checks the $ATTRIBUTE_LIST, finds an additional $DATA run, and treats it as a continuation of the first short $DATA run.

• https://github.com/sleuthkit/sleuthkit/blob/master/tsk/fs/ntfs.c#L43
• https://stackoverflow.com/a/30646166

In this case when reading the MFT, go-ntfs should also check if the $ATTRIBUTE_LIST contains an additional $DATA attribute, and add it to the runs in the reader, probably somewhere near https://github.com/Velocidex/go-ntfs/blob/master/parser/mft.go#L466 . There is already support in go-ntfs for parsing the $ATTRIBUTE_LIST but it just doesn't seem to be applied in this particular case.

I can confirm this issue on both v0.1.1 and master (b97c856),

I am working with a velociraptor offline collector and go-ntfs seems to break on my machine reproducible. I haven't done a deep dive into the code yet but this is my stack-trace. There seems to be an off by one error or similar thing happening.

panic: runtime error: index out of range [216] with length 216

goroutine 389 [running]:
www.velocidex.com/golang/go-ntfs/parser.(*NTFS_ATTRIBUTE).RunList(0xc003080ba0)
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/attribute.go:158 +0x313
www.velocidex.com/golang/go-ntfs/parser.joinAllVCNs(0xc00bf20360, {0xc001646100?, 0x3, 0x4})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/easy.go:393 +0x1d3
www.velocidex.com/golang/go-ntfs/parser.OpenStream(0xc00bf20360?, 0xc0015d5980?, 0xc00263e180?, 0xea1a?)
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/easy.go:356 +0x207
www.velocidex.com/golang/velociraptor/accessors/ntfs.(*MFTFileSystemAccessor).OpenWithOSPath(0xc00263c4f0, 0xc0006d6ea0?)
        /velociraptor-build/velociraptor/accessors/ntfs/mft.go:139 +0x127
www.velocidex.com/golang/velociraptor/vql/filesystem.(*ReadFileFunction).Call(0xc000ee7ae0?, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?}, 0x2149a20?)
        /velociraptor-build/velociraptor/vql/filesystem/filesystem.go:353 +0x2eb
www.velocidex.com/golang/vfilter.(*_SymbolRef).callFunction(0xc0011600c0, {0x26c1e20?, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0}, {0x26b76e8?, 0x34ff2d0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1691 +0x583
www.velocidex.com/golang/vfilter.(*_SymbolRef).Reduce(0xc0011600c0, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6ea0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1545 +0x1c6
www.velocidex.com/golang/vfilter.(*_Value).Reduce(0xc00034cc00, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6ea0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1433 +0x14f
www.velocidex.com/golang/vfilter.(*_MemberExpression).Reduce(0xc001a55e00, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1120 +0x56
www.velocidex.com/golang/vfilter.(*_MultiplicationExpression).Reduce(0xc001a55e40, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1369 +0x53
www.velocidex.com/golang/vfilter.(*_AdditionExpression).Reduce(0xc001a55e80, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1284 +0x53
www.velocidex.com/golang/vfilter.(*_ConditionOperand).Reduce(0xc001fe1410, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1322 +0x85
www.velocidex.com/golang/vfilter.(*_OrExpression).Reduce(0xc000592040, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1246 +0x56
www.velocidex.com/golang/vfilter.(*_AndExpression).Reduce(0xc000592200, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1214 +0x4a
www.velocidex.com/golang/vfilter.(*_AliasedExpression).Reduce(0x0?, {0x26c1e20?, 0xc0013e4880?}, {0x26dc670?, 0xc0006d6ea0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:679 +0x9f
www.velocidex.com/golang/vfilter.(*_SelectExpression).Transform.func2({0x26c1e20, 0xc0013e4880}, {0xc00690c330?, 0xa?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:916 +0x5b
www.velocidex.com/golang/vfilter.(*LazyRowImpl).Get(0xc0001c8e00, {0xc00690c330, 0xa})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/lazy.go:60 +0x7d
www.velocidex.com/golang/vfilter/protocols.(*AssociativeDispatcher).Associative(0xc002252f98, {0x26dc670, 0xc0006d6fc0}, {0x1ecd480?, 0xc0001c8e00}, {0x1d97a20?, 0xc00263c130?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/protocols/protocol_associative.go:52 +0x3f5
www.velocidex.com/golang/vfilter/scope.(*Scope).Associative(...)
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:276
www.velocidex.com/golang/vfilter/scope.(*Scope).Resolve(0xc0006d6fc0, {0xc00690c330, 0xa})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:551 +0x176
www.velocidex.com/golang/vfilter/protocols.(*AssociativeDispatcher).Associative(0xc002252f98, {0x26dc670, 0xc0006d6fc0}, {0x2149a20?, 0xc0006d6fc0}, {0x1d97a20?, 0xc00263c120?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/protocols/protocol_associative.go:46 +0x482
www.velocidex.com/golang/vfilter/scope.(*Scope).Associative(0xc00690c330?, {0x2149a20?, 0xc0006d6fc0?}, {0x1d97a20?, 0xc00263c120?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:276 +0x53
www.velocidex.com/golang/vfilter.(*_SymbolRef).getFunction(0xc001161f80, {0x26dc670, 0xc0006d6fc0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1505 +0x4de
www.velocidex.com/golang/vfilter.(*_SymbolRef).Reduce(0xc001161f80, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1534 +0x65
www.velocidex.com/golang/vfilter.(*_Value).Reduce(0xc00034cd00, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1433 +0x14f
www.velocidex.com/golang/vfilter.(*_MemberExpression).Reduce(0xc000592740, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1120 +0x56
www.velocidex.com/golang/vfilter.(*_MultiplicationExpression).Reduce(0xc000592780, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1369 +0x53
www.velocidex.com/golang/vfilter.(*_AdditionExpression).Reduce(0xc000592a00, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1284 +0x53
www.velocidex.com/golang/vfilter.(*_ConditionOperand).Reduce(0xc001fe15c0, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1322 +0x85
www.velocidex.com/golang/vfilter.(*_OrExpression).Reduce(0xc000a6e740, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1246 +0x56
www.velocidex.com/golang/vfilter.(*_AndExpression).Reduce(0xc000a6e800, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1214 +0x4a
www.velocidex.com/golang/vfilter.(*_CommaExpression).Reduce(0xc000a6eac0, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1177 +0x5a
www.velocidex.com/golang/vfilter.(*_Select).processSingleRow(0xc001393220, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0011a7680}, {0x2113680, 0xc0021cb180}, 0xc0020650e0)
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:585 +0x229
www.velocidex.com/golang/vfilter.(*_Select).Eval.func3()
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:548 +0xe5
created by www.velocidex.com/golang/vfilter.(*_Select).Eval
        /go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:533 +0x2ca

I did some testing with extraction of files from NTFS. I figured out, that PagedReader can be a bottleneck if configured incorrectly. A page size of 1024 is slowing down copies of large files.

I'm now using parser.NewPagedReader(r, 1024*1024, 100*1024*1024) which yielded quite good results. I've not thoroughly tested implications on memory though.

Is there any reason for 1024 being used basically everywhere (https://grep.app/search?q=NewPagedReader)?

It may be possible to get into a recursion loop by expanding ATTRIBUTE_LIST_ENTRY into an MFT_ENTRY with another ATTRIBUTE_LIST.

See CCXLabs/CCXDigger#13

Hello :),
I'm using the cloned and compiled version of go-ntfs and it works fine even with partition offset in "ls command".
I noticed that the current release doesn't work the same way.
There is an idea of when we will be able to have the new release based on the current code replacing the current one from 2020 :)

Thanks & Best regards

If the pageSize is not aligned with the content (e.g. content: "abcd", pagesize: 3) the data from the last page is not read by the PagedReader.

r, _ := NewPagedReader(bytes.NewReader([]byte("abcd")), 3, 100)
buf :=  make([]byte, 1)
c, err = r.ReadAt(buf, 3)

// returns: c=0, buf=[]byte{0x00}, err=io.EOF
// should be: c=1, buf=[]byte{0x64}, err=nil

Are you planing to write some documentation?

I would love to reuse the package, but apparently I will have to read all the code.

@scudette
Thank you for sharing the code of this library. As a newbie to MFT, could you write an example? I would greatly appreciate it if you could read the $MFT file directly without copying it, and then parse out the file name, file size, modified time of each file.

Adhoc errors like "Not found" from here https://github.com/Velocidex/go-ntfs/blob/master/parser/mft.go#L123 are hard to check against.

Instead we should use or wrap the standard io/fs errors like ErrNotFound in cases like that. https://pkg.go.dev/io/fs#pkg-variables.