Giter Club home page Giter Club logo

sveinbjornt / searchfs Goto Github PK

View Code? Open in Web Editor NEW
107.0 11.0 8.0 109 KB

Command line tool to quickly search by filename on entire APFS and HFS+ volumes using the file system catalog.

Home Page: https://sveinbjorn.org/searchfs

License: BSD 3-Clause "New" or "Revised" License

Objective-C 67.86% Roff 9.09% HTML 12.28% Makefile 1.61% Python 9.16%
volume catalog-search macos macosx apfs-volumes apfs hfs-filesystem search searchfs command-line-tool

searchfs's Introduction

License Language Build

searchfs

searchfs icon

searchfs is a macOS command line tool to quickly search by filename on entire APFS and HFS+ volumes. Searching takes place at the driver level using the file system catalog. This means the volume's directory tree can be scanned much faster than with a standard recursive filename search using find.

Search is case-insensitive by default. Matching files are printed to standard output in the order they are found in the catalog. See the man page for details.

KatSearch is a native graphical macOS application built on top of searchfs.

Download

Build & Install

  • git clone https://github.com/sveinbjornt/searchfs.git
  • cd searchfs
  • make
  • make install

Installs binary into /usr/local/bin/. Man page goes into /usr/local/share/man/man1/.

Performance

According to my benchmarks, searchfs runs about 35-50% faster than find on APFS filesystems and many times faster on HFS+.

The following are benchmark results on a 2012 Retina MacBook Pro with an Apple-supplied 512 GB SSD running an APFS file system containing about 2 million files:

searchfs

$ time searchfs "something"
0,01s user 33,15s system 32% cpu 1:23,59 total

find

$ time find / -name "*something*"
9,53s user 67,64s system 49% cpu 2:37,39 total

Although I have yet to test this properly, searchfs is probably much faster than find on hard disk drives, which have higher seek times. It is also very fast indeed on file systems with a small number of files.

History

Apple added file system catalog search to Mac OS with the introduction of the Hiearchical File System (HFS) back in 1985. HFS replaced the previous flat table structure in the old MFS file system with a catalog file using a B-tree structure. Unlike Windows' FAT file system, HFS (and later, HFS+) thus arranged the entire directory tree into one large file on the disk, with interlinked nodes that did not match the hierarchical folder structure. This meant that volumes could be searched very quickly regardless of size.

The Classic Mac OS exposed this functionality via the FSCatalogSearch() function, which iterated efficiently over the nodes, thus minimizing disk seek times. In the pre-SSD era, this gave the Mac a significant performance advantage over Windows when it came to full-volume search. For a long time, FSCatalogSearch continued to be available in Mac OS X / macOS via the Carbon APIs but it has now been deprecated and does not support APFS, Apple's new file system.

However, catalog search for both HFS+ and APFS is available in Darwin's low-level system libraries via the searchfs() function. The searchfs program makes use of this function.

TODO

  • Add support for macOS 10.15 Catalina
  • The searchfs API supports searching the catalog for files based on size, owner, group, creation, modification or access date, finder flags, deprecated old-school file and creator types, and so on. Add that.

Version History

11/05/2019 - 0.3

  • New -l flag lists all mounted volumes that support catalog search.
  • Volume to search can now be specified via device name as well as mount path.
  • Regex modifiers ^ and $ can now be used to match only at the start or end of a filenames.

26/04/2019 - 0.2

  • Fixed issue which prevented searchfs from working on older versions of macOS.
  • Now fails silently when path lookup fails for a file system object ID.
  • Now runs on macOS 10.7 or later.

14/07/2018 - 0.1

  • Initial release

BSD License

Copyright © 2017-2020 Sveinbjorn Thordarson <[email protected]>

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

searchfs's People

Contributors

gingerbeardman avatar sveinbjornt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

searchfs's Issues

Please support DESTDIR in Makefile

Please support the standard DESTDIR variable in your Makefile.

If I run

sudo make install DESTDIR=/tmp/root

then I would expect the files to get installed in

/tmp/root/usr/local/bin/searchfs
/tmp/root/usr/local/share/man/man1/searchfs.1

Note that your existing Makefile variable DEST_DIR, which serves an unrelated purpose, then becomes confusing. You may wish to rename that variable.

Please support alternate installation prefixes

The Makefile is hardcoded to install into /usr/local. Please provide a Makefile variable (perhaps PREFIX) that a user could specify at the command line to install to a different location.

Compiler warnings on Mojave

On macOS 10.14.4 with Xcode 10.2.1, I git pulled to update my clone, and did make. I saw the following warnings that I don't remember seeing before:

version-min=10.7 main.m
main.m:419:24: warning: 'fsgetpath' is only available on macOS 10.13 or newer
      [-Wunguarded-availability-new]
        ssize_t size = fsgetpath(buf, buflen, fsid, obj_id);
                       ^~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/sys/fsgetpath.h:46:9: note:
      'fsgetpath' has been marked as being introduced in macOS 10.13 here, but
      the deployment target is macOS 10.7.0
ssize_t fsgetpath(char *, size_t, fsid_t *, uint64_t) __OSX_AVAILABLE(10...
        ^
main.m:419:24: note: enclose 'fsgetpath' in an @available check to silence this
      warning
        ssize_t size = fsgetpath(buf, buflen, fsid, obj_id);
                       ^~~~~~~~~
1 warning generated.
strip -x searchfs

The build succeeded despite the warnings.

Support other file systems?

This is not really a "issue", and I realise that it's not even a feature request, but I still wanted to discuss the possibilities.

Is it possible to add support for other file systems?

Are there any other file systems which have Catalog of files? I guess NTFS does. (as Everything uses it for search)

But do you know if ext3/4 or any other linux file systems have this?

Only scan a specific directory?

one of my common workflow scripts searches for all .app files in /Applications and then processes them.

Currently I do this:

mdfind -onlyin /Applications "kMDItemFSName == *.app"

real 0m1.009s
user 0m0.023s
sys 0m0.027s

an slower alternative would be:

find "/Applications" -iname "*.app" -exec echo {} \;

real 1m20.565s
user 0m6.577s
sys 0m29.990s

however searchfs is even slower than this as i have to scan the whole volume and filter the results:

searchfs ".app" | grep -E "^/Applications.+app$"

real 2m7.277s
user 0m0.108s
sys 0m46.177s

So it would be useful if searchfs could operate on the cwd or a directory passed as an argv?

"Unable to get path for object ID"

(snip)
/Applications/iTunes.app/Contents/MacOS/iTunesHelper.app
Unable to get path for object ID: 3430632Unable to get path for object ID: 3460180/Applications/Video/Adapter.app
/Applications/Video/MediaInfo.app
(snip)

This happens multiple times in the results, for different object IDs.

Let me know if/how I can provide more logging or diagnostics.

Unable to search /System/Volumes/Data ?

Catalina 10.15.3

searchfs seems unable to search the r/w mountpoint where all user data is stored. Tried v0.3 from https://sveinbjorn.org/files/software/searchfs.zip as well as compiling from master - same result:

$ ./searchfs -l
/dev/disk1s5 (apfs): /
/dev/disk1s1 (apfs): /System/Volumes/Data
/dev/disk1s4 (apfs): /private/var/vm
/dev/disk2s2 (hfs ): /Volumes/TimeMachine

$ ./searchfs -v / USBDriverKit.framework$
/System/DriverKit/System/Library/Frameworks/USBDriverKit.framework
/System/Library/Frameworks/USBDriverKit.framework

$ ./searchfs -v /System/Volumes/Data foo
Not a volume mount point: /System/Volumes/Data

$ ./searchfs -v /dev/disk1s1 foo
Not a volume mount point: /System/Volumes/Data

Not sure if it matters, but I also tried granting the binary Full Disk Access, which did not make any difference. Am I doing something wrong?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.