Currently -i switch doesn't work on Windows.

Error when replacement string is longer than matched string: failed to write whole buffer
Error otherwise: failed to persist temporary file: Access is denied. (os error 5)

Writing the stream to a tempfile first, then rename it to the input file, would reduce the risk of corruption.

Ref: https://github.com/TankerHQ/ruplacer/tree/bf39991b#subvert-mode

While I use sd day-to-day for common replacement tasks, I find myself falling back to ruplacer whenever I need its --subvert functionality.

It would be amazing if sd supported this (via a new option or flag) so I could fully switch to sd for all my replacement needs!

Tools like ripgrep and fd have a "smart" case, where regex matches are case-insensitive unless the pattern contains uppercase characters.

Arguably, this behavior should be the default.

However, then we would also need a way to force case sensitivity/insensitivity as necessary for the more rare cases. If you have any thoughts or ideas, I would welcome them below.

-b flag, which creates a inputfile.bak backup before replacing.

Issue

In benchmark, I've noticed that you weren't using -p option for sd to print everything to stdout. Nevertheless, sed-commands print everything to stdout.

Also, once sd "(\w+)" "$1$1" dump.json >/dev/null is performed, every word in file is deleted. This happens because $1 is replaced by shell with (empty string) and sd performs 'in-place' (or 'inline') replacement.

Experiment

Here is my run for a simple thing

➜  ~/tmp echo '{"hello": "world"}' > test.txt

➜  ~/tmp cat test.txt
{"hello": "world"}

➜  ~/tmp hyperfine 'sd "(\w+)" "$1$1" test.txt'
Benchmark #1: sd "(\w+)" "$1$1" test.txt
  Time (mean ± σ):       6.7 ms ±   1.1 ms    [User: 3.2 ms, System: 1.8 ms]
  Range (min … max):     5.6 ms …  12.2 ms    245 runs

➜  ~/tmp cat test.txt
{"": ""}

Please pay attention to the second cat output.
This is the reason why almost every run of sd is so fast (except the first one) — it doesn't do anything but just reading the file.

The following command should be used to compete with sed:

hyperfine 'sd -p "(\w+)" "\$1\$1" test.txt > /dev/null'

Please note the escaped groups \$1 and the preview option -p

Experiment Results

Here are my results for a 120 MB file

➜  ~/tmp l dump.json
.rw-r--r--@ 120M sergey  2 Aug 22:20 dump.json

➜  ~/tmp hyperfine \
'sed -E "s:(\w+):\1\1:g" dump.json >/dev/null' \
"sed 's:\(\w\+\):\1\1:g' dump.json >/dev/null" \
'sd -p "(\w+)" "\$1\$1" dump.json >/dev/null'
Benchmark #1: sed -E "s:(\w+):\1\1:g" dump.json >/dev/null
  Time (mean ± σ):      5.724 s ±  0.056 s    [User: 5.489 s, System: 0.146 s]
  Range (min … max):    5.656 s …  5.849 s    10 runs

Benchmark #2: sed 's:\(\w\+\):\1\1:g' dump.json >/dev/null
  Time (mean ± σ):      2.614 s ±  0.034 s    [User: 2.493 s, System: 0.084 s]
  Range (min … max):    2.569 s …  2.676 s    10 runs

Benchmark #3: sd -p "(\w+)" "\$1\$1" dump.json >/dev/null
  Time (mean ± σ):     12.590 s ±  0.216 s    [User: 12.087 s, System: 0.303 s]
  Range (min … max):   12.403 s … 13.150 s    10 runs

Summary
  'sed 's:\(\w\+\):\1\1:g' dump.json >/dev/null' ran
    2.19 ± 0.04 times faster than 'sed -E "s:(\w+):\1\1:g" dump.json >/dev/null'
    4.82 ± 0.10 times faster than 'sd -p "(\w+)" "\$1\$1" dump.json >/dev/null'

➜  ~/tmp l dump.json
.rw-r--r--@ 120M sergey  2 Aug 22:20 dump.json

Thoughts

Even if we fixed the benchmark, I do think that we are capped with pipe throughput.

UPD: Ok, apparently pipe is not a problem.

Platform

MBP 2015, 2.7 GHz Intel Core i5

$ echo 123.45 | sd '(\d+)\.\d+' '$1'

Expected output: 123
Actual output:

Traceback (most recent call last):
ruby: No such file or directory -- script/server (LoadError)

fasd is fantastic, but one of its default aliases conflicts with this tool's name. I don't necessarily expect you to rename this project, but noting this in documentation would be nice!

D:\openvpn\config>sd a b *.txt
The filename, directory name, or volume label syntax is incorrect. (os error 123)

I'm using the Windows version 0.65 and this is the command. It's working with many other tools but not with sd.

set pattern="(.*)Call ((List|Text)\d+_Key(Down|Up))\((.*), (vbKeyReturn|13), (.*)\)"
set replace="$1Call $2($5, GetFM20ReturnKey(), $6)"
sd.exe %pattern% %replace% "Test.txt"

Test.txt contains this Text

Call Text3_KeyDown(Index, vbKeyReturn, 0)
    Call Text3_KeyDown(Index, vbKeyReturn, 0)
        Call Text3_KeyDown(Index, vbKeyReturn, 0)
Call Text3_KeyDown(Index, vbKeyReturn, 0)

It gets replaced with

 Text3_KeyDown(Index, GetFM20ReturnKey(), vbKeyReturn)
 Text3_KeyDown(Index, GetFM20ReturnKey(), vbKeyReturn)
 Text3_KeyDown(Index, GetFM20ReturnKey(), vbKeyReturn)
 Text3_KeyDown(Index, GetFM20ReturnKey(), vbKeyReturn)

I'm Missing the first capture group and the Call word.
Can you tell me if I'm doing something wrong or if it's a bug.
As other tools are working with the same pattern and replace strings, I think it's a bug in sd.

I am trying to do sd '^// ' ' /// ' myfile.rs to turn regular comments into doccomments, and its not replacing any of the beginning of line //.

I think its because ^ has ambiguity with the not operator?

mkdir play
cd play
touch foo.txt
touch bar.sh
chmod a+x bar.sh
git init
git add *
git commit -a -m "example"
sd -i a b foo.txt bar.sh
git diff

This is the output

diff --git a/bar.sh b/bar.sh
old mode 100755
new mode 100644

replacements.txt:

value replacement
test TEST

sd --replacements replacements.txt file.txt

Sorry - this was a misunderstanding...

In ZMV there is the -n option flag that shows you what the output would be, without actually doing anything. It is very handy for fine-tuning the regexp before actually commiting the replacement to disk.

However, whenever I want to use it I can't seem to remember that it's -n, so maybe it's not the best choice. The expanded --dry-run would make more sense.

I assume the majority of users who come across sd are likely also using other popular Rust-based CLI tools like fd and rg:

• sharkdp/[email protected]#L70-L75
• BurntSushi/[email protected]#L1175-L1178

For consistency with the above, it would be neat if sd's options were consistent with these to make context switching a bit easier.

A manpage would be awesome to have. sd -h is good for command line switches, but a page with examples as given in the quick guide is still missing when offline.

In #7, it was argued that sd should support smart case (which is great!) and should enable it by default, because it is also used in ripgrep and fd.

I personally really like smart-case and did therefore choose it as a default in fd. However, I would like to present an argument for why is should (potentially) not be the default in sd.

Suppose I want to rename a library/module in a given codebase. The name of the module will likely appear in many different variants:

• awesome_library: snake_case for things like namespaces, imports, build scripts, etc.
• AwesomeLibrary: PascalCase for things like class/struct names, names of types, etc.
• AWESOME_LIBRARY: ALL CAPS for things like include guards, constants, etc.
• (awesomeLibrary: camelCase for normal variables, unit test names, etc.)

If I am not careful, and use

sd -i awesome_library new_name …

first, I will also replace all AWESOME_LIBRARY strings with the new lower case name. In this particular example, I am probably better of with case-sensitive as a default.

In other words: I think I hardly ever want to replace both "old_name" and "OLD_NAME" by "new_name". In most scenarios, I rather want to change "old_name" to "new_name" and "OLD_NAME" to "NEW_NAME".

I am trying to use sd to remove some values from a NUL-separated file, but it does nothing to the file.

# zsh
sd --flags m --string-mode $'\0'"$i" '' "$attic"

# cat -v $attic
^@this is 1.
this is 2.
this is 3.
^@hi
^@blue boy

# cat -v <<<"$i"
this is 1.
this is 2.
this is 3.

Cool project! This is sort of an opinionated issue, but bothers me enough when trying out sd that I figured I'd file it just so it's at least filed for the future.

To put it quite bluntly, the -i flag is redundant in the interface you have. You could easily go from this:

cat file.txt | sd -s '((([])))' ''

To this:

sd -s '((([])))' '' file.txt

As it stands, you need -i file.txt, which I think is an unnecessary flag as it's quite common to see default positional arguments done this way (for example, cat works exactly as the above suggestion). In the few minutes of me trying this tool out, I forgot the -i at least 3 times.

I think potentially removing the argument in favour of the positional makes for a little bit of a nicer experience. If you're committed to compatibility, you can probably keep -i around as an undocumented flag to avoid breaking existing workflows.

Thoughts? I'm happy to try and file a PR for this when I get a few minutes, if that helps.

I'm having some trouble installing sd on macOS High Sierra (version 10.13.6)

I'm using rustc 1.25.0 (84203cac6 2018-03-25) and I got the error message below when issuing cargo install sd --verbose:

Caused by:
Could not compile sd.

Caused by:
process didn't exit successfully: rustc --crate-name sd src/main.rs --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C metadata=eeddc918bbb61feb -C extra-filename=-eeddc918bbb61feb --out-dir /var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps -L dependency=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps --extern rayon=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/librayon-50e0ecc9ad3d9d0e.rlib --extern regex_syntax=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/libregex_syntax-4ecf4d4c9b0429f8.rlib --extern regex=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/libregex-0ea6cc294b773273.rlib --extern structopt=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/libstructopt-a1bdebfe50528146.rlib --extern unescape=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/libunescape-830b9c63499749ae.rlib --extern atomicwrites=/var/folders/p2/kcd723157t98hxc8sd1rj3k40000gn/T/cargo-install.Nlorfacc8Jcq/release/deps/libatomicwrites-e8bbeb4fdf1f5771.rlib --cap-lints allow (exit code: 101)

Awesome project, anyway. 👍

I think it would be great if sd would silently ignore directory arguments. This would allow easier integration with shell globs and external tools like find/fd.

Suppose I have the following structure:

▶ mkdir dir; echo "foo" > file; echo "foo foo" > dir/other_file
▶ tree
.
├── dir
│   └── other_file
└── file

Now I want to replace foo by bar, recursively. It'd be great if I could just call

▶ sd foo bar **/*

However, this currently fails with:

Error: Is a directory (os error 21)

Interestingly, everything works with --in-place mode:

▶ sd -i foo bar **/*

Admittedly, --in-place is almost always what I want if I supply any arguments, but I wonder if the normal (not in-place) mode should behave the same.

Thinking of this, would it be an option to enable --in-place by default (if sd is not reading from STDIN)? Or would you consider this to be too dangerous?

Awesome tool, by the way - Thank you!

If the replacement string has a different length than the search pattern, sd either fails with an error "failed to write whole buffer" (when the replacement string is longer) or even leaves behind corrupt files (when the replacement string is shorter).

How to reproduce:

▶ echo "foo" > dummy

▶ sd "foo" "fo" dummy

▶ hexdump -C dummy
00000000  66 6f 0a 00                                       |fo..|
00000004

Notice the additional zero byte (00) at then end.

Dear Gregory,
Could you be so kind to generate .exe for the rest of us who are mere Windows users w/o compiler?

See here for a crate that can help: https://crates.io/crates/ignore

EDIT: whoops, for some reason I thought sd was taking a while because it was applying things recursively to my directory tree... which makes no sense in retrospect.

I recently wound up wanting the s flag, which makes . match \n. Is the s flag something that sd could support?

It might be nice to support all of the regex crate's flags, unless particular flag because it doesn't work in the context of sd.

PS Thank you for sd! I like it a lot ^_^

How would you contrast sd with ruplacer?

Let's say I want to rename a variable/function/class/etc. I want to change the name from foo to bar. But obviously, I don't want to replace any occurrences of the word fool by barl (can't think of a better example right now). I can solve this by adding \b word boundary markers around the pattern:

sd '\bfoo\b' bar …

However, since this is such a common scenario and such an easy thing to get wrong, I think it would be great if sd would feature a command-line option to do this automatically (maybe -w/--word or -b/--boundary). I think this would look much better and would be easier to type:

sd -b foo bar …

Hi @chmln ,

I'm trying to create a package for sd in the nixpkgs repo and I'm a bit stuck. Could you or anyone here help me out with this pull request here NixOS/nixpkgs#54761

First of all, thanks for the great tool! This is slowly becoming my default go-to instead of sed.

I used sd today for the first time in a while, so I ran sd --help to refresh on the (admittedly intuitive) syntax.

There was one line in the help dialog of note:

    -i, --in-place
            (Deprecated). Modify the files in-place. Otherwise, transformations will be emitted to STDOUT by default.

If I'm not mistaken, this line is out of date. By default, sd does modify files in-place.

I would personally vote for this tool to not modify in-place to more closely match the behavior of sed, awk, and other similar tools. I would argue that the most common use case for a tool like this is to run a few test cases through to ensure that you're doing what you think you're doing, iterate a few times, and then modify the file in place.

However, I respect that you decided to move to the default of modifying in-place. I would ask, however, that we change the help text to make it very obvious that, by default, this tool modifies files in place. It can be very surprising to someone coming from sed, awk, or a ton of other tools. I was lucky that I had recently committed my file to git, otherwise I could have lost a lot of changes--my replacement did not behave the way I thought it would!

I'm willing to submit a pull request if you'd be willing to merge such a change :)

Thanks again for the great tool!

Version 0.6.1 on Fedora Rawhide, rust 1.35.0:

BUILDSTDERR:    Compiling sd v0.6.1 (/builddir/build/BUILD/sd-0.6.1)
BUILDSTDERR:      Running `/usr/bin/rustc --edition=2018 --crate-name sd src/main.rs --color never --emit=dep-info,link -C opt-level=3 --test -C metadata=db7c97756d2f4430 -C extra-filename=-db7c97756d2f4430 --out-dir /builddir/build/BUILD/sd-0.6.1/target/release/deps -L dependency=/builddir/build/BUILD/sd-0.6.1/target/release/deps --extern memmap=/builddir/build/BUILD/sd-0.6.1/target/release/deps/libmemmap-b3f2069516387a5d.rlib --extern rayon=/builddir/build/BUILD/sd-0.6.1/target/release/deps/librayon-a99e4203c6332a21.rlib --extern regex=/builddir/build/BUILD/sd-0.6.1/target/release/deps/libregex-98ffbc1cf6224c5b.rlib --extern structopt=/builddir/build/BUILD/sd-0.6.1/target/release/deps/libstructopt-01b05acb492cc451.rlib --extern tempfile=/builddir/build/BUILD/sd-0.6.1/target/release/deps/libtempfile-b901b63516f8d697.rlib --extern unescape=/builddir/build/BUILD/sd-0.6.1/target/release/deps/libunescape-fd1a3ef56a6eb871.rlib -Copt-level=3 -Cdebuginfo=2 -Clink-arg=-Wl,-z,relro,-z,now -Ccodegen-units=1`
BUILDSTDERR: error[E0308]: mismatched types
BUILDSTDERR:    --> src/input.rs:181:58
BUILDSTDERR:     |
BUILDSTDERR: 181 |         assert_eq!(std::str::from_utf8(&replacer.replace(src)), Ok(target));
BUILDSTDERR:     |                                                          ^^^ expected slice, found str
BUILDSTDERR:     |
BUILDSTDERR:     = note: expected type `&[u8]`
BUILDSTDERR:                found type `&'static str`
BUILDSTDERR: error: aborting due to previous error
BUILDSTDERR: For more information about this error, try `rustc --explain E0308`.
BUILDSTDERR: error: Could not compile `sd`.

It may be useful to have the following regex flags:

• m - multi-line matching
• i - case-insensitivity
• c? - case-sensitivity

Rust version: rustc 1.32.0-nightly (13dab66a6 2018-11-05)

Error message:

error: `unescape` import is ambiguous
 --> /home/wenxuan/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/sd-0.2.0/src/utils.rs:2:9
  |
1 | / pub(crate) fn unescape(s: &str) -> Option<String> {
2 | |     use unescape::unescape;
  | |         ^^^^^^^^----------
  | |         |
  | |         shadowed by block-scoped `unescape`
3 | |     unescape(s)
4 | | }
  | |_- may refer to `self::unescape` in the future
  |
  = help: write `self::unescape` explicitly instead
  = note: in the future, `#![feature(uniform_paths)]` may become the default

The idea is to be able to search and replace some patterns, excepted those with another pattern in front or after the one we look for. I found some solutions with (?!) but this seems currently unsupported.

Which regex flavour does sd use?

I was wondering if it it would be possible to create a MUSL build of sd along with the GNU build. I ask only because I'd like to use sd on a system (supercomputing cluster) that hasn't quite kept up with the latest Linux for stability reasons:

(1332) $ sd --version
sd: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by sd)

This happens a lot with all the nice rust apps I use now (ripgrep) and the usual fix I have is to grab the MUSL build instead of the GNU build. (Or I guess, I bug..uh..ask them to add MUSL a la starship...and here! 😄 )

When trying some search & replace where I wanted to match something only if the line was starting with it, I wanted to use the usual ^ to flag the beginning of the line. The result was a fail of the match wail getting rid of it resulted in fine matching.
I tried to escaping it, guessing that as $ is used for variables, it probably needs some escaping to flag the end of a line the same way, but this didn't work as well.

I guess that ^and $ should somehow be usable to flag beginning and end of the line.

My CLI Swiss Army knife includes ripgrep, fd and fzf. And now sd. The first three can be installed on Windows with Chocolatey, which makes it easy to tell my friends and colleagues to check them out. It would be great to spread the word about sd too :)

apt install sd doesn't work .

I have found that cargo is need before install sd.
Do you have a plan publish sd to ubuntu apt ?

It would be nice to able to do a search and replace on an entire tree of files.

Command would be

sudo xbps-install sd

$ echo hello | sd j ''

$ echo hello | sed 's/j//g'
hello

This is new behavior that was introduced after upgrading this package via homebrew today. I'm not sure which version I was running before, but probably the immediately previous release or two.

The behavior used to match sed in this case. I'm not sure what's intended, but I wanted to file an issue incase it was helpful.

VSCode July 2019 (version 1.37) release note says:

Thanks to some upstream work in ripgrep, you can now use these features without enabling a special setting. ripgrep will fall back to the PCRE2 engine automatically if the regex uses a feature that isn't supported by the Rust regex engine

I have been using ripgrep & sd extensively for last few days. As per my understanding, sd can't do somethings which sed can do for example Convert Case. We can overcome a lot of those stuff by using ripgrep and setting --engine auto . along with some other tweaks, for example:

extended replacement string syntax that you can enable by including PCRE2_SUBSTITUTE_EXTENDED with the matching options when calling pcre2_substitute().

And addressing some issues.

So, can we use modified version of ripgrep to fallback to pcre2.

After running sd on a file

sd "ProjectPaths" "Settings" -i $somefile

This is an example diff:

-                    "INTERNALERROR: invalid html: {:?}\nERROR:{:?}",
+                    "INTERNALERROR: invalid html: {:?}
+ERROR:{:?}",

It looks like sd is interpreting the \n and using it when re-writing the file. I'm worried that other character escapes are also being treated in this way. I recommend doing extensive testing using a suite similar to the toml inputs

https://github.com/alexcrichton/toml-rs/blob/master/test-suite/tests/valid/string-escapes.json

I do roundtrip tests like this in stfu8, for example:
https://github.com/vitiral/stfu8/blob/master/tests/sanity.rs#L84

I also recommend running sd against unicode examples and asserting they don't change. You can see them in the .txt files here:
https://github.com/vitiral/stfu8/tree/master/tests

It would be nice, if you add this program to homebrew repository, so it could be accessible via brew install sd command.
Here is Formula-Cookbook how to do this (not too complicated)

printf "# foo: bar\n" > test1
printf "# foo: bar" > test2

sd '# foo: \w+\n?' '' test1 test2
memory map must have a non-zero length
memory map must have a non-zero length

cat test1 | sd '# foo: \w+\n?' ''
cat test2 | sd '# foo: \w+\n?' ''

As seen in the above example, piping the contents of test1 and test2 to the sd command works as expected, whereas passing the files themselves results in an error.

I would love a better way to preview changes, something more than just not using the -i flag.

Use case: sd 'find' 'replace' $(fd -e rs)

This just dumps all of my files un-colored to the terminal which isn't helpful.

I propose there be a -p flag for preview, which only shows lines that match along with file and some short amount of context. Like what grep -C3 prints out. For each of the lines that match, show the matched part in red, and the replacement right next to it in green, inline. Or maybe use strikethrough and bold, although those are less widely supported.

Bonus points if the styles are configurable somehow.
Even more bonus points for coloring the output like bat