It would be nice with an option to automatically stage changed lines (only) to git.

Thanks for a great tool!

Setup

echo "aaabb" > file

Test

fastmod '(a)*b' '${1}c' file

Expect

aaacb

Actual

a:1
- aaabb
+ acb

Accept change (y = yes [default], n = no, e = edit, A = yes to all, E = yes+edit, q = quit)?

I am wondering if it would be possible publish fastmod to Docker Hub in some official capacity, so that it could be used verbatim in the Sourcegraph batch changes spec?

This whole batch changes codemodding framework is container-based, similar to how Google Cloud Build configs work:

https://docs.sourcegraph.com/batch_changes/quickstart#write-a-batch-spec

Is this something Facebook ever does for other open source tools? I realize this can be done unofficially fairly easily, but an official image would be nice.

For example, https://github.com/comby-tools/comby/ is published to https://hub.docker.com/r/comby/comby (well, no wonder, since the main author is at Sourcegraph :) )

Add "find and replace tool" into repository description. Otherwise there is absolutely no way for search algorithms to discover the tool. And add some topics there like https://github.com/topics/find-and-replace https://github.com/topics/find and https://github.com/topics/replace

terminal.rs explicitly uses terminfo, which isn't going to work on Windows conhost terminals.

Expected:

$ fastmod --version
fastmod 0.1.0

Actual:

$ fastmod --version
fastmod

A missing functionality (if I'm not mistaken) is the ability to add a line.
Good use case would be that I want to add a 'feature'. For most of the code, a new line needs to be added similar to an existing feature.

One way to solve this would be to allow adding (\r)\n characters like

fastmod -m '(^[\s]*)(feature4\n)' '${1}${2}${1}feature5\n'

But \n is just printed raw as \+n

Hey folks,

The DX is a bit confusing when there are no matches, as the exit code is indeed 0 which is great, but the output for an example command:
fastmod -- '--palette-(dark)-([\w-]+)' '--theme-${1}-palette-${2}' -g '!base.css' -g '*.{tsx,ts,js,jsx,css,pcss}' ./src

would be:

Warning: -g: No such file or directory (os error 2)
Warning: !base.css: No such file or directory (os error 2)
Warning: -g: No such file or directory (os error 2)
Warning: *.{tsx,ts,js,jsx,css,pcss}: No such file or directory (os error 2)

which took me quite some time to figure out that indeed it run smoothly, the selectors just did not match anything.

Am I doing something wrong myself maybe?

Edit:

I get this only on times that I need to prepend with -- to avoid throwing with --palette for example

when I was trying to remove a matched set of configurations, I realized that fastmod default replacing the config with an empty line instead:

fastmod 'some-matching-config.*\n.*\n.*\n' ''

It would be nice to have a --delete flag that, instead of replacing existing matches with an empty line, it would just delete the matching lines.

I can't stress how awesome this project is for a Windows refugee who was addicted to Far Manager and its macros for far too long. :D

Feature request

First of all, thank you very much for opensourcing fastmod! I find it quite joyful to use!

At the moment, it does not have support for positive and negative lookaheads and lookbehinds.

These can be very much required with complex searches.

Security

Note that fastmod is usually run over trusted code, not over untrusted user input, thus an opt-in --pcre2 --fancy flag should not pose any unacceptable security risk.

Prior art

Note that ripgrep (which does search, but not replace) has optional support for switching its regex engine to use PCRE2.

Among other things, this makes it possible to use look-around and backreferences in your patterns, which are not supported in ripgrep's default regex engine. PCRE2 support can be enabled with -P/--pcre2 (use PCRE2 always) or --auto-hybrid-regex (use PCRE2 only if needed). An alternative syntax is provided via the --engine (default|pcre2|auto-hybrid) option.

Pivot from rust-pcre2 to fancy-regex

See #49 (comment). grep::pcre2 is unlikely to expose pcre2_substitute any time soon, due to obvious maintenance overload.

The fancy-regex crate is the go-to escape from rust's regex crate where-ever lookarounds are uncircumventable.

Implementation

https://docs.rs/pcre2/ should be usable as a base for this feature.

This crate is recommended by and instead of https://github.com/BurntSushi/ripgrep/blob/master/crates/pcre2/README.md

However, since fastmod already uses grep (rg as a library), it might just as well enable the pcre2 cargo dependency flag and use grep::pcre2 just as ripgrep does: https://github.com/BurntSushi/ripgrep/blob/327d74f1616e135a6eb09a0c3016f8f45cfc0cfc/crates/core/search.rs#L199

Enum-dispatched regex matcher and replacer based on regex and fancy-regex.

Hi folks,

Could we please get a new tag/release? v0.4.0...master

Thanks.

At times, a line contains multiple matches.

Fastmod thus presents the line multiple times, once for each match and replacement within the line.

It can be difficult to see where exactly something is being replaced, especially when replacing matches in very long lines. (think minification test fixtures, but much shorter lines can also benefit from word highlighting)

A rather extreme example (pixellated, but the point should become obvious):

The individual match and replacement should be highlighted within the red and green lines.

The following comes from my comment on #3 but never got turned into an issue, so here it is in an issue.

First of all, it's been a while since this PR came out and we've since made at least 1 change to the advancement logic, so the conflicts are real and it definitely can't be merged as-is. (Sorry for the delay! It took a while to flush out other issues and then ruminate on them.)

Second, I want to fix all our issues with advancement at once rather than doing one case at a time. We need an advancement policy for each of the actions the user can take:

• accept replacement (y)
• reject replacement and continue (n)
• open editor (e)
• accept all replacements (A)
• apply replacement and open editor (E)

I'll just talk through each of these cases below, keeping fastmod's philosophy in mind:

fastmod's major philosophical difference from codemod is that it is focused on improving the use case "I want to use interactive mode to make sure my regex is correct, and then I want to apply the regex everywhere"

Accept replacement (y)

Here, our current policy is to advance 0 characters if the replacement has length 0 and 1 character otherwise. We've already rejected the policy "advance 1 character always" (see 501a8b9 and 95ae97a). Advancing 1 character has the advantage that it allows you to do further editing on replacement text, but given fastmod's philosophy of driving toward automated application of your regex everywhere, this advantage has minimal value.

Proposed new policy: advance to the next character after the end of the replacement text.

Reject replacement (n)

We have two policies to choose from when a replacement is rejected: 1) We can advance 1 character and try again (existing behavior), or 2) we can advance to the end of the regex match and try again (@steven807's proposal in this PR). I think the answer to the tradeoff has to come from fastmod's philosophy.

If you say n at fastmod's interactive prompt without quitting, you're already outside our core use case. We're left trying to guess if your intent is "hmm, my regex looks wrong, but I want to keep searching to make sure I continue to see nonsense" (which would suggest policy 2) or "I do want to do this replacement, but the start position is wrong, so let's manually move over 1 character and try again" (which would suggest policy 1). Given our philosophy, I think @steven807 is right that we should move to policy 2 in the rejection case.

Proposed new policy: advance to the next character after the end of the match.

Open editor (e)

Our current policy is to advance 1 character and keep going.

The fundamental problem with the open editor option is that the user could arbitrarily change the document if they wanted to, and since this is outside our core use case, we're never going to have a great picture of their change. I propose that we should assume that the user is "working with us" by only editing within the matched region, may or may not have actually applied any changes, and wants to continue to the next match when they start interacting with fastmod again. Given these assumptions, I think our existing policy makes sense.

Proposed new policy: no change.

Apply replacement and open editor (E)

Our current policy is to advance 1 character and keep going. Since interactive editing is not our core use case, I think we should not worry about the difference between E and e.

Proposed new policy: no change.

Accept all replacements (A)

Our current policy is to act as though the user pressed y repeatedly. There is an infinite loop bug with A on master for both codemod and fastmod:

$ echo 'foo foo' > /tmp/test.txt
$ cd /tmp
$ codemod -m foo barfoo --extension txt
press A to accept all interactively

The problem there is that the replacement matches the search pattern, so we continually replace our own replacement (and grow it).

However, given the proposed policy changes for y, I think we can keep the policy for A as-is.

Proposed new policy: no direct change (act as though y was pressed at every subsequent prompt), but inherit the changes in the y policy.

Next steps

The next step is to implement the new policy. @steven807, if you want to do that, that's great, but I totally understand if the moment has passed.

Originally posted by @swolchok in #3 (comment)

I am working with some hack/php code and it has a lot of $ symbols. I have not found a way to include $ in the replacement code.

The below command

fastmod 'function unfurl_(\w+)\(.*message.*\)' 'function unfurl_${1}(team_t \$team, \$item, UnfurlContext $context)' file.php

Produces this

- function unfurl_unknown(team_t $team, $item, $message, bool $do_link_unfurling, bool $do_media_unfurling): legacy_ret_t {
+ function unfurl_unknown(team_t \, \, UnfurlContext ): legacy_ret_t {

First of all – this tool is sooo great! Thanks for putting this together. It replaces like 10 other tools I've cobbled together in the past.

I was wondering if it'd be possible to support going backwards? Like an undo command. Sometimes you say Y to the wrong change and it'd be really cool to be able to go back and change it.

The solution to this is probably to support -g/--glob like ripgrep, since we use the ignore crate to implement --extensions anyway and we could just implement -g with passthrough to ignore.

I want too see more context in the diff. It would be nice to be able to specify N lines before/after, like with grep

It seems like there's no way to supply a fixed string as the substitute string. Can this be added? When a fixed string is used as the regex, it's not uncommon that you would want to use a fixed substitute string as well. Thanks!

Problem

This came up when I was trying to do a version bump this way:

fastmod --hidden --ignore-case '(poetry.*)1\.2\.2' '${1}1.3.2'

I set the --hidden flag in order to also match asdf's .tool-versions file (needs to be included in the version bump).

However, I found fastmod matching files in the .git/ folder - which is something I definitely do not want to mess with (it was matching some of the previous commits where I performed the version bump to 1.2.2):

./.git/.graphite_cache_persist:128
        }
      ],
      [
        "poetry",
        {
          "validationResult": "VALID",
          "parentBranchName": "main",
          "parentBranchRevision": "3deb75c540a0daeb29986e3ef3cc36e07b8f07b0",
          "branchRevision": "fce724219a729bfb03b8bdd93035872c56f26dfd",
          "children": [],
          "prInfo": {
            "title": "upgrade poetry to 1.2.2",
-           "body": "Run this to catch places where our current poetry version is mentioned after the string \"poetry\":\n\n```\nfastmod --hidden --ignore-case '(poetry.*)1\\.2\\.1' '${1}1.2.2'\n```\n",
+           "body": "Run this to catch places where our current poetry version is mentioned after the string \"poetry\":\n\n```\nfastmod --hidden --ignore-case '(poetry.*)1\\.2\\.1' '${1}1.3.2'\n```\n",
            "number": 280,
            "url": "https://app.graphite.dev/github/pr/exponential-hq/expo/280",
            "base": "main",
            "state": "OPEN",
            "reviewDecision": "REVIEW_REQUIRED",
            "draft": false
          }
        }
      ]
    ]
  }
Accept change (y = yes [default], n = no, e = edit, A = yes to all, E = yes+edit, q = quit)?

(In this case it was a graphite file, rather than a vanilla git file, but I don't think it matters.)

I suspect that I could somehow use the --glob flag to exclude the directories I wanted, but it's not obvious to me how to do it in an elegant and reliable way.

Potential solution 1

Wdyt about having a flag to explicitly exclude some directories?

Similar to how comby has --exclude and --exclude-dirs: https://comby.dev/docs/cheat-sheet

Potential solution 2

Or have a mode where only files tracked by git would be modified by fastmod? I presume that .git is implicitly ignored by git when doing source control, so that could be also implicitly assumed as well.

I know that Facebook uses Eden now, but maybe you can consider being aware of git too, as nod to the open source community.

Right now fastmod ignore hidden files unless we specifically list them in the file list.

For example running fastmod 'target' 'targets' on this very repo will skip out on https://github.com/facebookincubator/fastmod/blob/master/.gitignore file.

Please provide an option to match hidden file.

The documentation states:

fastmod is supported on macOS and Linux.

Is there a reason for this support policy? I tested out fastmod on windows and it seems to work fine. I also skimmed the code base for any unixisms and the only one I could find was assuming $EDITOR was set/defaulting to vim.

This link in the readme is dead: https://docs.rs/regex/regex/index.html#syntax

This link works: https://docs.rs/regex/#syntax

When compiling, I get the following error

error: edition 2021 is unstable and only available with -Z unstable-options.

error: could not compile `crossterm`

Caused by:
  process didn't exit successfully: `rustc --crate-name crossterm --edition=2021 /home/amxx/.cargo/registry/src/github.com-1ecc6299db9ec823/crossterm-0.24.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="default"' -C metadata=e0e88cdfb7629f77 -C extra-filename=-e0e88cdfb7629f77 --out-dir /tmp/fastmod/target/release/deps -L dependency=/tmp/fastmod/target/release/deps --extern bitflags=/tmp/fastmod/target/release/deps/libbitflags-b1998aea2bb93e0b.rmeta --extern libc=/tmp/fastmod/target/release/deps/liblibc-a0aa1b259a998b18.rmeta --extern mio=/tmp/fastmod/target/release/deps/libmio-c753201384e66968.rmeta --extern parking_lot=/tmp/fastmod/target/release/deps/libparking_lot-3f7e58475b5e1462.rmeta --extern signal_hook=/tmp/fastmod/target/release/deps/libsignal_hook-5fa19eae6b6a1422.rmeta --extern signal_hook_mio=/tmp/fastmod/target/release/deps/libsignal_hook_mio-8af779fa7545dfd1.rmeta --cap-lints allow` (exit code: 1)
warning: build failed, waiting for other jobs to finish...
error: build failed

Hi and thank you for open sourcing fastmod!

I was trying to use it to replace

try {
  await thing.start()
} catch (e) {
  console.error(e)
}

with just

await thing.start()

I assume giving enough effort with the multiline regex I could solve it, but I was wondering there would be interest to have a parameter that would accept a simpler pattern in a file?

Something like fastmod -p pattern-file "replacement \${1}"

pattern-file could contain something like

try {
  $1
} catch (e) {
  console.error(e)
}

Everything else would remain the same 😄

In our codebase we have some committed .env files (it's a separate question whether that's a good idea or not...).

➜  fastmod-dot-env ls -al
total 16
drwxr-xr-x   4 konradkomorowski  staff  128 25 May 14:52 .
drwx------@ 25 konradkomorowski  staff  800 25 May 14:50 ..
-rw-r--r--   1 konradkomorowski  staff    4 25 May 14:50 .foo
-rw-r--r--   1 konradkomorowski  staff    4 25 May 14:52 foo
➜  fastmod-dot-env cat .foo 
bar
➜  fastmod-dot-env cat foo 
bar

I noticed that fastmod will ignore those:

➜  fastmod-dot-env fastmod --accept-all bar baz
➜  fastmod-dot-env cat .foo 
bar
➜  fastmod-dot-env cat foo 
baz

things work fine if I'm directly targeting that file:

➜  fastmod-dot-env fastmod --accept-all bar baz .foo 
➜  fastmod-dot-env cat .foo 
baz

But sometimes I don't know what the filenames will be (as in the case of the codemod we were running), and we are 100% confident that we want to just replace all substrings.

I tried options like -g **/* and -g */** but that didn't work.

What's the motivation for this behavior? Is there a workaround that I'm not aware of?

PS: here's my version

➜  fastmod-dot-env fastmod --version
fastmod 0.4.1

👋 Hello! This is a very exciting project. What is your timeline for releasing this at a 1.0 version? We're evaluating this for use on large-scale projects and would prefer to utilize a stable tool that is production ready. Not to mention semver before a 1.x can be a headache.

If there is anything I can do to help, I'm happy to contribute as well so don't hesitate to reach out.

I am trying to do a multiline substitution. -m works for matching, but \n in the replacement string is being inserted as literally backslash-n.

Hi there! Is there a way to exclude a subdirectory from being searched, like ack's --ignore-dir= option?

fastmod 0.2.6 on OS X

Create a file with the word "bacon" in it

Run the following:

fastmod 'ac' 'bacon'

Expected result: Substitution is processed once, changing the text to bbaconon

Actual result: Substitution is processed infinitely. If you select "Yes to all" you are treated to a very trippy experience in your terminal as fastmod expands the string as quickly as it can.

I'm not sure whether it's a welcome comparison, but codemod 'ac' 'bacon' produces the desired result, only processing the match once.

When I run the following commands:

git clone https://github.com/ZcashFoundation/zebra.git
cd zebra
git checkout b618f5b5
yes n | RUST_BACKTRACE=full fastmod --extensions rs,toml,md 'sapling' 'canopy'

Which modify the line (note unicode quotation marks):

/// test network, the Human-Readable Part is “zviewtestsapling”.

I get a panic:

thread 'main' panicked at 'byte index 24841 is not a char boundary; it is inside '”' (bytes 24840..24843) of `//! Key types.
//!
//! "The spend authorizing key ask, proof authorizing key (ak, nsk),
//! full viewing key (ak, nk, ovk), incoming viewing key ivk, and each
//! diversified payment address addr_d = (d, pk_d ) are derived from sk,
//! as described in [Sap`[...]', /build/rustc-1.45.2-src/src/libcore/str/mod.rs:2052:47
stack backtrace:
   0:     0x5596d4a5ff43 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8e44e3bad104136e
   1:     0x5596d4a912dd - core::fmt::write::he8cb6d64ed166147
   2:     0x5596d4a73cf7 - std::io::Write::write_fmt::hf25ce96005919ce6
   3:     0x5596d4a6ca50 - std::panicking::default_hook::{{closure}}::hf8bcda2c877e2dcc
   4:     0x5596d4a6c764 - std::panicking::default_hook::h0602fc6a3744f2c1
   5:     0x5596d4a6d077 - std::panicking::rust_panic_with_hook::h7b83b0fe7900eb7a
   6:     0x5596d4a6cc7b - rust_begin_unwind
   7:     0x5596d4a90fc1 - core::panicking::panic_fmt::h61e03e91a1a8868a
   8:     0x5596d4a9909a - core::str::slice_error_fail::hefd104cb0d72240c
   9:     0x5596d48c5f49 - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}::h58d54a1cca6b09df
  10:     0x5596d48ce3e4 - fastmod::fastmod::hab3ce024a00b140c
  11:     0x5596d48cee73 - fastmod::main::h61f268b40af6022f
  12:     0x5596d48db943 - std::rt::lang_start::{{closure}}::hccb3d6d4b12ae437
  13:     0x5596d4a6d427 - std::rt::lang_start_internal::h7d1b27a1580794aa
  14:     0x5596d48cf052 - main
  15:     0x7ff62902bcbd - __libc_start_main
  16:     0x5596d48b71fa - _start
  17:                0x0 - <unknown>

If you haven't built fastmod using cargo build, seven out of the thirteen tests fail. Those are all the tests that invoke Command::cargo_bin().

I'll look into this more closely and hope there is a way to express that the binary needs to be built prior to running cargo test within the Cargo.toml, because even if the idea is that fastmod be run, it probably implies that the latest version ought to be run.