Bug: Bulk update changes date-modified for random files even if there was no linting done in them about obsidian-linter HOT 14 OPEN

Hey @jvdevlab . I will need more information in order to be able to do anything about this. Are you able to reproduce this consistently with the bulk update? If so, could you verify that the data modified in the file system is within 5 seconds of the date modified in the frontmatter? If so, then there likely is a bug, but if not, it is going to be very hard to track down this issue.

Also, I would like to rule out syncing across multiple devices being the problem. You are not syncing these files from one device to another are you? If so, the Linter has no way to know where a change comes from except when it makes a change itself. So it is not really feasible to just update date modified when the Linter updates the file (the file system is considered to be the source of truth according to the Linter).

Thanks for helping me better understand what is going on in your case!

from obsidian-linter.

@pjkaufman

Are you able to reproduce this consistently with the bulk update?

Yes.
I take the vault with 1000+ notes that was already linted according to the enabled rules.
I lint the whole vault again (without changing anything) I get 178 changes. I commit to git.
Repeat. I get 170 changes. Commit
Repeat. I get 133 changes. Commit
...
117
I can repeat it and see if it goes down to 0

You can see how with each run I get less and less files. Each next run is a subset of files from previous run. At first I though maybe these files formatted weirdly or there is something special about them. I compared them to "normal" files (for which date-modified is not changing as expected) but I do not see difference. I used diff to show special chars and whatnot but nothing stands out.

I have another vault with all the same plugins, settings, rules etc. except it has 200 notes and it is not happening there.

If so, could you verify that the data modified in the file system is within 5 seconds of the date modified in the frontmatter?

I checked bunch of offending files.
They all shows modified on ‎Saturday, ‎January ‎27, ‎2024, ‏‎9:50:33 AM in Windows File Explored

In Frontmatter these files range from: 2024.01.27 09:50:20 to 2024.01.27 09:50:27

Also, I would like to rule out syncing across multiple devices being the problem. You are not syncing these files from one device to another are you?

I'm using Obsidian Sync.
I suspected it as well but I think it is safe to rule out. The above runs that I did (where number of changed files keep decreasing) were done during the time when the other devices were offline. So unless there is an issues with single device and Sync we should be good here.

Update:
I just paused the Sync to make sure. Did 2 runs.

• 108 files changed
• 100 files changed
  They issue still there and the count keeps going down...

from obsidian-linter.

Interesting. Is it affecting a subset of the same files? If so, does the same happen if you try linting 10 or so of them at random from the list of previously updated files on mass lint? Make sure that when you do a single file lint you change the log level for the debug tab to trace and enable capturing logs when running the linter on save. You can then post the Linter's logs here which should be able to help identify the issue.

I am guessing based on what you are listing as the date modified time is that the larger the amount of files are that get linted, the farther out of sync the date modified file metadata and the file frontmatter get.

I am definitely open to ideas on how to fix this if we can confirm that is the case. However, I am not 100% sure that is the case. Nor do I have a test setup large enough to test this scenario since it sounds like I need to lint more than a couple hundred files for this to happen.

Sorry that this is getting long. I do want to make sure that you are not using custom commands as a part of the way you use the Linter if you are using it in conjunction with the date modified logic since the Linter is unable to account for modifications made by custom commands which could be causing the date modified update to happen since custom commands could modify the file more than 5 seconds from when the Linter sets the date modified (that is not really fixable at this time).

Thanks for the info so far.

from obsidian-linter.

Is it affecting a subset of the same files?

Yes, the next run even so smaller contains files from previous runs. I didn't compare every single file but it is very much look that way.

If so, does the same happen if you try linting 10 or so of them at random from the list of previously updated files on mass lint?

When I open them one by one and do not change anything and hit ctrl+s it is linting them. Says 4 chars added and 4 removed. (minutes and seconds)

Make sure that when you do a single file lint you change the log level for the debug tab to trace and enable capturing logs when running the linter on save. You can then post the Linter's logs here which should be able to help identify the issue.

See attached file.
log.txt

I am definitely open to ideas on how to fix this if we can confirm that is the case.

I didn't look at your code. But if you tracking all the changes before committing the save can you have a postprocessing routine that would check: If the only linting change is date-modified and date-modified already exist with a value then do not commit changes.

I do want to make sure that you are not using custom commands as a part of the way you use the Linter

I do not

Thank you for the plugin and your help in troubleshooting the issue.

Update

From the log,

Running YAML Timestamp
yaml-timestamp: 0.2999999523162842 ms

Changed the timestamp but it's not clear why...

I have it like so

Update

• As I edit and add new notes, the count of affected notes went up. So it's not something wrong with some set of notes. The issue persist and the count will not go down to 0.

from obsidian-linter.

Interesting. This is not something I am seeing anything in the logs that would indicate what is going on. I am thinking that I need the logs with a modified version of the plugin to make it include information about why it is updating the values.

from obsidian-linter.

Could you reproduce the issue again while using the main.js from the following zip file and grab the debug logs for the file?
main.zip

To install the main.js:

• Unzip the zip file
• Copy the main.js from the output to path_to_vault/.obsidian/plugins/obsidian-linter/
• Then either start Obsidian or turn off and back on again the Linter plugin

Thanks for helping better understand what is going on in this situation.

from obsidian-linter.

Hey. Just wanted to check back in and see if there was any update on whether or not there will be an attempt to reproduce the issue with the custom plugin logic to help better understand the problem. It has more logging info that should clarify why the logic thinks an update needs to happen. If you do not plan on reproducing the issue with the custom logic, that is fine, it just leaves the actual cause of the problem here as a bit of a mystery.
Thanks!

from obsidian-linter.

@pjkaufman Sorry it was on my todo all week. And I didn't get to it. I will do today.

from obsidian-linter.

@pjkaufman,

Here are the results.

First time I ran the linting on the whole vault it modified date for 544 files. Second time -- 445.
Then I cherry picked a file.

date-created: 2024.01.29 16:34:04
date-modified: 2024.02.11 10:13:08 (this was actually from linting the whole vault)

After manually linted it

date-modified: 2024.02.11 10:27:06

Here I are the relevant part from the log.

---

Running YAML Timestamp
Key found
Text modified already? false
modified date time from frontmatter is undefined? false
modified date time is valid? true
current modified date metadata: 2024.02.11 10:13:24
frontmatter modified date: 2024.02.11 10:13:08
time difference between now file modified at time and frontmatter value? 16
yaml-timestamp: 0.5 ms
---

And then it sets it to date-modified: 2024.02.11 10:27:06

Here is the full log.
obsidian-linter.txt

from obsidian-linter.

I was really hoping it was not the time issue. There is really not that much that can be done to avoid that issue unless I allow the user to specify what time in seconds they think that the metadata should be within or figuring out why it is taking 16 seconds between generating the time for the frontmatter and the saving of the file.

from obsidian-linter.

@pjkaufman can you please expose it as advanced setting? It could have 5 sec default (as you mentioned above) and whoever runs into the issue can set a higher number.

from obsidian-linter.

I am not sure if it will get exposed as a setting. It is a bit tricky since some users will go ahead and bork things by changing it. I will have to think about this and see if it is something that makes sense to do. I would much rather address the root cause of the issue without making a settings change. But that will be difficult given the size of the data set needed to test this scenario.

from obsidian-linter.

Hi @pjkaufman,
There are number of large vaults online that can be downloaded. Quick search gave me this one with 1300 notes. There probably bigger ones. Tricky thing is it might not just be the vault size but the rest of the environment like OS version, etc. So not sure if this will help with finding the root cause.

Can you point me to the constant value in the code? (hopefully it's in one place) I can change it locally. And keep changing every time I take a plugin update. I don't know if the Obsidian plugins allow you to externalize constants on the code level. In a way that you can update the plugin but constant will remain the same. This way it will be truly advanced settings as it will not be exposed in the UI.

from obsidian-linter.

The constant in the code is located here. It is not really a constant. It just is used in place since it is only needed there to represent 5 seconds.

from obsidian-linter.