Comments (8)
arthurkiller
commented on August 17, 2024
Thx, I will take a look at this
from rollingwriter.
arthurkiller
commented on August 17, 2024
@chaomai 非常感谢细测试代码并提出问题，开源社区社区因为您的付出而更加美好！
不过关于您的问题，目前尚有些疑问：
m.lock.Lock()
// ...
m.lock.Lock()
- 这里有提到需要在 manager 中增加一个 lock ，而这个 lock 用于 guard 生成日志文件名的逻辑，这个地方不太理解为何会有 race ？因为 manager 本身不需要加锁的原因是只应该有一个 manager 实例，造成冲突是由于多个实例并发执行，那么这是使用方式的问题。
- 其次，假设真的发生了这样的冲突，那么后果其实也是符合预期的，因为文件名本身就是跟时间相关的，如果出现了 1 秒内频繁切分文件的需求，那么文件名的后缀就需要更加长以适应这种需求。因此如果文件名自定义格式使用 unixnanosecond 做后缀，理论上就不会有冲突发生。
感谢回复
from rollingwriter.
chaomai
commented on August 17, 2024
最后的代码手误，lock打错了。。。
// GenLogFileName generate the new log file name, filename should be absolute path
func (m *manager) GenLogFileName(c *Config) (filename string) {
m.lock.Lock()
// [path-to-log]/filename.log.2007010215041517
if c.Compress {
filename = path.Join(c.LogPath, c.FileName+".log.gz."+m.startAt.Format(c.TimeTagFormat))
} else {
filename = path.Join(c.LogPath, c.FileName+".log."+m.startAt.Format(c.TimeTagFormat))
}
// reset the start time to now
m.startAt = time.Now()
m.lock.Unlock()
return
}manager只有一个实例没错，但是GenLogFileName在执行时候，由于github.com/robfig/cron直接就在goroutine中运行了，因此可能存在多个goroutine都在跑GenLogFileName（如果前一个被cron调度的未执行完毕），进而出现多个goroutine共享同一个manager的现象。
from rollingwriter.
arthurkiller
commented on August 17, 2024
我明白你的意思了，这个cron其实我是想把它内置了的，因为本身依赖的不多，import比较重。我想直接内置进来修复
Have an awesome day
✉️✉️✉️✉️✉️✉️✉️✉️✉️✉️✉️✉️
Arthur lee
Sent from my iPhone
… On 28 Apr 2019, at 11:55 PM, Chao Mai ***@***.***> wrote:
最后的代码手误，lock打错了。。。
// GenLogFileName generate the new log file name, filename should be absolute path
func (m *manager) GenLogFileName(c *Config) (filename string) {
m.lock.Lock()
// [path-to-log]/filename.log.2007010215041517
if c.Compress {
filename = path.Join(c.LogPath, c.FileName+".log.gz."+m.startAt.Format(c.TimeTagFormat))
} else {
filename = path.Join(c.LogPath, c.FileName+".log."+m.startAt.Format(c.TimeTagFormat))
}
// reset the start time to now
m.startAt = time.Now()
m.lock.Unlock()
return
}
manager只有一个实例没错，但是GenLogFileName在执行时候，由于github.com/robfig/cron直接就在goroutine中运行了，因此可能存在多个goroutine都在跑GenLogFileName（如果前一个被cron调度的未执行完毕），进而出现多个goroutine共享同一个manager的现象。
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or mute the thread.
from rollingwriter.
arthurkiller
commented on August 17, 2024
@chaomai 不过我觉得目前来说应该不会有使用上的问题，因为这个只是生成一个字符串，没有数据竟态。
138 filename = path.Join(c.LogPath, c.FileName+".log.gz."+m.startAt.Format(c.TimeTagFormat))
所以，按照我 2 所说
假设真的发生了这样的冲突，那么后果其实也是符合预期的，因为文件名本身就是跟时间相关的，如果出现了 1 秒内频繁切分文件的需求，那么文件名的后缀就需要更加长以适应这种需求。因此如果文件名自定义格式使用 unixnanosecond 做后缀，理论上就不会有冲突发生。
不过这个问题我会记录下来，后面改造 cron 时候就内置进去了。再次感谢
from rollingwriter.
chaomai
commented on August 17, 2024
但GenLogFileName中，对m.startAt是有读有写的。
另外，在writer中也有一个问题。具体复现方法是，
1. 首先做如下修改
diff --git a/rollingWriter.go b/rollingWriter.go
index 3f2fa6f..ab561ec 100644
--- a/rollingWriter.go
+++ b/rollingWriter.go
@@ -99,13 +99,13 @@ func NewDefaultConfig() Config {
LogPath: "./log",
TimeTagFormat: "200601021504",
FileName: "log",
- MaxRemain: -1, // disable auto delete
+ MaxRemain: 5, // disable auto delete
RollingPolicy: 1, // TimeRotate by default
- RollingTimePattern: "0 0 0 * * *", // Rolling at 00:00 AM everyday
+ RollingTimePattern: "* * * * * *", // Rolling at 00:00 AM everyday
RollingVolumeSize: "1G",
WriterMode: "lock",
BufferWriterThershould: 64,
- Compress: false,
+ Compress: true,
}
}
diff --git a/writer_test.go b/writer_test.go
index 1d525f6..62fd1d4 100644
--- a/writer_test.go
+++ b/writer_test.go
@@ -155,7 +155,7 @@ func TestVolumeWriteParallel(t *testing.T) {
}
func TestWriteLockParallel(t *testing.T) {
var writer io.WriteCloser
- var c int = 1000
+ var c int = 10000000
var l int = 1024
t.Run("locked", func(t *testing.T) {Compress: true可改可不改，改后更容易触发
2. 运行func TestWriteLockParallel(t *testing.T)
会报，
=== RUN TestWriteLockParallel/locked
=== PAUSE TestWriteLockParallel/locked
=== CONT TestWriteLockParallel/locked
2019/04/29 11:22:48 error in compress log file remove test/unittest.log.gz.201904291122.tmp: no such file or directory
2019/04/29 11:22:49 error in compress log file remove test/unittest.log.gz.201904291122.tmp: no such file or directory
2019/04/29 11:22:50 error in compress log file remove test/unittest.log.gz.201904291122.tmp: no such file or directory
2019/04/29 11:22:55 error in compress log file remove test/unittest.log.gz.201904291122.tmp: no such file or directory
2019/04/29 11:22:55 error in compress log file remove test/unittest.log.gz.201904291122.tmp: no such file or directory
我做了fix，想提一个pull request到dev，不过我发现dev落后master好多，要不提到master你看看？
from rollingwriter.
arthurkiller
commented on August 17, 2024
@chaomai 嗯，这个确实有问题，PR 可以先 rebase 下 master，然后再提
我已经更新了dev 的代码
from rollingwriter.
arthurkiller
commented on August 17, 2024
closed via #10
感谢贡献😆，别忘了 star✨这个仓库 @chaomai
from rollingwriter.
Related Issues (20)
- Buffer writer concurrency unsafe HOT 3
- Schedule for v2.0 HOT 3
- 当长时间没有写日志操作，就会导致Writer中fire阻塞，后续写日志的时候，容易归档到错误的文件里面 HOT 3
- 能更新一下tag吗？ HOT 1
- How can I rotate every 3 seconds? HOT 2
- Error when renaming file HOT 2
- Dead lock detected? HOT 5
- FileName use template string HOT 2
- Volume rolling fails on windows
- Release tagged version (1.1.3) with latest fix HOT 1
- goroutne leak HOT 4
- 归档后，还是会往归档文件中持续写入日志 HOT 6
- Why we need _asyncBufferPool for async writer? HOT 4
- async模式的buffer大小，存在严重的性能问题 HOT 10
- write error: can't rename log file: rename / /-2023-04-26T11-45-46.770: device or resource busy HOT 4
- 日志分割有问题 HOT 1
- 按时间分割，文件已关闭错误 HOT 2
- onRotate callback function, file extension option. HOT 3
- 小建议
- rollingwriter: typo HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rollingwriter.