Comments (4)
HYLcool
commented on June 10, 2024
嗨 @noforit ，感谢你对data-juicer的关注与使用！
stopwords_filter在实现时的本意为根据停用词比例筛除一些搜索引擎处理过的文本。一般情况下，搜索引擎为了提升搜索效率等原因，会将一个文档中的停用词删除后再建立索引，但删除停用词的文档的语义信息会被破坏，在LLM的训练数据中会被认为是质量相对较低的文本。所以这个算子会将停用词比例较低的样本过滤掉。
但你说的也是正确的，停用词比例较高的样本也是质量较低的。stopwords_filter算子其实功能上有一个互补的算子，叫flagged_words_filter，它的本意是将敏感词比例过高的样本滤除。它们俩都可以指定词表，因此它们的更广泛用法为将感兴趣的某类词的比例过高或者过低的样本滤除。比如针对你所说的情况，我们可以加上一个flagged_words_filter，词表设置为停用词词表，这时就会把停用词比例过高的样本也筛除了。
如你还有进一步的疑问，欢迎随时与我们交流~
from data-juicer.
noforit
commented on June 10, 2024
@HYLcool 好的好的，谢谢你
from data-juicer.
github-actions
commented on June 10, 2024
This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.
from data-juicer.
github-actions
commented on June 10, 2024
Close this stale issue.
from data-juicer.
Related Issues (20)
- DJ-v.0.2 docker image update HOT 1
- DJ-v0.2 API page enhancement
- Video content compliance and privacy protection operators (image, text, audio)
- [Bug]: video split by duration mapper return non-exist video
- support panda's student captioner model in our captioning mapper HOT 4
- [Bug]: Video_split_by_scene_mapper create non-exist video_keys
- [Feature Request] Implement more streamlined interfaces for users seeking minimal functionality (data_juicer.op.functional) HOT 3
- Request a sample code demonstrating the use of image_captioning_from_gpt4v_mapper.py HOT 3
- Can not download the data quality classifier models. HOT 1
- alphanumeric_filter算子清洗疑问 HOT 5
- Absolute path to relative path for multi-source
- [Bug]: process on ray occur "TypeError: 'str' object cannot be interpreted as an integer" HOT 8
- filter是否支持batch处理，以及怎么设置batch_size？ HOT 6
- hash calculate in ray deduplicator HOT 5
- 为什么大部分的refined recipe都是用simhash去重？ HOT 3
- [Bug]: 运行tools/analyze_data.py报错，出现 KeyError: 'text' HOT 2
- [Question] Can't find evalutor.yaml on the path of `/workspace/data-juicer/demos` HOT 1
- A Compatibility Issue in Environment Installation of DJ-Sandbox HOT 1
- 报”error: Unrecognized arguments: -B -S -I -c“ HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from data-juicer.