Comments (6)
因为如果参数为 all ,即使无法执行,SQL层面上match也算对;而exec必须保证SQL能执行且执行结果和Gold完全一致才行,条件比all苛刻,所以会低2个点。
from mac-sql.
from mac-sql.
感谢您的回复,这对我非常重要。请问MAC中的实验结果使用的是all还是exec呢?NL2SQL普遍使用的是哪种呢?
并且我调用的公司聊天GPT4O(带有上下文可能会对实验结果产生影响),复现了MAC-SQL发现在SPIDER上,EM分数较低(20左右),这是否和您当时得到的实验结果一致?如果一致的话,您知道其中的原因是什么吗?
from mac-sql.
MAC-SQL 用的exec,这点可以从代码脚本中看到。NL2SQL普遍采用 exec,因为随着大模型出现,EM指标显得很鸡肋,无法反映出模型的真实能力,因为SQL的写法有很多,只要答案对了即可,而EM指标要求写法完全按照Gold的来,很显然这不合理。MAC-SQL在EM较低是正常的,现在大模型方案在EM上都低,所以现在NL2SQL论文都已经摈弃EM指标了,都用的EX指标。
from mac-sql.
FlyingFeather/DEA-SQL#1 (comment)
你知道DEA-SQL这篇论文吗,这个问题中提到您之前提交的代码好像和现在不同。
from mac-sql.
Spider数据集上是有点调整,你可以用现在的版本。
from mac-sql.
Related Issues (20)
- How much does it cost to use GPT-4 8k to complete the evaluation in the paper? HOT 4
- Code flow HOT 5
- Are instruction-tuned models available HOT 3
- Improving Dataset Accessibility on Google Drive HOT 1
- Help wanted! I can't reproduce your results on Spider dev as shown in the paper HOT 7
- Decomposition on Spider HOT 2
- Help!I want to know in the agents name 'parse_single_sql' is not defined. i don't find it in other code. please!!! HOT 3
- The sql-llama link in readme is not avaible HOT 5
- Clarification on EX and VES Scores without Oracle Schema in Table 1 HOT 1
- Selector few-shot prompts HOT 1
- Using MAC-SQL for a specific databse HOT 2
- Help wanted! Question about Bird dataset results on the paper. HOT 2
- result in spider-realistic
- 关于微调数据的一些疑问 HOT 1
- Cannot Reproduce SQL-Llama Result HOT 1
- 请问,MAC-SQL是使用了OpenAI的API了,在内网/局域网或者离线状态下能够使用么? HOT 2
- core下面没有api.py?? HOT 1
- 请问,MAC-SQL项目能否用于用户自己设计的数据库? HOT 2
- IceKingBing/SQL-Llama-v0.5模型得到结果与论文中结果差距较大 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mac-sql.