阿里巴巴论文_Can LLM Already Serve as A Database InterfaceA BIg Bench for Large-Scale Database GroundedText-to-SQLs.pdf

通讯员

1045

20页

18次

2023-06-08

免费下载

Can LLM Already Serve as A Database Interface?

A BIg Bench for Large-Scale Database Grounded

Text-to-SQLs

Jinyang Li

1,♣ ‡

, Binyuan Hui

2,♣

, Ge Qu

1,♣

, Binhua Li

, Jiaxi Yang

, Bowen Li

, Bailin Wang

Bowen Qin

, Rongyu Cao

, Ruiying Geng

, Nan Huo

, Xuanhe Zhou

, Chenhao Ma

Guoliang Li

, Kevin C.C. Chang

6 †

, Fei Huang

, Reynold Cheng

1 †

, Yongbin Li

2 †

The University of Hong Kong

DAMO Academy, Alibaba Group

Tsinghua University

Massachusetts Institute of Technology

The Chinese University of Hong Kong (Shenzhen)

University of Illinois at Urbana-Champaign

jl0725@connect.hku.hk, ckcheng@cs.hku.hk

binyuan.hby@alibaba-inc.com

Abstract

Text-to-SQL parsing, which aims at converting natural language instructions into

executable SQLs, has gained increasing attention in recent years. In particular,

Codex and ChatGPT have shown impressive results in this task. However, most of

the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema

with few rows of database contents leaving the gap between academic study and

real-world applications. To mitigate this gap, we present BIRD, a BIg benchmark

for laRge-scale Database grounded in text-to-SQL tasks, containing 12,751 pairs

of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning

37 professional domains. Our emphasis on database values highlights the new

challenges of dirty database contents, external knowledge between NL questions

and database contents, and SQL efﬁciency, particularly in the context of massive

databases. To solve these problems, text-to-SQL models must feature database

value comprehension in addition to semantic parsing. The experimental results

demonstrate the signiﬁcance of database values in generating accurate text-to-SQLs

for big databases. Furthermore, even the most popular and effective text-to-SQL

models, i.e. ChatGPT, only achieve 40.08% in execution accuracy, which is still far

from the human result of 92.96%, proving that challenges still stand. Besides, we

also provide an efﬁciency analysis to offer insights into generating text-to-efﬁcient-

SQLs that are beneﬁcial to industries. We believe that BIRD will contribute to

advancing real-world applications of text-to-SQL research. The leaderboard and

source code are available: https://bird-bench.github.io/.

1 Introduction

Text-to-SQL parsing [

], which focuses on transforming natural language into SQL

queries, has attracted signiﬁcant research interests from both academia and industry. This attention

stems from its potential to empower non-expert data analysts in automatically extracting desired

information from ubiquitous relational databases using natural language. Recent advances in neural

♣ Equal contribution.

‡

Work done during an intern at Alibaba DAMO Academy.

†

Corresponding authors.

arXiv:2305.03111v2 [cs.CL] 30 May 2023

External Knowledge Reasoning

Large and Realistic Database Values

SQL Execution Efficiency

What is the winning rate of Boston Celtics in 2000?

SELECT COUNT(won) / ((COUNT(won) + COUNT(lose))

FROM teams WHERE team_name = ‘Boston Celtics’

AND year = 2000;

External Knowledge:

winning rate = # won / (# won + # lose)

What is the average salary of the worst performing managers?

SELECT AVG(CAST(REPLACE(SUBSTR(T1.salary, 4), ',', '') AS REAL)) FROM

last_name

Milgrom

… …

em_id

0000

… …

Employees

US$57,500.00

first_name

salary

2222

6543

Adams

Wood

Milgrom

Sandy

Emily

… …

US$19,500.00

US$69,000.00

… …

Reasoned Database:

employee AS T1 JOIN position AS T2 ON T1.positionID = T2.positionID

WHERE T1.performance = 'Poor' AND T2.positiontitle = 'Manager'

Among the coaches who have served more than 2 NBA teams, during

which coach‘s period of coaching, a team has the least numbers of

games lost in the post-season games?

SELECT coachID FROM coaches WHERE lgID='NBA’ AND post_wins !=0

SQL

: normal semantic parser

AND post_losses !=0 AND coachID IN

(SELECT coachID FROM coaches WHERE lgID='NBA’ GROUP BY coachID

HAVING COUNT(tmID)>=2) ORDER BY post_losses ASC LIMIT 1 ;

Run time: 22.4s

What is the average salary of the worst performing managers?

SELECT AVG (CAST(REPLACE(SUBSTR(T1.salary, 4), ',', '') AS REAL)) FROM

last_name

Milgrom

… …

em_id

0000

… …

Employees

US$57,500.00

first_name

salary

2222

6543

Adams

Wood

Santa

Sandy

Emily

… …

US$19,500.00

US$69,000.00

… …

Reasoned Database:

employee AS T1 JOIN position AS T2 ON T1.positionID = T2.positionID

WHERE T1.performance = 'Poor' AND T2.positiontitle = 'Manager'

SQL

: efficient semantic parser

SELECT coachID FROM coaches WHERE lgID=‘NBA’ AND post_wins !=0

AND post_losses !=0 AND EXISTS (SELECT 1 FROM coaches AS coaches1

WHERE (coaches1.lgID=‘NBA’) AND (coaches.coachID=coaches1.coachID)

GROUP BY coaches1.coachID HAVING count(coaches1.tmID) >= 2

ORDER BY NULL ) ORDER BY coaches.post_losses ASC LIMIT 1

Run time: 4.0s

How many accounts are eligible for loans in New Yo rk City?

The condition of loans is that

the type of the account should

be “OWNER”.

SELECT COUNT(*) FROM account WHERE account.type

= ‘OWNER’ AND city = ‘NY’;

External Knowledge:

(a).

(b).

(c).

How many accounts are eligible for loans in New Yo rk City?

The condition of loans is that

the type of the account should

be “OWNER”.

SELECT COUNT(*) FROM account WHERE account.type

= ‘OWNER’ AND disp_id = ‘NY’;

External Knowl edge:

List account id who chooses weekly issue issuance statement?

‘POPLATEK TYDNE’ stands

for weekly issuance.

SELECT account_id FROM account WHERE account.frequency

= ‘POPLATEK TYDNE‘;

External Knowl edge:

How many accounts are eligible for loans in New Yo rk City?

The condition of loans is that

the type of the account should

be “OWNER”.

SELECT COUNT(*) FROM account WHERE account.type

= ‘OWNER’ AND disp_id = ‘NY’;

External Knowledge:

List account id who chooses weekly issue issuance statement?

‘POPLATEK TYDNE’ stands

for weekly issuance.

SELECT account_id FROM account WHERE account.frequency

= ‘POPLATEK TYDNE‘;

External Knowledge:

What is the average salary of the worst performing managers?

Figure 1: Examples of challenges in our BIRD benchmark. 1) databases contain values of noisy data

types [

]. In the left example, the average salary could be fetched by processing the

data type from string (

TEXT

in SQLite) to ﬂoat (

REAL

in SQLite) after deleting the special tokens,

"US$"

and

","

. 2) external knowledge and reasoning are required. In the middle example, models

must handle that only

"OWNER"

accounts are eligible for loans. 3) query execution efﬁciency needs

to be considered. In the right example, the adoption of more efﬁcient SQL queries leads to signiﬁcant

gains in speed, which is of great value in industries.

models, including those based on large language models (LLMs), have led to impressive performance

on existing benchmarks such as Spider [

] and WikiSQL [

]. For instance, the execution accuracy

of the top-performing model in Spider leaderboard has increased from 53.5% [

] to 85.3% [

] over

the past three years. The latest SOTA parser [

] in Spider beneﬁts from the powerful understanding

and coding capabilities of the large language model (LLM), and such excellent performance leads us

to ask a question: Can LLM already serve as a database interface ?

The answer is no, as shown in Figure.1, we discovered that current state-of-the-art models still struggle

to generalize to more realistic situations characterized by large database sizes and noisy content.

Besides, the mysteries hidden behind the huge database values require external knowledge and

reasoning to reveal. Furthermore, existing benchmarks do not account for SQL execution efﬁciency,

which holds signiﬁcant practical importance in real-life applications, notably in the case of large

databases. Motivated by these observations, we aim to develop a new text-to-SQL benchmark that

better represents real-life scenarios and narrows the gap between experimental and practical settings.

In this work, we propose BIRD, a BIg Bench for LaRge-Scale Database Grounded in Text-to-SQLs

for real-world applications. BIRD contains complex 12,751 examples of querying information over

95 big databases with a total size of 33.4 GB spanning 37 professional domains. For training, we

collected 80 open-source relational databases from real analysis platforms (Kaggle, Relation.vit);

for evaluation, we curated 15 additional relational databases. Given these databases, we rely on

crowdsourcing to collect natural language instructions and the corresponding SQLs. First, our

database experts create a description ﬁle explaining all column names, abbreviated values, value

types, and external knowledge for each database to help annotators better understand the database

contents. Then we hire and train native speakers to ask questions facing these databases on one

side; on the other side, a SQL annotation team consisting of data engineers and database students

is recruited to generate SQLs to answer questions. To accommodate efﬁciency, we propose a new

metric Valid Efﬁciency Score (VES) to evaluate the efﬁciency of generated SQLs in addition to the

standard execution accuracy. To the best of our knowledge, BIRD is the ﬁrst text-to-SQL benchmark

to incorporate efﬁciency, promoting more efﬁcient query methods within the context of massive and

noisy database contents.

We evaluate the performance of state-of-the-art text-to-SQL parsers using two popular methodologies:

ﬁne-tuning with T5 [

], and in-context learning with large language models (LLMs) such as Codex

[

] (

code-davinci-002

) and ChatGPT [

] (

gpt-3.5-turbo

). Our experimental results

reveal that the current models struggle to generalize well. Speciﬁcally, the Spider SOTA model,

which depends solely on the database schema, achieves execution accuracies of only 25.88% and

28.95% on the development and test sets, respectively. In comparison, the performance still lags far

behind human performance, which we also provide in this benchmark. We encourage further research

to address the more realistic settings presented in this benchmark.

of 20

免费下载

关注

评论