Abstract

LM은 적은 수의 예제와 텍스트 지침을 이용해서 몇 태스크에 뛰어난 성과였다.

teach themselves to use external tools via simple APIs and achieve the best of both worlds

how to best incorporate the results into future token prediction

Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks,

1. Introduction

all of these models have several inherent limitations that can at best be partially addressed by further scaling

related tendency to hallucinate facts

difficulties in understanding low-resource language

a lack of mathematical skills to perform precise calculations

A simple way to overcome these limitations of today’s language models is to give them the ability to use external tools such as search engines, calculators, or calendars.

제한된 몇 툴을 극목하기 위해서 we propose Toolformer, a model that learns to use tools in a novel way, which fulfills the following desiderata:

The use of tools should be learned in a self-supervised way without requiring large amounts of human annotations. - costs associated with such annotations, find useful

able to decide for itself when and how to use which tool

we let a LM annotate a huge language modeling dataset with potential API calls. We then use a self-supervised loss to determine which of these API calls actually help the model in predicting future tokens.

우리의 방법론은 데이터를 효과적으로 처리할수 있고, GPT-J 모델을 기반으로 Toolformer에서 tool을 사용하여 downstream task 실험을 수행한다.

2. Approach

API는 text sequences를 나타내고 special tokens의 시작과 끝(QA처럼 API 토큰이 들어갈 시작과 끝) 을 잘 나타낸다.

<API>와 </API>는 special tokens으로 C는 데이터셋으로 구성된다.

Sampling API Calls

We first sample up to k candidate positions for doing API calls by computing

Executing API Calls

we execute all API calls generated by M to obtain toe corresponding results.

Filtering API Calls

the input and the output of this call makes it easier for the model to predict future tokens

Model Finetuning

샘플링과 필터링된 API 호출로 augmented 된 데이터에 대해 언어 모델이 fine-tuning

3. Tools

text sequences로 5가지 툴을 사용

a Wikipedia search engine

a calculator

a calendar

a machine translation system

4. Experiments

유용함과 zero-shot을 평가

4.1 Experimental Setup

Dataset Generation CCNet의 subset을 언어모델 M으로써 GPT-J 사용, 충분한 예제를 보장하는지를 thresholds로써 고름

Baseline Model

• GPT-J
• GPT-J + CC - GPT-J finetuned on C
• Toolformer - GPT-J finetuned on C ∗
• Toolformer (disabled API call)

4.2 Downstream Tasks

구체적인 task들을 풀기 위해 어떻게 tool을 적용했는지 데이터셋의 예제로써 나열

decoding strategy를 사용, API 토큰이 포함되어 있으면 k를 10으로 설정

4.2.1 LAMA

LAMA의 subset인 SQuAD, Google-RE, T_REx 데이터 사용하여 평가

4.2.2 Math Dataset

ASDiv, SVAMP, MAWPS사용했고, zero-shot setup

4.2.3 Question Answering

metric is the percentage of times the model’s generation, capped at 10 words, contains the correct answer.

4.2.4 Multilingual Question Answering

4.2.5 Temporal Datasets

4.3 Language Modeling

4.4 Scaling Laws

smaller models achieve similar performance both with and without tools

5. Analysis

Decoding Strategy

k value(API가 생성될때를 찾는 K) 에 따른 성능 비교, k가 10일 때 98.1% to 100%

Data Quality

Filtering에 사용된 $L_i^- - L_i^+$를 이용하여 데이터 useful 측정

7. Limitations

• finetuning에서 사용할 chained tool이 없음
• 맞다 틀리다를 결정할 때, 입력과 똑같은 단어가 들어오는 것에 민감하게 반응한다.
• 데이터 부족으로 특정 API Call은 다른 것들에 비해 비효율적
  • 관련해서 bootstrapping 방법론을 사용할 수도 있음
• Tool-dependent를 고려하지 않아서 비용이 높음