Korean Indirect Speech Act Detection

A research workbench for detecting Korean indirect-speech-act spans and classifying their communicative function under severe label imbalance.

Research task

Given a Korean utterance or dialogue context, the project studies two related outputs:

1. Span detection: BIO tags (O, B-ISA, I-ISA) for the text that realizes an indirect speech act.
2. Utterance classification: whether the utterance expresses an indirect directive or no target act.

The dataset snapshot documented in this repository contains 984 utterances, including 223 ISA-positive items. ISA tokens account for about 4.4% of tokens, making majority-class accuracy a poor primary metric.

At a glance

Component	Scope
Language	Korean
Task	Span detection and utterance-level classification
Models explored	KLUE/RoBERTa, T5-family models, KoAlpaca, Llama, and prompted baselines
Primary span metric	Non-O precision, recall, and F1
Research emphasis	Class imbalance, context, architecture, and evaluation design

Recorded experimental results

The historical summary in results/experiment_summary.md reports the following selected outcomes:

Model/run	Reported Non-O F1	Important qualification
RoBERTa-base	0.000	Collapsed to the majority O class
RoBERTa-large	0.0146	Very low recall under the recorded setup
KoAlpaca 5.8B QLoRA	0.830	Reported precision 1.00 and recall 0.709
GPT-4o-mini prompt baseline	1.000	Only eight evaluated examples; not comparable to the full runs

These numbers come from heterogeneous training and evaluation setups. They should be read as an exploratory model inventory, not as a controlled causal comparison of model size or architecture. In particular, the repository does not establish that decoder-only architecture caused the reported performance difference.

Start here

• Historical experiment summary
• Dialogue evaluation results
• Benchmark evaluation plan
• Architecture-validation proposal

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python scripts/prepare_span_data_v2.py
python scripts/train_span_classification.py

For gated Hugging Face models, provide credentials only through the environment:

export HF_TOKEN="<your-token>"

Do not place credentials directly in scripts or configuration files.

Repository map

scripts/       Preprocessing, training, evaluation, and mission runners
experiments/   Experiment-specific implementations
results/       Metrics and model-comparison reports
docs/          Analysis notes and proposed controls
prompts/       Frozen or historical task prompts
config/        Experiment configuration

Reproducibility and limitations

• Non-O F1 should be reported alongside precision, recall, and support.
• Results based on different sample sizes or prompt regimes must not be ranked as if they came from one controlled benchmark.
• Several historical scripts contain machine-specific paths and require portability cleanup before clean reproduction.
• Raw datasets, checkpoints, W&B run files, and operational status scripts are not part of this curated public snapshot.
• Dataset splits and possible train/test overlap should be independently audited before promoting the historical numbers as benchmark results.

Data and license

The public snapshot does not redistribute the derived research datasets. It currently has no repository-level license; verify upstream dataset terms and obtain the required permissions before reuse.

See PUBLIC_RELEASE.md for the snapshot boundary and provenance.

Collaboration

The strongest next contribution would be a clean, split-audited benchmark run with one frozen preprocessing pipeline, comparable baselines, and confidence intervals for the minority-span metrics.