CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has been truncated

[g-yziquel]

Bug report

Bug description:

CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has long header fields.

This issue is lifted up here as it was initially raised in the jc cli github repo. Here:

kellyjonbrazil/jc#465

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Linked PRs

• [3.15] gh-110128: Ignore a truncated final row in csv.Sniffer #155037

[sunmy2019]

You'd better provide a repro code that only utilizes Python stdlib.

---

You raised a problem with csv.Sniffer.sniff. You think one of its sniff results is inaccurate.

But, it's just a guess. And guess cannot be perfect.

What new algo will you raise and how will it affect the current use case?

[g-yziquel]

Sure. Any more accurate sniffer is arguably not to be provided out of the box by the stdlib.

It remains that one of its sniffs is inaccurate, and I could reproduce it with the stdlib. (If that would be genuinely welcome).

I would however be very wary of making changes to the sniffing algo without a series of tests on a corpus of such CSV files.

As of today, I have no clue as to how this should be done. But I do believe the CSV files I gave should, in an ideal world, be part of such a testing corpus.

Feel free to close the issue. I raised it out of personal ethos. If it's just noise to you, please do close it.

[ws-garcia]

Hi @g-yziquel, the research on the subject is on its way! Please take a look to the Table Uniformity Method (TUM), also implemented in Python using the built-in csv module.

[serhiy-storchaka]

The problem is not the length of the header fields, but the truncation of the sample: jc sniffs a fixed-size prefix, and the old code counted the incomplete last row as a real one, so the tab was no longer consistent and sniff() raised csv.Error. DummyData.Right.csv is 1024 bytes and fits in the sample entirely, DummyData.Wrong.csv is 1046 bytes and does not; the "right" file fails too if the sample is cut at 512 bytes.

Fixed in 3.16 by the csv.Sniffer rewrite (gh-83273): the last row is not counted if the sample can be cut off in the middle of it. The maintenance branches are still affected, and a targeted fix for them is welcome.