Disable auto transfer mode by default

[jgoerzen]

Prior to this change, "show file" shows, among other things:

Transfer mode: automatic
File patterns: automatic (SHOW PATTERNS for list)
Default file type: binary

With this change:

Transfer mode: manual
File patterns: automatic (but disabled by TRANSFER-MODE MANUAL)
File type: binary

The modern assumption is a byte-accurate transfer of files. We have had a proliferation of file types, extensions, and complicating circumstances since the earlier days of Kermit.

By changing this default, we disable the heuristic for attempting to guess the type of files, and convert the existing binary default into a binary setting.

This can always be changed by the user, but the idea is to not violate the principle of least surprise. If the user asks to transfer a file, we assume the user wants an exact transfer of the file unless stated otherwise.

Additionally, some platforms (eg, HP48 calculators) have wildly different behavior depending on whether a text or binary transfer is requested. By defaulting to manual mode, the user is in charge and further surprises that may be caused by "set file type" being ignored can be avoided.

[jaltman]

I asked @kermitfdc about this proposal and this is what he wrote in response.

The default transfer mode is "auto", where it uses an time-tested algorithm
to look at the contents of each file to see if it contains only text
characters, in which case the xfer mode is text, otherwise it's binary.
Text is sent as ascii lines terminated by CRLF, and all other Kermits
understand this and convert to their own text format, e.g. Unix (CR instead
of CRLF), IBM OS/MVT (which also converts from ASCII to EBCDIC (did I forget
how to spell that?)); this scheme has worked smoothly for decades and
changing it would be nasty surprise for most users, would break countless
scripts and init files, etc.

It's especially important to keep this behavior when sending a group
of files where some are binary and others are text. Kermit automatically
switches mode for each file. (I'm sure you know all this, so I'm just
being careful.)

[jgoerzen]

I appreciate Frank's perspective!

But, we are in a different era now than when ckermit was initially written. Performing those changes, silently, by default, is interpreted as corruption by modern systems even though, in a different era, it was helpful. Cryptographic signatures will fail to verify, hashes will change, etc.

I would feel quite embarrassed to make a big announcement of a new version of file transfer software that is known to corrupt files by default.

Yes, even if it's a .txt or .c file or whatever, changing the file size and bytes that make it up will be interpreted as corruption by lots of systems. A property where files that are transferred by sftp, zip, tar, 7z, etc. have one content, and those sent by kermit have another, is also strongly undesirable for systems that do deduplication (backup systems and filesystems like btrfs and zfs) and those that do content-based hardlinking.

My point here is that even if it contains only text characters, changing the content of the file is a big deal on modern systems and with modern security systems.

I would absolutely think this change is something to be highlighted in the release notes. Fortunately it is very easy to workaround with a single line consisting of SET TRANSFER MODE AUTO in .kermrc or other scripts.

The fact that even I did not realize that SET FILE TYPE BINARY was insufficient to prevent corruption even after years of using kermit is telling.

Additionally, it looks like behavior around this has changed in the past. https://www.kermitproject.org/faq-c-bix.html says "Both Kermit and ftp transfer files in text mode by default." That's not the case in the current Kermit 10 betas; I don't know when it changed.

That page also says "You can tell Kermit to skip all conversions and transfer the file literally, as-is, with the command SET FILE TYPE BINARY", which is not correct because the trransfer mode can override that by default.

This has clearly been a source of confusion for many people for many years.

I confirmed just now, that using the latest Kermit 95 to the latest ckermit on Linux, sending a text file with a .txt extension -- even with the file type set to binary -- produces output that has fewer bytes on Linux than it did on Windows.

I myself have appreciated the text transfer mode and of course this doesn't take it away. My HP-48GX calculator did something wildly different when in text vs. binary mode (basically, sending the source code of a program vs. bytecode of it) and of course sometimes the automatic CRLF conversion is helpful. It shouldn't go away! Just that the defaults should be suitable for a modern environment, recognizing that the passage of time has changed the default-conversion approach from helpful to corruption.

[jgoerzen]

I also just observed that with the default settings, a file saved in UTF-16 on Windows had its BOM stripped and was converted to an 8-bit encoding (unclear of it was UTF-8 or latin-1 or what, since it didn't use any non-ASCII characters) on Linux.

When transferred back to Windows, it was not converted back to UTF-16. The round-trip, therefore, was also lossy.

SET FILE TRANSFER MODE MANUAL prevents this issue as well.

[jaltman]

https://www.kermitproject.org/faq-c-bix.html says "Both Kermit and ftp transfer files in text mode by default."
That page is circa C-Kermit 5A and has not been updated since. C-Kermit 5A pre-dates SET FILE TRANSFER MODE.

SET FILE TRANSFER MODE is documented on page 216 of Using C-Kermit 3rd Ed. https://www.kermitproject.org/onlinebooks/usingckermit3e.pdf

The behavior you are observing when transferring between Windows and UNIX is expected because the platforms are different, the permitted characters in file names are different, the permitted character sets are different, etc. Note that TEXT and BINARY are not the only transfer modes. There are also IMAGE (aka BLOCK) and LABELED modes. LABELED mode is used for OS/2 to OS/2 transfers and VMS to VMS transfers.

The lists of well-known file extensions in cluusx.c has not been updated since 2003. Please propose additional file extensions for UNIX (including macOS) and Windows. The Microsoft office XML formats are obvious additions.

[jgoerzen]

Let me back up a moment to the big picture.

To assume all the above-documented features enabled by default is a good idea, we have to believe all of these propositions:

1. The primary purpose of sending files to a different platform is to interact with them on that platform
2. The identity of the sending and receiving platforms is sufficient to determine the proper line-ending encodings
3. Interaction with text files from a different platform requires assistance in transforming them
4. The file transfer tool is the appropriate place for this, and default-on does not violate the principle of least surprise
5. The benefit of these outweighs the harm

None of these are clearly true, and most of them are clearly false.

The primary purpose of sending files

In many cases, the primary purpose of sending text files between platforms is not to interact with them on the target. For instance, if you are running Windows and have mounted a network share, that network share might be hosted on Windows or Linux (samba), and the files may never ever be accessed except on Windows clients.

Likewise if ckermit is used to send text files to a file server, or a backup server, or whatever, it is quite possible -- and, these days, probably even likely -- that the intent is not to interact with them on that system, but simply to store them on that system. For instance, I transfer files to a hosting directories for web, Gopher, and Gemini on a server. I never look at that on that server itself; they are only viewed by clients (which may be of many platforms)

Most transfer and synchronization software tries to be platform-independent and preserve your content regardless of the client and server platforms.

Determining line-ending and character set encodings

ckermit currently appears to assess what line endings to use based on what platforms are at the sending and receiving end. This is incorrect, because any given system may contain files that originated on any other given system, and moreover may contain files intended for any other given system, in any combination.

My file server contains text files originating on DOS, modern Windows, MacOS, and *nix systems. It would be an impossibility for ckermit to correctly determine what encoding to use for a file placed in a given directory. It may be able to guess the encoding of a file by examining it, but even that gets dicey with character sets.

Interaction with text files requires assistance

Every modern editor for text files, from vim on up, supports autodetection and preservation of line endings. Granted, cat on Unix may do the wrong thing with a file from Windows or Mac, but these issues are both routine and easily solved these days.

The appropriate place for this, and the principle of least surprise

I submit that the transfer tool is the wrong place for this (by default). Nobody expects that these days. Here is a partial list of software commonly used to exchange files that does not perform this translation:

• rsync
• x/y/zmodem
• UUCP
• NNCP
• scp
• sftp
• SMB/samba/CIFS
• NFS
• Dropbox
• Syncthing
• Nextcloud
• Google Drive
• Amazon S3
• tar
• dar
• xz
• rdiff
• unzip (has optional text file conversion, off by default)
• curl, wget, other http clients (curl supports ASCII mode for FTP but it defaults off)
• gzip, bzip2, zstd, etc.

In fact, the only other one I can think of that does this by default in some circumstances is some very old FTP clients. Modern ones like lftp and ncftp use binary mode by default.

So this /definitely/ violates the principle of least surprise. Tools like dos2unix are widely understood to perform the conversion for those cases where it's needed, but it rarely is anymore.

The benefit vs. the harm

The harm is significant.

It will cause severe breakage to:

• content-addressible (hashing) systems, such as git-annex
• Cryptographic signature verification tools
• Tools that create or apply binary deltas (rdiff, xdelta, dar)
• Some version control systems
• rsync (will cause it to perceive a file to be different and generally require a full retransmit since the typical line size is less than the typical rsync block size)
• syncthing
• many other syncing tools may perceive a conflict

There is utility to this conversion, especially for very old systems or ones that don't have what amounts to the typical notion of a file or one of the big three line-ending approaches (Microsoft, Mac, and Unix). But I don't see much utility, and a significant amount of harm, to it being enabled by default. As someone that's had to get data off an AS/400, which neither uses ASCII nor has a standard notion of a file, I get it. But I'd rather request that mode explicitly than have an algorithm decide it for me -- especially when that algorithm may change between releases.

As for an extension map, I think that is a losing battle. If that is truly the desire, libmagic is probably a better bet (it's been around for a long time but may not be on all platforms)

Conclusion

Is kermit going to be an 80s tool, locked in 80s conventions and primarily for interacting with 80s systems? Or is it something that can be more relevant in 2025? I believe the latter and use it that way. This change doesn't block the ability to use it for all those 80s things; it just gives it a default behavior that adheres to modern and Unix expectations (don't modify the file unless I specifically tell you to)

[jaltman]

The example set of software commonly used to exchange files you provided is quite broad ranging from tar/zip (archive formats), rsync (a program designed for synchronizing file trees on a single operating system: Linux), to object stores (S3 buckets), to distributed file systems, to file transfer protocols (X/Y/Zmodem) [well I'm not going to comment further, you know where to find references to the failings of those protocols.

I departed the Kermit Project in December 2002 and found a new home in 2004 as a leader of the OpenAFS distributed file system and in particular was the lead developer of a native Windows client implemented as an Installable File System driver. I know a bit about Windows and how file systems are required to expose file names, extended attributes, alternate data streams, and other meta data. I also contributed to the development of the macOS client.

Internally Windows represents file names using UCS2 (a variant of UTF16) and a particular version of Unicode. And there are explicit rules about how names must be normalized. Windows requires that normalization to "composed characters" must be performed. If the normalization is not performed correctly, the file name will display appropriately in a shell or in the File Explorer but when the user attempts to copy/paste the name and open the file or perform a search using the name, it will fail to find a match.

macOS X requires that all file names be stored as utf8 but the normalization rules for that platform require that all composed characters be decomposed and there are subsequent rules for how decomposed characters must be constructed in order to avoid the possibility that two file names with the same display name can appear in a directory.

Linux permits but does not require the locale to be UTF8. The Linux philosophy about file systems is that everything is a binary blob and it doesn't matter to the file system that its possible for the same display string to be represented in UTF8 in a dozen different ways. As far as Linux is concerned, its the user's problem if the end users are vulnerable to homograph attacks.

In a distributed file system, if every client is permitted to store file names using their local encoding rules, the end result is that in most cases the file names can be displayed on other operating systems but it might not be possible to open the files, or worse the wrong file is opened, or a homograph of the desired file is created.

The way that AFS makes this work is by implementing its own rules for UTF8 name canonicalization. Windows and macOS clients translate file names to that form and translate it back to the form required by the local OS. Of course, on Linux there are no rules and play with fire.

Beyond the Unicode name form issues, every OS also has a set of characters which are reserved and may not appear in a file name. The set of characters is quite broad for Windows.

How is this relevant to C-Kermit and automatic file transfer mode? Because Kermit FTP doesn't only decide if the data stream should be converted, its also responsible for deciding if and how the file names must be converted.

UNIX/Linux, macOS, VMS, OS/2, DOS, Windows, etc are each assigned a SYSID. The SYSID along with file capabilities are exchanged between the two endpoints. In most cases, if the SYSIDs match then the OS native transfer mode is used and there is no character set translation applies to the data stream.

Many people think of macOS as UNIX but from a file system perspective there are significant differences. Not only are file names required to be a particular canonicalized form of UTF8 but files also contain resource forks. Transferring a file without its resource fork is damaging the file. Therefore, Kermit FTP has a different transfer mode for macOS to macOS then is used for Linux to Linux. Disabling automatic transfer mode, will prevent the inclusion of resource fork contents in data transfers.

I don't remember two what extent support for Alternate Data Streams was added to K95. On Windows NT, a file may contain more than one data stream. There is a primary unnamed stream and secondary named streams. When transferring a file on Windows NT, all of the data streams should be transferred and not just the primary stream.

One mistake that I think was made in 1995 when OS/2 C-Kermit was ported to Windows is that a single SYSID was assigned covering both the Windows 95/98/ME family and the NT4/2000/... family. The file system properties of the two worlds are quite different. Windows 95/98/ME is best described as FAT-32 restrictions whereas DOS is described by FAT-16 restrictions. Windows NT requires quite a bit more flexibility because there underlying properties of the active file system must be used because there isn't a global set of properties which applies to the whole OS.

I believe what you are asking for is that Windows NT and Linux be considered as equivalents. Or at least sufficiently equivalent that character set translation of data stream contents should be disabled. Regardless, disabling the C-Kermit automatic transfer detection logic with the current knobs is not the answer. It might "fix" the Windows to Linux transfers which you care about but its also going to break many other use cases that C-Kermit end users have been relying upon since C-Kermit 6.x.

I've been away from Kermit for more than 20 years and my knowledge of the inner workings of the Kermit FTP is quite rusty. I do not yet understand the repercussions of adding a new SYSID for NT; nor do I remember how to add capabilities such as "case sensitivity" to the negotiated properties. This was Frank's area of expertise but unfortunately he is no longer available to participate in such a technical discussion.