                            WHAT'S IN A PROTOCOL?
                                By Greg Gilley

                  (Reprinted from the Austin TI Professional
               Computer Users Group Newsletter, February, 1987)

             A file transfer by any other name would take as long

        This is the first of two parts on microcomputer file transfer
        protocols.  I will attempt to give an overview of the protocols
        and some of their good and bad points.  This part will cover
        XMODEM and its derivatives: MODEM7, TELINK, and YMODEM. The
        second part will cover KERMIT, ZMODEM AND WXMODEM.  I have been
        meaning to write something for the newsletter for some time now.
        I just never could come up with a topic that I felt would be
        interesting to the readership.  Someone finally suggested that I
        consider file transfer protocols and that sounded like a good
        idea.

        XMODEM

        XMODEM is the granddaddy of the microcomputer file transfer
        protocols. It was invented by Ward Christensen, the same person
        who created microcomputer bulletin boards. XMODEM was originally
        developed on a CP/M based micro-computer in 1977.  Many of its
        characteristics are a direct consequence of that operating system
        and early microcomputer hardware.  XMODEM transmit data 128 bytes
        at a time (the size of a disk sector under CP/M) along  with a
        block number and a one byte checksum for error detection. It has
        no provision for passing the file name between the machines;
        originator and receiver must both specify the name.  Since CP/M
        did not support time and date stamping of files,  no provision
        was made to transfer the file's date and time.  CP/M only
        provided system calls to do I/O a sector at a time; therefore,
        all file sizes were a multiple of 128 with a CTRL/Z character to
        mark the end of a text file. Because of this XMODEM always rounds
        the size of the files being transferred to the next 128 byte
        boundary.  This results in up to 127 garbage bytes added to the
        file.  This can wreak havoc with files on other systems that do
        not conform to this convention.

        XMODEM is essentially a half-duplex protocol.  It assumes that
        the computers cannot send and receive data at the same time.
        Most of the microcomputers around when it was invented could only
        do polling to see if they could send a character or to find out
        if one had been received.  Most of today's computers use
        interrupts to stop what the computer is doing, send or receive
        the next character, and then resume the processing.  With
        polling, the computer must continually ask the I/O device if it
        is ready to send or receive a character. Another reason for the
        half-duplex operation was that most of the older machines could
        not receive data and write to the disk at the same time.  If data
        was sent continuously, all that arrived while the program was
        writing the last block to disk would be lost.  XMODEM will send a
        block of data and wait for the receiver to acknowledge receipt of
        the block. This gives the receiver a chance to write the data to
        disk before more arrives.  If the receiver does not receive the
        block correctly, it NAKs the block and the sender will retransmit
        it.  Because of its half-duplex nature, XMODEM is susceptible to
        and in fact causes problems with delays over packet switching
        networks such as TYMNET, TELENET and those used by CompuServe.
        It can usually be tweaked so that it can live with network
        delays, but transfers are excruciatingly slow.

        Some other problems with XMODEM are that it only uses an 8 bit
        checksum for error detection.  This means that we could expect 1
        out of every 256 packets with errors to have a valid checksum and
        go undetected.  This was corrected with XMODEM/CRC which uses a
        16 bit CRC (cyclic redundancy check) to detect transmission
        errors.  Using a 16 bit CCITT standard polynomial equation, CRC
        error detection is theoretically accurate 99.9965 percent of the
        time.  Another problem with XMODEM is that it is easily corrupted
        because of its simplicity.  Phone lines noise or extra delay can
        causes transfers to abort.  So, with all of these problems, why
        is XMODEM so popular?  Because it is easy to implement and
        because it is public domain with many sample implementations
        readily available.  Without XMODEM, the current explosion in
        microcomputer telecommunications would have been much longer in
        coming.

        MODEM7

        MODEM7 is a simple extension of XMODEM. One problem with XMODEM
        is that the file name must be specified by both the sender and
        receiver of the file.  XMODEM can also transfer only one file at
        a time.  MODEM7 corrects this problem by passing the file name
        from the sender to the receiver before sending each file.  This
        also allows wildcard transfers to occur such as "*.EXE" which
        would send all files with an EXE extension.  However, the file
        name is transferred one character at a time and its transfer is
        unreliable.  Since MODEM7 also originated on CP/M systems, it
        still did not transfer the file date and time.  It suffers from
        all of the same problems mentioned  for XMODEM.  MODEM7 is
        popular only on CP/M based machines.

        TELINK

        The TELINK protocol was created by Tom Jennings, author of the
        FIDO bulletin board system.  It is an extension of MODEM7. In
        addition to the file name, TELINK also provides for the transfer
        of the file's date and time.  The first block transferred after
        the name contains the file size in bytes and the file date and
        time.  Because it transfers the file's exact size, the receiving
        computer can truncate any garbage bytes that the protocol appends
        to the last block.  It also allows maintaining the file's
        original date and time.  A problem with TELINK is that it still
        uses MODEM7's convention for passing the file name. Another is
        that the file date and time is in an MS-DOS specific format and
        does not take the different time zones into account.  The format
        could be transferred to another machine's format, but it would be
        awkward and transfers between time zones would still be wrong.
        TELINK is, in general, not a widely supported protocol. FIDO
        bulletin boards and MINITEL, a comm program also by Tom Jennings,
        support it as well as TIgger and TERM.

        YMODEM

        YMODEM is an XMODEM extension created by Chuck Forsberg.  It
        allows batch transfers and also supports transferring the file's
        exact size, date and time.  It does not use the MODEM7 convention
        for transferring the file name, but instead puts the file's name
        and other data in the first block to be transferred.  The date
        and time are stored as a 32 bit quantity signifying the number of
        seconds since Jan 1, 1970 in GMT (Greenwich Mean Time).  This
        allows receivers in different time zones to adjust the file's
        date and time correctly.  YMODEM also provides an extension to
        the block size of XMODEM. YMODEM allows transfers to use 1K
        blocks (1024 bytes) for data.  This reduces the number of times
        the sender has to wait for the receiver to respond to a block.
        This help reduce the overall transfer time.  On noisy phone
        lines, however, YMODEM can actually increase the time of the
        transfer because the blocks take longer to retransmit.  It is
        especially vulnerable at lower baud rates where the probability
        of noise occurring during the transfer of a block is higher.
        YMODEM also has a non-batch mode where it does not transfer the
        file name, date and time, but does allow using 1K blocks to
        transfer the data.  YMODEM is gaining popularity, but
        unfortunately most implementation are non-batch  and therefore
        lose most of the benefits.

        CONCLUSION

        XMODEM helped to lay the foundation for the microcomputer
        telecommunication revolution.  It is dated, however, and should
        be replaced. MODEM7, TELINK, and YMODEM while correcting some of
        the problems with XMODEM, still retain many of its fundamental
        characteristics and are not a long term solution.  Next time,
        I'll talk about some of the possible successors to XMODEM:
        KERMIT, WYMODEM AND ZMODEM.  TTFN.

                             <<<<<<<<<<>>>>>>>>>>


                      WHAT'S IN A PROTOCOL?  THE SEQUEL
                                By Greg Gilley

     (The second installment of the file transfer article which appeared
       in the March issue of The TImes.  Reprinted from Austin TI Pro.
                 Computer Users Group Newsletter, March '87.)

        Last time, I talked about XMODEM, MODEM7, TELINK and
        YMODEM.  This time  I'll try to give you some background in
        KERMIT, WXMODEM, and ZMODEM.  There are a number of other
        protocols available, but most are proprietary and really aren't
        worth discussing.  Now that the social amenities are out of the
        way, lets jump in with both feet.

        WXMODEM -  WXMODEM was developed by Peter Boswell to try to
        overcome problems caused with XMODEM by network delays.  It
        uses the "sliding window" to reduce network delays.  This means
        that instead of waiting for each packet to be acknowledged by the
        receiver, up to four packets may be outstanding before an
        acknowledgement is required.  This allows time for the normal
        network delays that would occur for the "ACK" packets.  The ACK
        packets include a window index to tell which packet is being
        ACKed.  (Note: ACK stands for "acknowledged", NAK for "not
        acknowledged". ed.)  The data packets are normal XMODEM packets
        with the CRC (cyclic redundancy check) error checking extension.
        The characters XON, XOFF, SYN and DLE are quoted to allow
        XON/XOFF flow control so the network can keep up with the
        sender and receiver.  It does require an 8-bit data channel to
        operate.  WXMODEM does help reduce network delays, but is
        susceptible to noise.  It also does not support batch (wildcard)
        transfers, exact file size, or file date and time.  This is
        unjustifiable in a recent protocol definition.  WXMODEM is sure to
        be a short lived protocol.

        KERMIT - The KERMIT file transfer protocol was originally
        developed at Columbia University by Franca da Cruz and Bill
        Catchings.  It was designed to handle a specific problem,
        transferring files between microcomputers and mainframes.  It does
        a good job of solving the problem, even with brain damaged IBM
        front ends that run only at half-duplex.  It is, necessarily, very
        complex.  It provides for transferring 8-bit characters across serial
        lines that support only 7-bit data transfers.  It does this by using
        a quote character to precede each character that has its 8th bit
        set.  Some quick figuring should tell you that a file with all
        characters greater than (ASCII) 127 would double in size.  Control
        characters (less than 32: ie. backspace, carriage return, linefeed,
        etc.) are also quoted to keep the network or operating system from
        eating them.  KERMIT has three error checking types:  6-bit
        checksum, 12-bit checksum, and 16-bit CRC.  The 6-bit checksum is
        the normal default for file transfers, but use of the CRC is
        recommended to reduce the likelihood of undetected errors.
        KERMIT knows the difference between binary data files (like
        executables) and text files.  There is a standard format for
        transmitting text files which allows conversions to be performed
        between systems.  A DEC VAX, for instance, uses a zero to
        terminate lines; UNIX uses a line feed; MSDOS, a carriage return
        and a line feed.  KERMIT makes transfers between these systems
        much simpler.  It also has provisions for supporting many different
        types of file attributes such as fixed length records or encoding in
        EBCDIC (a character encoding scheme like ASCII that was used for
        punched cards).  It supports exact file size, file date and time
        transmission, and wildcard transfers.

        KERMIT does have some drawbacks, however.  KERMIT data
        packets are restricted to a maximum of 95 characters.  This small
        packet size combined with the character quoting means that
        KERMIT has a lot of overhead to slow the transfer down.  Since
        every packet must be acknowledged by the receiver before the next
        one can be transmitted, KERMIT is unbearably slow over a network.
        One way to reduce the network delays is the addition of "sliding
        windows."  Sliding windows simply means that there is a "window"
        of up to 32 packets which may be transmitted before an
        acknowledgement is required.  This means that ACKs which would
        be delayed by the network aren't waited for, but are processed as
        they arrive.  This means the sender can continuously transmit
        packets and the speed of transfer increases accordingly.  Another
        extension that helps to reduce overhead is to transmit characters
        with their 8th bit set unquoted on 8-bit data channels.  Since most
        microcomputer connections are with 8-bit data channels, this (with
        sliding windows) makes KERMIT more viable as a general purpose
        file transfer protocol.  Another proposed extension would allow
        data packets to be longer than 94 bytes.  This would allow
        throughput increases for those systems which cannot support the
        complexity of the sliding window mechanism.

        KERMIT has another nice feature.  It has a server mode where a
        client (that's you) issues commands, such as file transfers, and the
        server executes them.  This means the file names are typed in only
        once.

        ZMODEM - ZMODEM was developed by Chuck Forsberg, the
        inventor of YMODEM.  It really has no relation to XMODEM other
        than the last five characters of its name.  ZMODEM was developed
        at the request of some network services to help reduce delays
        inherent in file transfers.

        ZMODEM is based on ideas derived from X.PC and KERMIT.  It
        allows continuous transfers with ACKs/NAKs occurring only when
        an error is detected.  This permits ZMODEM to transfer data
        without network delays at almost the full speed of the modem.  It
        supports variable block sizes to allow it to adjust to the speed of
        the connections.  It uses larger blocks at higher speeds to reduce
        overhead.  ZMODEM has a lot of features which means that it is a
        fairly complex protocol.  It supports wildcard transfers, exact file
        size and file date and time.  It can perform selective updating,
        transferring only those files with more recent dates than those on
        the receiving system.  It also supports restarting transfers that are
        aborted in the middle.  It can perform remote command execution
        and transfers the results back to the sending system.  This works
        best in an environment like UNIX or XENIX, but could also be used
        on a DOS system with appropriate software.

        ZMODEM performs error checking with a 16-bit CRC.  A recent
        addition to the protocol allows the use of 32-bit CRC checking for
        transfers that require the highest integrity.

        CONCLUSION - Before I wrap all of this up, let me confuse the
        issue a little bit more by talking about a couple of other protocols.

        One that you might be familiar with is the protocol used by
        CompuServe:  CIS-B.  This is a proprietary protocol, although they
        have recently released example source code implementation of the
        protocol.  I guess they realized their fortunes were being made
        from people who transferred files, and not from selling their
        VIDTEX software packages.  It is developed specifically to reduce
        problems with transfers over networks by using packets up to 1024
        bytes in length and ignoring delays between characters within a
        packet.  The CompuServe host transfers the name to the sending or
        receiving system which means that the name needs to be specified
        in only one place.  It also transfers files with the exact size,
        although date and time stamping are not supported.  For transfers
        on CompuServe, I would recommend using CIS-B.  It is much more
        reliable and faster than doing XMODEM transfers with CompuServe.

        Something else I should mention is X.PC.  X.PC is a microcomputer
        implementation of X.25, a mainframe networking protocol.  It
        supports multiple simultaneous "channels."  This means that on
        systems that support X.PC (like TYMNET, the creator of X.PC) you
        can be connected to more than one system at a time.  X.PC is not
        a file transfer protocol, it is a data transport layer.  It is a means
        of getting data from one place to another.  It handles error
        correction, and retransmission of error packets.  All you do is feed
        it data and it makes sure it arrives at the other end.  X.PC is
        slowly gaining a little popularity, but it still has a long way to go.
        It is a very complex protocol, and, although TYMNET supplies C
        source code for a copying fee, is still a significant effort to
        implement, especially on non-standard (read TIPC) hardware.  It
        may gain more popularity in the future if someone comes up with a
        microcomputer bulletin board that supports it.  It will require a file
        transfer protocol to operate in conjunction with it.  ZMODEM
        would make an excellent candidate.

        Another data transport layer is MNP.  MNP operates in much the
        same manner as X.PC:  Feed it data and it gets it where you want
        it, error free.  It is, however, completely proprietary.  The only
        thing going for it is that the people who created MNP have sold it
        to modem manufacturers and they have implemented it in their
        modems.  This means that if you connect two MNP modems
        together (must be one on each end of the connection), then you
        have essentially an error free connection.  Very nice for those of
        us with noisy phone lines.  It is, however, not a file transfer
        protocol, and would require one to work in conjunction with it.
        Once again, ZMODEM comes to mind.

        "So, what does all this mean!?!" you may scream!  Well, let me drag
        things out a little further by adding that you still can't transfer
        data any faster than the modem can.  The absolute minimum time
        to transfer one megabyte of data at 2400 baud is about 70 minutes,
        at 9600 baud this is reduced to about 18 minutes.  Although some
        file transfer protocols help approach the theoretical limits of the
        transfer rates, they won't do as much to help as data compression.
        Data compression will reduce the transfer time by reducing the
        number of bytes that need to be transmitted.  ARC, SQ (CP/M
        squeeze), and ZOO will do more for reducing transfer times than
        any of the protocols.  In fact, some of the so-called high speed
        modems simply have a 68000 in them doing compression like one of
        these programs.  Try running an ARCed file through one of them
        and see what the transfer time really is.

        In conclusion (<CHEER>), my OPINION about all of these file
        transfer protocols is this:

        For micro to mainframe transfers, transfers between systems
        with strange file formats, or transfers over 7-bit data
        channels, the winner is KERMIT.  The sliding windows
        extension makes this protocol one that should survive for
        some time to come.

        For best all-around micro to micro transmission:  ZMODEM.
        This is a very robust protocol and seems to work well even
        under noisy environments.  It has hooks for extendibility and
        I see it going no place but up.

        What does the future hold?  I know better than to try to predict
        anything that has to do with the computer industry.  Those
        "industry analysts" who get paid for throwing darts aren't doing
        well, and I'm sure not going to join them.  I hope I have given
        you enough to go on to draw your own conclusions.  I know I
        haven't been completely objective, but I have tried to give the
        reasons that I wasn't.  In case you were wondering, the next
        release of TERM should have support for CIS-B, ZMODEM, and
        KERMIT with sliding windows.  Nothing like covering all bases.

                        <<<<<<<<<<<<<<<>>>>>>>>>>>>>>>
