The Wikipedia article Ancestry.com has a table under the section heading "FTM version history" which lists the Banner Blue software as the earliest versions from 1989-1994 (pre-Broderbund), running under DOS and Windows 3.1 (FTM on the DOS version, FTW on the Windows version). (Note that the authors of the Wikipedia article are unaware of a Mac version from Banner Blue; the earliest listed is 3.02 Mac for PowerPCs under OS9 from Broderbund. So no clues there.)
One of the references in that article points to a post on alt.genealogy from October 1996 which says that
Exporting from Family Tree Maker for DOS uses the Data Exchange
Utility. The DEU ships with Family Tree Maker version 3 and 4 for
DOS. It had to be purchased separately for earlier versions.
WorldCat.org has an entry for the Family tree maker : data exchange utility version 2.0 : reference manual. (2nd ed. published in 1991) which is in the collection at the Allen County Public Library in Fort Wayne, IN. Perhaps the librarians could tell you whether the specs for the FTM file format are in the manual.
There was also a utility for reading the older files from the Family Archive CDs and downloaded World Family Tree files, called the Family Archive Viewer, which later versions of FTM can't read. But the pages on Genealogy.com don't say what file format those files were in, so it's not clear whether that reader would also open your old Banner Blue files.
Another option, depending on whether you have access to an emulator or an older computer, is to search eBay, Amazon, or other sites on the net for the older versions of the software. As of this writing, I can find listings for copies of the Broderbund version of FTM (for Win3.1/Win95 and (apparently) for Mac OS 7.0. Copies of FTM that were closer in time to your 2005 edition might be able to read or convert your old files.
Since you've been able to recover some of your data from your uploaded copy of your tree with the Family Archive Viewer, you now have a list of some of the names that appeared on your tree. If you enter these names into the 2005 version of FTM, you'll have a copy of your tree in the modern FTW format. Could you then use a Hex editor to compare the files side-by-side? If the names are in the clear (a big assumption, I know), it might yield some clues about the file format of the old version.
Edited to add this section for GenBridge users:
On 29 July 2014, the developer of The Master Genealogist announced that he has decided to discontinue the Master Genealogist. He says:
In the interest of preserving users’ data, I have released a document
that details TMG’s internal file structure and I will make GenBridge
available for free to developers who wish to produce a direct import
from TMG insofar as their programs support the same features.
Support forums will continue, but official support for TMG will end at the end of 2014.
Since many sections of the site Genealogy.com have been retired by Ancestry, the link to the page about the File Archive Viewer has been changed to an archived version on the Internet Archive's Wayback Machine (snapshot taken 31 Aug 2014). This allows us to read the instructions and notes showing the names of the executables, but the link to download the executables is broken.
Family Archive Viewer update
Download Family Archive Viewer from the Wayback Machine here. Be sure to read the tips on the download page. (This appears to be the most recent screen capture from February 27, 2015.)
Download Family Archive Viewer from S&N Genealogy Supplies (the retail sister site for TheGenealogist.co.uk).