# wvWare 2 Design Document

## Design Goals

- The filter should depend on as few libraries as possible. Right now it depends on libgsf-1, which in turn depends on glib-2 and libxml2. Additionally wv2 needs a working iconv installation (which is present on almost all systems).
  Apart from that we need a working C++ compiler (e.g. gcc-2.95.2 or newer). Developers might also need Perl 5 or newer (to regenerate the code), reasonably new versions of automake, autoconf, and libtool. If you have some Doxygen version you can generate the API documentation yourself, using the doxyfile in wv2/doc.
  To use the regression test script you need a working Python installation.
- We try to get the filter as portable as possible (without major pain). It has been tested on various architectures like x86, PPC, Alpha, and Sparc and on several operating systems. The library compiles without any problems with recent gcc versions, with Intel's C++ compiler, and MS Visual C++ 7. Unfortunately I wasn't able to test it with other compilers, but any modern compiler with a working STL should do.
- To allow different use cases (like importing a file to some word processor or just outputting a bit of plain text for a preview) we try to provide a layered API (low level/high complexity vs. high level). Right now we concentrate on the low level API, later on we'll add the high level interfaces.
- Albeit the Word file format is crazy and broken we still aim to have readable and sane code. The API should be easy to use and well documented (using Doxygen tags). We also fix the buggy SPEC whenever we detect bugs or ambiguities. If we are not sure about our changes and fixes of the SPEC we don't change the HTML file, but add an entry to src/generator/spec_defects.
- The "old" WinWord filter of KOffice reads the information "just in time" which makes the code a bit crowded and hard to follow. wv2 reads the important information like stylesheets, list information,... before we start to process the main document body. This way the code should get a bit simpler (but we'll use a bit more memory while parsing the document).
- We assume the filter is used as in-process component and the API is designed for this purpose. In order to make it possible to dlopen/dlclose the filter we try not to use lots of static objects (as there are problems with destructing them on unloading). Please inform us if you experience problems in that area, it's very likely that we can fix them.
- For all important components of this library self-checking unit and function tests are recommended. It's also recommended to check the code using Julian Seward's tool valgrind. Additionally the library comes with an automatic regression test utility in the `tests` directory.
- We aim to reuse code and share the parser/utility code between parsers for different Word versions. This goal mainly is a design task and requires a lot of knowledge about the formats.
- The library is not designed to work in a multi-threaded environment. If you need non-blocking filtering you will have to use a separate process (note that I'm no threading expert, but I definitely know that a thread safe library doesn't just "happen").

## Directory Structure

Not a huge package, but in case you get lost:

Path | Contents
wv2 | Holds some build system stuff and general build information.
wv2/doc | Here we keep some information for developers and a Doxygen file to generate the API documentation.
wv2/src | Contains 99% of the sources. As we don't want to have a build-time dependency on Perl we also added the generated Code to the CVS tree.
wv2/src/generator | Two Perl scripts, some template files, and the available file format specification for Word 8 and Word 6. This stuff generates the scanner code. If you finished reading this document you might want to check out the file format spec in this directory.
wv2/tests | Mainly self checking unit tests and function tests for the library. Use "make check" to build them.

## Design Overview

Viewed from far, far away the filter structure somehow looks like that:

A Word document consists of a number of streams, embedded in one file. This file-system-in-a-file is called OLE structured storage. We're using libgsf to get hold of the real data. The filter itself consists of some central "intelligence" to perform the needed steps to parse the document and some utility classes to support that task. During the parsing process we send the gathered information to the consumer, the program loading the Word file (on the right). This program has to process the delivered information and either assemble a native file or stream the contents directly to the application.

#### OLE Code

The interface to the documents is a C++ wrapper around the libgsf library. libgsf allows -- among many other things -- to read and write OLE streams from and to the document file. It would be rather inconvenient to use it directly, so we created a class representing the whole document (`OLEStorage`), and two classes for reading and writing a single stream (`OLEStreamReader` and `OLEStreamWriter`).

`OLEStorage` holds the state of the document and allows to travel through the "directories." It also provides methods to create `OLEStreamReader` and `OLEStreamWriter` objects on the document.

`OLEStream` is the base class for `OLEStreamReader` and `OLEStreamWriter`, providing the common functionality, like seeking in the stream and pushing and popping the current cursor position.
The `OLEStreamReader/Writer` classes provide a stream-based API, although we don't use the stream operators (operator<< and operator>>). Using the stream operators would be very inconvenient, as we often would have to specify the exact type we want to read or write to/from a variable of a different type.

This part of the code contained in the ole* files is generally straightforward, but as libgsf is a lot stricter than libole2 some of the functionality is gone (e.g. you can't browse the contents of a directory in a file you write out, you can't open an OLE storage for reading and writing,...).

#### API

The external API for the users of the library should consist of at least two, but maybe more, layers. Ranging from a low level and fine grained API where lots of work is needed on the consumer side (with the benefit of high flexibility and enormous amounts of information) to a very high level API, basically returning enriched text, at the cost of flexibility.

Another main task of that API is to hide differences between Word versions if that's feasible. In any case even the low level layer of the API shouldn't expose too much of the ugliness of Word documents. For the time being we chose to make every document look like it's a Word 8 (aka Word 97) one to the consumer. For Word 6 or newer this seems to work, and I think it's possible to do the same for older Word versions. In the unlikely case that Microsoft releases a more recent file format specification (e.g. the specification for Word 2002) we should think about "updating" the API, to provide as much information as possible to the consumer.

Technically the API is a mixture of a good old "Hollywood Principle" API (Don't call us; we'll call you) and a fancy functor-based approach. The Hollywood part of the API can be found in the handler.h file, it's split across several smaller interfaces. We are incrementally adding/moving/removing functionality there, so please don't expect that API to be stable, yet.

The main reason to choose this approach is that the very common callbacks like `TextHandler::runOfText` are as lightweight as possible. More complex callbacks like `TextHandler::headersFound` allow a good deal of flexibility in parsing, as the consumer decides when to parse e.g. the header (also known as stored command). This helps to avoid nasty hacks if the concepts of the destination file format differ from the MS Word ones. The consumer just stores the functor objects and executes them whenever it feels like. For an example please refer to the KOffice MS Word filter in `koffice/filters/kword/msword`.

#### Parser

The core part of the whole filter. This part of the code ensures that the utility classes are used in the correct order and manages the communication between various parts of the library. It's also quite challenging to design this part of the code. Various versions contain similar or even identical chunks, but other parts differ a lot. The aim is to find a design which allows to reuse much of the parser code for several versions.

Right now it seems that we found a nice mixture of plain interfaces with virtual methods and fancy functor-like objects for more complex structures like footnote information. The advantage of this mixture is, that common operations are reasonably fast (just a virtual method call) and yet we provide enough flexibility for the consumer to trigger the parsing of the more complex structures itself. This means that you can easily cope with different concepts in the file formats by delaying the parsing of, say, headers and footers till after you read all the main body text.

This flexibility of course isn't free of costs, but the functor concept is pretty lightweight, totally typesafe, and it allows to hide parts of the parser API. I'd like to hear your opinions on that topic.

The main task in the parser section is to find a design which allows to share the common code between different file format versions. Another important task is to keep the coupling of the code reasonably low. I see a lot of places in the specification where information from various blocks of our design is needed, and I really hate code where every object holds 5 pointers to other objects just because it needs to query some information from every of these objects once in its lifetime. Code like that is a pain to maintain.

For the code sharing topic the current solution is a small hierarchy of `Parser*` classes like this one:

`Parser` is an abstract base class providing a few methods to start the parsing and so on. This is the interface the outside world sees and uses. `Parser9x` derives from that base class and implements the common parsing code for Word 6, Word 7, and Word 8. Whenever these versions need a different handling there are two possibilities: smaller differences are solved via a conditional expression or a if-else construct, bigger differences are solved by an abstract virtual method in `Parser9x` and the appropriate implementation in `Parser97` and `Parser95.`
Therefore `Parser9x` does the main work. It's hard to argue that this is a normal Is-A inheritance, but with a little bit of phantasy it's pretty close.

The whole parsing process is divided into different stages and all this code is chopped into nice little pieces and put into various helper/template methods. We take care to separate methods in a way that as many of them as possible can be "bubbled up" the inheritance hierarchy right to `Parser9x` or even `Parser`.

To keep the coupling between the blocks of the design low the parser has to implement the Mediator pattern or something similar. It is the only block in our design containing "intelligence" in the sense that it's the only block knowing about the sequence of parsing and the interaction of the encapsulated components like the OLE subsystem and the stylesheet-handling utility classes.

#### String Classes

We agreed to use Harri Porten's `UString` class from kjs, a clean implementation of an implicitly shared UCS-2 string class (host order Unicode values). In the same file (ustring.h) there's also a `CString` class, but we'll use `std::string` for ASCII strings.

The iconv library is used to convert text stored as CP 1252 or similar to UCS-2. This is done by the `Textconverter` class, which wraps libiconv. Some systems ship a broken/incomplete version of libiconv (e.g. Darwin, older Solaris versions,...), so we have a configure option `--with-iconv-dir` to specify the path of alternative iconv installations.

The main classes `UString` and `std::string` are well tested and known to work well. Take a lot of care when using `UString::ascii`, though. The buffer for the ASCII string is shared among all instances of `UString` (static buffer)! As we need that method for debugging only this is no problem. `UString` is implicitly shared, so copying strings is rather cheap as long as you don't modify them (copy on write semantics).

Older Word versions don't store the text as Unicode strings but encoded using some codepage like CP 1252. libiconv helps us to convert all these encodings to UCS-2 (sloppy: 16bit Unicode). We don't use libiconv directly from within the library, but we use a small wrapper class (`Textconverter`) for convenience.

#### Utility Classes

To reduce the complexity of the code we try to write small entities designed to do one specific, encapsulated task (e.g. all the code in styles.cpp is used to read the stylesheet information contained in every Word file, lists.cpp cares about -- surprise -- lists,...). These classes are, IMHO, the key to clean code. Classes for the programming infrastructure like the `SharedPtr` class also belong to this category.
We use a certain naming scheme to distinguish code which works for all versions (at least Word 6 and newer) or just for one specific category. All the *97.(cpp|h) files are designed to work with Word 8 or newer, files without such a number should work with all versions (note that there are some exceptions to that rule, e.g. `Properties97` as I was too lazy to mess around with the files in CVS, losing the history).

This part of the code also consists of a number of templates to handle the different ways arrays and more complex structures are stored in a Word file (e.g. the meta structures PLF, PLCF, and FKP). If that sounds like Greek to you it's probably a good idea to read the Definitions section at the top of the file format specification in wv2/src/generator.

#### Generated Scanner Code

It's a tedious job to implement the most basic part of the filter -- reading and writing the structures in Word documents. It is boring, repetitious, error prone, so we decided to generate this ultra-low level code. We're using two Perl scripts and the available HTML specification for Word 8 and Word 6. One script called `generate.pl` is used to scan the HTML file and output the reading/writing code and some test files. The other script, `convert.pl` generates code to convert Word 6 to Word 8 structures. We need to do this, because we want to present the files as Word 8 files to the outside world. The idea behind that is to hide all the subtle differences between the formats from the user of this library. For Word 6 this seems to be possible, no idea if that will work out for older formats.

The generated code mentioned above consists of several thousand lines of code. The design of this code is non-existent, it's just a number of structures supporting reading, writing, copying, assignment, and so on. Some of the structures are partly generated only (like the `apply()` method of the main property structures like `PAP`, `CHP`, `SEP`, and others). Some structures are commented out, as it would be too hard to generate them. These few structures have to be written manually if they are needed.

Generally we just parse the specification to get the information out, but sometimes we need a few hints from the programmer to know what to do. These hints are mostly given by adding special comments to the HTML specification. For further information on these hints, and on the available tricks, please have a look at the top of the Perl scripts. The comments are quite detailed and it should be easy to figure out what I intend to do with the hints.

Another way to influence the generated code is to manipulate certain parts of the script itself. You need to do that to change the order of the structures in the file, disable a structure completely and so on. You can also select structures to derive from the `Shared` class to be able to use the structure with the `SharedPtr` class.

The whole file might need some minor tweaking, a license, `#includes`, and maybe even some declarations or code. This is what the template files in wv2/src/generator are for -- the code gets copied verbatim into the generated file. Never manipulate a generated file, all your changes will be lost when the code is regenerated!

If you think you found a bug in the specification you can try to correct the HTML file and regenerate the scanner code using the command `make generated`. In case you aren't satisfied with the resulting C++ code, or if you found a bug in the scripts please contact me. If you aren't scared by a bit of Perl code feel free to fix or extend the code yourself.

Please note that using the C++ `sizeof()` operator on these structures is dangerous. You should never rely on their memory layout. The reason for that is that the structures in the Word file are "packed", this means there are no padding and alignment bytes between variables. In our generated code we can't achieve that in a portable manner, so we decided not to use it at all. Due to that reading the whole structure in at once doesn't even work on little endian platforms, let alone big endian machines. The solution are the generated `read()` methods. In case you need to know the in-file size of a Word structure, you can add a `sizeOf` variable in the HTML spec (please check the code generation script for more information). It should be obvious that casting memory chunks from a Word file to structures or casting among different structures is also a bad idea. If you really want to create a certain structure from some memory block, please add a `readPtr` special-comment in the HTML spec.

#### Unit Tests

A vital part of the whole library are self-checking unit and function tests, to avoid introducing hard to find bugs while implementing new features. The goal is to test the major components, but it's close to impossible to test everything. Please run the unit tests before you commit bigger changes to see if something breaks. If you find out that some test is broken on your platform please send me the whole output, some platform information, and the document you used for testing.

It's a bit hard to test the proper parsing of a file, the best thing I came up with is a kind of record and playback approach. The Python script `regression` can be used to compare the filter output with some previously recorded output. This tool should be run with the `-r` option before you do any major changes. The created files are quite a detailed recording of the parsing process. After the changes are implemented you re-run the script without the `-r` option. If the result differs you might want to check, whether the difference is intended.

Code-wise there's not much to say about the unit tests. If you add new code please also add a test for it, or at least tell me to do so. The header test.h contains a trivial test method and a method to convert integers to strings (as `std::string` doesn't have such functionality).

If you decide to create a unit test please ensure that it's self checking. That means if it runs till the end everything is alright. If it stops somewhere in between something unexpected happened. Oh, and let me repeat the warning that `UString::ascii()` might produce unexpected results due to the static buffer.

## Pending Design Issues

Currently the filter is in a pretty usable state, it is able to read the text including properties and styles, it handles fonts, lists, headers/footers, footnotes and endnotes, sections, fields (to some extent, it's close to impossible to do anything useful without knowing the target application), and tables. This functionality is tested for Word 97, but I'm lacking test documents for Word 6 and Word 95. In theory most of the mentioned features should work there too, but I doubt that lists work without any problems.

This section of the design document lists my plans for features I'd like to implement next and some ideas about their design.

#### Images and Graphic Objects

Embedded images and graphic objects are a hard topic. According to Shaheed there are approximately 9 different ways to have images embedded in a Word file, and the documentation is very brief. In newer Office versions (anything from Office 97 on) Microsoft decided to share the graphics embedding code among Word, Excel, and Powerpoint. This project is called Escher and some documentation can be found here. Older Office versions are known to embed bitmaps directly in the files, e.g. stored as .dib or .tiff image, or as .wmf drawing.

Apart from raster images it's also possible to embed drawing objects (lines, rectangles,...) in a Word file. These can be stored in an Escher container, or directly in the Word file (in older files). Due to OLE it's also possible to embed e.g. AutoCAD drawings in a word file, but I didn't check how that's done yet. For Far East versions of Word it seems to be possible to have a drawing grid for far east characters, but I have no idea how that works as I have never seen a FE Word nor speak any far east language.

One thing that seems to be common among all the embedded images and drawing objects (regardless of the Word version) is that they are anchored using a special character (`SPEC_PICTURE = 1`) and of course the `fSpec` flag is set. For this character it should be possible to find and construct the PICF.

For Word 8 the important structures seem to be PICF and METAFILEPICT, the rest should be embedded in Escher containers. For Word 6 we have the PICF, METAFILEPICT, DO and DP* (for the drawing primitives).

## Questions

Finally some questions that still make my head ache, from a design point of view:

- What to do with embedded documents (like Excel documents)?
- How to handle embedded image/clipart/wmf/... files?
- How can we handle bugs in the SPEC as effective as possible? It shouldn't be necessary that two programmers lose their hair on the same bug...

Please send comments, corrections, condolences, patches, and suggestions to Werner Trobin. Thanks in advance. If you really read that document till here I owe you a beverage of your choice next time we meet :-)