WordDown: Word to HTML5 conversion tool

HTML5 Case Study

WordDown: Word to HTML5 conversion tool

Document details

Author:

Peter Sefton

Author’s ID (URI):

http://nla.gov.au/nla.party-541658

Document Type:

http://purl.org/ontology/bibo/Report

Date:

2011-09-22

Version:

0.1

File Name:

QA

Notes:

This is a first draft – I will post it to my blog, using the tool it describes, to seek feedback.


Rights

This work has been published under a Creative Commons attribution-sharealike 2.0 licence. 

Acknowledgements

UKOLN is funded by the Joint Information Systems Committee (JISC) of the Higher and Further Education Funding Councils, as well as by project funding from the JISC and the European Union. UKOLN also receives support from the University of Bath where it is based.


Contents Page


About This Case Study

Target Audience

The main target for this work is tool developers building authoring systems, repositories and publishing infrastructure for academic documents.

It may also be useful to committed academic authors who are comfortable with HTML already and have some technical skills, who would be able to install a bookmarklet and possibly run Python code on a Windows machine (ie not at this stage the broad academic community).

What Is Covered

This case study examines ways that academic authors working with word processors such as Microsoft Word, the OpenOffice.org family and Google Docs would be able to produce compliant Scholarly HTML5. Due to time constraints, the tool developed for this project handles Microsoft Word documents only, but the principles outlined here apply more broadly.

Word processors are used very widely in academia for all sorts of document authoring, yet the articles, essays, theses, course materials and so on produced in Microsoft Word et al in the Higher Education sector in huge volumes are not easy to convert to good quality, clean, semantically rich HTML 5 of the type being discussed in the jiscHTML5 project.

This is a critical piece of work for the overall project of bringing scholarship to the (semantic) web. If the tools still being used to create academic content do not create HTML natively then who will do the markup? What new tools are needed if any? This case study will consider these questions, as well as produce some demonstrations of what is possible with the demonstration application, WordDown.

What Isn’t Covered

The questions posed above about tools for creating HTML5 are even more pressing if we are targeting XML for scholarly documents – XML is in its teens now, and there have been no widely available tools produced to create XML for DTDs such as DocBook or TEI that have gained any kind of traction with large user groups. This is an interesting question for the Higher Education sector but it is out of scope for this case study.

Use Case

There are no formal studies that we are aware of that show the usage rates of different academic authoring tools by country or discipline, but it is quite clear that Microsoft Word (along with other word processors) is a very widely used document creation tool in many, many disciplines at our Higher Education initiations and research organisations. For example on this study the UKOLN team have requested that case studies be submitted in Word format, using a template supplied by UKOLN. While the template gives the case study documents some structure, using ‘Save as HTML…’ from Word will not produce good quality HTML – far from the kind of structured documents that are being produced as exemplars in the jiscHTML5 project, Word’s output is focussed a decade-old approach of trying to match paper formatting.

The use case here is using Microsoft Word in the the production of any academically oriented document that’s destined for the web, including articles, theses and other student work, reports, course materials, academically included blog posts or other web pages, and reports such as this one.

[TODO: more review of other approaches – Lemon8XML etc]

Solution

The solution to the use case use Word to create HTML5 is a tool called WordDown created for this JISC project. This is a JavaScript application which runs in a web browser and processes Word documents into clean HTML5. It takes Word’s HTML output as an input.

The wiki at the jiscHTML5 project at Google code covers how to run the application and the basics of how to format documents.

Background to WordDown

The Word 2000 HTML format has been a feature of Microsoft’s flagship word processor since Word 2000 and has been the target of much criticism. At the time it was introduced it was capable of rendering almost all features of Word documents into a kind of HTML, using a combination of extended CSS formatting and islands of very obscure non standard markup. It was, however, actually not very far away from XML, and could be processed into XML with a small transformation program1.

The solution presented here revisits the format with modern tools by loading it into a modern web browser and using the jQuery framework to interrogate various aspects of the formatting. Recent versions of web browsers are all coded to deal gracefully with the ‘mutant markup’ in Word’s HTML output, hiding Word specific code in comments, because HTML5 parsing rules take account of all kinds of legacy issues like Microsoft’s non-standard markup.

The application WordDown is inspired by the success of lightweight wiki-style markup languages which allow users to create HTML (and PDF in some cases) from simple text files . One of the foremost examples of this class of language is the MarkDown format, used  by the Pandoc processing framework. (Peter Krautzberger has a useful introduction to Markdown and Pandoc for academic authors and explains how it may be considered what we might call ‘the new LaTeX’ for academic authoring.)

In Markdown, one way of making a heading is to preface some text with #.

# Introduction  (turns into <h1> in HTML)

## About markdown (turns into <h2>)

In WordDown, to accomplish the same thing the author uses the built in heading styles – Heading 1 and Heading 2 respectively. These styles are the only widely used standard way of structuring documents in the word processing world– other elements such as quotes or lists have no equivalent standard implementations.

To make a block-quote in Markdown, you use a greater-than character:

> This is a block-quote.

In Word Down, just indent the paragraph either using the formatting tools on the Word ribbon, toolbar or menus, or define  a style – but (at this stage at least) the WordDown processor does not use style names other than for detecting some headings, it uses indenting. So any indented style which is not a list or a heading will be treated as a block-quote.

The algorithm WordDown uses is being documented on the Google Code site, initially via the code. In essence it is designed around the assumption that the user wants to create clean HTML, not to recreate the look of a paper document so in a similar way to the lightweight wiki markup languges, it uses formatting and indenting as structural cues. The main device used is to look at the left margin:

  • With a plain paragraph that is not a heading or a list item, an indent greater than the preceding non-heading paragraoh means block quote.

  • Paragraphs in a monospace font are maped to the <pre> (pre-format) element.

Features summary

WordDown has the following features, described in more detail on the Google Code site for the software:

  • Creates HTML from Word documents saved using “Save as HTML…” on Word for Windows versions from 2000 to 2010. The code runs in a web browser and is packaged both as a bookmarklet and as a small Python web server that user needs to run from their documents directory.

  • Works with Zotero citations and embeds them inline using best-practice Scholarly HTML5 conventions.

  • Can create rich semantic HTML5 with embedded microdata, given microformats in the source document.

Demonstration – screenshots

The simplest way to run WordDown is to manually save documents as HTML, load them into a web browser and use the WordDown bookmarklet as documented on the Google code wiki. A slightly easier workflow (which is harder to set up) is to run the WordDown server:

Figure 1: Browsing local files using the WordDown web server

When the user selects a word document, the WordDown server runs Microsoft Word in the background, saves the document as HTML, inserts Javascript into the head and serves the result back the user’s browser. The result is that the user is presented with an HTML version of the Word document using a stylesheet derived from the one used by the W3C for their standards documents:

Figure 2: A word document (this one) converted to HTML5 by WordDown running the browser

The resulting document is HTML5 – and can be saved by the user for reuse. Alternatively, using another JavaScript application developed for the jiscHTML5 project, parts of the document can be copied and pasted via the Show5ource bookmarklet.

Figure 3: The Show5 bookmarklet identifies the HTML5 sections in a document and lets the user click to see copy and paste-ready source or grab the whole document as a Zip file with all images.