ptsefton http://ptsefton.com This seems to be a workblog Tue, 09 Mar 2010 03:12:49 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Back to the wordprocessor http://ptsefton.com/2010/03/09/back-to-the-wordprocessor.htm http://ptsefton.com/2010/03/09/back-to-the-wordprocessor.htm#comments Tue, 09 Mar 2010 03:12:49 +0000 ptsefton http://ptsefton.com/2010/03/09/back-to-the-wordprocessor.htm

It’s been quite a while since I have looked at word processing here. Over the next couple of weeks I’m going to revisit some of the ongoing themes I cover on this blog about document formats and Scholarly HTML and so on, working with my colleague Ron ”If The Encodings Don’t Get You The Namespaces Will” Ward, a veteran word processor wrangler, the man who wrote most of the rendering and conversion code in the ICE application. (I’m still working on the metadata stores thing, too).

So, in no particular order, some things we’re going to look at.

  1. I’ll come back to the wonderful world of Custom XML in Word 2007, which may be a ex-feature, and the work that Microsoft Research did on various authoring plugins that used it, and try to find out what’s happening there. We have a visitor coming this week who worked with MS Research on their ontology plugin. More about that soon.

  2. I’m going to revisit the Word 2007 interface, which frankly still frightens me, I dread that process of hunting through weird interface widgets for something I used to know how to do in muscle memory.1

  3. Interop between Microsoft Word 2007 and Openoffice across .doc, .docx and .odt. This is not just a technical issue, there’s politics involved. For instance I think, Sun’s evangelical policy used to be “Over our dead body will Writer save into that filthy Microsoft .docx format, although we will let you read it in, in order that your documents might be saved.” (That’s saved as in saved from eternal damnation as well as saved in the one true open format). Dear Larry, we have the dead body part, can we have Save as … .docx now2?

    Closer to home there are political issues at USQ with ICE and how it works with some classes of documents. Over in the Faculty of Engineering I gather people are being forced to use ICE for maths-heavy courses, Bron Chandler has been working hard to help the course-maintainers in engineering, to find an acceptable way for them to use Word or Writer plus ICE but there are limits to how far we can push these word processors for heavy duty technical documents. I’ll say it again: I don’t think it makes sense to mandate the use of a tool like ICE at a university; we should talk about performance based standards for materials and remain pragmatic; if we can’t find a cheap way to make HTML versions of maths-heavy materials then we may have to settle for PDF and we may have to let people use LaTeX, or DocBook or whatever (I’ve never heard of one wanting to use DocBook but there are plenty of LaTeX enthusiasts).

  4. Ron is going to look at how we might be able to use Word instead of or in addition to OpenOffice.org in ICE. We picked OpenOffice.org as the main engine-room for ICE several years ago because it is available cross-platform but what if we could do one or more of the following on Windows?

    • Use Word to generate PDF – should improve the WYSIWYG fidelity (Ron’s already got this automated, although not hooked in to ICE yet.)

    • Use Word rather than OpenOffice to render the image parts of a page, again to improve the way things look. ICE uses a two-part process to convert to HTML. It converts OpenDocument format XML into HTML, but to get images such as charts or equations, it also calls OpenOffice’s Save as HTML feature (initial experiments look promising).

    • Maybe replace OpenOffice.org altogether on Windows by automating it to save as .odt behind the scenes. (If that works, there is still the issue of how to make books out of individual course documents. At them moment this I a very complex process which automates Writer, as the master document feature doesn’t work for what we want.)

    • And if the above fails, maybe rewrite the ICE renderer for Office Open XML rather than Open Document Format. Big job we’d rather not have to resource.

Along the way, this is going to force me to spend some time in Windows for the first time in a long time, guess I’ll have a chance to learn about Windows 7.

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

1 Look, I’m not afraid of learning new things, why at the moment I’m more or less learning to play the mandolin which is tuned in 5th s rather than 4th s like most of a guitar – that’s a whole lot of new fingerings, but it makes sense and it’s fun.

2 Yes I can save as .docx here in Ubuntu, but that’s a special build with some Novell code in it, I believe.

]]> http://ptsefton.com/2010/03/09/back-to-the-wordprocessor.htm/feed 1 More details on a metdata store for data in/alongside VITAL http://ptsefton.com/2010/03/04/more-details-on-a-metdata-store-for-data-inalongside-vital.htm http://ptsefton.com/2010/03/04/more-details-on-a-metdata-store-for-data-inalongside-vital.htm#comments Thu, 04 Mar 2010 00:55:03 +0000 ptsefton http://ptsefton.com/2010/03/04/more-details-on-a-metdata-store-for-data-inalongside-vital.htm

Here’s another post about the ANDS metadata store work I’ve been doing. I was at the university of Newcastle last week working with Vicki Picasso and Dave Huthnance, with calls to Teula Morgan at Swinburne. Together we fleshed-out a model for how Newcastle might run a data-registry alongside their VITAL repository.

I want to use this blog post to expand on the last blog post and refine the details of what will become an ANDS project plan for someone to develop this stuff. I’m going to talk about the diagram from the last post, in particular the application which has become known as “The Red Box”.

graphics1

1 Requirements

The main requirements for this system, as collected from the stakeholders are that it:

  1. Can provide a university-wide registry of research data, (the data collections will reside wherever they currently reside under access control, and I believe there is a policy in development that open data will all be served from the research storage service) with two main inputs:

    • Form-based deposit, with dead-simple workflow using a system which is as easy for the library and research office to customise as the current VALET system, with an option to port the existing ingest workflows as well (that’s why there’s a dotted red line around the current forms system).

    • Discovery of data files and data collections on the university’s new research storage facility.

  2. Work alongside of the VITAL institutional repository by storing metadata (and potentially small collections themselves) in Fedora, which is the underlying storage layer for VITAL

    • VITAL can be used as the portal for access to research collection metadata or not, as appropriate. Teula Morgan told us that at Swinburne they would likely connect the research data collections into their discovery layer rather than using VITAL. The following models need to be supported:

      • VITAL as the authoritative source for digital objects, with RedBox deleting objects which it has handed off to VITAL.

      • RedBox as the authoritative source for digital objects, with VITAL or another portal acting as the discovery interface for research data collections. With the addition of a portal, this configuration could form the basis of a standalone metadata store.

    • The system needs to stay out of VITAL’s way as much as possible to avoid the risk of unapproved material ‘leaking’ through VITAL and to avoid affecting performance.

    • The system should provide for batch-changes to repository content, in order to assist in cleaning up the document data that is already in there prior to adding research data. (This component came up both in discussions about how we can work in a linked-data way – adding URIs for people mentioned in the repository and as a requirement from Swinburne where they lament the lack of batch-editing tools in VITAL).

    • The system needs to have no dependencies on the proprietary parts of VITAL, to allow the possibility of switching repositories.

  3. As far as possible deal only with the collections side of things, without attempting to become management system for activities (research projects) or parties (people). Following from this, to be a “Linked Data” application where Activities and Parties are referred to by URIs, and other terms also use well-define URIs rather than strings. (There is a potential side-project to this one to upgrade the NicNames system – more on that soon).

Caroline Drury the CAIRSS/ANDS liaison person pointed out that we should discuss how these requirements go beyond encouraging people to register data via the ANDS ‘register my data’ service for Research Data Australia. The big things are:

  1. This allows an institution to have additional management metadata that is not in the RDA system, such as details of how long data should be held. We’re working with Simon Porter from the University of Melbourne on this. The idea as that when this is implemented with University X they will work out the metadata they need and the developer would assist in creating the input forms and mappings from one metadata format to another.

  2. This provides for curation by a data librarians who can leave some submissions in the queue while they sort out details of the metadata or wait for data to be made available on the storage facility.

  3. This also encompasses data which is not destined for or not ready for listing on RDA.

  4. But most importantly it will allow the institution to meet its obligations under The Code.

2 Implementation

The deliverable for the work we’re doing at USQ for ANDS is an ANDS project plan. That makes sense, as it makes starting up the next phase of work straightforward. But to put forward a complete plan, we need to make some assumptions about the design, mainly in what technologies we would use. So here’s a proposed broad-brush architecture for a metadata stores solution work with VITAL. Remember this is only a proposal an the reason it’s up on this blog is so you can comment on it.

The large-scale assumptions are:

  1. RedBox will use Fedora 3 as an internal storage component – with data synchronised to VITAL as needed. This meets the requirement that VITAL is functioning as a portal and will reduce stress on the VITAL repository as much as possible. Fedora is an obvious choice for ARROW/VITAL sites. We are choosing to work with the latest version for the RedBox component, but it will need to synchronise with Fedora 2 which underpins the VITAL product.

  2. The application will be developed in Java, building on The Fascinator platform which was originally sponsored by the ARROW project in 2008 and which has been under development at USQ since then. Benefits include:

    • Being Java it can sit in the same Tomcat web-server as Fedora and the Apache Solr indexer used by most Fedora repositories these days.

    • The ingest component we’re developing for The Fascinator, while incomplete, meets the requirement that the system be configurable in a similar way to VALET – where extending the forms and integrating them with external systems like CrossRef is trivially easy. Existing VALET forms can also be ported to the new system (it’s a manual process, but not difficult).

    • It’s highly modular and so can be used, for example, without a portal, a role which VITAL, or a discovery service can take on. One of the most important modules will be plugin harvest technology to pick up content that’s on the storage system and provide a view to researchers and data librarians to begin describing it. The Fascinator has as extensible system of plugins and we already have file-indexers and a framework for extracting metadata from files, which can be extended to work with new kinds of research data as they appear.

    • Our developers know the system, meaning we can be up and running with this application very quickly.

    We are aware of other Fedora software components, but none that meet all of the above criteria. The closest would probably be Muradora – if there are ANDS contributors using that who want help setting it for research data then I think that would be in scope for us to look at in our ANDS work,

  3. While there was some discussion about using VALET or VITAL as the foundation for the RedBox early in the project neither of these systems has an architecture which can work on a university-wide scale if all data sets are to be described, or the ability to harvest metadata about files on the research storage service.

2.1 Components

I’ll take a brief look at some of the components in turn.

2.1.1 Batch editing

The Fascinator already has the basis for a batch-change tool. The Fascinator, whether it is sitting on top of Fedora or our simple file back-end has indexer component which can either watch a queue for changes in a repository or do a complete re-index.

This indexer has an extensible set of rules which are small scripts that can be fired off to deal with various kinds of content. In our desktop work we use this for things like generating web-ready versions of video content and HTML versions of documents, but it could also be used to transform datastreams in the repository to make bulk changes.

Here’s an example of how it might be used. I talked previously about setting up a system with the NLA or locally using NicNames to assign unique IDs to researchers before we start the major part of this project. Once we had a set of name identities, they could be put back into the repository by writing an ‘indexing’ rule that for each document does a look up to the name authority system and puts it back into the MARCXML datastream in an appropriate field. Then you tell the system to re-index.

For a batch edit, you’d back everything up, then set this up and run it on a copy of the repository and swap in the new data and test thoroughly with VITAL, before either running the process on the live repository or swapping in a new Fedora database underneath.

2.1.2 OAI-PMH feeds

One of the key deliverables for this project is to get data flowing to Research Data Australia. Part of the project scope will be to make sure that we have fully-functional OAI-PMH feeds, complete with support for deletions, working from both VITAL’s copy of Fedora and from the RedBox itself to meet the requirement that the system is able to run in “headless” mode. Xiaobin Shen from ANDS in Melbourne has been working with OAI-PMH providers, so we don’t expect this part to be hard., just a matter of careful selection, configuration and testing.

2.1.3 VITAL configuration

One of the major tasks is to create customisations for VITAL to display metadata records about data and make sure that the result fits in to the rest of the VITAL repository portal.

2.1.4 The Forms interface

While we’re not proposing to use the Squire forms interface sponsored by ARROW directly, the code we will be using is based on and informed by the same VALET model. This system consists of:

 
"stages": [
    {
            "name": "init",
            "security": ["owner", "admin"],
            "visibility": ["guest", "owner", "admin"],
            "template": "${user.home}/.fascinator/workflows/templates/basic-init.vm"
        },
        {
            "name": "live",
            "security": ["owner", "admin"],
            "visibility": ["guest", "owner", "admin"],
            "template": "${user.home}/.fascinator/workflows/templates/basic-live.vm"
        }
    ]

2.1.5 Research storage harvest & grants database triggers

The University of Newcastle has a research storage facility (a SAN) and policies are under development which will likely see lots of RDA-ready data made available on the facility. One of the features of the RedBox application will be to he able to harvest metadata about such files; at the very least, file-paths, sizes and dates, and any other metadata that can be automatically extracted. For data that’s on the storage facility the idea is that researchers can add it to the metadata store by finding the data files themselves and clicking “Describe this data” to fill out a form.

The Fascinator has a number of features in this area such as the ability to generate thumbnail versions of files and web-ready renditions of various formats, which is of peripheral interest to the metadata stores activity.

The harvester system is highly configurable, so it could be set up to index, or watch other kinds of storage service; one that Newcastle require is the ability to harvest an email account with messages from the grants database about grant completions; which are events that should trigger a metadata librarian to chase-up data from researchers.

3 Risks

There will be a detailed risk assessment for ANDS, but here are some notes about the main risks:

Risk

Mitigation Plan

The software we develop here might end up being only used at one or a handful of institutions, which would then bear the maintenance load.

  • Work to promote this solution to other ARROW sites.

  • Release all code as open source with tested documentation.

  • Use components of this solution as part of the standalone solution we have also been asked to look at, broadening the installed base for the the RedBox application.

  • Consider funding a program to port VALET workflows to the new system for ARROW sites to build a sustainable community.

  • Work with the University of Melbourne and collaborators to see if some of the component developed for the RedBox application could be used at their sites, and to ensure compatibility with VITRO as a data store.

  • Document the metadata storage system and batch transformation system so that new Fedora-compatible ingest or portal tools can be swapped-in later.

University X unable to supply full metadata schema on time; there is no clear consensus on best practice for describing research data collection to meet the demands of The Code, while being able to serve RIF-CS to Research Data Australia.

  • Stakeholders to vigorously encourage ANDS to produce metadata guides, possibly after workshops.

  • If all else fails implement RIF-CS in the repository., with the possibility of doing a batch-update later.

Linked Data infrastructure including URI endpoints for parties and activities doesn’t come on-line in time for implementation

  • Work with the PIP project to make sure that this does not happen (at least for names) possibly entering name data into People Australia semi-manually.

  • Work with ANDS to set up interim systems where possible.

  • If all else fails store strings in metadata fields just as we have been doing in IRs for years and update to URIs later when the infrastructure is available.

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

1 I’m not convinced that using the ANDS handle infrastructure is a good idea. Firstly, ANDS is only funded for a short time and while ANDS staff express their aspirations for keeping things going there are no guarantees, secondly, data has to have URL pointing to where it is stored, and that has to be maintained as well, with redirects and so when things change. In a lot of use-cases I think that using Handles just increases complexity, cost and risk.

]]> http://ptsefton.com/2010/03/04/more-details-on-a-metdata-store-for-data-inalongside-vital.htm/feed 10 ANDS metadata stores: Describing metadata collections in VITAL http://ptsefton.com/2010/02/23/ands-metadata-stores-describing-metadata-collections-in-vital.htm http://ptsefton.com/2010/02/23/ands-metadata-stores-describing-metadata-collections-in-vital.htm#comments Tue, 23 Feb 2010 08:03:54 +0000 ptsefton http://ptsefton.com/2010/02/23/ands-metadata-stores-describing-metadata-collections-in-vital.htm

I’m at the University of Newcastle visiting repository rat extraordinaire Vicki Picasso (actually at this bushland campus she should be a repository wallaby or something) and her colleague Dave Huthnance from IT. We are working on a model for how research data collections destined for Research Data Australia might be described and managed in the local institutional repository.

(Please ANDS can we have some advice on this metadata issue? Some of you say to use RIF-CS and some say that’s a bad idea.)

Vicki presented a model of how metadata about research data could be ingested into the VITAL repository they use at Newcastle at eResearch Australasia 2009; it featured the VALET system which is a very simple repository ingest tool and what Vicki calls “Institutional Data Triggers” such as events in a grants database which would fire-off a metadata ingest workflow.

The new model

Today we refined that diagram. Like it says the blue bits represent the current Nova repository infrastructure (VITAL + VALET + Fedora) which feeds data to (amongst other things) the National Library’s harvesting systems.

The red bits are new proposed infrastructure, to be developed, to enable collections metadata to be captured and feeds of RIF-CS metadata to Research Data Australia. The new red box labelled “Research Data Collections”, should it be built, will be a more sophisticated version of VALET, probably written in Java so it can work in the same Tomcat web container as Fedora – it would have a VALET-style simple forms interface for walk-up submissions (this could be used to replace the existing publications ingest and staging workflows too, as shown by the dotted red line – if this were a requirement).

Green is for external services. One of the very interesting green bits is the Research Storage system which is being provided by university IT and administered by the Research Office. I gather that this is essentially a file-store; we are proposing to add an interface that lets researchers see their files in the new (red) ingest system and add metadata to them, and flag them as candidates for RDA. I think Newcastle’s policy will be that if you want data to be available via Research Data Australia then it is desirable this it goes in the Research Storage System. Sounds good to me. To bridge the gap between files on a storage system we are proposing a bit of middleware to link the file view of data to a web/repository view.

As discussed before here, the ANDS stakeholders in this project are keen for us to take a linked-data approach to metadata (slogan: Less typing, more linking!). I talked a bit about how this might work in the previous post on name identities; potential integration with services like the NLA’s PIP/People Australia and possible services like an ARC website for grant information are shown in green at the bottom right of the diagram (I have some input from Basil at the NLA I need to process, but at this stage I think we’re looking at having NicNames in there so institutions can manage their own metadata.

graphics1

One assumption we’re making here is that the core class of item we’re describing here is a collection, which should fit with the kind of data that is already in the repository, which is research outputs, like data.

There are some questions, of course.

  1. What metadata schema to use for describing data collections?

  2. And where would the ISO2146 notion of Services fit in? The services listed in the RIF-CS documentation are all repository-type search/feed services so it seems appropriate to either tie them in to the OAI-PMH ‘identify’ verb or to let repository managers simply enter them in to an ANDS system directly.

Going further

One idea that has come up is that VITAL sites might want to use Fedora and the OAI-PMH feeds available off it but not expose them via a web portal at all. In conversation with Teula Morgan from Swinburne today, Vicki proposed a model where there is no portal interface. I call this a ‘headless’ approach; there would be local management interface for research data collection metadata (the red box) but it could be that the primary discovery mechanism is outsourced to RDA. This is pretty common for university web sites – USQ uses Google for our website search service for example.

I am also exploring the idea that this ingest tool, which will be able to put records into Fedora (which as far as I know nobody has ever been fired for acquiring) could form the basis for our major deliverable on our ANDS metadata stores project; a specification for a stand-alone metadata -about-research-data-collection system.

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2010/02/23/ands-metadata-stores-describing-metadata-collections-in-vital.htm/feed 10 ANDS Metadata Stores: Integrating VITAL with the NLA's Party Infrastructure Project http://ptsefton.com/2010/02/23/ands-metadata-stores-integrating-vital-with-the-nlas-party-infrastructure-project.htm http://ptsefton.com/2010/02/23/ands-metadata-stores-integrating-vital-with-the-nlas-party-infrastructure-project.htm#comments Tue, 23 Feb 2010 02:28:27 +0000 ptsefton http://ptsefton.com/2010/02/23/ands-metadata-stores-integrating-vital-with-the-nlas-party-infrastructure-project.htm

Here is some more news on the metadata stores project for ANDS (see previous posts) and how we might build links between VITAL/Fedora and an identity service for people (parties in ANDS-speak). This is potentially a step on the way to a linked-data future not just for research data but also for institutional repositories.

Natasha Simons said, on the CAIRSS list:

An exciting new project, which was mentioned at the CAIRSS Community Day last December, has commenced at the National Library of Australia (NLA). The Australian Research Data Commons Party Infrastructure Project (ARDCPIP) is an Australian National Data Service (ANDS) funded project being carried out  by the NLA to enable improved discovery of research outputs and data through assigning public and persistent identifiers to Australian researchers and research organisations using the NLA’s People Australia Infrastructure. The project will involve consultation with the research sector and more information about it will be made available to you shortly. 

This is good news for the metadata stores work we’re doing with ANDS. I was in Canberra in early February and we caught up with Basil Dewhurst and Natasha; seems like the timing is good to try to get a link-up between the work we’re doing to sketch the design of a metadata store for data on top of the VITAL repository.

Why is this important?

One of the problems we have been talking about with ANDS is how to represent people. In VITAL names are typically not subject to authority control, and even if they are it is via some kind of people-driven workflow, not a feature of the software. (Any exceptions to this? Please let me know in the comments.)

The linked-data semantic web ideal would be for there to be some kind of resolvable identifier stored each time a party is mentioned. You want at least two bits of data (using the example that Basil gave in the comments here last time).

  1. The name as it appears on a particular work that’s a string, like Cappo, Michael Charles.

  2. A URI – such as http://nla.gov.au/nla.party-471077

  3. And maybe a canonical, normalized way of referring to the name for a particular service – the heading at the NLA’s trove site is Cappo, Michael.

  4. Possibly more URIs that refer to the same party.

  5. If we had this it would meant that submitting data about data collections to the Australian Research Data Commons would be much more concise and accurate – it should not be necessary to include any party data at all in a feed from a metadata store to the commons, just URIs.

  6. But, in the case of University X, not all the researchers have persistent identifiers. So, I have suggested that we try to create persistent identifiers for all parties as the first step in building a metadata store on top of VITAL. This is good several ways:

  7. Having an early adopter on board will help the PIP project at the NLA.

  8. This will bring together three ANDS funded projects and promote general metadata harmony.

  9. University X will get a significant upgrade to the integrity and usefulness of their existing data.

How?

What we’re looking at is a pre-load stage to populate PIP/People Australia with the identities of people already mentioned in the repository.

  1. Harvest name data from VITAL - could be via MARC / DC / or some other format derived from those, over OAI-PMH or via some other batch load. Contextual information is likely to be limited to subject and affiliation data pretty much – you won’t get birthdates. Vicki Picasso tells me that at Newcastle they record which authors belong to their institution, so for that institution this process could concentrate on the authors they know are theirs.

  2. Records are auto-matched in PIP using whatever algorithms it has.

  3. A person (probably at University X) uses the NLA’s private People Australia tools to sort out the name references and associate the various publications with the right person ID.

  4. Once the name data has been checked (step 3) the developer writes some code to inject the ID data back into Fedora, in MARC (if that’s possible) or into a new datastream. (If my team got the job we would do this by writing some custom index rules for our Fedora indexer which is part of modular system that is The Fascinator.)

  5. The developer creates an ingest system (like VALET). As part of the form-filling process it would look up name IDs at point of ingest and trigger creation of new parties when one is not found. We will write up more detail about how this would look, using Newcastle as an example.


There is a variant on this where NicNames might be used in between 1 & 2 and PIP would be fed cleaner data.

Once the intial load was done, many names associated with data collections would already have URIs so these would not have to be re entered; for ‘new’ identities there would have to be some kind of system for minting a new ID then flagging it for attention from a name-curator at a later time.

Linked data more generally

I have described a process here for party identities; it would also be good to have the same kind of approach for activities (in the ISO-2146 sense) which means research projects in this context. Andrew Treloar of ANDS Monash branch tells me that there has been talk of having the major funding bodies like ARC provide a PIP-like service for grant funded projects, providing some kind of URI that a human or a machine could look up, getting a web page or an RDF record as appropriate. The RDF record could be used to extract trusted data like project names at both the repository and the ARDC registry but nobody would have to re-type that data into a form.

Does anyone out there know anything about this potential service for Activities? Can you comment?

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2010/02/23/ands-metadata-stores-integrating-vital-with-the-nlas-party-infrastructure-project.htm/feed 0 ANDS Metadata store: starting point http://ptsefton.com/2010/02/02/ands-metadata-store-starting-point.htm http://ptsefton.com/2010/02/02/ands-metadata-store-starting-point.htm#comments Tue, 02 Feb 2010 05:41:07 +0000 ptsefton http://ptsefton.com/2010/02/02/ands-metadata-store-starting-point.htm

This is the second post about the ANDS-funded metadata store work we’re doing at ADFI. The project now has a Trac site where we will be tracking progress and keeping notes; the site will be open to the public to read, to make it as easy as possible to reach a wide range of stakeholders, although there will be a few documents we have to keep under tighter control. The Trac site will be mainly used as a project wiki – but I will put in some job tickets and use the milestones feature to track what happens between our (mostly) weekly project meetings. This week’s milestone is up now.

In this post I’ll reveal some detail about our starting point (though not the name of the institution(s) we’ll be working with) and follow up on some of the feedback I got from ANDS staffers to my previous post. Scott Yeadon raised quite a few points, some of which I will get to in future posts and/or project plans and wiki pages.

Starting point

ADFI staff met with ANDS stakeholders late in 2009, and we have agreed that a good starting point will be to focus on one of the ‘additional’ deliverables first. The core deliverable is a project plan to build a stand- alone metadata store, with an option to write extra plans for add-ons or customisations to existing repository software such as DSpace or Eprints should any organisations want to keep metadata about data collections in their IR. It happens that there has been a fair bit of work at least one institution (University X) in Australia where they do plan to keep metadata about collections (and maybe parties, and so on) in their IR; they are running the VITAL software which was associated with the ARROW project. So that’s our starting point: a project plan to write some open source software and supply configuration files and customisation, and documentation for an ARROW repository so that University X and other sites running the ARROW suite of software can participate in the data commons, and submit data to research data Australia via their IR.

The VITAL software is a web interface to Fedora, with configuration to index a Fedora repository and display. You can see it in action at the home of ARROW, Monash University. Some sites use bit of free software called VALET to put things in to the repository.

VALET has a simple design which I like – it allows you to design a form or forms to capture as much metadata as you like and configure a set of really simple approval steps. When a user starts adding metadata about a new object, the system saves the in-progress data by the simple expedient of serialising the form data to XML. Moving to the next step of a workflow just requires the application to put the saved data back into the form. When a user with the correct rights adds the item to the repository, pre-configured XSLT stylesheets run automatically to transform the serialised form data to whatever is required, usually MARCXML and Dublin Core.

The ARROW project sponsored a replacement for VALET called Squire which I reported on in 2008, but so far nobody has used it in anger. For this project I think it might be good to use Squire, or a something like it rather than VALET; we’re discussing the pros and cons with ANDS and University X.

Over the next week I will be putting together a skeletal draft of a project plan based around the proposed architecture ready for stakeholders at ANDS and the IR and eResearch communities to comment now that people are back from their summer holidays.

Issues

Some of the things which will need to be resolved are:

  1. Which OAI-PMH provider to use?

    Metadata will get from the VITAL repository to Research Data Australia via an OAI-PMH feed, but there are a few open source toolkits to choose from. We will need to support at least one, maybe more. As ANDS staffer Xiaobin Shen reminded me in the comments of my last post, one consideration will be whether or not the provider supports deletions, not all do. This will require careful testing before we commit to one provider or another.

  2. What data model will be used?

    At the moment, all the VITAL repositories in Australia that I know about have a very simple data model. Each item in the repository has a ‘master’ metadata record usually in MARCXML, sometimes MODS (they’re effectively the same) with derived metadata in Dublin Core. There may also be datastreams, usually PDF files. At this stage there is no formal content model, so the datastreams could be called anything and it’s up to humans to make sense of them; there’s no guaranteed way to tell whether a PDF is an abstract, or the whole record, a preprint or a published version of a paper, for example.

    It’s a bit hard to get information about VITAL unless you’re a customer; the product page is currently sporting a copyright statement from 2008 and the brochure (PDF) is big on tropical fish and short on specifications so what I report here may need correcting.

    The version of VITAL which I think most sites are running in Australia is 3.x. It uses Fedora 2 which doesn’t have formal content models. Fedora 3 has a formal mechanism for describing content models which means that you could describe the parts of an object, and their role in the object. From what I can gather VITAL 4 which I saw demonstrated in late 2008, and was released in March 2009 has content models too, but they are more about how to display an object than describing the relations between its parts. Perhaps someone could elaborate or correct this in the comments?

    My working assumption is that for this development, the idea will be to stick with the way VITAL 3.x works, without worrying about content modelling which is fine here because this is not about complex objects with lots of data, it’s about metadata about data collections where the collections themselves will usually reside elsewhere, which brings us to the sub-question.

    What goes in the repository and what is stored elsewhere?

    There’s a real chicken and egg problem here. I gather that eventually the NLA will be running a party-identifier service based on People Australia, so when that’s established we won’t be typing names into metadata forms any more, we’ll be linking to an ID. So in the abstract model behind RIF-CS the party management just goes elsewhere. I wonder if the same could happen with activities (every project has some kind of web site now, so why not point to an RDF endpoint hosted on the project website or just to the project web site as an identifier) and services (not sure what to do about these, but then I’m not really sure yet what services are in this context).

    Question is, if we want to get a system running now, what’s the best way to identify parties in a future-proof way? I discussed a related issue in a blog post for CAIRSS about NicNames and People Australia. Maybe NicNames can play a role here in the short term?

    One of the design patterns I mentioned in that post, using an index to associate names in a repository with an identity service like NicNames via an index is expanded in a paper I wrote for the New Review of Information Networking. At the moment I can only link to this version which is not open as the publisher has not responded to my questions about the OA policy so I have yet to deposit a version in Eprints.

  3. Which metadata format to use?

    Scott Yeadon made it clear in the comments to my last post that RIF-CS was designed as an interchange format only and that it is not yet stable, which sounds like good reasons not to use it as a storage format. but I have confirmed reports that others in ANDS are thinking otherwise, and are encouraging IR managers to put RIF-CS in the repository; I’d like to hear their side of the story too. Stability aside, if RIF-CS has what it takes to describe a collection of data then it might be an OK storage format.

    I’m not aware of all the alternatives but one that I have heard mentioned by and ANDS person is the Dublin Core Collections Application Profile. What else is out there in use for describing data collections and are there other data-collections registries harvesting from those descriptions in the rest of the world?

    So this is an open issue for now; I hope we can get some consensus on a good data model for storing metadata about data collections (and the other entities).

  4. What configuration is needed in VITAL? I think we need the following:

    • Display for items (are we going to have parties and services and actions as items as well?)

    • Index configuration for VITAL’s Solr Index.

    And If the data collection resides in the repository should there be a collection object and a collection-description object or just one object with both collection and description?

  5. What kinds of APIs do we need?

    VALET can be used to integrate with other systems via XML files which are deposited in a directory and picked-up by the ingest workflow of VALET, so they can be curated by data librarians, a technique which I think was developed by Simon McMillan at UNE in the RUBRIC days. We can certainly implement this, but should we have a web (or other) API for a system such as a grants database to add a new item as well? Should it be AtomPub or a simple post, or SWORD or something else?

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2010/02/02/ands-metadata-store-starting-point.htm/feed 1 New Project: Metadata for data collections http://ptsefton.com/2010/01/15/new-project-metadata-for-data-collections.htm http://ptsefton.com/2010/01/15/new-project-metadata-for-data-collections.htm#comments Thu, 14 Jan 2010 21:37:40 +0000 ptsefton http://ptsefton.com/2010/01/15/new-project-metadata-for-data-collections.htm

[This is a re-post. I put this up a couple of days ago but my hosting provider lost the server for a day and had to restore from backup.]

At the Australian Digital Futures Institute we’ve landed a contract with ANDS (the Australian National Data Service) to write some software specifications for key bits of infrastructure for the data commons. I’m going to be doing most of the initial work on this one, and I’ll use my blog and Twitter to communicate what I’m up to as I go, as per the communication plan for the project. I know that some of my regular readers are interested in this stuff.; please comment if you have anything to add.

This first post will look at the scope of the consultancy, and talk about some of the major issues that I think we’ll need to resolve.

Scope & Deliverables

Our contract is to provide specifications for software applications to help institutions manage metadata about data collections and project plans to build them. There are two models for these applications. The first is a stand-alone system, while the second is using the existing institutional repository to store metadata – here’s what the project plan says:

The proposed [agreed now] process is for ADFI staff to prepare at least one scoped and costed software specification, with an agreed development methodology for a:

[…] standalone metadata store that can be used to augment either an existing institutional repository or data store, and which will manage metadata about data objects and data collections. This metadata will need to complement the existing object-level metadata. The interface should be designed to be web-based and easy to use by repository managers without specific technical skills. The metadata store should be able to generate collection descriptions as RIF-CS (http://www.globalregistries.org/rifcs.html) and make these available for harvesting using OAI-PMH and/or direct harvest of XML.

Source: ANDS-internal document.

Additionally we will develop one or more specifications for providing metadata capture solutions based on existing institutional repository software.

So the main thing we’ll be looking at is a kind of repository or registry-like thing where institutions can describe their data collections, and publish those descriptions to Research Data Australia. If you visit that site you can see the data model that’s being used by ANDS with four classes of thing. Here’s a snapshot of how it looks today:

Collections (877)

Where a collection is a useful grouping of physical or digital items.

Parties (260)

Where a party is a person or organisation that has some relationship to a collection, service, activity, or party.

Services (1)

Where a service is a mechanism for gaining some kind of access to or information about a collection (or items within a collection).

Activities (2)

Where an activity is an undertaking or process related to the creation, update, or maintenance of a collection. [I think projects are activities - PS]

This model, which is expressed in a schema known as RIF-CS, is based on a yet-to-be-approved ISO Standard (ISO2146). RIF-CS implements quite a different approach from the way most Institutional Repositories (IRs) are structured, where the repository items themselves are the only primary objects.

In an IR you typically have metadata which refers to parties and services and so on using text-strings. Parties, services and activities are not things in the repository (speaking in general here, there are some exceptions, and increasing people have some status as objects in repositories like Fez). A publisher, which is a party, will be described in an XML document conforming to some metadata schema (in Australia this is typically Dublin Core, MODS or MARC XML) using its name expressed as a string. If two records refer to the same party using different strings then things start to get messy. And as we all know, there is one kind of party around which strings proliferate and cause confusion: people. There is an ANDS effort under way to provide services for describing people outside of a repository so you can refer to them using some kind of ID, but that service is a way off still.

RIF-CS is described here as an interchange format (it is, after all the The Registry Interchange Format – Collections and Services) The which I gather was originally cooked up to support the Global Registries Initiative. But I think some of the people are thinking of using it as a storage format within repositories. If we decide do that we will have to do so carefully. For example, you would not want to store this example RIF-CS as a single repository object, trust me, as the file contains a number of things that really should be treated as discrete objects in a repository, including metadata about people and projects.

Issues for a new standalone metadata store

If the goal for the standalone metadata store is to support the abstract model behind RIF-CS then one of the challenges of this project will be working out how to support this kind of a model. How do make sure that parties and so on are described once, and as accurately as possible? And where that fails, make sure there is infrastructure to assert that two differently described parties are the same, and two identically described parties are in fact different. If you want to refer to a party that is not described yet how will we support that? Will people and objects and so on be primary objects?

In discussions so far, we have tentative agreement between our ANDS stakeholders and ADFI and on one key point; as far as possible we’d like the stand-alone metadata store to be Linked (Open) Data ready. This means that all Collections, Parties, Services and Activities would have URIs, and the metadata store would allow data entry that uses the URIs behind the scenes. (I checked with Scott Yeadon at ANDS and he tells me that RIF-CS ‘keys’ can be URIs and are expected to be globally unique, even though this is not entirely clear from the schema documentation). This is a user interface challenge, but if we can pull it off we should be able to avoid stuff like this, where a mistyped string results in two parties where there should be one:

http://services.ands.org.au/home/orca/rda/list.php?group=&class=Party&page=2

  • Australian Institute od Marine Science 

  • Australian Institute of Marine Science 

So this is all pointing to a repository which uses RDF to describe things, drawing on appropriate vocabularies for relations and values, such as FOAF for parties, the bibliogrpahic ontology for documents, and maybe the Dublin Core Collections for data collections . We could, of course build a classic database-backed system with a database schema that reflect the abstract model directly, but that’s very inflexible and not easily extensible.

(We don’t have a lot of experience with large-scale RDF at ADFI, but I am thinking that in addition to the RDF, and a triple store to keep it in you would probably still have a high performance index of the repository using Apache Solr to provide the search/browser interface.)

Issues with adapting an IR

If we work on storing data collection metadata in IRs then there will be additional challenges. All institutional repositories in Australia already support Dublin Core metadata, at least in their OAI-PMH feeds. But a lot of repository software in use in Australia is limited to storing metadata as plain strings. Because of this, most of the repository people are used to thinking in terms of flat metadata models serialised in XML, not networks of relations, RDF style. Anything we do to augment IR software will have to fit in with the way IRs are used now, mainly for document content. There are lots of design challenges there for the information model and user interface. We have started thinking about this in detail for a particular repository platform at a particular uni, more on that soon.

Copyright Peter Sefton, 2010. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2010/01/15/new-project-metadata-for-data-collections.htm/feed 4 Bye Bye Word 2007 Custom XML ? http://ptsefton.com/2009/12/23/bye-bye-word-2007-custom-xml.htm http://ptsefton.com/2009/12/23/bye-bye-word-2007-custom-xml.htm#comments Wed, 23 Dec 2009 00:42:29 +0000 ptsefton http://ptsefton.com/2009/12/23/bye-bye-word-2007-custom-xml.htm

I have argued here repeatedly that building applications using Microsoft Word’s Custom XML feature is a spectacularly bad idea, but it turns out that I missed the best reason of all not to use custom XML: It may well be going away very soon, because, I gather, it infringed someone’s patent. In reaction to this news, Tim Bray is reiterating his case that rolling your own XML is a bad idea:

People like me, who had experience with the extreme difficulty of doing this kind of customization, the extremely limited number of places where it made sense, and the high proportion of failure among people who tried to do it, shouted “That’s a bug!” Given that the number of organizations that deploy Office is huge, I bet Microsoft can trot out a few customers who’ve got good results with Custom XML. But I also bet that, first of all, the proportion who try is tiny and, second, that among those who do, few succeed in getting much business value.

I’m one of these people like Tim and like him, I told you so. Let’s revisit some of this:

  • Earlier this year I agreed with Glyn Moody that using custom XML in a collaboration between Science Commons and Microsoft was promoting lock-in to Word 2007 and later versions. If the feature is removed from future versions then would MS Research like to consider the suggestion I made in March to embed ontological annotations using links. The ontology plugin in question used the simplest of schemas, so simple that there was really no need for custom XML at all.

  • I have also expressed doubts about the usefulness of a more complicated plugin that also comes out of Microsoft Research; Article Authoring Add-in for Microsoft Office Word 2007. This was supposed to let you author documents that were complaint with the complex NLM document Schema using Word. Again, it would have worked to lock documents to Word 2007 but I also found it to be pretty well unusable, and having seen a few of these classes of application over the years I didn’t give it much chance of survival in the wild. Now, I guess its future is in doubt, and I’d still love to find the resources to try NLM authoring using the ICE system or a similar style-based system. I’m assuming that MS won’t be under an injunction to drop styles support any time soon, although they have done their best to bury them under the new user interface and make it less likely that people will use them.

  • I counselled my colleagues at Cambridge working on the Chem4Word plugin to be cautious, citing lock-in, usability and maintainability. I wonder if they have heard anything about what might happen to this tool now? (I still think the best approach would be to use OLE to embed the chemical editor and simple features like links, fields and styles for the rest, which would mean that the tool could interoperate between different word processors including older versions of Word).

So, if Microsoft is going to pull Custom XML out of Word, at least in the USA, then I was right, it was a trap, it just turned out to have even nastier teeth than I thought. Now, I’d be really happy to have our group help any creators open source academic word processor plugins that have been snared. We’re happy to share our experience on how to build interoperable, simple, robust authoring tools.

Copyright Peter Sefton, 2009. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2009/12/23/bye-bye-word-2007-custom-xml.htm/feed 2 DRFIC, Tokyo http://ptsefton.com/2009/12/09/drfic-tokyo.htm http://ptsefton.com/2009/12/09/drfic-tokyo.htm#comments Wed, 09 Dec 2009 06:35:20 +0000 ptsefton http://ptsefton.com/2009/12/09/drfic-tokyo.htm

Suggestions for collaboration based on hard-won experience

On Tuesday last week I took to the stage at the University of the Sunshine Coast Innovation Centre for the CAIRSS community day, I talked about seven things we’re doing wrong in the world of repositories. Negative? Maybe, but there’s no point in spending all our time patting ourselves on the back; I thought I might spark some discussion, and it seemed to work. My talk was in the session on software, so most of what I had to say was to do with technology, but some of it was policy and governance. That presentation will be posted on the CAIRSS site soon with commentary on the CAIRSS blog.

By Friday I was in Tokyo, at the Tokyo Institute of Technology in a building rejoicing in the name “Tokyo Tech Front”, another nice new venue. This time I was presenting a similar but more positive and less technically oriented list that I put together with Kate Watson, my CAIRSS colleague. The theme of the conference was Open Access now and in the future, from the global and Asia-Pacific points of view so we put together a summary of where we think Australian university Institutional Repositories are up to from an on-the-ground perspective, with some suggestions for future collaboration based on our own experience of four years or so of IR development in Australia.

There was a lot of emphasis on the Open Access movement at this conference, along with discussion of how to recruit repository deposits. I think the first lot of Australian IRs were mostly motivated by Open Access, but Katie and I note that while this is still a concern, the biggest driver for an Australian university to have a repository is the good old ERA: The Excellence in Research for Australia Initiative. Open Access might get you cited, extend the reach of your research and chip away at rationalising the scholarly communications process, but the ERA will get government cash. I think that the ERA is having a galvanising effect on populating repositories in Australia, at least with metadata and ‘dark’ publisher’s versions of articles, maybe at the cost of openness, for now.

I won’t try to summarise everything on the program, but Alicia López Medina  of COAR/UNED had a strong thread about OA running through her talk, as did Salvatore Mele from CERN, which is unsurprising given that those physicists are very OA, using ArXiv as they do. Salvatore also talked about SCOAP3. As the site says a “A consortium facilitates Open Access publishing in High Energy Physics by re-directing subscription money.” (Alicia pointed out that the percentage of literature covered by SCOAP3 is very small so we should not relax our efforts to recruit deposits just yet).

On the recruitment side, David Shulenburger (Association of Public and Land-grant Universities) talked a lot about repository deposit mandates. The chair, Syun Tutiya asked me to comment on the state of repository mandates in Australia, and I said that I think that the whole open access thing has gone off the boil a bit in Australia, probably because of the ERA, especially since some of the requirements for the ERA are the opposite of OA, people have been figuring out how to limit, rather than promote access to some content. But I couldn’t help noting that we have mandatory income tax in Australia, and some people go to great lengths to avoid it, (that’s why I think we need to re-brand it as ’sharing’ which we are all taught from an early age is a good thing, an idea that resonates with the success of services like slideshare in academia).

This was my first time presenting with a translator, so I was glad that the presentation was relatively simple; I took it one point at a time and paused for the Japanese version. The talk following mine, by Hideki Uchijima (Kanazawa University) summarised the state of play in Japan; I hope the presentation turns up online, as I have lost my printed proceedings. From memory about a quarter of circa 800 (!) universities have repositories, although that sounds better if you look at unis that award PhDs; and while there are metadata standards to which most repositories adhere, I gather that the Japanese experience of harvesting multiple repositories is like ours in Australia ’some normalisation required’ via ‘crosswalks’.

I was interested to see Susan Gibbons presenting on the outcome of University of Rochester’s approach of Studying Users to Design a Better Repository, resulting in their new IR+ software, written by Nathan Sarr, which Tim McCallum is installing in our CAIRSS sandbox so Australian IR people can kick the tyres.

Mikiko Tanifuji of the National Institute for Materials Science pointed out to me in a break that the situation in Japan is like Australia in that we have CAIRSS which is for universities, and they have the Digital Repository Federation which is likewise focussed, and wouldn’t it be nice if we could be a bit more inclusive. Sounds like a good idea to me. Anybody care to fund it?

My 3 votes for the posters went to: 16, 17 and 10.

The slides follow below, thanks to a new feature in The Fascinator that allows blog posts (and soon repository deposits) to be composed of more than one digital item.

Copyright Peter Sefton, 2009. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

Slides

Page 1

Page 2

Page 3

Page 4

Page 5

Page 6

Page 7

Page 8

Page 9

Page 10

Page 11

Page 12

Page 13

Page 14

Page 15

Page 16

Page 17

Page 18

Page 19

Page 20

Page 21

]]> http://ptsefton.com/2009/12/09/drfic-tokyo.htm/feed 2 ICE Week, background http://ptsefton.com/2009/11/25/ice-week-background.htm http://ptsefton.com/2009/11/25/ice-week-background.htm#comments Wed, 25 Nov 2009 01:52:26 +0000 ptsefton http://ptsefton.com/2009/11/25/ice-week-background.htm
PDF version

This week at the Australian Digital Futures Institute we’re taking a look at ICE: the Integrated Content Environment, an open source software system which is now part of the core infrastructure at USQ.

The ADFI team are spending all week exploring new stuff and considering what ICE 3 might look like and how it might fit with or merge with our other project, The Fascinator.

The week kicked off with a couple of sessions with the USQ community. The first session was by invitation, we asked a number of USQ staff who have engaged with ICE over the last few years along, including some who have been vocal critics. The second session was an open-invitation ADFI event. In both cases we went through this agenda:

  • How did we get here? (and what did we learn?)

  • Your concerns

  • What should we do next?

Bron Chandler has summarised some of the points already, It’s half way through the week now and I have been jotting down my thoughts in this post since Monday. This is a bit long winded, but I hope it is a useful record of some of the lessons learned from the last few years of content-creation systems at USQ. It’s on my blog, so it’s my opinion, comments are welcome to expand, clarify, correct or argue.

Remember that USQ is a distance education-specialist and produces lots of courseware, book-length study materials that are delivered in print/PDF and/or HTML to thousands of students. Teaching and learning is not all about courseware, of course, but courseware is a big part of what we do, and that’s what ICE was developed for.

History

FrameMaker

Adobe FrameMaker has been used since the 1990s at USQ to produce courseware for print. It’s still used for some print-only courses (largely ones with lots of maths I think). The problem with our use of FrameMaker is all the manual steps involved. Authors submit manuscripts in Word, these are converted to Frame by operators, largely by hand, and then proofing is done by the academics in Word with track changes, or on paper with corrections. Staff in Electronic Publishing Services then have to update the Frame documents change-by-change. This legacy process is being phased out, but these things take time.

USQ has never had a fully automated web-conversion process for FrameMaker documents although there have been a couple of attempts, and there is now a process for converting to ICE in a semi-automated way. In a decidedly un-automated process from 1999 or so until 2003-ish USQ used to send FrameMaker documents over to NextEd at the USQ Toowoomba campus to have courses converted to HTML by hand. This was a process which I recall costing circa $5000 US per course payable every semester, because of the almost complete lack of automation they had to be redone with each offering. Apparently some of the HTML courses from the NextEd days are still extant and being maintained using DreamWeaver and the like.

Bron Chandler and Ron Ward and I from the current ADFI team all worked for NextEd, and we still go to the support-group meetings from time to time but I think we’re getting better.

At NextEd I devised a word-processor based publishing system built around FrameMaker which would have slashed the cost of producing courses for both print and HTML at USQ. The system used Frame’s structured editing mode using XHTML as a document model, so you could produce high quality print documents and export them straight to the web – and because it used styles in FrameMaker you could import word processing documents and have FrameMaker automatically add structure. NextEd got a pilot system up in a few weeks, with help from Allette Systems. I thought it was rather a good idea but USQ was not interested in that, or our other publishing system the CPS, because they had other plans, in the form of an XML system.

Lessons learned?

I think it was pretty broadly recognised that all the manual work involved in producing courseware this way was not sustainable and not sensible which is why USQ tried to build an XML publishing system. The main lesson here is that systems change can take a long time in a university and document conversion is always slow and expensive, which is why we still have some FrameMaker documents.

LaTeX

Before looking at the XML world, though, there’s another legacy system, LaTeX.

  • Used in some technical disciplines.

  • No universal web conversion (really, there isn’t).

There’s not a whole lot to say about this, the people who know how to use it continue to do so to produce courseware, and it continues to make no inroads into other disciplines, I hear, though, that in engineering they are starting to get new staff who don’t know LaTeX. The problem is that Word and OpenOffice.org are not great when it comes to very large amounts of Maths, and ICE uses OpenOffice.org to make PDF, which is a bit buggy for maths so there is still a place for LaTeX, even if it is embedding LaTeX in word processing documents and automatically rendering the maths at least until further investments are made in ICE to improve its MathML support. USQ needs to work this out.

Participants in the sessions this week emphasised some of the good points of LaTeX:

  • BibTex referencing is vastly superior to anything else, we heard. (I’m not an expert but I believe (via Bruce Darcus) if this is true it is only so for some scientific disciplines. ICE works with the university supported EndNote and with the open source Zotero).

  • LaTex’s rendering is better than we produce using ICE/FrameMaker. (Certainly true for maths, probably not that important for most of the rest of what we do – but even if you can produce PDF with links is that enough? For web use I’d like to see more usable fluid materials like the stuff we did in ICE-TheOREM with live, interactive chemical models embedded in web pages, or interactive maps for documents with geographical references, easy in HTML and not possible in PDF.)

We had a follow-up meeting about LaTeX (and online document editors, more of which later) and reached the conclusion that ICE might be able to help manage LaTeX files and bundle them as courses, if communities of users could agree on which LaTeX stlyes or macros to use. Some people insist that for maths-heavy courseware the only practical delivery medium is PDF.

Lessons?

We’ve learned that where a community is using a tool that meets their requirements it is best to leave them to it. Apparently, though there is pressure on some of the LaTeX users to use ICE as per USQ policy. Me, I think the policy should be based on performance outcomes not mandating how people should achieve the outcomes. I’d love to see a competitor/successor to ICE emerge from someone who’s a bit feral and won’t use the corporate tools (but it aint going to be LaTeX-based).

GOOD

The GOOD system (Generic Online Offline Delivery) was USQ’s was a very big, complicated bespoke XML system which never realised its claimed potential. The idea was to build a semantically aware highly structured courseware production system.

There were several issues with GOOD:

  • Converting content from FrameMaker was a very costly, slow process.

  • Despite the best efforts of the team to design the one true Document Type Definition (a DTD was an ancient kind of document schema they had back in the twentieth century) for USQ courseware, there was exception after exception as more disciplines came on board and wanted different conventions, referencing systems, extensions to the table model and so on and so on, with every discussion ending up in a big meeting to see whether a change was needed. Being a university we didn’t count the cost of those meetings, but I bet it added up, let alone the cost of all the changes.

  • It was slow to render documents, so changes like sorting out pagination or fixing typos were very painful.

  • But the biggest issue was that almost none of the lecturers used it, see below.

Lessons?

There were lots of things that USQ should remember from the GOOD experience, most of which were well understood in the XML/SGML community before the project started, but there’s nothing like experiencing these things for yourself:

  • Top-down mandating of system is risky, particularly if the people you’re telling what to do are academics in an Australian university. They’re not process workers who will take orders.

  • Don’t build a big system without sorting out basic issues like what editor you are going to use and testing to see whether the user community will be able to use it and actually do so.

  • The system produced HTML and PDF from the same source document alright but then so did the demo system we built at NextEd using FrameMaker at about 100th of the cost. I think maybe GOOD would have had more traction if all that semantic markup had been used to better effect, so people could see the point of the extra work and cost involved. When I joined USQ I worked with the GOOD team (particularity Oliver Lucido who now works with us in ADFI) to demo some of the potential but our proposals never made it into the learning management system which the students used.

CPS

The NextEd Continuous Publishing System was my baby at NextEd, sponsored by another current USQer Cameron Loudon who ran the conversion team. It was a word processor based HTML publishing system which many of our clients used and which was used for the company intranet. But even when customers didn’t fully embrace it for courseware, we were able to use it internally to dramatically cut costs.

  • Server based like ICE server.

  • Used an earlier version of ICE templates (inherited from Standards Australia where I helped set them up for writing standards).

  • No print output.

  • Banned at USQ’s Distance and e-Learning Centre. (True, even though there was a small user community who liked it, and it was open source it was considered a risk to use. I think one of the issues was whether the open source code was truly clean-room open, and then there was the GOOD system with which the CPS competed (except for the lack of print output)).

Lessons?

The CPS worked pretty well. Ron Ward did most of the work on the later versions, and together we learned a lot about modern software development and how to keep things as simple as possible. For example lots of the features we thought we needed it turned out we didn’t and even with the simplest possible model we could think of for document-reuse across semesters it was too hard for most users to get their heads around.

It was eventually made open source, but too late, meaning that all that work died with most of NextEd’s business. I resolved that if I worked on software for other people again I would either (a) get paid substantial amounts of money or (b) get to release the software under an open source license so I could build on it in future, never mind the other potential benefits of open source software.

The big thing that I miss about CPS which is so far not present in ICE is that it was very much driven by metadata, which means that courses self-assembled as you uploaded and described parts of a course. This is an area we are exploring with our ICE/Fascinator mashup which we hope will be used to serve all the universities policies and procedures in a faceted, browse/search interface before too long.

ICE

Regulars here probably know more than they want to about ICE and you can read about it on the website, and in various papers (Sefton et al. 2009; Sefton 2006; Sefton 2007).

  • Originally called GOOD-lite (2004) for (internal) marketing reasons and changed immediately to The Integrated Courseware Environment for (external) marketing reasons. (Then we changed Courseware to Content to go after a grant.)

  • Approved within DeC without the benefit of the kind of USQ governance we now have.

  • Grew organically from a user-base of one.

  • Core system in 2009.

Lessons?

Lots of lessons, but we’re still working out what they are. The big ones for me are:

  • Trying to build a replicated version controlled content management system as well as the core ICE function which was to make HTML and PDF courseware from word-processing files was a big mistake, cost us a lot for limited benefit. My fault, I think for getting carried away with architectural space-travel. We’re going to see if we can get away without using Subversion or anything remotely like it for future versions and focus on the things that ICE does that no other system does, mainly having good generic word processing templates and turning them into HTML. Yes it is strange that no other open system does this but no, we don’t know of anything comparable.

  • The agile, organic approach worked well to make the actual software but because we started the project under local governance, just before a big project to centralise uni IT services, by the time it was ready to roll out there was a whole new governance framework in place and it took longer than it should to navigate that. Future projects need to move into core mode much more smoothly.

Future directions?

I put up a slide for discussion with some bullet points, some brief notes here on each point:

  • Concept Maps? (Bron’s summary)

    Mark Phythian and co (Phythian & Das Gupta 2008) have been trialling Concept Maps as a learning and teaching aid. Mark started by wondering if Concept Maps could provide a navigation aid for courses, then started looking at how they might help learners. We had a meeting on Tuesday which affirmed that ADFI will keep helping with this line of research, with a view to building open tools for the use of concept maps in learning, teaching and research as indicated by ongoing evaluations like Mark has been doing.

    We also heard about Mind Maps (don’t get me started on that one) and something called Argument Maps which were new to me . Duncan is exploring how work on Aus-e-Lit might be used in our tools to build concept-map-like navigation or aggregation in our tools.

  • Efficiency:

    There were a few points that came down to ICE efficiency and performance; there are already processes under way to make ICE more responsive by getting some of the large video content out of it and into more suitable repositories, building towards university-wide media and courseware repository and discovery services. ADFI may have a role to play in developing some of this infrastructure, and we are gearing up to build and pilot some software along these lines in 2010.

    • Media repository?

      Yep, we know we need it and there are people looking at this. It is clear that we need some kind of repository of course content to remember what we served up to students; Bron Chandler is looking at the new version (2) of the Moodle learning management system to see how repository-like it is.

    • Drop versioning from ICE?

      It seems that the versioning is not one of ICE’s most used features and people would be happy to sacrifice it for extra speed. Some of the maths and computing people would lament the loss of subversion, but I figure they know how to type: svn add *; svn commit -m 'Finished for semester one!'

    • Syncing?

      Ditto.

  • GoogleDocs and other online editors?

    We had a group looking at online editors today. Stijn Dekeyser is particularly interested in either working with Google Docs and its APIs to do some ICE integration, or better yet designing a collaborative structured semantically aware editor. The latter sounds fun, but it would be a huge project and I think would be well beyond us without a very strong partner. We will look at opportunities for work in this area where we can. Via Anna Gerber I got a tip to look at Google Docs Base editor which uses the Google Docs APIs:

    AnnaGerber

    @ptsefton Have you seen Google Docs Base Editor? code.google.com/p/gdbe/

  • Google Wave (no)

    See my recent critique.

  • More LaTeX support?

    I wrote about this above. If the LaTeX users can organise themselves, there might be a case for extending ICE’s limited support for LaTeX to help with the course-management side of things. Look, if someone is telling a mathematician to dump LaTeX and use Word or openoffice.org just because university policy is to ‘use ICE’ then I think that’s wrong and I’m happy to help them fight their case. But I’m also going to fight against everyone who wants to just put PDF on the web and not take advantage of the web in every possible way, so I support PDF-only courses only where they are so maths-heavy there is no other practical way to deliver them right now.

  • Ebook delivery

    I have been using and loving an ebook reader on my android phone, which experience I will go into in more detail soon. There was interest in adding eBook publishing to ICE from both the library, where ex RUBRIC colleague Alison Hunter emphasised the importance of being able to deliver electronic books as part of the library of the future and from the Learning and Teaching Support Unit, where Michael Sankey thought eBook delivery would be important for open educational resources. (Which reminds me, need to make some more noise about Open Courseware and other open things we could be doing.)

    Linda Octalina and Cynthia Wong got ICE packaging books in EPUB format in less than a day and we’re ironing out bugs and testing in Stanza on the iPhone and Aldiko on Android. When that’s done we will had the plugin over to Michael at LTSU and see if we can get some trials going.

  • SiteCore/SharePoint?

    These are the corporate web CMS and intranet systems. It would be nice if they could understand ICE documents; we’re not going to tackle these this week, but I hope that USQ gets around to it one day. Given that we can produce high-quality HTML pages for courses from the tool that most people use for documents authoring (MS Word) it seems a pity not to extend that to more corporate documents. There is the forthcoming policies and procedure site which will show just how much better ICE documents are than standard ad-hoc unstlyed word documents.

  • Moodle tie-in?

    This is a big one, for which we don’t have a lot of data yet. Bron chandler is investigating Moodle 2.

  • Theses?

    Nothing much to report since August, but I hope to talk more with people at ANU about theses, as I write this my honours thesis is not function in ePub format, possibly because the title it too long: MAKING PLANS FOR NIGEL: Defining interfaces between computational representations of linguistic structure and output systems: Adding intonation, punctuation and typography systems to the PENMAN system.

  • Journals (OJS)?

    There’s a project to get OJS up at USQ, and we have had a long running dialogue with the PKP folks about ICE integration, but nothing new to report at this stage.

  • Annotations!!

    There was strong support for taking the kind of in-document in-browser annotation we have in ICE as an authoring service and making it more generally available. Ron Ward is working with Duncan Dickinson to see if we can get the open Dannotate system going in our systems with a view to having rich discussions in-context as part of the teaching and learning process. We are keen to get annotations going for eResearch too, for peer review, and for public participation in research in our work with the Public Memory Research Centre.

  • Various contributions about potential functionality from Michael Sankey (thanks!): syndication of audio and video content from YouTube, Facebook, inline quizzes, infopath/Sharepoint forms integration. Implementation of a global glossary and integration with and Mahara.

    We’re going to find out more about Mahara and how we might bridge between it and ICE and media repositories and Eprints and our eReserve system and library catalogue and so on.

Working parties

And we have 4 mini ’sprint’ developments/investigation happening in the tech team this week. These are designed to explore some of the many things that were brought up by our colleagues where we think we can get something to show quickly that will advance ICE significantly:

  1. Annotations: Ron Ward & Duncan Dickinson. The point here is to look for generic annotation services for academia covering comment, discussion, notes for personal use, peer review, marking and so on, using a common web based system across multiple web applications. The basis for this is Dannotate, to which we are adding some user-interface tweaks

  2. ePub-format books: Linda Octalina and Cynthia Wong have this mostly working – we’ll post some examples soon.

  3. Packaging of arbitrary collections of resources (ICE, flickr, images, data) using The Fascinator: Oliver Lucido is looking at a proof of concept here; to show a possible ICE-future.

  4. Moodle 2 integration possibilities: Bron Chandler. This is fact-finding at this stage, but if we can hook in any of the above into a demo then we will.

References

Phythian, M.W. & Das Gupta, J., 2008. Hyperlinked concept map enhancements for electronic study materials. Available at: http://eprints.usq.edu.au/4776/ [Accessed November 24, 2009].

Sefton, P., 2007. An integrated approach to preparing, publishing,

presenting and preserving theses. In ETD 2007. Uppsala. Available at: http://eprints.usq.edu.au/archive/00002653/ [Accessed July 2, 2007].

Sefton, P., 2006. The Integrated Content Environment for Research and Scholarship. ICE Website. Available at: http://ice.usq.edu.au/introduction/ice_rs.htm [Accessed April 30, 2007].

Sefton, P., Downing, J. & Day, N., 2009. ICE-theorem – end to end semantically aware eResearch infrastructure for theses. University of Southern Queensland. Available at: http://eprints.usq.edu.au/5248/1/ice-theorem-paper-OR09.htm [Accessed August 24, 2009].

Copyright Peter Sefton, 2009. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2009/11/25/ice-week-background.htm/feed 4 Promising developments in the Australian Access Federation http://ptsefton.com/2009/11/18/promising-developments-in-the-australian-access-federation.htm http://ptsefton.com/2009/11/18/promising-developments-in-the-australian-access-federation.htm#comments Wed, 18 Nov 2009 05:11:12 +0000 ptsefton http://ptsefton.com/2009/11/18/promising-developments-in-the-australian-access-federation.htm
PDF version

At the eResearch Australasia conference last week I attended a Birds of a Feather session, Authorisation community developments run by Markus Buchhorn (Intersect), Lyle Winton (Victorian eResearch Strategic Initiative), Clare Sloggett (Intersect) and Neil Witheridge (ARCS). Clare asked me to come along, possibly because she knew I had been critical of some aspects of the Australian Access Federation machinery and how it is supposed to work. I am pleased to report, though that it sounds like some things have changed for the better.

About a year ago, I addressed the ARROW repository community about the then-new Object Reuse and Exchange (ORE) standard. I started with a slightly belligerent rant about another standard, XACML (eXtensible Access Control Markup Language). XACML was supposed to be a part of how the Australian Access Federation (AAF) was supposed to allow all of us in the Australian Higher-Education and research community to access each other’s stuff based on our roles.

Back then I noted the difficulty with defining role attributes in a way that would work cross-institution:

I was always vaguely worried about how XACML policies were going to work but one day I met Kent Fitch who really nailed it. On the subject of these use cases for XACML where you, an anthropologist want to grant access to a repository to other anthropologists, he asked “What’s an anthropologist?”

This is a very, very good question. Does an academic working in the education faculty who self-identifies as a visual ethnographer qualify? What if she’s got an honours degree in anthropology? […] [fixed a spelling error]

(I finally have an answer for Kent about what an anthropologist is. Read on.)

My previous understanding of how the AAF would work was that my host organisation would have to be solely responsible for asserting stuff about me, who I was and what roles I had, because there was no publicly shareable identifier to allow other people to assert things about me, such as that they trusted me to look at some data or edit a document. This meant that simple, obvious use-cases like a group of ethnographers setting up a collaboration space by asking each other for their login names would not be possible; due to privacy concerns there would be a unique ID for each AAF person but there would be no sharing.

I’m pleased to report that things have changed and the AAF now does encourage the use of public shareable IDs. So a few points occur to me. The first couple I raised at the BoF, the last one came later.

  1. This means we can let people self-organise by adding white-lists of known IDs to their systems. An institution might not be a reliable way to sort the anthropologists from the ethnographers, but they can sort themselves out all right and form whatever working groups they want.

  2. It was noted at the BoF that some people would have more than one AAF ID, I suggested that it might be good to register these in the new researcher ID system that I believe is being set up at the National Library as part of the ANDS infrastructure. I think this new NLA system will be what Nick Nicholas calls an Identifier Assertion Hub.

  3. There was some talk of systems that could manage groups of AAF users. This answers the bigest problem I had with the AAF not being able to work with ad-hoc or user-defined communities, only with role attributes assigned by the home identity provider.

    Following from this I finally worked out how to answer Kent’s question “what’s an anthropologist”. An (Australian) anthropologist is someone who’s a member of the he Australian Anthropological Society.

    In an identity federation that does include public, shareable identifiers, societies could publish lists of members or run a group-server that could be used by other services.

Finally, I can’t help commenting that if we’re going to have a federation where a lot of the trust relationships are devolved to the users, allowing them to assemble groups, or to societies like the AAS then maybe we should consider allowing the use of OpenId as well as or instead of Shibboleth. I bet not everyone in the AAS has an institutional login at an AAF member, even once the AAF is fully operational with wall to wall Shibboleth, but anyone can get an OpenId.

Copyright Peter Sefton, 2009. Licensed under Creative Commons Attribution-Share Alike 2.5 Australia. <http://creativecommons.org/licenses/by-sa/2.5/au/>

HTTP://DBPEDIA.ORG/SNORQL/?QUERY=SELECT+%3FRESOURCE%0D%0AWHERE+{+%0D%0A%3FRESOURCE+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%2FBIRTHPLACE%3E+%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FRESOURCE%2FSYDNEY%3E+%3B%0D%0A%3CHTTP%3A%2F%2FDBPEDIA.ORG%2FONTOLOGY%2FPERSON%

This post was written in OpenOffice.org, using templates and tools provided by the Integrated Content Environment project and published to WordPress using The Fascinator.

]]> http://ptsefton.com/2009/11/18/promising-developments-in-the-australian-access-federation.htm/feed 2