Validation

Efforts so far around BibJSON have focussed on building a useful JSON representation of bibliographic metadata, with some typical key/value pairs that are common in or extended from bibtex. This started off simply, but we have seen increasing complexity to accommodate further functionality requests. There was some work on a JSON schema for validation against, but given the aim of being as flexible as possible, and with very few required keys, the function of validation of a BibJSON document would have very little effect.

Validating a document as properly formatted JSON is, of course, a good idea; but there are plenty ways to do this already – just try to parse it with any number of libraries for your programming language of choice.

But to reach the stage of actually supporting validation against a pre-defined schema, we must pre-define a schema – and that means becoming inflexible (or doing such little validation as for it to be essentially pointless).

An alternative to validation against a schema would be adoption of namespaces.

Namespaces

We do already have a namespace concept in BibJSON – it is just a key in the metadata, under which can be listed namespaces and a suitable prefix for them. However, this model is not widely known (because we made it up). To overcome this, we should adopt the JSON-LD method of using @context parameters. This way, it would be possible to specify the namespace in which your record keys are defined, and to share namespace information with other people / machines.

What is the point

Using namespaces, having schema, only become sensible when there is a concerted effort to share data with others. For internal use, they could be valuable for consistency, but the code we write internally adheres by definition to our own level of consistency anyway.

Therefore, it is not a function of BibJSON to perform validation – BibJSON is just JSON. Rather, it is the function of a community to make agreements and to conform to those agreements as required.

Where such a function must be supported, it should be done via mechanisms already available and maintained for that purpose – there is no point attempting to maintain our own; it is not our key strength or goal.

Recommendation

Change the BibJSON use of namespaces to conform to the method specified in JSON-LD, and that wherever consistency is required, agreement to share data via JSON and within a particular @context should be reached.

The fundamental basic keys in BibJSON – the default context – should remain as they are, and should not require contextualisation.

If contextualisation of the fundamental keys of BibJSON is required, then those keys should be contextualised into a schema by whomsoever has such a requirement.

Ramifications

• drop the “namespace” key in BibJSON
• continue using BibJSON as normal, but:
• reference JSON-LD for use of @context and other more complex LD functions as required
• wherever validation is required, perform it based on the use of namespaced keys (beyond scope of bibjson)

References

• http://bibjson.org
• http://json-ld.org/spec/latest/
• http://openbiblio.net/2011/05/05/follow-up-to-serialising-rdf-in-json/

Peter has been blogging for a while about BibSoup (see here for the basics and here for how to use it) and we’ve mentioned it in passing on this blog (for example this sprint post and explanation of Bib- terms)… But now it is time for the ‘official’ launch. Hurrah!

So, how to get involved?

• View the features of the BibServer software
• Sign up at BibSoup.net and try uploading and viewing your collection
• Check out the example collections that some people have created already
• Watch our videos and screencasts and look out for more
• BibServer is available for use via BibServer.org and the repository
• Sign up to the OpenBiblio principles
• Learn more about our bibliographic JSON format at BibJSON.org
• Watch this demo of BibSoup:

Setting up a Bibserver and Faceted Browsing (Mark MacGillivray) from Bibsoup Project on Vimeo.

We already have parsers available to get your data directly via either BibTex or RIS (or from BibJSON…), which means you can get data in from most major bibliographic tools already; you can even use the parsers programmatically if you like, at http://bibsoup.net/parse (although that functionality is in the process of improvement). We are open to suggestions for further parsers, and would be happy to guide anyone through making one.

(By the way, we are assuming you will have seen previous posts on this site and will therefore know what we’re talking about, but if not then please see this OKFN blog post for a fuller explanation of what BibSoup is for, why it’s great and what this overall project is all about).

So, what do you think? Let us know. There will be bugs, or areas we could improve, so please pass suggestions our way. Feature requests can be submitted via our issue tracker, and we batch those up into milestones to work towards the next release. Our current focus is on improving parser functionality and also on enabling editing.

We hope you like it and find all this useful… do add your collections so we can share them with the rest of the world, too. If you would like your own BibServer, go ahead and download the code, or contact us for help / support options.

Any proposal for discussion is published ahead of the meeting at which it is to be raised, with an email inviting everyone on the openbiblio-dev list to the call and linking to the etherpad (which contains or links to further details). This is to ensure materials are available in advance of calls.

Everyone is free to suggest direction, which is agreed by consensus. Technical lead is Mark, and Community lead is Naomi.

All discussion should be carried out openly, on the available mailing lists. Agreement before publication of blog posts or pages is not required – any early discussions off-list should be posted on the appropriate mailing list for discussion, then posted on the blog if they come to fruition. Strategic and management proposals should make it clear that they are for discussion until the team has covered the topic at the weekly catch-up (Wednesdays at 16.00 GMT).

Recently, a discussion on the Working Group List raised the subject of existing technologies that store and share reading / publication lists, and how BibSoup / BibServer compares to them.

Tom Morris said:

Perhaps it would be illustrative to compare and contrast with other existing widely known services and tools such as Zotero, Mendeley, CiteUlike, and the venerable emacs/Bibtex/LaTex. What is better, worse, or just different? Which sets of things are alternatives to each other and which complement each other? What are the things which make BibSoup/BibServer unique?

Of course, if this is already laid out in detail somewhere on a web page, just point me there.

There wasn’t, until now, so please refer to this wiki page http://wiki.okfn.org/Projects/jiscopenbib2/managementtools set up for this purpose and start comparing!

I have used Thad Guidry’s notes on Mendeley, as well as the first line of the Wikipedia entry, to populate that example. Please do edit and add to this page – we want to avoid a debate on which is better than which, so please keep your opinions in check, but hopefully this will be a good opportunity to get a sense of what is in use and how they compare with one another and BibSoup.

• The OCLC released Faceted Application of Subject Terminology (FAST) as Linked Data under an open licence. Their press release gives more information (please note, at present the full database is limited to API access, but OCLC will be making FAST available as a dump under the ODC-BY license in the near future).
• NISO is working on a standard for the coding of citations in ebooks, full details here.
• Our own Principles on Open Bibliographic Data have been translated into Hungarian (by Dudás Anikó) and Polish (by Karol Langner) so we hope this is the beginning of improved communication in both Hungary and Poland and with native speakers all over the world.
• OKFN has been working on various ways of improving the overall field of Open Data, including the Open Metadata Handbook which is designed to help make it easier to harvest and process bibliographic metadata from a variety of sources, for people with little or no technical background, explaining the metadata standards as used by different institutions.
• And of course, Peter Murray-Rust has been busy on his blog, exploring issues of US bill HR3699, how to present your list using BibSoup and generally promoting the world of open data.
• Finally – watch this space for an announcement about BibSoup, launching in beta later this week!

Many thanks to the terrific group of people who keep us updated with this information. If you would like to be among the first to hear about developments like these, get involved with the Open Bibliographic Working Group and let us know what you’re up to.

Datos.bne.es is an initiative of the Biblioteca Nacional de España (BNE) whose aim is to enrich the Semantic Web with library data.

This initiative is part of the project “Linked Data at the BNE”, supported by the BNE in cooperation with the Ontology Engineering Group (OEG) at the Universidad Politécnica de Madrid (UPM). The first meeting took place in September 2010, whereas the collaboration agreement was signed in October 2010. The first set of data was transformed and linked in April 2011, but a more significant set of data was done in December 2011.

The initiative was presented in the auditorium of the BNE on 14th December 2011 by Asunción Gómez-Pérez, Professor at the UPM and Daniel Vila-Suero, Project Lead (OEG-UPM), and by Ricardo Santos, Chief of Authorities, and Ana Manchado Mangas, Chief of Bibliographic Projects, both from the BNE. The attendant audience enjoyed the invaluable participation of Gordon Dunsire, Chair of the IFLA Namespace Group.

The concept of Linked Data was first introduced by Tim Berners-Lee in the context of the Semantic Web. It refers to the method of publishing and linking structured data on the Web. Hence, the project “Linked Data at the BNE” involves the transformation of BNE bibliographic and authority catalogues into RDF as well as their publication and linkage by means of IFLA-backed ontologies and vocabularies, with the aim of making data available in the so-called cloud of “Linked Open Data”. This project focuses on connecting the published data to other data sets in the cloud, such as VIAF (Virtual International Authority File) or DBpedia. With this initiative, the BNE takes the challenge of publishing bibliographic and authority data in RDF, following the Linked Data Principles and under the CC0 (Creative Commons Public Domain Dedication) open license. Thereby, Spain joins the initiatives that national libraries from countries such as the United Kingdom and Germany have recently launched.

Vocabularies and models

IFLA-backed ontologies and models, widely agreed upon by the library community, have been used to represent the resources in RDF. Datos.bne.es is one of the first international initiatives to thoroughly embrace the models developed by IFLA, such as the FR models FRBR (Functional Requirements for Bibliographic Records), FRAD (Functional Requirements for Authority Data), FRSAD (Functional Requirements for Subject Authority Data), and ISBD (International Standard for Bibliographic Description).

FRBR has been used as a reference model and as a data model because it provides a comprehensive and organized description of the bibliographic universe, allowing the gathering of useful data and navigation. Entities, relationships and properties have been written in RDF using the RDF vocabularies taken from IFLA; thus FR ontologies have been used to describe Persons, Corporate Bodies, Works and Expressions, and ISBD properties for Manifestations. All these vocabularies are now available at Open Metadata Registry (OMR), with the status of published. Additionally, in cooperation with IFLA, labels have been translated to Spanish. MARC21 bibliographic and authority files have been tested and mapped to the classes and properties at OMR. The following mappings were carried out:

• A mapping to determine, given a field tag and a certain subfield combination, to which FRBR entity it is related (Person, Corporate Body, Work, Expression). This mapping was applied to authority files.
• A mapping to establish relationships between entities.
• A mapping to determine, given a field/subfield combination, to which property it can be mapped. Authority files were mapped to FR vocabularies, whereas bibliographic files were mapped to ISBD vocabulary. A number of properties from other vocabularies were also used.

The aforementioned mappings will be soon available to the library community and thus the BNE would like to contribute to the discussion of mapping MARC records to RDF; in addition, other libraries willing to transform their MARC records into RDF will be able to reuse such mappings.

Almost 7 million records transformed under an open license

Approximately 2.4 million bibliographic records have been transformed into RDF. They are modern and ancient monographies, sound-recordings and musical scores. Besides, 4 million authority records of persons, corporate names, uniform titles and subjects have been transformed. All of them belong to the bibliographic and authority catalogues of the BNE stored in MARC 21 format. As for the data transformation, the MARImbA (MARc mappIngs and rdf generAtor) tool has been developed and used. MARiMbA is a tool for librarians, whose goal is to support the entire process of generating RDF from MARC21 records. This tool allows using any vocabulary (in this case ISBD and FR family) and simplifies the process of assigning correspondences between RDFS/OWL vocabularies and MARC 21. As a result of this process, about 58 million triples have been generated in Spanish. These triples are high quality data with an important cultural value that substantially increases the presence of the Spanish language in the data cloud.

Once the data were described with IFLA models, and the bibliographic and authorities catalogues were generated in RDF, the following step was to connect these data with other existing knowledge RDF databases included in the Linking Open Data initiative. Thus, the data of the BNE are now linked or connected with data from other international data source through VIAF, the Virtual International Authority File.

The type of licence applied to the data is CC0 (Creative Commons Public Domain Dedication), a completely open licence aimed at promoting data reuse. With this project, the BNE adheres to the Spanish Public Sector’s Commitment to openness and data reuse, as established in the Royal Decree 1495/ 2011 of 24 October, (Real Decreto 1495/2011, de 24 de octubre) on reusing information in the public sector, and also acknowledges the proposals of the CENL (Conference of European National Librarians).

Future steps

In the short term, the next steps to carry out include

• Migration of a larger set of catalogue records.
• Improvement of the quality and granularity of both the transformed entities and the relationships between them.
• Establishment of new links to other interesting datasets.
• Development of a human-friendly visualization tool.
• SKOSification of subject headings.

Team

From BNE: Ana Manchado, Mar Hernández Agustí, Fernando Monzón, Pilar Tejero López, Ana Manero García, Marina Jiménez Piano, Ricardo Santos Muñoz and Elena Escolano. From UPM: Asunción Gómez-Pérez, Elena Montiel-Ponsoda, Boris Villazón-Terrazas and Daniel Vila-Suero.

1. changed its licensing regime for Linked Data to CC0 which makes the data open according to the open definition,
2. has begun to publish the German national bibliography as Linked Open Data.

For background see the email (German) announcing this step. There it says (my translation):

“In 2010 the German National Library (DNB) started publishing authority data as Linked Data. The existing Linked Data service of the DNB is now extended with title data. In this context the licence for linked data is shifted to “Creative Commons Zero.

Until now, the majority of DNB title data is implemented as well as periodicals and series – the music data and holdings of the German Exiles Archive are missing. From now on, the RDF/XML representation of a title record is available in the DNB portal via a link. This is expressly an experimental service which will be extended and refined continually. More detailed informations about modelling questions and the general approach can be fund in the updated documentation.“

The English documentation (PDF) hasn’t been updated yet and only describes the GND authority data. On the wiki page about the LOD service it says: “Examples and further information about FTP-downloads will come soon.” An entry on the Data Hub has already been made for the data.

If, like me, you are not of a technical bent, do not despair – Mark has created a straightforward video guide on how to use it and how it’s useful:

Setting up a Bibserver and Faceted Browsing (Mark MacGillivray) from Bibsoup Project on Vimeo.

Mark Williamson, a post-doctoral researcher at Cambridge University, was introduced to BibServer and we filmed him talking about using it a (very) short while later:

Ingesting a personal collection of references into Bibserver (Mark Williamson) from Bibsoup Project on Vimeo.

Thanks to Peter for his camera and video-production skills, and of course his blog.

Interview with Mark MacGillivray, Openbiblio 2 project from Bibsoup Project on Vimeo.

Peter Murray-Rust Co-I Openbiblio project talks from Bibsoup Project on Vimeo.

Bibsoup: Interview with Etienne Posthumus (developer) from Bibsoup Project on Vimeo.

Interview with Ed Chamberlain, Openbiblio2 project from Bibsoup Project on Vimeo.

Thanks to Peter for the videos as presented on his brilliant blog!

• Sam and Mark discussed the interface between Open Biblio and the TEXTUS project, looking at text and image processing. Project Gutenberg was suggested as one possible avenue, exploring scanned archives being processed in order to provide searchable text. It was agreed that openphilosophy.org would be the best central point of reference for this data, as an instance of TEXTUS with BibServer support in the background.
• Mark and Ira discussed how to present CKAN / BibServer at events such as Dev8D – there is cross-over between the two, and we took the opportunity to learn more about both projects. CKAN is a purpose-built data catalogue with flexible addons and a mature open source product. Both are part of the OKFN and, combined, are an easy way to publish and find data and references. It was agreed that working more closely together would be of mutual benefit as well as to the wider community.
• Peter and Mark discussed BibServer in terms of where we could offer CKAN and other services to academic / research groups, as a stack of tools that would find beneficial to their work. There is a lots of talk just now about using dspace, e-prints or some as yet uninvented system for storing research data – JISC is funding some projects, and we will be having a discussion about this at Dev8D.
• Sam, Primavera and Etienne hacked some code and Etienne also continued his work on the parsers.
• Peter, Mark and I discussed BKN – the Bibliographic Knowledge Network – which is Jim Pitman’s project and the first BibServer… Follow-up happening next week.
• Peter and I interviewed Mark Williamson, a post-doctoral researcher at the Chemistry Department, about using BibSoup (which he’d only looked at for a few moments before we put him on the spot – thanks Mark!). Mark also gave us a demonstration of using BibSoup for the blog which is a good ‘how to’ for people who haven’t used it before. Peter’s excellent summary of BibSoup goes as follows: “BibSoup is a philosophy rather than a technology – ie having local control over bibliographic data. The idea is to get people to share data together and to sign-up to supporting it in 5 years’ time”. We will follow up soon with links to the videos we made.

The main aim for the past few days was to get all the people working on the project together, with the aim of Getting Stuff Done: do some coding, boot up some dataset demos, plan more demos and integrations, plan further community engagement and coding over the next six months, integrating BibServer with other projects, etc… Amongst all the lively discussions, I think it’s safe to say the aim was achieved!

Many thanks to all involved – if anything from this week strikes you as particularly interesting, please do get involved.