It struck me that as other JISC projects such as ours are required to make their output similarly publicly available, this discussion may be of benefit to others; after all, not everyone knows what open source software is, let alone the complexities that can arise from trying to create such software. Whilst we cannot help avoid all such complexities, we can at least detail what we have found helpful to date, and how OSS Watch view our efforts.

I provided Sander and Rowan a review of our project, and Rowan provided some feedback confirming that overall we are doing a good job, although we lack a listing of the other open source software our project relies on, and their licenses. Whilst such data can be discerned from the dependencies of the project, this is not clear enough; I will add a written list of dependencies to the README.

The response we received is provided below, followed by the overview I initially provided, which gives a brief overview of how we managed our open source development efforts:

==== Rowan Wilson, OSS Watch, responds:

Your work on this project is extremely impressive. You have the systems in place that we recommend for open development and creation of community around software, and you are using them. As an outsider I am able to quickly see that your project is active and the mailing list and roadmap present information about ways in which I could participate.

One thing I could not find, although this may be my fault, is a list of third party software within the distribution. This may well be because there is none, but it’s something I would generally be keen to see for the purposes of auditing licence compatibility.

Overall though I commend you on how tangible and visible the development work on this project is, and on the focus on user-base expansion that is evident on the mailing list.

==== Mark MacGillivray wrote:

Background – May 2011, OKF / AIM bibserver project

Open Knowledge Foundation contracted with American Institute of
Mathematics under the direction of Jim Pitman in the dept. of Maths
and Stats at UC Berkeley. The purpose of the project was to create an
open source software repository named BibServer, and to develop a
software tool that could be deployed by anyone requiring an easy way
to put and share bibliographic records online.

A repository was created at http://github.com/okfn/bibserver, and it
performs the usual logging of commits and other activities expected of
a modern DVCS system. This work was completed in September 2011, and the repository has been available since the start of that project with a GNU Affero GPL v3 licence attached.

October 2011 – JISC Open Biblio 2 project

The JISC Open BIblio 2 project chose to build on the open source
software tool named BibServer. As there was no support from AIM for
maintaining the BibServer repository, the project took on maintenance
of the repository and all further development work, with no change to
previous licence conditions.

We made this choice as we perceive open source licensing as a benefit
rather than a threat; it fit very well with the requirements of JISC
and with the desires of the developers involved in the project. At
worst, an owner may change the licence attached to some software, but
even in such a situation we could continue our work by forking from
the last available open source version (presuming that licence
conditions cannot be altered retrospectively).

The code continues to display the licence under which it is available,
and remains publicly downloadable at http://github.com/okfn/bibserver.
Should this hosting resource become publicly unavailable, an
alternative public host would be sought.

Development work and discussion has been managed publicly, via a
combination of the project website at
http://openbiblio.net/p/jiscopenbib2, the issue tracker at
http://github.com/okfn/bibserver/issues, a project wiki at
http://wiki.okfn.org/Projects/openbibliography, and via a mailing list
at openbiblio-dev@lists.okfn.org

February 2012 – JISC Open Biblio 2 offers bibsoup.net beta service

In February the JISC Open Biblio 2 project announced a beta service
available online for free public use at http://bibsoup.net. The
website runs an instance of BibServer, and highlights that the code is
open source and available (linking to the repository) to anyone who
wishes to use it.

Current status

We believe that we have made sensible decisions in choosing open
source software for our project, and have made all efforts to promote
the fact that the code is freely and publicly available.

We have found the open source development paradigm to be highly
beneficial – it has enabled us to publicly share all the work we have
done on the project, increasing engagement with potential users and
also with collaborators; we have also been able to take advantage of
other open source software during the project, incorporating it into
our work to enable faster development and improved outcomes.

We continue to develop code for the benefit of people wishing to
publicly put and share their bibliographies online, and all our
outputs will continue to be publicly available beyond the end of the
current project.

We will be making the new version available over the course of this week, and will have it up and running on http://bibsoup.net soon.

Below is an overview of the new functionality you can expect to see over the course of the next week; we will write some blog posts about the various new capabilities, and this will tie in with the focus of the next sprint – doing docs, tests and issues (no new functionality).

• editing of records and collections
• merging collections from multiple sources
• adding notes to records
• much improved search UI
• embed images in search results
• better visualisation of collections
• embeddable UI into other web pages via javascript
• asynchronous parsing – you don’t have to hang on the page waiting for it to complete
• feedback tickets from asynchronous parses
• sharing collection admin rights with other users
• new parser for NLM XML
• new parser concept – search term gets pages from wikipedia, pulls citations from pages
• capability to accept and run parsers written in different programming languages
• browse site users

Thinking of ways to show the possibilities of linked data, we have made a Wikipedia gadget, making use of a great resource the Wikimedia developers give to the community.

Wikipedia gadgets are small pieces of code you can add to your Wikipedia user templates, and allow you to add more functionality and render more information when you browse wikipedia pages.

In our case, we wanted to retrieve information from our bibliographica site to render in Wikipedia, and so as the pages are rendered with specific markup we can use the ISBN numbers present on the wikipedia articles to make consults to the bibliographica database, in a way similar to what Mark has done with the Edinburgh International Science Festival.

Bibliographica.org offers an isbn search endpoint at http://bibliographica.org/isbn/, so if we ask for the page http://bibliographica.org/isbn/0241105161 we receive

[{"issued": "1981-01-01T00:00:00Z", "publisher": {"name": "Hamilton"}, "uri": "http://bnb.bibliographica.org/entry/GB8102507", "contributors": [{"name": "Boyd, William, 1952-"}], "title": "A good man in Africa"}]

I can use this information to make a window pop up with more information about works when we hover their ISBNs on the Wikipedia pages. If my user templates has the bibliographica gadget, every time I open a wiki page the script will ask information about all the ISBNs the page has to our database.
If something is found, it will render a frame around the ISBN numbers:

And if I hover over them, I see a window with information about the book:

Get the widget

So, if you want to have this widget, first you need to create an account in the wikipedia, and then change your default template to add the JavaScript snippet. Once you do this (instructions here ) you will be able to get the information available in bibliographica about the books.

Next steps

By now, the interaction goes in just one direction. Later on, we will be able to feed that information back to Bibliographica.

OFS is a python library that seeks to capitalise on their similarities – providing a single, general API to put and get files from one of these services while hiding the specifics of the implementation from the user. This allows for local testing and development before transitioning to using one of the cloud services, services which typically cost real money and slows down testing due to the necessity of communicating with these services over an internet connection.

Characteristics of OFS:

• Uses a ‘bucket/label’ mechanism to identify individual files
• Provides a list of content in a given bucket (as best as that the service can provide)
• Provides per-file metadata in so far as the service can provide (key-value or JSON encode-able data)
• Current backend plugins:
  • Local storage – based on the pairtree specification that optimises file-distribution across a native file-system to handle large quantities of files. Uses JSON to encode arbitrary metadata about the files in a given bucket.
  • Remote storage (S3 and Archive plugins written by Friedrich Lindenberg (pudo) who has also made large contributions to the codebase):
    • Amazon S3
    • Archive.org
    • Riak (in progress)
  • Also in progress – a REST Client by Friedrich Lindenberg (pudo)
  • One key desire is to provide opaque sharding – breaking up very large files to spread across buckets or even systems to improve performance and the range of services or backend systems OFS can make use of.

It is plain that having the ability to write storage code in a common way, but make use of local as well as remote ‘cloud’ storage is of a great benefit. It encourages file storage to be codified in a distribute-able manner so that scaling later on is easier.

This is a work in progress, but the local implementation is intended to be both a reference implementation as well as useful testing or even production backend for storage. Other backends potentially will have less comprehensive metadata support for individual files, but these ‘limits’ will be included as optional warnings or exceptions once we have a handle on what they are.

Please comment or give feedback on this library. Also, we would welcome any patches for other backend support to the library!

http://bitbucket.org/okfn/ofs