“It remains to be seen what agent or agents will end up providing the best service for individual researchers, departments or universities to display the bibliographic data they generate, and how best such data will be aggregated for search and discovery. But don’t wait for the big publishers and information brokers to monopolize this function.”

Finally, Jim provides some useful tips on how to “push the publishers and subscription services to support open access to your research work and open bibliographic data”. Go here to read the whole post.

Glottolog/Langdoc is built upon a collation of 20 source bibliographies covering the whole world, from Alaska to Australia. The original bibliographies were parsed and enriched with machine learning techniques. This allows to formulate queries such as

• 
  “Give me all works about Zulu”
• “Give me all works by Dempwolff”

and combinations thereof such as

• “Give me all grammars of Semitic languages of Eurasia written before 1900”

Furthermore, an areally and genalogically balanced sample can be drawn.

All references have their own URIs. All resources are available as xhtml and rdf, and can be downloaded as bib, html, txt, or via Zotero. Dumps of references are available as a very large *bib and as a dump in rdf+xml.
Glottolog/Langdoc content is made available under CC-BY-NC. Intercultural issues upstream unfortunately prevent us from releasing the content under a more permissive license.

Glottolog/Langdoc uses DCMI, BIBO, FRBR, and ISBD ontologies to provide an interoperable resource. Glottolog is part of the Linguistic Linked Open Data cloud. We are working on a SPARQL endpoint, which will probably be made available in June this year.

We are happy to contribute our bibliographical resource to the Linked Data cloud and would welcome feedback under glottolog@eva.mpg.de.

Channels: Meeting was held via Skype and Etherpad

Participants

• Adrian Pohl
• Sebastian Nordhoff
• Jim Pitman

Agenda

Action points from the last meeting

• Adrian has posted to the KIM-DINI-LLD list regarding the national bibliography and a concersion to BibJSON. There are no answers yet.
• Until now Mark & Adrian haven’t looked at the data themselves and whether/how it can be converted to BibJSON.

ACTION: Adrian will personally ask people from the German National Library at BibCamp.

Langdoc dataset

Sebastian gave us some information about the bibliographic dataset for the world’s lesser known languages recently published at http://glottolog.livingsources.org/. (See also the thread on the mailing list.)

Data

• The dataset contains 180k references with 120K records tagged by language.
• Sebastian sees a possibility (and need?) for crowdsourcing to add missing data elements for 40K records.
• The data can be downloaded under http://glottolog.livingsources.org/meta/downloads.
• In addition to bibliographic data the website contains a “comprehensive catalogue of the world’s languages, language families and dialects (langoids)” which can be searched and browsed starting here.
• Example bibliographic record.
• Existing problems with the data include
  • several ad hoc BibText properties
  • 73 different data fields
  • Sometimes semantics of a property aren’t known, e.g. document_type={B}
  • These problems are due to heterogeneous data sources for this dataset
  • RDF for references still in need of improvement

Code

• There’s machine-learning code for automatically recognizing the language of a work involved
• written in Python
• Isn’t open yet
• The code is created by Harald Hammarström.
• NLTK, Natural Language Toolkit: http://www.nltk.org/

licensing issues

• How many parts of the data set make problems? – “Africa part”, “Australia part”
• What share of the data is problematic? – Approx. one quarter.

What’s Sebastian up to with the data set?

• SN is happy to provide data and help other people start with it.
• SN can probably convert existing data in BibJSON
• SN will host a SPARQL endpoint
• SN will not host a bibserver
• SN will not develop bibserver

ACTION: Sebastian will provide a post for openbiblio.net when the data set is officially released (sometimes in March).

Provenance

A short discussion about provenance – in regard to BibJSON – arose.

• Adrian pointed to the DCMI provenance group.
• There also exists a Provenance Working Group within the W3C.
• I would make sense to adopt some proposals for provenance from these groups.

ACTION: Ask members of these groups to provide a short post about it on openbiblio.net.

Resource and tasks of openbiblio activities at OKF

• Jim had some questions about governance and allocation of OKF resources to the working group’s activities
  • How to organize WGOBD to engage contributors to creation/maintenance of various listings relevant to OBD?
  • How to keep the enquiries to potential open data providers going?
  • …

ACTION: Write down core resources and tasks of the openbiblio group. (Adrian)

What is especially of interest regarding open bibliographic data is that the new Open Data Strategy for Europe includes some proposals by the EU commission for updating the Directive on the re-use of public sector information and expanding it to memory institutions. The press release says the commission proposes “massively expanding the reach of the Directive to include libraries, museums and archives for the first time; the existing 2003 rules will apply to data from such institutions.”

More from the press release:

“Brussels, 12 December 2011 – The Commission has launched an Open Data Strategy for Europe, which is expected to deliver a €40 billion boost to the EU’s economy each year. Europe’s public administrations are sitting on a goldmine of unrealised economic potential: the large volumes of information collected by numerous public authorities and services. Member States such as the United Kingdom and France are already demonstrating this value. The strategy to lift performance EU-wide is three-fold: firstly the Commission will lead by example, opening its vaults of information to the public for free through a new data portal. Secondly, a level playing field for open data across the EU will be established. Finally, these new measures are backed by the €100 million which will be granted in 2011-2013 to fund research into improved data-handling technologies.

These actions position the EU as the global leader in the re-use of public sector information. They will boost the thriving industry that turns raw data into the material that hundreds of millions of ICT users depend on, for example smart phone apps, such as maps, real-time traffic and weather information, price comparison tools and more. Other leading beneficiaries will include journalists and academics.

Commission Vice President Neelie Kroes said: “We are sending a strong signal to administrations today. Your data is worth more if you give it away. So start releasing it now: use this framework to join the other smart leaders who are already gaining from embracing open data. Taxpayers have already paid for this information, the least we can do is give it back to those who want to use it in new ways that help people and create jobs and growth.” See Mrs Kroes video quote here.

The Commission proposes to update the 2003 Directive on the re-use of public sector information by:

• Making it a general rule that all documents made accessible by public sector bodies can be re-used for any purpose, commercial or non-commercial, unless protected by third party copyright;
• Establishing the principle that public bodies should not be allowed to charge more than costs triggered by the individual request for data (marginal costs); in practice this means most data will be offered for free or virtually for free, unless duly justified.
• Making it compulsory to provide data in commonly-used, machine-readable formats, to ensure data can be effectively re-used.
• Introducing regulatory oversight to enforce these principles;
• Massively expanding the reach of the Directive to include libraries, museums and archives for the first time; the existing 2003 rules will apply to data from such institutions.

In addition, the Commission will make its own data public through a new “data portal”, for which the Commission has already agreed the contract. This portal is currently in ‘beta version’ (development and testing phase) with an expected launch in spring 2012. In time this will serve as a single-access point for re-usable data from all EU institutions, bodies and agencies and national authorities.“

This provides a simple search interface for openbiblio, using a network-addressable Apache Solr instance to provide FTS over the content.

The indexer currently relies on the Entry Model (in /model/entry.py) to provide an acceptable dictionary of terms to be fed to a solr instance.

Configuration:

In the paster main .ini, you need to set the param ‘solr.server’ to point to the solr instance. For example, ‘http://localhost:8983/solr’ or ‘http://solr.okfn.org/solr/bibliographica.org’. If the instance requires authentication, set the ‘solr.http_user’ and ‘solr.http_pass’ parameters too. (Solr is often put behind a password-protected proxy, due to its lack of native authentication for updating the index.)

Basic usage:

The search controller: solr_search.py (linked in config/routing.py to /search)

Provides HTML and JSON responses (content-negotiation or .html/.json as desired) and interprets a limited but easily expandable subset of Solr params (see ALLOWED_TERMS in the /controller/solr_search.py.)

JSON response is the raw solr response as this is quite usable in javascript.

HTML response is styled in the same manner as the previous (xapian-based?) search service, with the key template function formatting each row in templates/paginated_common.html – genshi function “solr_search_row”. Unless specified, the search controller will get all the fields it can for the search terms, meaning that the list of resuts in c.solr.results contain dicts with much more information than is currently exposed. The potentially available fields are as follows:

    "uri"          # URI for the item - eg http://bibligraphica.org/entry/BB1000
    "title"        # Title of the item
    "type"         # URI type(s) of the item (eg http://.... bibo#Document)
    "description"
    "issued"       # Corresponds to the date issued, if given.
    "extent"
    "language"     # ISO formatted, 3 lettered - eg 'eng'
    "hasEditionStatement"

    "replaces"        # Free-text entry for the work that this item supercedes
    "isReplacedBy"    # Vice-versa above

    "contributor"           # Author, collaborator, co-author, etc
                            # Formatted as "John Smith b1920 "
                            # Use lib/helpers.py:extracturi method to add formatting.
                            # Give it a list of these sorts of strings, and it will return
                            # a list of tuples back, in the form ("John Smith b1920", "http...")
                            # or ("John Smith", "") if no -enclosed URI is found.
    "contributor_filtered"  # URIs removed
    "contributor_uris"      # Just the entity URIs alone

    "editor"                # editor and publisher are formatted as contributor
    "publisher"
    "publisher_uris"        # list of publisher entity URIs

    "placeofpublication"    # Place of publication - as defined in ISBD. Possible and likely to
                            # have multiple locations here

    "keyword"               # Keyword (eg not ascribed to a taxonomy)
    "ddc"                   # Dewey number (formatted as contributor, if accompanied by a URI scheme)
    "ddc_inscheme"          # Just the dewey scheme URIs
    "lcsh"                  # eg "Music "
    "lcsh_inscheme"         # lcsh URIs

    "subjects"              # Catch-all,with all the above subjects queriable in one field.

    "bnb_id"                # Identifiers, if found in the item
    "bl_id"
    "isbn"
    "issn"
    "eissn"
    "nlmid"                 # NLM-specific id, used in PubMed
    "seeAlso"               # URIs pertinent to this item

    "series_title"          # If part of a series: (again, formatted like contributor)
    "series_uris"

    "container_title"       # If it has some other container, like a Journal, or similar
    "container_type"

    "text"                  # Catch-all and default search field.
                            # Covers: title, contributor, description, publisher, and subjects

    "f_title"               # Fields indexed to be suitable for facetting
    "f_contributor"         # Contents as above
    "f_subjects
    "f_publisher"
    "f_placeofpublication"  # See http://wiki.apache.org/solr/SimpleFacetParameters for info

The query text is passed to the solr instance verbatim, so it is possible to do complex queries within the textbox, according to normal solr/lucene syntax. See http://wiki.apache.org/solr/SolrQuerySyntax for some generic documentation. The basics of the more advanced search are as follows however:

• field:query — search only within a given field,

eg ‘contributor:”Dickens, Charles”‘

Note that query text within quotes is searched for as declared. The above search will

not hit an author value of “Charles Dickens” for example (and why the above is not a good

way to search generically.)

• Booleans, AND and OR — if left out, multiple queries will be OR’d

eg ‘contributor:Dickens contributor:Charles’ == ‘contributor:Dickens OR contributor:Charles’

The above will match contributors who are called ‘Charles’ OR ‘Dickens’ (non-exclusively), which is unlikely to be what is desired. ‘Charles Smith’ and ‘Eliza Dickens’ would be valid hits in this search.

‘contributor:Dickens AND contributor:Charles’ would be closer to what is intended.

• URI matching — many fields include the URI and these can be used to be specific about the match

eg ‘contributor:”http://bibliographica.org/entity/E200000″‘

Given an entity URI therefore, you can see which items are published/contributed/etc just by performing a search for the URI in that field.

Basic Solr Updating:

The ‘solrpy’ library is used to talk to a Solr instance and so seek that project out for library-specific documentation. (>=0.9.4 as this includes basic auth)

Fundamentally, to update the index, you need an Entry (model/entry.py) instance mapped to the item you wish to (re)index and a valid SolrConnection instance.

from solr import SolrConnection, SolrException
s = SolrConnection("http://host/solr", http_user="usernamehere", http_pass="passwordhere")
e = Entry.get_by_uri("Entry Graph URI")

Then, it’s straightforward: (catching two typical errors that might be thrown due to a bad or incorrectly configured Solr connection.)

from socket import error as SocketError
try:
    s.add(e.to_solr_dict())
    # Uncomment the next line to commit updates (inadvisable to do after every small change of a bulk update):
    # s.commit()
except SocketError:
    print "Solr isn't responding or isn't there"
    # Do something here about it
except SolrException:
    print "Something wrong with the update that was sent. Make sure the solr instance has the correct schema in place and is working and that the Entry has something in it."
    # Do something here, like log the error, etc

Bulk Solr updating from nquads:

There is a paster command for taking the nquads Bibliographica.org dataset, parsing this into mapped Entry’s and then performing the above.

    Usage: paster indexnquads [options] config.ini NQuadFile
Create Solr index from an NQuad input

Options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config=CONFIG_FILE
                        Configuration File
  -b BATCHSIZE, --batchsize=BATCHSIZE
                        Number of solr 'docs' to combine into a single update
                        request document
  -j TOJSON, --json=TOJSON
                        Do not update solr - entry's solr dicts will be
                        json.dumped to files for later solr updating

The –json option is particularly useful for production systems, as the time consuming part of this is the parsing and mapping to Entry’s and you can offload that drain to any computer and upload the solrupdate*.json files it creates directly to the production system for rapid indexing.

NOTE! This will start with solrupdate0.json and iterate up. IT WONT CHECK for existence of previous solr updates and they will be overwritten!

[I used a batchsize of 10000 when using the json export method]

Bulk Solr updating from aforementioned solrupdate*.json:

    paster indexjson [options] config.ini solrupdate*
    Create Solr index from a JSON serialised list of dicts

Options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config=CONFIG_FILE
                        Configuration File
  -C COMMIT, --commit=COMMIT
                        COMMIT the solr index after sending all the updates
  -o OPTIMISE, --optimise=OPTIMISE
                        Optimise the solr index after sending all the updates
                        and committing (forces a commit)

eg

“paster indexjson development.ini –commit=True /path/to/solrupdate*”

Channel: Meeting was held via Skype and Etherpad

Participants

• Adrian Pohl
• Jim Pitman
• Primavera De Filippi
• Karen Coyle
• James Robert Griffin III
• Thomas Krichel
• William Waites
• Lucy Chambers

publicdomainworks.net & bibliographica.org

Primavera from publicdomainworks.net attended the meeting. As publicdomainworks.net in the future wants to use bibliographica.net as data store she wanted to know how to query bibliographica.org to get JSON output. It was decided to move this particular discussion to the openbiblio-dev mailing list which has already happened.

The further discussion was about how to best model bibliographic data: First, which serialization to use and second which metadata elements. William said, regarding serializations the current lineup of JSON serialisations was being considered at bibliographica.org.

Furthermore, BibJSON was mentioned as a loose standard for describing bibliographic resources in JSON. FOAF/BIBO/DC are incorporated in BibJSON which also allows other name spaces.

Openbiblio Principles

• Since the last meeting mails promoting the openbiblio principles have been sent to many library-related mailing lists:
  • Karen: lita-l, code4lib, ngc4lib, ol-discuss, public-lld
  • Antoine: europeana, public-lod, public-esw
  • Adrian: Inetbib
  • Jim and Thomas haven’t sent out mails. ACTION: They will catch up on this.
• The number of signatories grew to 83.
• A Korean version of the Principles on Open Bibliographic Data was sent to Mark McGillivray. These need to be verified before publishing them on openbiblio.net.
• ACTION: Primavera will ask a Korean friend to take a look at it.

Open Data enquiries

• Adrian had a mail exchange with E-LIS.
• They confirmed that E-LIS metadata is open and are thinking about explicitely using an open license and signing the openbiblio principles. The enquiry is now resolved, see here.
• Nothing happened regarding the other running enquiries.

ORCID

• ORCID ist a non-profit organization “dedicated to solving the name ambiguity problem in scholarly research” by establishing “a registry that is adopted and embraced as the de facto standard by the whole of the community”. (See http://www.orcid.org/aboutus.)
• Momentarily the ORCID principles state that individual person information will be licensed CC0 but there is no such statement regarding bulk data.
• James Griffin and Thomas will give a talk at the next ORCID participant meetings, 18th May where they will promote open bibliography.

BibServer/Openbiblio/Bibliographica

• Jim announced that he agreed with Rufus and Mark about BibServer integration with Openbiblio/Bibliographica.

Microsoft Academic Search (MAS) API

• Microsoft Academic Search
• MAS has got clean well-structured data about academic articles.
• Thomas Krichel, Peter Murray-Rust and Jim Pitman are talking with MAS about making the service more open.

Upcoming events

Open Knowledge Conference: On 30th June & 1st July, 2011 OKCon 2011 will take place in Berlin. It would be great if the openbiblio working group would be represented there with a talk. Adrian said he could do this but he’d like present together with someone from the academic paper group (like PMR, Jim or Thomas) to paint a better picture of the group.

Action Collection

• Thomas will post to engineering librarian lists and lists Karen hasn’t covered.
• Jim will cover math/stat related lists and publications
• Primavera will ask a Korean friend to take a look at the openbiblio principles translation.

Unclear licensing status

As 3lib – like other services – is without question an interesting project in the context of bibliographic data, unfortunately the data it provides nonetheless isn’t clearly open in light of the Principles on Open Bibliographic Data. For most of the data aggregated within 3lib the licensing status is unclear as no explicit terms of use can be found. As we want to build an environment of shared open bibliographic data, the question is: How to best reach clarity about the licensing status of this data?

Reaching clarity

Here comes IsItOpenData into play, a service established by the OKFN Working Group on Open Data in Science. IsItOpenData provides a platform for enquiring data providers about their data’s licensing status. It seems to really fit well for our purposes and so we decided to use it. First, we made a public list naming all 3lib data sources and indicating their licensing status. Then we started with sending enquiries (1, 2, 3) to data providers who don’t make explicit a licensing policy on their web pages. Until now, there are no repsonses but we’ll follow up on this post when first answers come in.

As you can see, there are still some enquiries to be made. If you are familiar with some of the data providers, why don’t you make an enquiry?

How to make an enquiry?

Most helpful for making such an enquiry is Heather Piwowar’s guide IsItOpenData? tips. I’ll summarize the most important points adding information with respect to new features.

The basics

1. Register for a new account at the IsItOpenData? site
2. Make an Enquiry
3. Wait for a response
4. Follow up with a thank you!
5. Resolve the enquiry as soon as you know whether the data is closed or open.

More detailed tips

• Start with a well-considered email based on the template IsItOpenData will serve you.
• Try to compose your email such that it isn’t mistaken for spam. This probably means limiting links.
• Recognize that your email may be identified as spam anyway. Follow up with a short email from your personal email account, alerting the recipient that they have been sent an IsItOpenData email and it may be in their spam filter.
• In the main email AND the personal email, highlight (in a central place in the main body of the email) that responses will be made public on the IsItOpenData site. Emphasize this very clearly. It is important, easily missed, and potentially very embarrassing if not clear.
• Put the organization name in email subject. This will make your request easy to browse in the enquiry list.
• The “IsItOpenData” footer will automatically be appended to the bottom of your email.
• Send the orignal email through the IsItOpenData site using the Make Enquiry link. This email will be sent with an IsItOpenData reply address. You will receive a copy of this email as a bcc: recipient.
• If people reply to the original email, replies will be automatically posted onto the website. IsItOpenData will email you an alert that you received a response. Note that these alerts may be considered spam by your email program.
• If data providers write back to your personal email address, send them an email thanking them and confirming that you can post their email to the website. If yes, log back into the original query on IsItOpenData, and “FollowUp” with another post to them, thanking them, with their response appended to the bottom. This will archive the response at IsItOpenData.
• Be sure to sincerely thank the respondents. Articulating these policies is not easy.
• Change the enquiry status as soon as you have an explicit and definite statement about the openness of the respective data.
• Keep the tone respectful, since the goal of IsItOpenData is to understand current policy. Lobbying for more open policies is a different task.
• If you make an enquiry for a 3lib data set make a note on the etherpad.

Ask other data providers

Interested in using other bibliographic data sets but you couldn’t find any information about their licensing status? Just start another enquiry project with isitopendata.org, talk about it on the openbiblio group’s mailing list or on this blog. The Working Group on Open Bibliographic Data is happy to support your openbiblio project.

Anybody working on open bibliographic data projects is invited to publish on this blog, sharing his/her knowledge and experience or to start a discussion. Just approach someone from the group or write an email to openbiblio[at]okfn[dot]org.

The open library data movement in Germany has achieved a major success. In December, the German National Library of Medicine (ZB MED) announced the release of more than one Million bibliographic records into the public domain under a CC0 licence. Descriptions of the data as well as download links are available here. The data was converted to RDF using the Bibliographic Ontology (Bibo) and is now part of the Linked Open Bibliographic Data service lobid.org.

Flyer on open library data

The hbz has also published together with the Open Knowledge Foundation a German flyer on open library data which informs about the advantages of open data for libraries. The flyer was created based on feedback and discussions within the OKFN Working Group on Open Bibliographic Data. A pdf version of the flyer can be downloaded here. If interested in printed exemplars of the flyer just print your own, create a new one (it’s openly licensed) or write an email to the hbz.
A – slightly different – English version of the flyer can be viewed here.

Adrian Pohl is coordinator of the OKFN Working Group on Open Bibliographic Data and works at the North Rhine-Westphalian Library Service Center (hbz).

• Adrian Pohl
• John Mark Ockerbloom
• Rufus Pollock
• Peter Murray-Rust

Apologies

• Micah Altman

Minutes of former meetings

For the minutes of former meetings see

• 1st meeting: http://okfnpad.org/Qv1jMMQL1h

• 2nd meeting: http://okfnpad.org/wTnZOOxiRz

Agenda

Working group blog/website

We decided to use http://openbiblio.net/ as the Working Group’s blog. In the future we’ll publish minutes and informations about ongoing activities there.

We also reminded ourselves to use the group’s hashtag (#openbiblio) more frequently on twitter etc.

Flyer

We did some more work on the Open Bibliographic Data flyer. It’s now posted on the blog and we solicit wider comments.

Principles for Open Bibliographic Data

see http://okfnpad.org/openbibliography-principles

We did some more work on this text and decided to not only adress libraries but also publishers and other institutions producing bibliographic data. More discussion is encouraged but please move it into the discussion part at the bottom of the document.

Debrief from #jiscopenbib

We had no time left for this. So it will happen on list.

Infos from the lld-xg?

Nobody from the lld-xg attended the meeting.

Upcoming: Events and projects

John Mark Ockerbloom hopes to promote open data at BooksOnline10 in Toronto (Oct 26)
http://research.microsoft.com/en-us/events/booksonline10/

• http://packages.python.org/ordf/odm.html