Advanced Use of the Contribution Engine

The Internet Archive Contribution Engine supports some advanced functionality to enable high volume contributors to more easily (and less interactively) upload and import content into the Archive. The advanced functionality consists of several parts:

1. Uploading files into a directory for an item via FTP using your username and password
2. Each item to be imported must have an XML file describing the item and an XML file describing the item files
3. Calling a URL telling the contribution engine that you are done uploading a specific directory. Results are returned in XML for easy parsing.

In the following sections, the XML formats will be discussed and examples will be given.

Preparing the item metadata

First of all, every item in the Archive must have a unique identifier. This identifier is used to locate the item on our servers and provide the content to the users. When you upload a directory, the name of the directory is the unique identifier used. Identifiers can only contain letters, numbers, underscores, dashes, and dots (nothing else!)

For the sake of clarity, we will use an example of a movie called "My Home Movie". Assume the movie is one file named MyHomeMovie.mpeg, is encoded in the MPEG2 format, has a running time of 2:30, and was directed by "Joe Producer". The unique identifier is MyHomeMovie.

In order to avoid manual intervention with the website, you will need to specify all of the above information using two XML files. The files must be named IDENTIFIER_meta.xml and IDENTIFIER_files.xml where IDENTIFIER is the unique identifier (in our example these files would be called MyHomeMovie_files.xml and MyHomeMovies_meta.xml).

The item metadata file (MyHomeMovies_meta.xml) would look like this:

<metadata>
  <collection>opensource_movies</collection>
  <mediatype>movies</mediatype>
  <title>My Home Movie</title>
  <runtime>2:30</runtime>
  <director>Joe Producer</director>
</metadata>    

All items must have the collection, mediatype, and title field supplied. You can include whatever other fields you like. Some fields will be displayed on the website. For movies these include: date, producer, production_company, director, contact, sponsor, description, runtime, color, sound, shotlist, segments, credits, and country. For audio these include: creator, description, taper, source, runtime, date, and notes. Any other fields will be kept in the XML file and will, in the future, be shown on the website. For a good example check this metadata file.

You can also select a Creative Commons license for your item by adding a field called licenseurl to your metadata file. Choose the URL from the list below, based on the type of license you want:

So, for example, if you wanted the Public Domain license, you would put this line in your metadata file: <licenseurl>http://creativecommons.org/licenses/publicdomain/<licenseurl>

Preparing the file metadata

Additionally, you need to describe the files you are uploading so the system knows what types of conversions to do, and so that this information can be shown to users on the website. This file for our example would look like (MyHomeMovie_files.xml):

<files>
  <file name="MyHomeMovie.mpeg" source="original">
    <runtime>2:30</runtime>
    <format>MPEG2</format>
  </file>
</files>    

The format name is very important to choose accurately. Upper/lowercase is significant. By picking formats from this list, you will be assured to get the most possible "derivatives" created from the item's files. This list is dynamically generated so check back later for newly acceptable formats. The list of acceptable format names is:

If you are uploading multiple versions of the same content, the lower quality copies should have source="derivative" instead of source="original" and should also have an <original> tag(s) indicating what file(s) it came from. For a good example check this metadata file.

It is highly recommended that contribute as much metadata as possible about the item and its files.

Uploading the files

One you have created the metadata files, use your FTP client to upload the directory containing the item files and the metadata files. In our example the directory uploaded would look like this:

MyHomeMovie/
MyHomeMovie/MyHomeMovie_files.xml
MyHomeMovie/MyHomeMovie_meta.xml
MyHomeMovie/MyHomeMovie.mpeg

Telling the contribution engine to process the upload

Once you have uploaded the directory the contribution engine needs to be informed to process your contribution. Doing so is simple. Assume our example film was uploaded by user@users.com. A HTTP GET should be issued to:

http://www.archive.org/services/contrib-submit.php?user_email=user@user.com&server=movies-uploads.archive.org&dir=MyHomeMovie

Clearly you would adjust the parameters relevant to your contribution. Calling this URL will return an XML response. If the contribution was successful, it will look like this:

<result type="success">
  <message>Item successfully added</message>
</result>
<url>http://www.archive.org/movies/MyHomeMovie</url>    

Otherwise, if there was a failure, it would look like this:

<result type="error" code="meta_error">
  <message>Your metadata file is missing the collection field</message>
</result>    

If all is successful, the item will be added to the Archive and the URL tag in the returned XML will indicate the URL you can use to get to the details page for the item. Otherwise, the message will indicate what you need to fix.