When we launched the project, we offered to give NARA copies of the scans completely free, as well as free access from within the Archives facilities. We were told by Archives administration that they were not interested and they would not provide access to the documents we needed because we were charging a subscription. (We charge $15 per month for full access)

Talking about these contracts that NARA has partnered with, Footnote, Family Search, etc. the finished product is “poor” barely useable now, let alone in five years time. No one wants to look at microfilm on a viewer let alone on a computer screen. Scanning microfilm at 300 dpi, and then putting it online does absolutely nothing to preserve the document in its original format. And yes, projects like this do take money. Equipment, labor, etc. Since NARA is not willing to endeavor to digitize the material, then it is up to private industry to do the work and provide the access. And you are right in the fact that NARA has absolutely no desire to get servers that will hold the vast digital images necessary for public access. Those cost money as well. So until the Federal government contracts with private industry to digitize the material, and pay them for their work, to make the material available online free, subscriptions are the only way to recoup costs.

The current partnership with Family Search to digitize the Civil War Widow pension records is another example of poor standards. History happened in color and the color in documents is there for a reason. Family Search is digitizing the Civil War pension records with B & W digital cameras, doing 300 dpi, which is barely acceptable for OCR today. Instead of requiring that the material be digitized right the first time, with current standards, that of 600 dpi, full color, the Archives is once again barely meeting the criteria, which will make the project a waste of time. And once the project is done, the original documents will not be accessible to anyone, for the Archives will claim that they are accessible online.

These contracts are not non exclusionary. They are removing the original documents from scholars and researchers who will be left with bad microfilmed images. Oh, and NARA will offer to provide a digital scan of the original for $35 and higher. So what is the difference between a private sector company making the money, or a federal agency milking the American public who owns the documents from paying double jeopardy to gain access to what they should already have.

The Lincolnarchives project has received no federal funding nor grants. We currently have over 5000 documents online, with 100 per week being added, and 3000 per month being scanned. The cost for hosting is currently $135 per month. We have over 20 years experience with technology and work with the records at the National Archives and Library of Congress. But we are a small company and NARA seems to be only interested in forming digital partnerships with big names who are putting out poor quality products. Quick and dirty is not the answer to preserving these documents, but NARA just doesn’t seem to be interested.

http://freegovinfo.info/node/1799

I share Peter’s wish that NARA share protocols for reviewing for privacy. This would be useful information for those institutions interested in either doing scanning on their own or with partners.

I am less concerned about paper-based works not being digitized according to standards that were developed (digitally speaking!) some time ago. I don’t think any of us imagined (or did the math) on what it would cost to preserve these access copies in their full blown TIFF glory. See “Preservation in the Era of Large Scale Digitization” for a thoughtful discussion (http://www.clir.org/pubs/reports/pub141/pub141.pdf).

I think we have a lot more to worry about with born-digital materials, many of which will ironically be of lesser quality than the access scans we are making of paper-based materials. (Think of the “originals” from many digital camera, and hold those images up against the standards for scanning and you’ll see what I am talking about.)

TGN can, however, limit via contract what is done with the images that they reproduce. That means that they could sue Carl Malamud if he should download all of the images in violation of the license terms and put them on his web site. They could not, however, sue anyone who might take the images from Malamud’s web site and use them some place else (since there is no contract with that 3rd party).

The big plus of the TGN contract, and why it should serve as a model for all public/private digitization partnerships, is that the exclusive rights granted to TGN are limited in time, at which point NARA can make the material freely available on its web site or give it to the Internet Archive. And this, BTW, is no different than the agreement that NARA used to sign with commercial microfilmers (except that my recollection is that the period of exclusivity was longer).

If NARA had to give TGN perpetual exclusive rights, or if it had to prevent anyone else from digitizing the contents, then there might be a problem. But as far as the contract terms are concerned, it is a win/win for everyone – so long as NARA acts responsibly at the end of the 5 year term. The issue is not with the contract – but with whether we trust NARA to act in the public’s behalf.

FYI, I had other issues with the NARA contract that I submitted. First, I was disappointed to see that TGN was not required to follow the standards for digitization that NARA has established. If NARA is going to commit resources to make this happen, they should make sure that they get a quality product that they want to use and preserve at the end. Second, proposed agreement states that “NARA represents and warrants that all selections of the Archival Materials will be reviewed for privacy before they are made available to TGN for digitization.” This can be very difficult and expensive to do. It would be very helpful to the archival community, therefore, if NARA would release publicly its protocols for doing this so that others may learn from NARA’s experience. Third, while materials created by the Federal Government are in the public domain, materials received by the government are still copyrighted (for example, a letter sent by an individual to an agency). There are exemptions to copyright that allow NARA to copy and distribute these copyrighted works, but it is unclear to me if they would extend to TGN (or any other contractor).

Overall, however, this is better than any of the Google contracts that have been made publicly available. It is not as good as OCA’s contract with the Boston Public Library – but the level of investment that TGN is making is orders of magnitude greater than what OCA is doing. It seems like a fair trade.

If NARA infrastructure is the issue with future access, there’s a simple solution: contract hosting with the Internet Archive. The IA has worked with several governments on crawls of their domains (UK Archives comes to mind). They’ve collaborated with the Smithsonian and the Library of Congress. In other words, they have the interest and the expertise to deal with free access to large amounts of content, long-term preservation of that content, and they’re a *library*! So have TGN digitize the content and then host it at the IA. Problem solved

I’m still mulling about Konrad’s copyright comment. I think in the print world, publishers had a leg to stand on because their microfilm/bound volumes were seen as adding value to the content. There was nothing stopping publishers from putting out their products in the public domain, it was just legal and economic protection of their investment. For more on copyright in the digital age, readers should go visit Question Copyright (disclosure: I’m on the board of Question Copyright).

I don’t think that Konrad would have any problem legally if he wanted to put up a bunch of govt documents on the Web. There is in fact precedent for doing just that. Carl Malamud has been doing it for years, as have projects like Outraged Moderates as well as one-off “ransoms” of USGS map data by Jared Benedict. The 1000 flowers blooming is wonderful IMHO, but I hope that some non-profit organizations (like libraries!) would take it upon themselves to create systems for the long-term access and preservation of digital government information. Yes it’ll be expensive and ongoing, but wouldn’t it be better all around if libraries and other cultural institutions undertook this in a collaborative manner rather than leaving it to private companies to do it?

I’m glad we’re having this discussion, and believe that we all agree that access to and preservation of digital govt information is of the utmost priority. Let’s hope NARA is reading this and it’ll have a positive affect on future digitization projects.

I did not mean to imply that TGN would make money *solely* on content, but that, unlike LN, it would be (evidently) the only network source for digitized copies. It is that effective monopoly on the content that I object to.

I could be wrong, of course. If TGN is willing to post all the digitized content on an FTP site for downloading without licensing or DRM restrictions so that anyone can use or reuse the content, then I have misinterpreted their intentions. If that is the case, then they are not making money on the content, but only on the value they have added to the content.

Another possible solution to this problem is if NARA is willing (and contractually able) to give (or even sell at marginal cost) the complete content without restrictions for others to use, reuse, and redistribute. Then other libraries and archives could make the content available without fees, historians could examine whole bodies of content and republish without restrictions.

A good parallel example is court cases that Carl Malamud has made available. He had to pay for a copy, but after doing so was able to make them freely available. (See http://itc.conversationsnetwork.org/shows/detail3590.html) In other words, Carl’s model was to pay for digitization to free the content as opposed to the NARA/TGN plan that “pays” for the digitization at a cost of locking the content up.

Also, I did not mean to imply that public information would be available ONLY for a fee. If I did, I was imprecise. In fact it is precisely the inequity and inequality of access, which you describe and which the TGN deal creates, that is one of the problems I have with deals like this. Users who are willing to accept whatever licensing and DRM restrictions TGN imposes (to protect its investment and income) will have “easy” access, but users who want free, unrestricted access will be have to have the time and money to travel to a NARA repository. And even then, it is not clear to me from reading the contract if users will have unlimited unrestricted access to the digitized versions.

As you note, most government information is created electronically today, but I would ask you to consider if even this “born digital” information is equitably, freely available today (too much is locked into government systems that do not allow unrestricted access; see my other post http://freegovinfo.info/node/1783) and if it will always be available for historians? I think that we, as a society, face the same long-term problem for born digital and newly-digitized information.

For each digitization project, there will be costs. My argument is that we can pay those costs up front and make the information freely available, or we can construct deals in which we pay the cost of digitization by restricting access and control in the long run. I prefer the first.

in response to Konrad

I agree that there is a big problem of losing public domain information, particularly government information. I worry most these days about licensing restrictions and DRM locks that prevent use and reuse but avoid the copyright issue. I worry that they will, essentially, side step copyright law — including fair use. As noted in my comments here, that is one of the problems I find with the NARA/TGN deal

But the other problem I see with your argument is this idea that information that was publicly available is suddenly ONLY avaialble for a fee. That simply is not true. As before the digitization, anyone who can physically travel to any of the many government document repositories that collect that particular government information can have free access to it. Nothing has changed, except that with the digitization people have a choice on how to access that information. They can spend the money to drive and park (or fly, or take a bus, or train) to the nearest repository that has the government information they seek, or they can spend the money for the convenience of accessing that information electronically.

Today, of course, most government information is created electronically and available electronically. (So the main problem there is whether or not ERA will ever work – no matter how much money is handed to Lockheed/Martin – and whether NARA will be able to continue to preserve and make that electronic information available through time. Of that I am skeptical.) But we’re talking about all of the information that was previously created in a print-bound world. If left to NARA, this past information will likely never be digitized for the reasons already mentioned. And I think it is unreasonable to expect any institution to retrospectively digitize all of the information it created or maintains. So what is to be done? I’m sure I don’t know, but this imperfect solution is one I can accept as “better than nothing.”