Organizing Research: brief subscriptions or research trips

Question

Several of the big box providers like Ancestry, Find My Past, and British Origins offer limited-time subscriptions or limited free trials. I just took advantage of a month's cheap subscription at FindMyPast, and now I have a mountain of downloaded census and register images to sort out.

Usually I create a folder for these images that is named after the vendor, so I will know which online repository held those images. But if I'm not careful, the images can be saved to the wrong folder, or other things that I download in the same session can get saved to that folder by mistake.

What I would like to have is some kind of record of my download history. The ideal list would show the information that is in my Firefox download history -- the filename, the site it was downloaded from, and the date it was downloaded.

Searching Mozilla.org yields a lot of history cleaners, but I don't see anything that would allow me to export the download history to a file. Apparently the newer versions of Firefox only keep the history for a limited amount of time, so the window of opportunity to capture the data from there is small.

In the days of DOS, I knew how to pipe the contents of a directory to a file. If I had a list of the filenames, I could put them into a spreadsheet and then populate it with the information about where the files had come from, and eventually, with surname tags or other notes as a quick reminder of why I downloaded the file.

The reason I ask is that both Find My Past and The British Newspaper Archive keep logs of your records, but for the PPV records especially, the amount of time you can view them is limited. So I have a list of newspaper articles that I accessed last year, but since I didn't think to make notes on the BNA site, it isn't always evident what search terms made them appear, or why I collected them. So I spent a lot of time re-accessing some of the newspaper articles on Find My Past and downloading them again, trying to make notes as I went along.

My plan is to catalog and organize all the images and analyze them properly, but how do I preserve the ephemeral data about the repository in the meantime?

Answer 1

Another way of doing this is to use OneNote or Evernote and grab the image of the page with the appropriate clipping tool and that stores the url that the image came from. These notes can be orgainsed into notebooks by site the information came from, surname or whatever.

I am not an expert user of OneNote only recently having started using it but it is a great way organise scraps of data that you gather for your study.

Both OneNote and Evernote are free applications.

Answer 2

According to the mozillaZine article downloads.rdf

In Firefox (version 2.x and below) and Mozilla Suite/SeaMonkey, downloads.rdf is the (only) place that stores information (such as file name, URL, and date of download) about files downloaded. Its contents can be viewed and managed through the Download Manager and its associated settings. In Thunderbird, it is the (only) place that maintains a history of attachments opened or saved by double-clicking on files listed in the Attachments pane or by using the menu sequence "File -> Attachments -> [attachment name] -> Open". Thunderbird has no user interface for viewing or managing the contents.

Note: Starting in Firefox 3.0, download history is stored in downloads.sqlite instead of downloads.rdf.

The file downloads.rdf is readable with Firefox itself, and could be scraped to put the information into a spreadsheet or checklist.

The mozillaZine article on the profile folder says:

• downloads.rdf Previous to 3.0 Download history. Can be deleted to resolve slow downloads or hangs. [24]
• downloads.sqlite 3.0 and above; Previous to 26.0 Download history. [25] Removed in Firefox 26.0. [26]

An extension like SQLite Manager 0.8.1 would have allowed me to export the information in the file downloads.sqlite, but the current version of Firefox (29 -- I haven't yet upgraded to 30) doesn't use this file.

So where does Firefox 29/30 store this information? The Mozilla support forum:

Why did downloads.sqlite disappear from my profile when I upgraded to FF26?

The downloads.sqlite is no longer used in current Firefox releases.

Download history is stored as part of the normal browsing history (visited sites/links) in places.sqlite and a special annotation property identifies such an history item as a download item, so you can't separate them easily.

So it may not be possible to extract the information from the file places.sqlite. One possible solution posted on the MozillaZine forums last year is Export downloads history from Firefox SQLite DB.

But for me, creating a spreadsheet of filenames and copying the information by brute force (copy/paste) may be the most straightforward way to go.

---

Another strategy I've used is to capture pages with the Firefox extension Scrapbook, which keeps an editable screenshot of the page. If the service displays a message "you searched for [term 1 term 2]" then this saves your search terms along with the search results. The Scrapbook extension has a feature that will send you back to the live URL, so the actual URL for the captured page is retrievable.

Saving with Scrapbook is most useful for the hits from the British Newspaper Archives, but it doesn't solve the original problem of recording the source of the downloaded images and associating them with their captured transcriptions.