The Development of the Centre for Migration Studies' Irish Emigrations Database

In 1988 the Ulster American Folk Park (UAFP) near Omagh in Co Tyrone, Northern Ireland began to set up a computerised Irish Emigration Database (IED) in its library. This was a ground-breaking project at that time and was immediately beset by problems of all kinds, the details of which will be explained later. By 1997 the Folk Parks library had expanded to become the Centre for Emigration Studies and eventually the Centre for Migration Studies (CMS). The latter is now funded jointly by the Scotch-Irish Trust of Ulster in partnership with DCAL (Department of Culture, Arts & Leisure) and the five Education and Library Boards of Northern Ireland. In this paper I recount the experiences we had and the positive results that arose from them in the hope that this will help others who are planning to set up similar databases  [1].

EARLY HISTORY

The concept of a computerised IED had been discussed in the early 1980s at the UAFP. In 1987, Graham Kirkham, of the new University of Ulster in Coleraine, had completed a feasibility study in various archives which held material on Irish emigration. These were the Linenhall, Central & Queen's University libraries in Belfast, the Public Record Office, N. Ireland (PRONI) and the Public Record Office (now the National Archives), in Kew in London. This feasibility study covered only a small percentage of the material available as time allotted to do this research was limited to three months. Initially, the project was funded by a company named Digital and organized by the Department of Education. Digital donated hardware and provided software solutions for the project. John Gilmour, the Education Officer at the UAFP in that period, coordinated the project, and Coopers and Lybrand (C & L) in Belfast organized the setting up of the software. The late John Winters (who sadly died in May 2002) managed the Database project at UAFP and then subsequently became Database Manager at CMS. As Database Collection Manager, I have managed the Project in Belfast from its inception until the present day. Des McMorrow was in post as ICT Manager from October 2003-March 2004. A new ICT Manager is to be in place by July 2004.

SETTING UP AND STAFFING

The British Council funded Ms Joan Reedy, from Boston, Massachusetts, to work on the project in Belfast during 1989/90 and Dr Katharine Brown from Virginia, and Ms Maureen Macauley, also from Boston, during 1991/92. They all did some excellent work for the Database and were a tremendous asset to the project. All other workers have been appointed via unemployment training schemes.

After a year initially working at PRONI a new government training scheme was introduced to replace the old Ace Scheme. This entailed recruiting trainees who would earn only 10 per week, on top of their unemployment benefit. The prospects of recruiting well-qualified people to process the documents looked bleak. However, after a short while we began to build up a number of trainees who were interested in the work.

The main Training and Employment Agency was Enterprise Ulster - http://www.enterpriseulster.co.uk - whose head office is in Belfast. During the early years when unemployment was high there could be 1213 trainees at a time working at our Belfast office. Government regulations permitted them to stay for a year and the majority usually obtained jobs after 912 months with the project. Periodically they obtained extensions and stayed for 18 months to 2 years. The reduction in unemployment more recently has had an adverse effect on the ease with which trainees could be recruited. Due to new Government legislation their contracts last for only 6 months and they do not have long enough to gain the experience they would have acquired in the past. Most years there is an annual turnover of 1720 trainees.

The criteria for selecting trainees have always been an interest in history and a good command of English. Many of them are graduates, however, and reading and transcribing historical documents is a skill that is acquired over many years and the checking of all documents has to rest solely with the Project Manager. This can result in a huge backlog of transcribed documents waiting to be checked and edited. In recent years various archives have employed commercial firms, whose expertise is in computers rather than historical transcription, to digitize material. Although this may appear to be the quickest route to building historical databases, I would question the validity of the documents contained in them. Over the years, I have noted the mistakes that can be made with one simple word or phrase, changing the whole meaning of the text. Trainees who lack the experience of reading handwritten documents will often omit important words in the text. There will always be inaccuracies as no one is infallible, but hopefully these will be reduced in an historical rather than a commercial environment.

One of the most difficult problems on the project has been that trainees feel that they are doing a job and should be paid a fee for this. The EU training scheme stipulates that they should be occupied for at least 30 hours per week. At the start of the project trainees spent 7 hours per day on the project. After approximately a year, I reduced this to 6 hours per day. There is a break in mid-morning and mid-afternoon, and trainees are advised to look away from the screen after an hour or so for Health and Safety reasons. A very long day researching or sitting at the computer leads to boredom and reduction in the quality of work. Trainees can also spend part of their week on a training course of their choice. I also find it is important that trainees meet everyone on the project on a regular basis. It helps to know that others who are unemployed are experiencing the same difficulties as themselves. Over the years groups of trainees have built up friendships which still survive. One of the incentives offered to trainees when they start is that with hard work and good timekeeping etc., they will get a reference when they apply for jobs. They are shown a form which is filled in for potential employers which covers questions relating to the trainee concerning their dependability, attitude towards work, timekeeping etc. At times it is very difficult to obtain a happy medium of not appearing too strict with people who are earning very little, whilst running a project that involves serious work. Many of our trainees have found employment and I like to think that their time with us has helped. Obviously the constant turnover of trainees (due to them acquiring paid employment) results in constant training of new recruits. Over the years I have had several very competent trainees. While our major purpose is to get as many documents transferred into the Database as possible I do feel a sense of achievement if a trainee obtains paid employment even though this means more time spent finding and training a replacement.

I was employed as an Ace worker by EU at the beginning, but after approximately a year I signed a temporary contract, with EU and UAFP, which was to continue as long as the project survived. A work space was set up for us at C & L in the centre of Belfast. On my first day I contacted George Williams who was employed by C & L to direct our project. He showed me some of the equipment that we were to use. The plan was to photograph documents and then scan these into a database which was to be set up at UAFP. Two things puzzled me about this; (a) the price of processing, since the photographs would obviously run into thousands in number, and (b) the scanner was upright and could only take one page at a time. Many of our documents were contained in huge bound volumes. Within a week or two it was decided to abandon the idea of photographing the documents and a flat bed scanner was introduced. Eventually I was to move to PRONI, into a portakabin parked behind the main building. A Clerical Assistant was employed to help me process the documents and a Repository Assistant was employed to carry documents to us in the portakabin. The latter was a stipulation set up by PRONI as it was felt the extra work of carrying documents to us would be a burden to their Repository Assistants. While these arrangements were being set up, I spent that time researching in the Linenhall Library (LHL). The staff were very helpful and I was given the freedom to search the attic where many uncatalogued documents and published material were held. A long list of references was accumulated from the loose newspapers, periodicals, books on emigration etc. The next step was to process these documents. We were given permission by the library to take some bound newspaper volumes to C. & L. to be scanned. Problems immediately arose here. These volumes were very large and heavy. Many of the articles, i.e. shipping adverts, were very small. As the scanner was used like a photocopier it was almost impossible to pinpoint the article we wished to scan. The scanned document would be the size of an A4 page and would include irrelevant material taking up unnecessary space in the Database. I decided to photocopy, cut out and then scan the relevant articles. All of this took time especially as the photocopiers at C & L were also in use by their staff. This worked quite well for the next few weeks until we eventually moved to PRONI. When we moved to PRONI our Repository Assistant began working with us on the project. It was difficult to begin with, keeping him fully employed, as carrying documents to us took up a very small portion of his time. This was quickly rectified as I very soon saw many other areas in which he could become involved. I had done research for many years on Irish emigration in PRONI, so the system of processing began almost immediately. My first task was to compile a list of documents which we were to process. In PRONI I began to work through the Catalogue Books to ensure we missed nothing which related to emigration. This entailed getting out boxes of estate papers and family papers which very often contained emigrant letters etc., which were not catalogued. I began to make lists of:

• D references, which were the original documents
• T References, which were usually transcripts or photocopies of the original item
• Board of Guardian reports
• Microfilms which contained shipping lists

Very often other researchers in PRONI would find articles and inform me of them and these were added to our lists. These lists were compiled consecutively. While I did this, the Data Processor began to scan the relevant documents and the Repository Assistant printed microfilms of shipping lists on a microfilm reader which we borrowed from UAFP. It quickly became apparent that these printouts were very difficult to read and it was also very costly in terms of printing paper and ink for the microfilm reader. The only option was to engage our Repository Assistant in transcribing these lists by hand during his spare time.

C & L had set up a system whereby we sent all documents to UAFP where they were then typed in readiness for transfer to the Database. EU trainees were engaged in this task at UAFP. The numbering of documents was done by choosing the year, then the month, then the document number. For example - 9704001, 9704002, 9704003 etc. This worked quite well and helped us keep track of the number of documents we were processing each month. One major annoyance was the fact that we were typing or handwriting the majority of documents in PRONI and these were being sent to UAFP to be typed over again. Regardless of how accurately this was done in Belfast, typing the document a second time increased the possibility of errors. This also made operations more time-consuming.

The original plan had been that we would process all PRONI documents first then move on to the LHL or one of the other Belfast archives. It soon became apparent to me that the work in PRONI was unending and it would be possible to work in all the other archives simultaneously if we had enough trainees. All of the archives, with the exception of PRONI, where we worked in our portakabin, stipulated that only two trainees should work in them at any one time, due to lack of space. We began to work in all the archives and gradually built up to 12 trainees working on the Belfast Project. Periodically there were from 5-7 trainees processing documents at UAFP. During one particular period we had 17 trainees working on the project in both Belfast and Omagh.

My daily routine now consisted of mornings spent preparing documents to be sent to UAFP and helping trainees in PRONI. In the afternoons I did a tour of all the archives to check that the trainees there had no problems. In the PRONI portakabin, I found that to rotate trainees from one task to another on a daily basis kept them interested. These tasks consisted of:

• Transcribing shipping lists from the microfilms
• Typing documents
• Scanning or photocopying documents
• Sorting documents into different categories of processing

To do any one of these tasks endlessly caused boredom which in turn encouraged more mistakes. Meanwhile those trainees in the libraries were spending all of their time there, with little variety in their work. This was rectified when we acquired an office in Enterprise Ulster and they could then spend part of their time rotating between the libraries and the office. During this period we set up a full-sized photocopier on the 5th floor in The Main Library, Queen's University, Belfast. We researched and photocopied every Parliamentary Paper and Hansard Report that related to Irish migration to America and Canada, from 1800 to 1914. This took a couple of years. After two to three years PRONI was anxious that we should move from the portakabin, as they needed this space, and they wanted us to use PRONI in the same way as members of the public. The project base eventually moved to the EU office on Ormeau Avenue, Belfast. Our equipment now consisted of one Amstrad Word-processor. Trainees had the laborious task of writing documents by hand from the various archives and taking turns to type these on the Word-processor and sending these documents to UAFP. I learned about small hand-held photocopiers and we acquired approximately 4 of these to photocopy the newspapers in the libraries. We had many problems with these. They were constantly sent back to the suppliers as they could not cope with the amount of photocopying we were doing. It took practice to hold the photocopier steady enough to obtain a clear copy. The copies faded after a short while. Some of the newspaper columns were wider than the width of the photocopier so we had to photocopy horizontally and sellotape these together. Although this did not appear to be much of an improvement on writing by hand, it was a small step forward. We then acquired a Toshiba laptop. One trainee could now type documents in PRONI on the laptop and these were then checked, printed and sent to UAFP. When Apple Macs were introduced at UAFP, the old MicroVAX system there was rendered obsolete. This was brought to our office in Belfast along with approximately six terminals. We were loaned an Apple Mac computer which was connected to the MicroVAX system. Trainees could now type on each of the terminals, check each other's documents, transfer these to the Apple Mac where they were checked and transferred to floppy disc by me. However they still had to write articles by hand in the libraries.

During this period Professor Jack Smith of the Queens University of Belfast acted as advisor to the project. I suggested to him that if we could acquire a number of cheap word processors, one for each trainee, they could be used in either the archives or the office. The Toshibas, of which we now had three, proved very expensive to buy and also repair if anything happened to the hard disc. After some research I found three different types of small word processors, which were easily transported when we bought bags for them. They were Samsung, Smith Corona and Panasonic. After testing we found that the Panasonic proved the most efficient for our needs. We were able to buy six of these. These laptops proved to be sufficient for our needs and we managed to transfer documents between the Samsung, Smith Corona, the Panasonic, and a new PC which I had recently acquired.

The MicroVAX system was returned to UAFP along with the Apple Mac. The advantages were enormous. Documents could be processed in the archives by each trainee with no more laborious handwriting of material. We increased the number of documents to be transferred to the Database. It helped trainee morale as they were now increasing their word-processing skills. Several years later two Compaq laptops were supplied. Finally in March 2002 we moved into the 21st century by acquiring seven new PCs and four new laptops with Windows XP software. We were also fortunate at that time to have a trainee who had a first class honours degree in IT. He set up various programmes to simplify the process of transcribing original hand written manuscripts and shipping lists.

THE DOCUMENTS AND THEIR LAYOUT

Ever since the Database was established the focus has been to collect primary source documents relating to Irish emigration to the United States and Canada in the eighteenth and nineteenth centuries. It should be emphasised that these are full transcripts of documents. They are completely word searchable with the exception of small words such as and and the. There are several keyed fields which allow the researcher to search by date, type of document and the document number. The Database contains a variety of original material which includes emigrant letters, newspaper articles, shipping advertisements, ships' passenger lists, Hansard's Parliamentary Debates, reports of Government Commissions, and family papers. Diagram 1 shows the various proportions of these types of documents contained in the Database

To date there are approximately 32,000 documents in the Database and this number is increasing on a regular basis. These can vary in size from, for example, large journals to small shipping advertisements. Material for the Database is still collected from the several archives in Belfast previously listed, and also the Wesley Historical Society. It is hoped that the Database will eventually include information from various archives in the Republic of Ireland, mainland Britain, U.S.A. and Canada.

It is possible to list the titles of all documents that have been retrieved in the Database. After several years it was decided that these titles should be shortened to allow users to see at a glance the content of each individual document. The catalogues also needed to be set up in the new format consisting of Title, Source, Archive, Date and Type of document. As there were already approximately four thousand documents in the Database, in 1994, these had to be edited. Three competent and hard working trainees were assigned to do this task in Belfast and these corrected documents were transferred to the Database. For examples of the new format for catalogues see Appendix 1. At this stage, I could also see the need for guidelines for document titles. A format for all examples of these was set up and trainees have used this ever since. See Appendix 2. How best to deal with spelling, punctuation and grammatical practices that may not necessarily have been incorrect at the time of writing, was a concern. Documents were also often faded, in bad condition and the hand-writing difficult to read. Our first instructions were to leave these wrong spellings in the text exactly as seen. I pointed out that this would be useless for search purposes as the majority of people would use the correct spelling whilst searching the Database. In an attempt to rectify all these problems, a meeting was held at UAFP and a pattern for editorial policy adapted which we have used ever since. A handbook of rules was set up giving guidelines to transcribers over the years. Consistency has been important in this area because to change the pattern could cause confusion to researchers. Certain rules were set down. See Appendix 3.

With a high turnover of staff there is always a danger of documents being duplicated. If data processors are only permitted to stay six months or they get a job within this period they may forget to keep document lists updated. We have recently set up a Document Progress Database that monitors the production stage of each document. This is still reliant on methodical updating but is much more efficient than the older method of hard copy lists. If any duplicates do slip through the net, the Project Manager in Belfast has access by telephone line to the Database in Omagh and if in doubt can check to avoid duplication. Now when documents are transferred a simple procedure is followed checking by date and then checking titles to ensure none are repeated.

DONATIONS/LOANS OF DOCUMENTS

Increasingly owners of original documents lend them to us and with their permission we transcribe, scan and transfer them to the Database. One recent example is a collection of letters which were found in a house in Castledamph, Plumbridge, Co Tyrone. The owner contacted the Project Manager and arranged to leave one case full of documents and another of photographs at the CMS in Omagh for inclusion in the Database. These documents consist mainly of letters between James Alexander Smyth, who lived in Canada and Ireland, and his family in Castledamph. They are presently being transcribed and scanned by the Belfast team

SEARCHING CMSIED ONLINE

Diagrams 2 5 show the process of screens that appear in the Database online when the word Castledamph is the search term. In Diagram 2 Castledamph has been typed into the Query Box and a search in the Database for all references to Castledamph is underway. Diagram 3 shows the list of documents retrieved with a reference to Castledamph in them. In Diagram 4 a document has been selected. Diagram 5 shows the full transcription of the selected document and Castledamph has been highlighted. Diagram 6 shows Thumbnail views of all the images available for the original document. Diagram 7 shows the full-size image that was selected.

Where possible we scan printed documents through Optical Character Recognition, (OCR). However this can be time consuming if the document is a nineteenth century Parliamentary Commission or Hansard Parliamentary Debate. The programme very often does not recognize the older print and a lot of editing is necessary. This negates the advantages of using the OCR programme. With the owners permission (see below) we scan original handwritten documents through Adobe Photoshop and attach these images to the transcriptions. This allows researchers to check the typed version against the scanned image of the original if they are doubtful of the accuracy of any of the text.

There are many advantages to a historical database. Original documents may be scanned, where permission has been given by the owners, so preserving the original copy for posterity. It is also particularly important when owners retain the originals and may not have the right conditions for preservation etc. Historical analysis can be done more rapidly. Studying typed versions in the Database instead of wading through hard copy manuscripts is a tremendous advantage. This is especially beneficial for visiting academics whose time is limited. It is also excellent for comparative studies. Very often, especially with emigrant letters, it is difficult to find repetitive evidence in a large number of documents. This is now possible with the digitization of historical manuscripts.

THE TECHNICAL SYSTEM SET-UP

The Irish Emigration Database resides on a Windows NT Server at the Centre for Migration Studies. The data is contained within a database application program called Status IQ for Windows which is text based and word searchable by keyword. It was developed by Dataware Technologies (formerly Solcara) - http://www.dataware.com. The Centre is looking at the possibility of migrating to another database system in the near future as technology improves and the use of the Internet as a learning and research medium continues to grow. Visitors who come to the Centre can use the four client terminals that are available in the library to access the Database on the server across the Local Area Network.  [2]

An online version of the Database is available to a limited number of people. It resides on a Windows NT4 Server at the Queens University of Belfast. The server currently has Service Pack 5 on it and is running Internet Information Services 4 (IIS) as its web server application. A program called Net Answer for Status sits between IIS and the Status IQ Database. As there is only one server, when it receives multiple hits, its performance can be drastically reduced. To restrict the number of users a username and password is necessary in order to log on to the online version (see Diagram 8). The type of people who have been given access are students who are undertaking the MSSc. in Migration Studies at the Centre and academics with an interest in migration studies. The general public can also access the online version of the Database by visiting one of the main libraries throughout Northern Ireland. Staff there will log them on to the Database using one of the Public Access Terminals in the libraries. The Database is also available online to the public at PRONI where many of the original versions of the documents are housed. The plan is eventually to make the Database available to everyone on the world wide web. However there are technical, financial, legal and logistical hurdles to overcome before this is possible.  [3]

Although Service Pack 6a is available for NT4 Server, this cannot be installed as this would then require the Database and the web server to sit on two separate servers. Some of the services and patches that Service Pack 6a installs require the Database to be on a separate server from Net Answer. Microsoft has recently stopped supporting NT Server and this has further highlighted the need to look at other alternatives for the Database both in Omagh and the online version.  [4]

COPYRIGHT

Copyright has been a serious issue from the outset of the project. We had hoped to scan or photograph the original documents, transfer them to the Database and then transcribe them at a later date. Due to the Copyright Law we were permitted to transfer transcriptions only and not scanned images of documents from PRONI to the Database. On 17 June 1988, John Gilmour, Bryan Follis of PRONI, and myself had a meeting to agree certain points. The points which affected us were as follows:

(3) All documents which are more than 100 years old or which have appeared in a publication and if 50 years has passed since the date of publication, may be scanned and an image made. However such an image copy can only be made on the understanding that it is for private study and research and accordingly a Copyright Declaration form will have to be completed.

(4) Although the Copyright Act, 1956 prevents the making of an image copy for documents under 100 years old, the contents of such documents could be keyed into the computer. No Copyright declaration form would need to be completed for such an input of information.

(5) No visual image of the original document may be viewed by any member of the public at the Ulster American Folk Park nor can print-out copies be made. The only exception would be the shipping lists but PRONI must first clear this proposal with the owners of the documents. The UAFP may however make transcript computer print-outs of documents available to the public on an individual basis....  [5]

It became very obvious that we would have to decide on a particular course of action from the start and we have adhered to this over the years. In PRONI we type many original manuscripts and we always supply whatever contact address, phone-number, e-mail address etc there is available from the catalogues in PRONI. This enables researchers to write to the owners of documents to gain permission to use quotes from particular documents if desired. Copyright was obtained from the publishers of the Parliamentary Papers and Hansard Reports to allow us to transcribe some of their reports, commissions etc. More recently many collections of original documents have been lent to us from private owners (see above). With their permission we transcribe, scan and transfer them to the Database. We always state in the catalogue whether copyright is retained by these owners or not. In the last resort if material was ever sensitive to families and they strongly opposed their inclusion in the Database, we would delete the offending material. When in doubt about the copyright of any published articles, I write to the publisher and these reply letters are kept on file.

THE FUTURE

When the library at the Ulster American Folk Park, where the Database is housed, became the Centre for Migration Studies, the area of research there expanded to include Britain, Australia, New Zealand and elsewhere. The natural progression is for the Database to also include these areas. However, the problem arises as to whether they should be included in the existing Database or if specific Databases would be preferable. There was always the possibility that one Database would overlap with the other. Many emigrants who emigrated to America or Canada had already traveled to Australia or New Zealand and their letters were included in the existing Database. This could necessitate going back through all previous newspapers etc to search for material which was not relevant in the early days.

Initially the collection project was expected to last 6 months, but the amount of material available is growing rather than diminishing. It is now clear that it will continue well into the foreseeable future. Over the years funding for staff and equipment has been limited. The majority of projects are funded for a given period and on completion of that particular project the funding stops. The difficulties of finding funds for a project such as ours are immense. Having arrived at this point with our Database, there are many considerations to be addressed in the future. How to improve the search facilities for the user is always uppermost. In many of the letters, for example, there are various themes such as Return Migration and Agriculture running through the document. In a word searchable database it may not be possible to retrieve information on these subjects as the exact terms may not be used in the document. There are two ways that this could be handled. The first is to go through every document and transcribe into the notes section of a document themes that are contained within the document. A second option would be to set up Variant lists for certain themes such as agriculture i.e., cows, grain, crops, hay etc, words that are linked to the base word. The method of using key words in the note section would seem to be the best option but it would be a huge project for someone to systematically go through every document in the Database to do this.

CONCLUSION

In conclusion, the necessity of having the original document available in electronic and original form to the researcher is important. Human error is always possible in the transcription of our documents. Therefore to have both copies available whether in PRONI or the Database is important. If quotes are to be used it is always best to consult the original. However the numerous advantages of using a Historical Database far outweigh any negative issues and this is what has kept the project moving forward over the past sixteen years. In future there should be easier methods of collecting information. We have always tried to be innovative over the years and will continue to be so in the future.

Lorraine Tennant BA, M Phil. Irish Emigration Database, Project Manager (Belfast), Centre for Migration Studies.

May 2004. 4 The Mount, Belfast, BT15 4NA, N. Ireland. [email protected]

Images

Types of Documents

Search Screen

Retrieved List for "castledamph"

Selection of Document

The Transcription of the Selected Document

Thumbnail Views of Images

Image Selected to View

Password Screen

Notes

1. Tennant, Lorraine, The Irish Emigration Database, Journal of Scotch-Irish Studies, Vol 1, No 1 (Spring 2000) pp120-124

2. Des McMorrow Former ICT Manager at the Centre for Migration Studies

3. Ibid

4. Patrick Morgan of Morgan Software.

5. Bryan A Follis, PRONI, June 1988

Appendix 1

CATALOGUE DETAILS

Each document should be processed through either the Shipping List (SL) Programme or the Document Entry Application (DEA) Programme. This should be done by clicking on either the DEA Network or SL icons.

1. The Title should not exceed three-quarters of the width of the screen.
2. The Source may extend well beyond the right-hand margin and may be 2 lines in length. However it should be on one line for processing in preparation for the Database. Where margins are set, word wrapping onto the second line is acceptable.
3. The Archive should not exceed three quarters of the width of the screen.
4. The Date should be divided by colons in the format given below. Do not put a full stop at the end.
5. The Type should always be typed in capitals and should be 2 to 3 letters only. Do not put a full stop at the end.
6. A Blank Line should be left between the end of the catalogue and the start of the document.

CATALOGUE EXAMPLE 1

Start of Document

CATALOGUE EXAMPLE 2

Start of Document.

Appendix 2

Appendix 3