The pros and cons of personal diaries

Hi, I'm new to this forum and I am really impressed by the wealth of knowledge on display.

I've recently transcribed my grandfather's war diary from North Africa in 1943.

The events he describes matches his battalion's diary, and it has references to his family members - which won't be useful to anyone outside my extended family - but there are also his personal comments, and references to his comrades and superiors that I would imagine researchers of this campaign might find useful.

Have any of you either shared relatives' personal diaries from the war, or used personal diaries in your research - and how valuable did you find it?

 

Welcome to the forum Matthew.

Personal accounts are absolutely invaluable in research, and just by saving and transcribing your grandfather's diary you've contributed to the historical record. If you are willing to share your transcript here, no doubt forum members with knowledge of the NA theatre would be very interested to read it, and would be able to point you to more resources specific to the operations your grandfather experienced (I wouldn't have a clue, all my research is on India/Burma).

Every source can contribute a piece to the 'jigsaw'. Sometimes just a word or two in one source can clarify or confirm something that's unclear in another source. Each has to be evaluated on its own merits, of course, and you have to watch out for bias and assumptions and exaggerations. People often make the mistake of taking official documents like war diaries as gospel truth, but in reality diary keepers could (and did) make errors in dates and places, often had incomplete information on which to base their entries, and were liable to downplay any failings by the unit. One quick example that springs to mind is the diary of an Indian battalion in the Arakan. When a platoon failed to return from its patrol, it was recorded that the platoon had been ambushed and captured. Only by greatly expanding the scope of research will you start to find the truth - that the platoon had defected en masse to the Japanese. If you're so inclined, this leads you on to the intricacies of Indian politics, Japanese propaganda efforts, failings in leadership/recruitment/training, the murky world of field security, official cover-ups, post-war recriminations... There are an awful lot of rabbit holes to fall down!

With personal accounts, the danger is that while they can give you an accurate sense of what was happening on the ground, the author may be wildly wrong on other aspects. You often find that writers don't have a very clear idea of what is going on beyond their immediate surroundings; with more junior ranks, they frequently aren't even sure exactly where they are at the time. If it was written after the event, you always have to bear in mind how fallible our memory can be, sometimes without us even realising it - I've seen plenty of accounts where the writer has got the chronology completely wrong, and mixed up things that happened in different years. It's not surprising and it's not intentional, but you have to watch out for it. The more sources you have the better, as by comparing and contrasting them you can unpick clearer sense of the truth. Personal accounts from senior officers are usually better informed, but you have to remember that there may also be a degree of arse-covering and face-saving in their version of events. Holding a high rank, especially in a time of total war, doesn't make you immune from personal prejudices or vendettas, and if you want that gong or knighthood at the end of hostilities you'll need to shape the narrative in a way that justifies your failings.

Personally I find it fascinating to be able to view the same event from multiple viewpoints. Individual accounts and diaries can give a sense of things as they play out at the 'sharp end'. Reading them alongside the unit war diary gives you a wider view, like a camera zooming out. Then you can go sideways to the documents of other units involved - diaries of the supporting artillery and engineers for example, or ORBs from the RAF squadrons overhead. And then upwards, through brigade and division diaries, and outwards to the ancillary units like provost, medical and transport services. Further up you reach the corps and army diaries, and perhaps draft monologues and reports on the operation written by senior officers (the Indian Archive has been especially good at saving these). You can go further still, up into the realms of General Staff and politicians, where Cabinet Office minutes and GS planning papers can reveal the reason a particular operation was embarked upon, its original intentions, and the compromises enforced by limited resources and the demands of other areas.

All that to say... post it if you can, the NA nerds will love it!

Martin

 

Hi Matthew,

I have used personal accounts and diaries quite a lot in my research into the first Chindit expedition in 1943. These give an all round perspective to the campaign, especially when it comes to the views and experiences of the lower-ranked soldiers in my view. Martin has put the case far better than I could above.....as he says the members with a North Africa interest will be happy to hear about your grandfather's diary and thoughts.

Thanks for posting.

Steve

 

Cheers Martin and Steve for your encouraging words, and an explanation of the caveats.

I'm persuaded that I should share my transcription, and - while I have at least two people's attention - I want to raise another point: I am being slightly disingenuous when I say that I transcribed the diary; it was actually Gemini 3 that transcribed it. I edited it.

The diary has been in my family for decades, however, we are not a military family, and none of us fully understood its contents: it has messy handwriting, and the ink from one page sometimes bleeds to the next; it uses military jargon and abbreviations; and it has a lot of unfamiliar place names.

I had hoped that some military museums be interested, in a kind of quid pro quo arrangement: they get access to its contents; and I would - hopefully - get to understand it through their expert analysis. However, none of them got back to me, and I didn't pursue it any further.

A couple of weeks ago I read an article from a Canadian historian who said he was gobsmacked at Gemini 3's ability to transcribe handwriting and infer meaning from it - so I thought I would try it on my grandfather's diary. I gave Gemini my grandfather's name, battalion, and rank, and asked it to transcribe each entry; then give a glossary of terms; then explain its historical context.

The resulting output astounded me, and I found I was now able to comprehend it. However, I am familiar with the limitations of large language models and their tendency to hallucinate, so I checked its output. I found it was remarkably accurate; there are a few instances in which named individuals were incorrect, but I could fix that by referring to the battalion diary - a job for another day.

I would guess that artificial intelligence will play a significant part in opening up historical documents, particularly from people like me who have the curiosity - but not the knowledge - to understand documents in their possession.

That's my aside finished - which may or may not be useful to researchers.

In any case, I am happy to share the annotated transcriptions - and any of its high quality scans, on request.

The annotated transcription is in PDF format, and is about 100 pages long. It is roughy 2 MB. How would you recommend I share it e.g. which forum, and should I upload the file to the forum, or upload it to a public web server and link to it?

Matthew

 

Hi from Germany,


During my research for the tunnel escape of British officers from a local POW camp in 1941, I listened to local contemporary witnesses (teenagers in 1941) and diaries of many officers.
Without them, I could not have written my book about the escape, both, in German and English.
Stefan.

 

Not diaries, but my grandfather’s letters to my grandmother, some from the BEF in 1940, which I hope many with an interest in 1940 found helpful, then his later letters up to 1942.

I learned a lot from them, family stuff naturally, which in itself was eye opening, but quite a bit about the social history and military life at the time. An incident that I had read about earlier in Gun Buster’s book Return Via Dunkirk turned out to have directly involved grandad, which I’d never have known without his letter home!

I transcribed his letters and posted them here, which I believe some found interesting and informative, so yeah, I heartily second what the others have said. Go for it Matt! Look forward to reading them.

 

I work in the heritage sector, and digitising letters and diaries from soldiers who served in South Australia, during WW1 & WW2 is a high priority for the institution. Naturally, the letters had large, black "blocks" for the WW1 letters and cut-outs for those from WW2; as an historian myself, this type of material is gold - knowing the story of those who were involved first hand, in their own words, is so important to the end result.

What they thought, felt, endured and even what they did in their down time, makes for a far better story than dry reports of shelling, enemy movements and map co-ordinates. Reading how heavy the shelling was from someone who endured it, is far more interesting IMHO, than reading the line/s "heavy shelling from 2am until 4am" with a total number of shells used as a closing remark.

As an example:

My grandfather was involved in a convoy which had it being followed by U-boats. The war diary of the lead ship gave a lot of interesting information, but a letter written to a group of school children who had "adopted" his ship was far more interesting. That letter gave a human insight of the events, and whilst the writer did not reveal where they were, I knew instantly having already read the war diary, that he was describing the same events, with more feeling and insight than those diaries could ever give.

I'm sure many more members of the forum will find your efforts of great interest, even though they may not have commented here.

 

I have My Grandfathers diary from 1944 which I am transcribing, at the time he was attached to AFHQ and was stationed in offices in and around Naples. It makes for good reading but most of it is personal to him, what duties he was on, what they watched at the ENSA theatres, what he did in his spare time...but there are names given of friends, superiors etc which I think is interesting for research.

 

I have a couple of secondary concerns - which I should have listed in my original post: how to address copyright; and how to address living family members? My uncle is my grandfather's closest living relative, and I feel he ought to know what I plans I have for my transcription - out of courtesy. I wouldn't go ahead with publishing anything he was not comfortable with.

Possibly also my siblings, who have an equal interest on anything passed down by my father.

 

Thanks for posting about Gemini 3, Matthew. I have been a bit of a Luddite with regards to 'AI' so have fallen way behind in understanding it. If it can help with the hard work of transcription I'm all for it though, so I have finally forced myself to give it a proper go. Results: very impressive and very worrying.

I first fed it a random page from a hand-written document on the Burma Navy that I recently spent untold hours slogging through unaided. This is what I gave it:



And this is the start of what Gemini transcribed:

"Port, Hillington and Marine Gun. Warrant Officer Herrington taught the intricacies of lights, naval guns & Saturday mornings were laid aside for gun drill at Ponting Point where was an H/A [High Angle] L/A [Low Angle] apparatus on an (obsolete) 6 inch gun. Signalling lectures & practical instruction were given to apprentice men by the signalling instruction, Lieut. Duntley & Whitcutt, assisted by technical ratings who had had previous experience in civil life, & the Telegraph Department..."​

Not immediately promising. That handwriting is horrible, but Gemini made errors right away that seem obvious when you have a little knowledge of the subject. For the opening words it suggests "Port, Hillington and Marine Gun" when "Bren, Hotchkiss and Maxim guns" is plain as day to my eyes. And I know it's "Monkey Point" not "Ponting Point" because... well it just is, that's the spot in Rangoon the author will be referring to.

However, elsewhere it correctly guessed some words that I had struggled to decipher myself - very useful. Overall it did a fair job, better than any OCR program I have seen. What surprised me most was that, unprompted, it worked out the context and made good suggestions for some of the abbreviations it came across.

After that quick experiment I'm impressed by the potential, but what has me worried is the result I got with the next document: a Mentioned in Dispatches citation. This came from a file (WO-373-91-1) of assorted MiDs digitised by the staff at Kew and looks like an old photocopy of the original. Probably the only surviving copy, horribly degraded with that speckled photocopy 'noise' obscuring the text A very tough challenge. I can make out just enough words to get the gist of it, but only because it relates to a specific little niche of the Burma campaign that I have been researching for years. Here it is:



In short it concerns Bombardier Burt of 130th Field Regiment RA, surrounded by the Japanese at a village called Indin, who volunteered to go out in an Armoured OP carrier to recover a truck that had been abandoned outside the perimeter. I could already transcribe the first line pretty confidently: "In Arakan, Burma on 7th April, near INDIN information was received from the Durham Light Infantry that a 3 Ton Lorry loaded with rations had had to be abandoned..." From there on I can only get maybe half of the words.

Could Gemini work some magic and pick out the typeface among the noise better than I can? It immediately recognised the page as a recommendation for a military award - good start - then gave me this transcription:



Incredible. It is completely and confidently wrong in every detail. It hallucinated a totally irrelevant citation, and when I pushed back it disagreed and argued at length (with f-king gusto and diagrams!). Giving it specific corrections had it cheerfully admitting small errors, but it continued to spew nonsense. "Warrant Bombardier"??



Very post-truth, kind of "Well that's close enough, even if what I told you isn't strictly true then something like that happened, or could have happened, don't stress about it." Not good. I don't want vibes, I'm researching factual history. It seems that you need to babysit it and check its work thoroughly - if you don't it will merrily concoct an alternate history and present it convincingly as fact. For the field of historical research it strikes me as an amazing helper for certain things if heavily supervised, but an absolute menace if left to its own devices.

I gave up and fed it a tightly zoomed crop of just the citation text, and went through a few iterations of offering it corrections and extra info. Now we started to get somewhere, and it began get words that I had been unable to make out clearly. It continued to argue that the phrase "just before the Germans arrived in the area" was in there (this was Burma 1943...) and kept substituting things like "armed party" where I knew it was actually "Armd. O.P.". Still, by persisting and getting really specific, it gave me some useful outputs. I then zoomed way in on the image myself (and put my specs on) and went through word by word, heavily editing what Gemini had produced. Using it as a tool in this way let me create a transcription that, other than a few words, I'm confident in. Cracking result, and way better than I had managed alone using any other method, but I really had to fight for it. It was like working with an infuriating colleague, skilled in some areas but with a massive ego that won't let them admit when they have got something wrong or they don't know something, with the confidence and charisma to lead you up the garden path if you're not fully switched on. The AI model seems to favour a wrong answer over no answer, which I find very concerning.

To be fair I don't know enough about LLMs to structure my questions efficiently, and I guess the model will be getting better all the time. Clearly it is already a useful tool in the box if you are transcribing difficult documents - with the caveat that you really need subject knowledge and attention to detail to spot when it is fabricating stuff. I plan to learn more, so thank you again for the introduction to it.

None of that waffle is intended in any way to disparage your use of Gemini in the transcription! I think it's an excellent idea, and if it unlocks an impenetrable document and gives you some decent output as a starting point for further editing then that is a brilliant use of technology. The only danger is that if you don't have good familiarity with the subject, you may not spot all of the inventions and errors creeping in.

So what to do with your diary next? Museums are great but so under-resourced that they are unlikely to be able to help with a detailed interpretation; realistically, the diary would just get catalogued and boxed up in their archive, then quite possibly never see the light of day again, preserved for the future but effectively forgotten. IMHO your best bet is with active and enthusiastic amateur/semi-pro researchers on the internet - like us spods who have gravitated to this forum. Are you planning to get deeper into researching your grandfather's war yourself, or have you reached an endpoint and want to pass the resource on to others who might find it useful?

If I was a North Africa researcher I would be thrilled to see it. The ideal for me would be to have your edited transcription (so it is easy for humans and search engines alike to read) along with photos of the original that could be referenced if anything didn't look quite right. You could have all the photos collected as a single zipped folder or combined into a single PDF file (with the resolution/quality reduced to keep file size down). I'm sure that if that went onto the sub-forum 'Theatres of War > North Africa & the Med' and you asked for help, other members would gladly read it and offer corrections (if any are needed), or give you more context on the precise events your grandfather wrote about.

Copyright is tricky. You could embed a copyright notice or watermark on your files if you wanted, and if you want to keep slightly tighter control put it on your Google Drive or other cloud storage and share a link rather than actually upload it to the forum (would also avoid the file size limit for forum uploads). On the family angle, definitely good to discuss with others if it is a family heirloom rather than something passed solely to you, but I would make the point to them that sharing your grandfather's words is an act of remembrance and helps keep his memory alive. If any entries include family details that you find uncomfortable or too revealing, you can always redact sentences in the transcript (and put a black box over them in the photos if necessary).

Bloody hell that's a long post! Hope there's something of use, and feel free to ask for help, this is a friendly place.

Martin

 

Hi Matt,
What regiment did your grandfather fight in?

 

Good advice. I did run the question of posting the letters on here by my older brother, who is, effectively, the “head” of the family now, so to speak. He was fine with it, but I’d have felt uncomfortable doing it without that chat.

I did also ask myself whether sharing what are private letters was the right thing to do, but went with it in the end for the historical interest. However, tbh, I’m still asking myself the question even now. Some things in the letters I did redact at the time of posting, where there was a privacy line that I just didn’t want to cross. That was a very subjective matter, of course.

I’d not thought about adding a Copyright to the letters, hadn’t occurred to me, that.

 

Thanks everyone for your messages of encouragement.

My uncle has given me permission to publish online for non-commercial use, and I am waiting on my siblings; I doubt they will object but I want to give them the opportunity to read it before anyone outside the family does.

I will probably use the time to proof read Gemini's transcription again because I think some of the officer names are incorrect, having had a brief look at the battalion diaries - which I've only just discovered. I imagine that many more inaccuracies will surface when forum posters get an opportunity to read it.

Someone asked who my grandad served with: it was 6th Battalion Grenadier Guards; the diary covers the first six months of 1943.

Martin - thanks for showing your successes and failures with Gemini. I suspect that you were using version 2.5 (Flash) because I added the same scans to version 3 (Pro) and the results were closer to your suggestions. In any case, the point you made about LLMs cheerfully peddling their output with no hesitation or doubt remains a legitimate concern.

Matthew

 

My family have approved the publication of my transcription, and I have posted a link to it in this thread, as suggested.