Roam Research: A new software for historical research

Background:

This is the first time I’ve ever tried to present a new software tool to a general-(ish) public. In my managerial positions I promoted new large-scale work environments, but thankfully I’ve put my managerial career behind me and am back to historical research. Although there was that one time, between managerial positions, when I spent a year or so with a start-up working on a way for researchers to organize their data and ideas. I dropped out when I went to run Israel’s National Archives in 2011. When I left in 2018 I briefly toyed with the idea of re-starting the start-up, since no-one had yet offered what we’d been intending, but didn’t.

Recently I stumbled across a new piece of software, Roam Research. It took about 30 seconds to see that their basic idea was pretty close to ours. So I started using it. A month later, I’m here to report.

In 1991 Geoffrey A Moore published Crossing the Chasm. He explained that selling a new product to the small market of folks who like new things, doesn’t mean there’s a large market. 

(Linked from here). 

My impression is that right now Roam is exciting early adopters, many of them technology-minded millennials. Perhaps some of them will read this post and tell me of better ways to be using the software for my purposes. So that’s one motivation for writing this post. The second is to interest historians or similar researchers. The more people use Roam, the greater the likelihood its development will move in directions we can use.

I have no commercial connection to Roam Research, beyond purchasing their product.

The Core Idea:

The central feature of Roam is the simple idea that any content that’s likely to repeat itself in your material, gets its own page. Creating pages is extremely easy, and there’s automatically a link from where-ever you repeat something back to its page.

For example, look at my description of an historical document:

Page 195: Important General Cabinet decisions from April 11th, 1984: Adjoining expansion of settlements which have already been authorized by Inter-ministerial Planning Committee does not require additional authorization by that committee unless there are differences of opinion. The expansion does require authorization by Pliah Albeck's unit, and by the planning unit of the Settlement Division.

Cabinet decisions are a type of source, as are court rulings or research articles, say. Adjoining expansion is a tag, or a keyword, representing a concept. That inter-ministerial thingy is an organization, as is the Settlement Division, and Pliah Albeck is a specific woman’s name. April 11^th is a date, and we expect of dates that they can be arranged in progression or segments of progression.

Notionally, all of this can be done on the fly in Roam, in an unstructured format, and any of it can later be retrieved. If you’ve got hundreds or perhaps even a few thousand items, perhaps it really can. Anyone who has lived with large databases knows, sadly, that it’s unlikely to work that way with them.

The Research Project:

My project is to research the history of Israel’s settlements in the occupied territories. Millions of words have been written about them, mostly in the media, but there’s also quite a bit of academic research, in sociology, geography, political science and even theology. It’s striking, however, how little solid historical research has been published – mostly because the relevant archival material was sealed until recently and some of it still is. About two years ago I joined up with the Taub Center for Israel Studies at NYU. Our goal is to get the full archival record into the public domain, and to facilitate research on it. I personally intend to do some of the research.

It’s a broad project. 50 years, for starters. Hundreds of governmental agencies and non-governmental organizations. Many hundreds of historical figures. More than 300 settlements, some tiny two-shacks-and-a-pickup-truck affairs, others towns with almost 100,000 residents. I’ll need a large number of keywords. There are thousands of sources, archival files and others. Long before I have any results to publish, it will be take two or three years just to go over the materials. Eventually I expect I’ll have tens of thousands of notations.

 My Roam solution:

In the center of the screen is a detailed description of a file. On the right is my list of keywords, which I often refer to, but sometimes I put other things there. On the left is the list of categories I’ve created to keep control of what I’m doing. The materials themselves, more than a million pages of scanned documents, are in a separate, cloud-based system.

The centerpiece of my version of Roam is the list of categories. I’ve got eight of them.

Chronology:

Note that I insert dates only when I come across a document I want to describe. I saw no need to create in advance a list of all the 19,000 days that have passed since the Six Day War. Thus, at this early stage of using Roam, I’ve only cited one event in 1988, on June 14^th, but lots of events in 1989. This is a quirk of the files I’ve looked at so far. Having this table enables me to follow events gleaned from multiple sources, in their chronological sequence, and also to see separate threads of simultaneous events.

There’s a list of settlements. The flexibility of the software allows me to nest settlements under their region, and alternate names under the official one. Each settlement has its own page – that’s the item all its mentions throughout the database link to – and that’s where I’ll put standard data: year of founding, population stats and so on. I haven’t started doing that part yet, and there’s no need to hurry. The ability to link all mentions of a town to the town’s page is unaffected by the wealth of data on that page.

Historical figures: Here I stumbled across a cool possibility. 

Look at Ariel Sharon, a key figure in the story of the settlements, mostly as their champion, sometimes as their nemesis. I’ve created a top-level page for him, and underneath it I’ve nested pages for his various positions. When I come across him as the Minister of Agriculture I index to that specific page. Down the road, I’ll be able to pull up everything about him, or limit myself to his actions in a particular function. Any historical figure who moved around will get this feature.

Organizations:

Organizations are more complicated than people, since they evolve, change names, migrate between ministries, some privatized and moved completely from the government administration into the private sector, and so on. As my DB grows larger and richer, and assuming I’ll be able to keep a handle on all of these developments, the page listing the organizations, coupled with the descriptions on the specific pages, should come to have considerable value, over and above its power as an index.

Keywords:

Of course I need keywords. The reason my list of them is often open on the right side of the screen is to be sure I’m not inventing new ones all the time. Each time I have a need for a keyword I start by looking at the list of existing ones; often, there’s already one that’s good enough. Better to use the same keyword for lots of similar cases, than to have endless narrow-and-precise terms and lose the ability to see then together.

Types of info looks like technical information. It isn’t. The ability to distinguish between types of information creates the possibility of telling particular strands of the story. What was the Cabinet deliberating, is an obvious example. What were the courts saying? How do people present events in their memoires?

Finally, I have a page called Templates.

 This is where I force structure on my descriptions. Whenever I create a new page, I copy and paste into it the structure of information I need to describe it. Roam advertises itself as a flexible, free-of-advance-structuring system. My experience since the 1980s is that planning your structure is a key to thinking.

Information Retrieval:

The point of the entire exercise is to extract useful information. Or perhaps, to extract specific useful information.

The easiest way to do this is to go to the page of an entity and see all the items that are linked to it. If there are too many of them, they can be filtered. If you wish to see what information comes from the overlap between entities, Roam offers the usual search operators: And, or, not, & between (for dates), as well as combinations of them: (a+b) or c but not d, between 1^st date & 2^nd date. The one function I’ve been told is there but I’m not managing to see, is searches that include nested entities. Thus, to return to a previous example, a query that includes Sharon should also bring results that were tagged Sharon, Minster of Agriculture. If any reader wishes to help me here, I’d appreciate it.

The larger issue is the organizing of large quantities of results. What do you do when a precise query gives you 450 results? I don’t see tools to organize the results once I’ve got them. A spreadsheet can do that, if you built it right. Here I’m left hoping. Roam is a new and developing product, and I hope this is on their development roadmap.

Finally, a thought about writing. I’m not going to write articles or a book in Roam. Microsoft Word has a 25-year head start, and it will always offer better writing functions. However, Roam does have product management functions. Quite a bit of the buzz focuses on them. I have no need, except in one context. I expect to use a few of these functions to mark the items I need for specific articles or chapters, and then mark which I’ve already used and which remain to be used. That should be helpful to ensure I don’t forget to use important documents, but don’t use them repeatedly.

Gmail: yaacov dot lozowick

Twitter @yaacovlozowick