on the keywords, databases and inclusion/exclusion criteria to use?
Partly through trial and error. I found that it was only by running a particular search you could really understand how certain terms are used. The early searches were therefore probably a waste of time, but did they at least helped me understand the searching process.
To give an example, I was interested in Acceptance and Commitment Therapy delivered as a training intervention in the workplace. But what is ACT? Is it just what someone calls it? Is it something specific about the content? If so, what? Is it defined by outcomes, for example ACT is always designed to develop psychological flexibility, is it about that? All of this has implications for search terms.
146 explanation).
3.2 What challenges did you face and how did you overcome them?
The technology was probably the biggest challenge, but defining terms was surprisingly complex. Potentially the technology was difficult because I had been away from academia so long, so it took me a while to get to grips with 1) accessing the Universities systems and navigating my way around and then 2) familiarising myself with the workings of each database.
Although the university staff were very helpful, I often waited a few days to hear back from them, which slowed the process down and made it more disjointed.
I also found several exceptions in terms of relevant papers to the searches themselves. For example, I knew about a particular paper which I thought would meet the criteria, but it did not appear in my searches. I would then need to understand why my search had failed to capture it. In most cases there was a good reason, in some cases it remained a source of confusion.
However, the nadir as far as this part of the process (and the technology) was concerned was definitely Refworks. The process of de-duplication within RefWorks was laborious and frustrating. Although I followed online tutorials and advice from Library staff to the letter, I kept on losing the merged folders. Only later did I find out that this was a common glitch. By
147 In terms of narrowing down (i.e. down to abstract sifts and then onto full papers), there was a reassuring convergence on both mine and Rachel’s searches. In terms of abstracts quite a few of the papers turned out to be PhD dissertations, which were hard to obtain. I had absolutely no help from the interlibrary loan system at the university library and several e-mails were ignored. However, I had a much better record of getting papers sent to me by the authors themselves.
In the end I did manage to get all of the papers. In the final analysis however, I decided that the quality of some of the PhD papers was too varied – it would have made the SLR far less focused and I would have been distracted by the need to standardise papers that were often ambiguous. So, in the end I decided to add published papers to my list of inclusion terms. This still left me with 14 studies which felt about right.
The second challenge was probably defining the search terms, and I was surprised how a seemingly straightforward tasks can lead to philosophical debate. ‘What is Acceptance and Commitment Therapy’ is not an easy question to answer, though I was reassured that this debate is mirrored within the Association for Contextual Behavioural Science (ACBS), which is the main research association for ACT. In fact, some of the founders and key researchers within the ACBS regularly debate what makes an ACT intervention. There is broad agreement
148 working as its guiding principle. This means that any intervention that helps someone unhook from the content of their internal experience and focus on the most workable action is ‘doing’ ACT. This is the basic definition of psychological flexibility, therefore I reasoned that it is this that distinguishes an ACT intervention and that that should be a key search term. The rest of the search terms were deliberately left broad, as I felt that this was more art than science, so I would need to use my own judgment using inclusion criteria.
3.3 How did this process differ from your expectations/plan?
In all honesty I am not sure it did. With the technology I was expecting the worst and it was indeed hard to understand at first. Robert Elves was very helpful and some of the online advice you can get is also good. However, with the databases especially it was a question of trial and error.
The interlibrary loan system was a mystery – I received no reply to several requests for articles and yet I know colleagues had success with the system.
The Refworks de-duplication process simply didn’t work.
The search terms were perhaps harder to define than I thought, though to be honest I didn’t have too many expectations for this part of the process. I was in new territory.
149 of conducting research than before, so the difficulties at least feel worthwhile. However,
whilst the technology is undoubtedly improving, it continues to be a little clunky for users and to lag behind.
3.5 What would you do differently if you were to go about conducting systematic searches again?
I would use my own manual system as a backup to Refworks and do the search more deliberately as a result, documenting each step.
Invest time upfront in the basics. For example, follow the proformas provided more closely and use them as a thinking tool. Also I think I would have gone into Kingston a bit more often, especially to discuss the interloan library system. Too much of what I did was by email.
Although I found including the PhD dissertations lowered the quality of the searches and added too much variety in terms of content, I think I would still repeat the process of looking at PhD dissertations initially, as I felt I learned a lot about the wider literature from doing this. Related to this point, I feel like another lesson is that the peer-review publication process is probably quite effective, in that it sifts out many of the more variable papers and you are left with more standardised, higher quality papers. It therefore felt like adding ‘peer review’ as an inclusion criterion was valid. It helped tighten and standardise the SLR, certainly in my case.
150 thread of my research in very short, focused bursts. At the time I had a (new) 4-month old
baby (along with a now 2-year old) as well as a full-time job. The worst times were when I had to leave my tired wife alone to cope with the baby, so I could focus on my SLR, but I then had failed to make any progress (for whatever reason). Dark times!