The Trial Protocol Tool: the PRACTIHC software tool that supported the writing of protocols for pragmatic randomized controlled trials

Article Outline

• Abstract
• 1. Background
• 2. Tool specification
• 3. Content development
  • 3.1. What should be in a good protocol?
  • 3.2. Other sections
• 4. Translation of content into Spanish
• 5. Resources
• 6. Technical information
• 7. Comment
• 8. Conclusion
• 9. Availability and requirements
• Acknowledgment
• Appendix 1. The PRACTIHC management group
• Appendix 2. The 12 full protocols in the Protocol Library
• References
• Copyright

Abstract 

Objective

To develop a tool that would make it easier for researchers, especially those in low- and middle-income countries, to write research protocols for pragmatic randomized controlled trials.

Study Design and Setting

A series of focus groups was held at the inaugural meeting of the Pragmatic RAndomized Controlled Trials in Health Care (PRACTIHC) project in 2001 to develop a desired specification for the Trial Protocol Tool. A working group of five individuals from the PRACTIHC group was formed to develop content for the tool.

Results

The Trial Protocol Tool was developed in English and Spanish as a Microsoft Windows HTML help system. A Web-based version is also available. This main body of the tool provides information, advice, and resources about the major headings that should be part of every research protocol. Illustrative examples are used throughout and are taken directly from the tool's protocol library. Additional resources include checklists, programs (e.g., a sample size calculator), and example documents (e.g., patient information leaflets).

Conclusion

The Trial Protocol Tool packages all the key requirements for the development of a research protocol into one resource. We believe that the use of the tool will help researchers to design effective trials and to write high-quality protocols.

Keywords: Randomized controlled trials, Pragmatic, Study protocols, Software, Protocol resources, Low- and middle-income countries

Back to Article Outline

1. Background 

The randomized controlled trial (RCT) is the widely acknowledged design of choice for evaluating the effectiveness of health care interventions [1], [2]. Pragmatic RCTs are evaluations of health care effectiveness carried out under real world conditions, in typical settings and on typical users. Effectiveness of the intervention is usually assessed by its impact on simple outcomes of importance to users of the intervention such as death, major disability, cost, and quality of life. Pragmatic trials are often more relevant to policymakers than efficacy trials because they generally use the same criteria of effectiveness as those used by policymakers—namely, user perceptions, important and visible outcomes, usual health service planning entities, and typical service limitations.

The first step in running a high-quality pragmatic RCT is to produce a good research protocol. This is true both for obtaining funding to run the trial and for ensuring that the trial itself will produce meaningful results. Despite the existence of guidelines and individual tools to help with this, these are spread across hundreds of journals, Web sites, books, conference proceedings, and software tools. Access to this dispersed information can be a problem, especially in low- and middle-income countries [3]. Moreover, inexperienced researchers may not appreciate the full range of issues that need to be addressed in a protocol and may, therefore, fail to look for resources that address these issues. In this paper, we report the development of a software tool (known as the Trial Protocol Tool), which packages this information in a single, easily accessible, and practical software tool. A screenshot of the current version of the Trial Protocol Tool is shown in Fig. 1.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 1 

  A sample screenshot from the Trial Protocol Tool.

The Trial Protocol Tool was developed as an output of a European Commission–funded partnership between European Union, African, and Latin American trialists known as the Pragmatic RAndomized Controlled Trials in Health Care (PRACTIHC) project (www.practihc.org). The aim of PRACTIHC was to promote the global science of pragmatic RCTs with special emphasis on trials conducted in low- and middle-income countries. PRACTIHC partners based in Africa and Latin America considered the lack of a single tool to support the production of high-quality protocols to be a significant barrier to meeting this aim. Given the focus of PRACTIHC, the examples included in the Trial Protocol Tool are primarily from trials mounted in low- and middle-income countries. Further information on the wider PRACTIHC project can be found on the project Web site.

Back to Article Outline

2. Tool specification 

A series of focus groups was held at the inaugural meeting of the PRACTIHC project (which was held in Cape Town, South Africa, in 2001) to develop a desired specification for the Trial Protocol Tool. Attendees at the meeting included experts in trial design, clinical researchers, and trial managers. The focus groups were asked to prioritize (a) the information they would like the tool to contain, (b) the resources they would like to be available within the tool, and (c) any desired design features. In addition to these functional specifications, attendees were also asked to indicate any information technology constraints, which might be faced in their individual clinical settings given that the tool would be primarily used in low- or middle-income countries. Discussion of the results of the focus groups was held in a plenary session at the end of the meeting, and the following consensus requirements were identified:

Back to Article Outline

3. Content development 

A working group of five individuals from the PRACTIHC group was formed to develop the Trial Protocol Tool content (E.A., M.C., K.McC., C.R., S.T.). These five individuals represented a range of expertise including trial design, trial management, statistics, Information Technology, and clinical application. All had significant experience in the design and conduct of RCTs and the development of successful trial protocols.

There are six main sections in the Trial Protocol Tool:

Fig. 2 shows screenshots of the opening pages of these six sections.

• 
  • View Large Image
  • Download to PowerPoint

• Fig. 2 

  Screenshots from the six main sections in the Trial Protocol Tool.

3.1. What should be in a good protocol? 

This is the main body of the Trial Protocol Tool, and it lists the major headings that should be part of every research protocol.

An Internet search was initially undertaken to inform the development of the structure of a “good protocol.” A number of protocol “checklists” were identified in the Internet search and reviewed within the working group (examples of which can be found in the reference section) [4], [5], [6]. Following discussions, a consensus structure was agreed upon. This was passed to the PRACTIHC management group for wider review and final approval. The PRACTIHC management group consists of experts from 10 countries all experienced in the design and conduct of RCTs (see Appendix 1). The section headings for “What should be in a good protocol?” are shown in Table 1. Each heading generally contains several subheadings. The “Overview of RCTs and trial design” section, for example, has six subheadings: Parallel trial, Cluster randomised controlled trial, Multiple arm trial, Factorial clinical trial, Crossover design, and Equivalence design.

Table 1. Section headings for “What should be in a good protocol?”

Each section was developed to a common structure:

When drafting the “introduction” and “things to consider” sections, the aim was for the text to be instructional as well as informative. For example, when describing the range of possible trial designs, care was taken to not only outline the features of each design, but to guide the user as to when the use of each design might be appropriate. For example, when describing a crossover design, both the characteristics of the crossover design and the situations where its use might be appropriate are outlined:

“In a crossover design, each participant is randomized to a sequence of two or more treatments…Crossover trials produce within participant comparisons, whereas parallel designs produce between participant comparisons…Crossover trials can be used to investigate chronic conditions, such as asthma, where the objective is to investigate the participants' short term response to therapy. The condition must also be stable, so that the circumstances at the beginning of each period are more likely to be the same. Clearly, not all interventions can be studied in crossover designs. For example, comparing surgical procedures or evaluating long term outcomes such as 5 year survival where it is impossible to crossover to another intervention…”

Illustrative examples are taken directly from “real-life” research protocols (which are available in full in a protocol library reference section). The user can therefore see how other researchers addressed this part of the protocol and then, if desired, he or she can go to the full protocol in the Protocol Library. The library currently contains 12 full protocols of which five describe trials done in low- and middle-income countries, and three of the protocols are also available in Spanish (see Appendix 2). Additional resources include checklists, images, programs, example documents (e.g., patient information leaflets), guidelines, and links to Web sites. Resources that are part of the Trial Protocol Tool are differentiated from Web resources by means of an icon before the resource name.

A lead individual from the working group was identified for each section of the protocol. The lead person was determined primarily by the content under review (e.g., statistical analysis was led by the statistician, trial management was led by the trial manager). This lead person took responsibility for preparing the first draft of the content for that section. (For each section, an Internet search was also undertaken [by K.McC.] to identify any relevant literature and/or resources to inform the development of the content. Reference lists and bibliographies of key documents were also searched. This draft was then circulated around the larger working group for comment and revision. This process was repeated until the working group had no further comments. Once the content was “finalized” by the working group, each section was passed to an editorial subcommittee (made up of three researchers in Aberdeen) who assessed all sections for style and consistency. Following this, all sections were passed to the PRACTIHC management group for wider review and final approval.

3.2. Other sections 

The remaining four sections (Teaching resources, Useful documents, Web resources, and Glossary) contain text and links that support the information given in the Protocol checklist. The teaching resources are a series of PowerPoint presentations that can be used for teaching or self-study. The Glossary is based on the glossary in the Cochrane Collaboration Handbook and is used with permission.

Back to Article Outline

4. Translation of content into Spanish 

Following final approval, each section of the “What should be in a good protocol?” component was sent to the PRACTIHC partner in Argentina (E.A.) for translation into Spanish. The translation was done by a linguist using Trados Workbench (a translation software package) and TagEditor (a translation tool interfacing with Trados Workbench). After editing, linguistic and stylistic review, the translation was done by a second linguist to check for completeness, accuracy, style, grammar, terminology, and consistency. Translations were finally checked by E.A.

Back to Article Outline

5. Resources 

A sample size calculator incorporating both individual and cluster randomized trials was developed for inclusion in the Trial Protocol Tool. It uses standard formulas for sample size calculations [7] and provides sample size calculations for both the comparison of proportions and the comparison of means. A basic Gantt chart builder was also developed. This allows the different stages of the trial to be presented visually on a monthly timeline from the proposed trial start date. Both tools were developed using Visual Basic v6.

Additionally, we included as resources a wide range of checklists, leaflets, brochures, and articles. In particular, we included checklists and text written by David Sackett on RCTs for “Clinical epidemiology: how to do clinical practice research” [8]. All extra resources are included with permission.

Back to Article Outline

6. Technical information 

The Trial Protocol Tool was developed as a Microsoft Windows HTML help system, the standard help system used with Windows 95 and above. This choice gives the tool a familiar Windows interface, does not require an Internet connection to run, and provides a range of off-the-shelf functionality. The software for running Windows HTML help, including Internet Explorer upon which it relies, is part of the Windows operating system. This means that the size of the Trial Protocol Tool is decided by its content and not the delivery system. The basic HTML help development tool, Microsoft HTML Help Workshop, is available free from Microsoft's Web site.

For the content, HTML versions of Word documents were created using Mozilla 1.4 for Mac OS X, although any HTML editor could be used. The HTML documents were then linked to the help system in HTML Help Workshop. The HTML documents themselves can contain links to the full spectrum of electronic document formats and can, therefore, include not only textual information but also graphics, documents created by other programs, sound, video, links to Web sites, and other programs. Once the HTML documents have been added or updated, the collection is compiled into a single file (a .chm file). This file can then be distributed by email, from a Web site, or on CD.

If an HTML document contains a link to a pdf or Word document, for example, the document is included in the compiled .chm file. Although convenient, this has a number of disadvantages; in particular, it quickly leads to a large .chm file. To avoid this, such documents are not embedded but kept as external files in the same folder as the compiled .chm file. When the user clicks on the link the file will open in the program to which it is associated. For example, clicking a link to a .pdf file will launch an instance of Acrobat Reader. However, if a user does not have the appropriate program installed, he or she will not be able to open the file. To minimize this problem, the external files within the Trial Protocol Tool are either programs themselves or use programs that are free such as Acrobat Reader.

The current version (5/18/06) of the tool comprises a compiled .chm file (520kB), an external library of protocol examples (2.6Mb), a set of teaching resources (3.7Mb), and a collection of other supporting external material (8.6Mb). The .chm file is usable without the external files and meets the PRACTIHC requirement for small size. The tool is, however, much more useful with the external files. The combined size of these resources is large, although compression and splitting the download into smaller chunks will make things more manageable. We have also developed a Web-based version using chm2web distributed by A!K Research Labs (http://chm2web.aklabs.com/) that is platform independent and works well with only a modest connection speed.

Back to Article Outline

7. Comment 

The first step in running a high-quality trial is to produce a good research protocol. Despite the existence of guidelines and individual tools to help with this, these are spread across many diverse locations and different media. The Trial Protocol Tool presented in this paper packages this information into a single, easily accessible, and practical resource. It uses a familiar Windows or Web-based interface, is available in English and Spanish, and contains the full spectrum of electronic document formats.

The Trial Protocol Tool has been evaluated twice by the PRACTIHC group and was used as the core component of a training course in RCTs held in South Africa in April 2004 and 2005. The feedback from the evaluations and the training courses was overwhelmingly positive, although a number of small modifications were proposed. The tool has been modified to take account of this feedback and we expect this development to continue albeit at a reduced pace now that the PRACTIHC project has formally finished.

The PRACTIHC project has a number of other outputs that have been operationalized as practical tools within the Trial Protocol Tool. One important example is the training material on RCTs for teachers and students. This would allow a student to read some lecture material on sample size calculations, then look at the examples within the Trial Protocol Tool before using the calculator to do his or her own sample size calculation. A trial simulator that can be used to explore aspects of the design, conduct, and analysis of RCTs is another example. The intention is to provide a tool that will help people to design effective protocols without the need for them to do huge amounts of independent work.

The current Trial Protocol Tool meets most, but not all, of the PRACTIHC group's original requirements for a protocol tool. Although the core tool (the .chm file) is small, the external resources are not. Separating these into a series of smaller chunks would make it easier for individuals with slow modem connections to obtain them, although this will not be a satisfactory solution for all potential users. Other options we are exploring include breaking down the external resources into individual documents (e.g., a single protocol) and distributing the Trial Protocol Tool on CD in low- and middle-income countries. We have also developed a Web-based version, which makes the Trial Protocol Tool a platform-independent tool that can be viewed in any modern Web browser.

The Trial Protocol Tool is now being promoted through the PRACTIHC network, conferences, and local meetings and is available for download from the PRACTIHC Web site. The Trial Protocol Tool is also discussed with medical and other health science students and with health care professionals via the teaching and training commitments of members of the PRACTIHC partnership. We encourage researchers to use the tool in their own teaching and research work.

Back to Article Outline

8. Conclusion 

This paper presents the Trial Protocol Tool, which packages all the key requirements for the development of a high-quality research protocol into one resource. We believe that the use of this centralized resource will help researchers, particularly in low- and middle-income countries, to design effective trials and to write high-quality protocols.

Back to Article Outline

9. Availability and requirements 

PRACTIHC is happy to have its products used at no charge in free educational programs. However, when tuition is charged, PRACTIHC's royalty fee is 10% of gross receipts. If you are planning to use the Trial Protocol Tool within a for-profit organization in any way, please contact us to discuss a donation to our work.

Back to Article Outline

Acknowledgment 

The development of the Trial Protocol Tool was supported by the European Commission's 5th Framework INCO programme, contract ICA4-CT-2001-10019. The Health Services Research Unit, University of Aberdeen, UK, is funded by the Chief Scientist Office of the Scottish Executive Health Department. The views expressed are those of the authors and not necessarily those of the funding body.

Back to Article Outline

Appendix 1. The PRACTIHC management group 

Back to Article Outline

Appendix 2. The 12 full protocols in the Protocol Library 

Back to Article Outline

References 

1. Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. BMJ. 1996;313:570–571
   • View In Article
   • MEDLINE
   • CrossRef
2. Pocock SJ. Clinical trials: a practical approach. Chichester: Wiley; 1996;
   • View In Article
3. Smith R. Closing the digital divide. BMJ. 2003;326:238
   • View In Article
   • CrossRef
4. ICH Harmonised Tripartite Guideline: Guideline for Good Clinical Practice E6(R1) (Current Step 4 Version dated June 10, 1996). Available at http://www.ich.org/cache/compo/276-254-1.html. Accessed March 2003
   • View In Article
5. HIV Prevention Trials Network . HPTN protocol template. Available at http://www.hptn.org/network_information/network_templates.htm. Accessed March 2003
   • View In Article
6. Centers for Disease Control and Prevention . Developing a protocol: a guide for CDC investigators. Available at http://www.cdc.gov/niosh/nas/mining/pdfs/Protocol%Checklist.pdf. Accessed March 2003
   • View In Article
7. Campbell MJ, Julious SA, Altman DG. Estimating sample sizes for binary, ordered categorical and continuous outcomes in two group comparisons. BMJ. 1995;311:1145–1148
   • View In Article
   • MEDLINE
   • CrossRef
8. In:  Haynes RB,  Sackett DL,  Guyatt GH,  Tugwell P editor. Clinical epidemiology: how to do clinical practice research. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2005;
   • View In Article