Getting Started as a Researcher: 
Designing and Conducting Research Studies in Instructional Technology

How do you get started in planning a research study?  The main problem for many beginning researchers is putting the cart before the horse, which in research parlance translates into selecting methods, a design, or worse, a statistical analysis procedure (as in “Won't I need to use analysis of variance to get my dissertation approved?”) before clearly deciding what questions they want to investigate. The usual result of such “backward planning” is a collection of methods that lack a unifying purpose of direction. To help you avoid this error, the following pages will present a seven-part model that will guide you through a logical sequence of planning steps.

1. Selecting a Topic

What general topic area interests you?  Is it the use of computer agents for instruction, serious gaming, the benefits of animation, how computers are used in the classroom, the design of distance education, or some other focus?  Think of a research study as something that you will live with for awhile, perhaps a year and probably more (not too much more, we hope, if you are doing a thesis or dissertation!). A “shot-gun” marriage with a topic area that doesn't interest you, but your advisor imposes or colleague suggests is almost certain to be an unpleasant experience. Research can be fun and involving when you have a natural curiosity about the subject; it can be drudgery when you don't.

2. Identifying the Research Problem

The next step is to determine what problem or concern to investigate within your topic area. Sometimes the problem will be immediately identifiable based on your present knowledge and interests. In other instances, it may be only a general idea that needs to be more carefully considered and sharpened. For example, a few years ago, one of us became interested in children's solving of math word problems. The particular idea was making word problems more interesting to students to increase motivation and learning. The basis for a research problem was thus established, but some library time was needed to sharpen the premises and specific focus. Through a review of the literature, the problem evolved into one concerning whether meaningful learning could be enhanced by using a computer to personalize problems for each student. In selecting your topic, a key concern is the importance of the problem to the field.

Conducting research requires too much time and effort to examine trivial questions that do not expand existing knowledge. Experienced researchers are usually attuned to important topics, based on their knowledge of the literature and current research activities. Novices, however, need to use care when establishing support for their idea from recent research and issues-oriented publications (see Step 3). For experts and novices alike, it is always a good practice to use other researchers as a sounding board for a research focus, before getting too far into the study design (Steps 4 and 5). Regardless of your focus, a thorough literature search is essential here and/or at various other places in the planning process. We turn to this activity next.

3. Conducting a Literature Search

Although there are several approaches to doing a literature review, a few key ideas and techniques can help you improve your efficiency. A starting point for a search is electronic databases such as Psychological Abstracts, or ERIC. To gain the most efficiency in using these indexes, you must start with one or two key terms. You can start with some key terms and see if the search is productive. If not, try another set of terms. Some databases will suggest alternative key words. For example, while searching a database for abstracts recently, we searched the term “instructional technology” did not find any results. The results, however, suggested we try “educational technology” which produced a series of abstracts. As a last resort, you can look through a thesaurus for specific database to find other key words. Articles listed in these indexes are classified according to established lists of terms. Thus, a few minutes spent locating key terms can make your search time more efficient. A second approach is to find a citation in one of the indexes for a key article you have read and note the terms used to classify the article. These terms are your starting point for the literature review and may need to be revised as you make progress.

The Internet (e.g., Google Scholar) and electronic databases have made it much easier to locate a host of articles related to our research topic. The key is to find the correct search terms. You can start by using terms your own key words or those from a thesaurus or “known” article. As you find relevant articles, you can scan the abstract for other terms to help you either broaden or narrow your search for relevant articles.

Applications such as EndNote (www.endnote.com) provide an excellent tool for both searching and compiling bibliographies for your research. You can use these applications to search different indexes, or you can collect the data and import it into the application. Once you have compiled a set of references, you can use the application to help write your paper and create the reference list.

The next step is to retrieve the articles your search has identified. The easiest articles to retrieve are those that list full text with the database entry. If you cannot find the article in your library, we often use a search engine such as Google and search for the title. Often, we find the article posted on one of the author’s web pages. If you are using a paper that was not published in a journal, make sure you include the URL for citing the document (you can find current information on citing Internet resources at www.apa.org/journals/webref.html). If you have access to a database that includes full text articles, you can download it, have it emailed to you, or print the article or paper. As a last resort, you may need to visit the library and retrieve the article from a bound volume or email the author and ask for a copy.

Your initial search will probably identify one or more key articles. One way to expand the search is to read the references cited in those articles. This approach, however, only provides you with older articles. Another option is to search through the Social Sciences Citation Index or the web version, Web of Science, to see if your key articles have been cited by others. This method provides more recent articles on your topic. To broaden your search, a third strategy is to examine indexes of journals in related fields and of related international journals. A final strategy to locating some of the newest papers is search through convention programs such as AECT and AERA for recently presented papers that have not yet been published.
Problem Articles. There may be times when you cannot locate a particular journal or ERIC paper. There are seven strategies for obtaining these elusive papers. First, check with library to see if another library in the area has the journal. Many libraries have such a list and exchange agreements with other libraries. Second, order a copy through inter-library loan. This process may take a few days and possibly cost a few dollars, but it is quite reasonable. Third, write to the author and request a copy. Current addresses are available through association directories such as AECT, AERA, and APA. Most authors are pleased to know that someone is reading their research and may even include additional articles. You can increase the chance of receiving the article if you include a self-addressed-stamped-envelop that makes it simple for the author to place a copy of the paper in your envelop and put in the mail. Fourth, try to order the article through companies such as Carl UnCover or ISI Document Solution. Fifth, ask your librarian to help you locate the journal publisher and then request to purchase the article or a single back issue. Sixth, if you are searching for a paper in ERIC, check Psychological Abstracts and CIJE for a three to five-year period after the paper was presented to see if it was published in a journal. Seventh, search the Internet to determine if the article is available electronically.

4. Stating the Research Questions

We have now reached what is probably the most critical step in the planning process. Once defined, research questions provide the basis for planning all other parts of the study–design, materials, and data analysis. Unfortunately, it is sometimes difficult to convince beginning researchers of this sequence. As stated earlier, a common tendency is to focus much too early on which treatments to compare or what design (e.g., experimental, correlational, or qualitative) to use. These decisions are obviously important, but they cannot be reasonably addressed until you know exactly what you want to find out. Research questions convey those goals.

At this point you have identified a problem focus (Step 2) and are familiar with the literature (Step 3). Based on this background, what specific questions do you want your study to answer? As you think of questions, write them down in a list. Don't worry about their exact phrasing, whether others will like them, or having too many. Before you actually begin the study, there will be many opportunities (perhaps a lot more than you want!) to present your ideas to others and obtain reactions. Chances are that you will also search the literature again to read more about a specific theory or prior study. Thus, this initial list will probably not be the final one, but will serve as a useful starting point for planning how the study will be performed. As an example, Table 1 lists the questions used in the personalization study referred to earlier. In the following pages, we will return to this table to illustrate how research questions directly guide subsequent planning steps.

Note, however, that if you are conducting a qualitative study, you will want to induce (derive) hypotheses from your observations, not deduce the answers to predefined hypotheses . Thus, qualitative research uses an open, flexible, and inductive orientation.

Table 1:  An Example of Research Questions and Procedural Planning for an Actual Study

Questions	Design	Instrumentation	Statistical Analysis
1. Will personalizing contexts improve problem solving relative to using concrete or abstract contexts?	Experimental: Three Groups	a. Achievement tests	One-way Analysis of Variance
2. Will personalizing contexts improve attitudes toward the lesson?	Experimental: Three Groups	b. Attitude survey	One-way Analysis of Variance
3. Are attitudes and problem-solving performance related?	Correlational	a and b	Pearson Correlation
4. What are the logistics of personalizing problems?	Descriptive	c. Experimenter journal	Narrative report (Qualitative)
5. What are teachers’ reactions about the practicality and value of the personalized strategy?	Descriptive	Interview	Narrative report (Qualitative)

5. Determining the Research Design

Your research design will be primarily determined by two factors: (a) what the research questions require (of course!), and (b) what is feasible given the resources or conditions at hand. Space restrictions limit how much can be said here about different design approaches, but, as a brief introduction, we'll note three general categories. Experimental-type designs are used to test hypotheses regarding causation, for example, that a particular instructional strategy leads to better student performance. Making causal inferences requires a high degree of experimental control, such that all conditions for the “treatment” group and the control group are identical except for the particular strategy being tested. A true-experimental design, in which subjects are randomly assigned to treatments, is best able to achieve this control, but is oftentimes impractical. A quasi-experimental design, which uses intact groups (e.g., two existing third grade classes), is a less rigorous design, but generally easier to implement. Two additional design categories are the correlational and descriptive approaches. Correlational studies examine how variables relate to one another (e.g., amount of computer experience and attitudes toward word-processing) rather than whether one causes the other. Descriptive studies depict conditions as they exist in a particular setting (e.g., the number of teachers at different grade levels who use computer-based instruction). With descriptive studies, you may use qualitative data sources (field notes from observations, research journals, interviews), quantitative sources (frequency counts, percentages, descriptive statistics), or both.

Equipped with a basic knowledge of design approaches, the next task is to determine what is needed for each research question. Your study may therefore include several approaches, for example, an experimental part and a descriptive part. In fact, there is growing support among educational researchers for the application of “mixed methods” studies that include both quantitative and qualitative data collection.  To illustrate, the second column of Table 1 lists the designs used to address the research questions in the personalization study. Note that where treatment groups were compared on some dependent (outcome) measure, an experimental design was specified. Question 3, concerned a relationship between variables, naturally required correlation. Questions 4 and 5 were descriptive in nature, concerning the researcher's experiences in implementing the personalization strategy and the teachers' reactions to its instructional use. Keep in mind that as a beginning researcher you do not have to be a research design expert; colleagues (or your advisor and committee) can provide assistance in this area.

6. Determining Methods

Methods of the study include (a) participants (subjects), (b) materials and instrumentation, and (c) procedures. Let’s start with participants. What kind and how many participants does your research design require?  For the present example (Table 1), it was decided that to manipulate the three treatments (Questions 1 and 2), we would need about 60 elementary school students. Two or three teachers would also be needed for the strategy evaluation (Question 5).

Remember, the challenging part of data analysis is selecting appropriate methods and using them in the right ways. Qualitative analysis requires careful and systematic coding and reliability checking; be sure to reference a qualitative research source or seek assistance from a knowledgeable qualitative researcher if you are unfamiliar with these methods. The same recommendation applies to quantitative analysis; Table 2 is provided here to give you a head start.

Next, consider materials and instrumentation. Sometimes the specific resources needed will be obvious. When uncertainties exist, construct a listing on a question-by-question basis. This procedure is illustrated in Table 1 (column 3) for the identification of assessment instruments for the personalization study. Basic to all five-research questions was obviously the need for a computer-based mathematics lesson that would convey the personalization and control strategies. A specific lesson idea and target grade level evolved through discussion with teachers and examination of curriculum materials. Because the lesson would be unique to our study, we were faced with designing our own achievement test (a common situation in educational research) and attitude survey to inquire about the particular materials used. As you plan your own research, a good rule to follow is to try to obtain existing validated instruments where you can; when existing measures provide a poor fit with your research questions, you will be better off

developing your own using the guidelines described in most educational measurement texts.

The last step in planning the methodology is to develop a procedural plan. What logical sequence of steps will provide the information needed to answer the research questions?  What materials will be used or information given to participants at each step?  You may find it helpful to begin by constructing a general outline. Once you become more confident about the basic procedures, the important task will be to describe them in narrative form (as required in a prospectus or proposal). Our best advice:  Write the narrative so that another person could conduct the study for you just by reading it (the supreme wish of every doctoral student). The more precise you are, the better basis you'll have for conducting the study and your committee will have for evaluating it.

7. Identifying Data Analysis Procedures

Some beginning researchers can't wait to identify the statistics they will use, doing this step (unfortunately) as the first rather than last part in the planning process. Others would be most happy if the statistical component somehow magically disappeared. For nearly all quantitative studies in instructional technology, some statistical analysis will be needed. The good news is that if you have gone through the preceding six steps, specifying an analysis plan is very straightforward and easy. If statistics aren't your strength, no need to worry. Show someone who knows statistics (usually there's such a person on your committee or ask a colleague) a clear set of research questions and design descriptions, and he/she will be able to readily identify what analyses are needed. Your task is to learn from a practitioner's standpoint why those analyses are appropriate, what types of information they provide, and how to interpret those results. Interested readers can examine in Table 1 (last column) a listing of the analyses selected for the personalization study. Note that each is matched to a specific research question (getting the picture?); also note that mixture of quantitative and qualitative analysis procedures was used.

If you are conducting a qualitative study, a common misconception is that data analysis is easier or less complicated than is the case for quantitative research.  In fact, many would argue that the opposite is true!  Qualitative analysis requires careful organization and synthesis of notes and transcripts to derive patterns and ideas to form the basis for conclusions.  Readers of qualitative studies need to be convinced that the data analysis was done thoroughly, systematically, and reliably.  Numerous textbooks and other reference materials, along with computer software, are available to help the qualitative researcher achieve these standards.

A Final Thought

As you design your study, you need to consider the procedures for seeking human subjects approval. Universities typically require that researchers complete training and submit a proposal for approval prior to collecting data. Universities might require you to complete an online training program (e.g, http://cme.cancer.gov/clinicaltrials/learning/ humanparticipant-protections.asp) or they might have their own training related to collecting data from human subjects. You may also be required to submit an application or proposal to your human subjects or Institutional Review Board (IRB) before collecting data. Since approval is seldom received “overnight”, you should plan for this approval process early to maintain your timeline.

 In summary, we again caution you about one of the most common mistakes made in planning research. It is thinking about treatments first (as in, “I think it would be good to compare a higher-order questioning strategy vs. an on-line coaching strategy using animated or static images in diagrams. That would even get us a 2 x 2 multifactor design!”) to the likely sacrifice of any meaningful research questions. The result is often a collage of uninteresting, disconnected findings that fail to advance either theory or practice. Using a systematic planning approach, such as the present seven-step model, will help you to identify and investigate problems that interest you and are currently of interest in the field. More importantly, a systematic approach is likely to result in a higher quality study and a much more positive experience for you as a new researcher. And, if you begin now, you may soon be able to take advantage of our fourth chapter on how to write research reports for publication in professional journals such as ETR&D.