Friday, November 5, 2010

Chapter 3
Bullying at work

The concept of dignity at work
                People have a right to be treated well and with respect in the workplace. Many, though, are not in this position. They wake up, often from fitful sleep, and go to work with fear and dread that this is going to be another day during which they are going to be treated badly.
                My sense is that by the time you have finished reading this chapter, each one of you will recognise someone either from your past or in your present who is bully.  Although this book is about bullying in the workplace it does of course not mean that these behaviours happen only in the workplace. Children bully each other in school, playgrounds or classrooms, friends bully each other people in close interpersonal relationships bully each other. That, however, is for another book!
                The key idea to bear in mind when reading this book is that bullying is not about intention, it is about impact. In other words, it is about the effects that the behaviours of one or more people have on other people.

What is bullying?
The working definition I use for bullying is the one used by the Manufacturing Science and Finance (MSF) Union:
                Persistent, offensive, abusive, intimidating, malicious or insulting behaviour, abuse of power or unfair penal sanctions, which make the recipient feel upset , threatened, humiliated or vulnerable which undermines their self-confidence and which may cause them to suffer stress.

                According to the Industrial Society, “Harassment can be defined as any improper, offensive and humiliating behaviour, practices or conduct, which may threaten a person’s job security, create an intimidating and unwelcoming and stressful work environment, or cause personal offense or injury.”
                People who are being bullied often do not recognise that they are being bullied. They often think that they are only one who is constantly getting it wrong and that there is something wrong with them. They often feel ashamed and that they cannot cope. Moreover, they fear that if they tell anyone they will not be believed but seen as weak and inadequate.

Who Bullies?
Most of us are capable of bullying behaviours. Thankfully, most of us prevent ourselves from acting out in this way. How many of us can say that in certain circumstances, such as when we are in a hurry and the person behind the counter is being slow or incompetent, we haven’t wanted to say something but, instead, kept quiet. It is not bullying to feel frustrated, angry or upset at another person’s behaviour; but it is bullying behaviour when we act on our feelings in an inappropriate way.
              The self righteous bully is someone who cannot accept that they could possibly be in the wrong. They are totally devoid of self awareness and neither knows nor cares about the impact of their behaviours on others. These are people professional development coaches have difficulty working with to effect behaviour change. They are always right and others are always wrong. One woman whom I worked with refused to accept any responsibility for the effect that her behaviour was having on others; she wrote them off as wimps unable to take fair criticism. Another said she could work with these people again as she did not ‘bear grudges’. Yet another said, ‘Of course i didn’t bully him. I merely said performance.
                It would be incorrect to think that bullying takes place only downwards in the hierarchy. There are many examples of bullying by junior members of a team. For example, many lecturers have complained of harassment and bullying by their students. If you know what is good for you, you will rethink the grade you gave me for my last assignment. My father has known the Chancellor for years and pays for one of the college trophies.
                Another example was that of an organisation in which it appeared that the whole office was living in fear of upsetting the receptionist who had worked here longer than anyone else. She acted as unofficial time keeper, making notes of hours worked, sickness absences and holidays even though this was not part of her job. Her own attendance and holiday details were of course ex-directory.
                Bullying is not necessarily an act of one person against other. A team of people can bully one or two individuals, for instance where a team resents former colleague’s promotion or where a new manager comes in and wants to make changes. One freelance female computer programmer was brought in to manage a team of men. They would not accept instructions from her and made constant personal remarks about her – all within her hearing. This bullying caused her to leave, especially as she felt the company would not be interested in dealing with it because she was ‘only’ a contractor.
                When teams of people get together to bully a new manager, the underlying reason can be that one member of the team, who is the psychological leader, wants the manager’s job and encourages the team support out of fear rather than loyalty.
                 One of the most common themes of bullying behaviour can be observed when someone in a management position bullies their second-in-command. They do things such as arrange important meeting in the employee’s absence and then blame them for missing agenda items or for non-attendance. I have come across this many times, especially after a manager gets back from vacation to find that his job has been done perfectly  in his absence. Adams (1992) describes this as work envy, when someone in a more junior position can do their job as well as, or even better than, they themselves can do it.
                People with some power who take more are usually the people who bully others. Teachers can bully children, as can other adults who are in a position of power, as we saw in the examples of young teenage models who were bullied by photographers (Chapter 2).
                Sometimes, when sexual relationships creep in, the balance of power can change and people become bullies. An example of this was the secretary who became the mistress of the chairman of the company and suddenly started to bully colleagues who had been her friends.
                Bullying seems not be associated with the gender. Field (2001) found the split between men and women who bully roughly 50/50.



The bully
                In my experience there are two main types of bully. The first type is the self-righteous bully who comes from an ‘I’m OK – You’re Not OK’ position. These people are vulnerable to the criticism of others, have low self-esteem and project this feeling of inadequacy onto other people. Their behaviour is destructive to others and they appear to take great pleasure in seeing their victims suffer from being afraid of them. Organisations have a duty of care to protect their staff from these people.
                The other main kind of bully is someone who functions from an ‘I’m Not OK – You’re Not OK’ position. These people are afraid, vulnerable and behave in an appropriate way in an attempt to protect themselves from what they perceive as danger. As a result, these bullies see themselves as victims and actually feel that it is they who are being persecuted. They don’t trust people and are actually scared of being found out for their inadequacies. These people need training. They are easier to deal with and are usually willing to take feedback in the form of coaching.
                A third type is the person who behaves in the way that they believe is expected, or are told is expected, of them. They may be demonstrating behaviours that the previous managers modelled or they may have had their own job security threatened in the past for being ‘too soft’ on their team. This is another group for whom coaching works well and they can be trained in more appropriate ways of managing and behaving.
                Bullies have favourites – ‘flavours of the month’. In the bully’s eyes, the favoured person can do no wrong and the bully places them on a pedestal, elevating them above everyone else. This can make the rest of the team resentful, especially as they tend to be blamed for any mistakes that the favourite might take. The situation can shift as quickly as it began when, suddenly, the bull’s ‘pet’ is replaced by someone else. This fall from grace can be followed by vicious nit-picking by bully until the pet leaves, confused and unpopular through little fault of their own. Many people have said that they are afraid of ever becoming the pet, as it usually means the next stage is to be the victim. This thought can be so unbearable that people prefer to leave the company.
                Bullies have their special victims. They frequently pick on a member of the team who is not likely to answer back but is likely to blame themselves and not want make a fuss by complaining. Other members of the team watch these behaviours going on and do not say anything for fear of being the next victim. Some people are more vulnerable to being a victim of bullying: often the immediate assistants who are contractually obliged to work with the bully. Part-time workers or juniors often feel especially defenceless and, because they have less recourse to action, don’t say anything. People with mental or physical health problems and anyone from an ethnic minority background are also vulnerable. The bully has a habit of finding out from people what their weaknesses are and then using this information as ammunition. The victim invariably feels confused and they say ‘I don’t understand it, she was so nice to me at first – really warm and friendly.’
                Bullies can also use threatening behaviours, either by threatening someone physically (‘Make sure you don’t leave this place alone or someone will be waiting for you’) or in terms of blackmail (‘What would your wife say if she knew about your “friendship” with X?’)
                Bullies like to bring in their own team and frequently want to weed out anyone that they personally did not employ. The established staffs are excluded from social events, kept out of information loops or, in some cases, actually told ‘There is no longer any room for you here and it would be in your own interest to look for another job.’
                Bullies generally have two distinct sides. Cruel and calculating in the office, they are also entertaining company and great fun to be around at social occasions. Frequently in my work with victims they have said to me ‘He’s not always like this and is so nice after hours when we go out socially.’ One set of behaviours doesn’t outweigh the other; it just serves to confuse people.
                Bullies are often intelligent people who are clever at manipulating and misrepresenting facts. They can twist what people say and confused them until they are too upset to even try and stand up for themselves. They are not good role models, they tend to be the first to criticise the behaviours of others, even while they are guilty of the same misconduct.
                Bullies ‘rubbish’ their staff to other people: ‘I have a real bunch of stupid ****s in my team.’ No wonder we don’t make sales target with that load of useless ****s – they haven’t a clue.’
                Bullies tend not to encourage discussion and more likely to adopt a dictatorial approach. One man, who was the sole owner of his empire, went round the staff desks and destroyed any bits of paper that had anything personal written on them, such as a telephone number for the dentist. When challenged, he said it was his right as he owned the paper and no one was allowed to even think about their life outside of the company when he was paying their wages.
                People who bully often invite themselves to meeting or conferences outside of their area of responsibility. Their colleagues are usually too scared to actually challenge them – particularly if the bully holds a senior position in the organisation. Another favourite tactic is to take over the running of the meeting and change the agenda to suit their purposes – or even to announce that they think that the meeting is a waste of time, and then leave, causing everyone else present to feel deflated, incompetent and upset.
                Bullies pick on the shy and vulnerable. One woman returned from sick leave after undergoing treatment for cancer to find her chair had been swapped for a rickety old one, her desk had been moved into a dark corner away from the window and her personal things removed. She also found that a new manager had in instigated these changes and he made it clear to her that there was no longer room for anyone of her age, and with her health record, in her new department.
                Parody can be a subtle form of bullying. It can be so common place that its inappropriateness is not fully acknowledged. One large team of public-sector Asian women workers were constantly upset when their colleagues mimicked their accents over the telephone. What the women found even more difficult to understand was why these people, who lived in the far north of the country, were mimicking them when they had their own very distinctive accents.
                Bullies frequently behave like children. They can be snide, spiteful and vindictive when they do not get their own way. They have temper tantrums, stamp their feet and throw things around. An example of this is a picture-framer who, whenever he loses something, turns and makes nasty comment to his assistant. On one occasion he threatened to splash her with paint when he could not remember where he had put his phone. On another occasion he threatened to hit her with the broom because she had not swept the front of the shop.
                Adams (1992) talks about bullies suffering from envy. In this context she means work envy, where the bully fears that a colleague could do their job as well, or even better. Typically, in cases where they feel threatened, because the bully has some power they can resort to official actions such as unfair appraisals. Or, because the outrank their colleagues, they can cut across or ignore them at meetings, steal their ideas or – as in one example told to me – simply interrupt a colleague’s presentation and take over.
                These actions, together with constant criticism and never any praise (even when tight deadlines are met) are a source of deliberate bullying designed to demotivate and wear down the target individuals.

Bullying behaviours
                Bullying behaviours are rarely isolated incidents. Furthermore, the first incident usually signals the beginning of a soon-to-be established pattern. While single incidents may be seen as too subtle to be considered bullying, added together and viewed as a series, the pattern emerges.
                Galen (1991) reports on an IBM employee who was forced to take early retirement of tarnish an unblemished career with an unsatisfactory job rating. The court found that he had indeed been forced out of his job and decided in his favour. He was awarded a sizeable settlement.
                There are two types of bullying behaviours: gross and obvious or a more subtie variety.

Gross behaviours
                First I will give you examples of gross behaviours; in fact, I could write an entire book describing behaviour so bad that the reader might be forgiven for thinking it the basis for a fictitious horror movie.

Undermining behaviour
A new managing director took over the overall running of charity. In position as the officer manager was a highly competent, popular and efficient middle-aged woman. From day one he decided that he no longer wanted her or anyone to answer the telephone using their own names, only ‘Mr X’s office’. He listened in to all conversations and criticised any gesture of warmth or pleasantries. The office manager was no longer to sign letters I her own name. He constantly undermined any authority she previously held, he was rude to her publicly and, privately, asked her when she was leaving, since he could find someone better and cheaper. He managed to reduce all of the team to tears at various times and eventually succeeded with the office manager. Clearly delighted with himself, he said, ‘Oh good! I was wondering how long this would take.’
                This woman, like many others in her situation, eventually felt that she really was no longer competent and began to believe that she was making many errors where before she had run the office smoothly and efficiently. When she finally went off sick, the managing director asked her colleagues if she had any family- or age-related problems.

Changing deadlines
                I have frequently been given examples of a manager insisting that a piece of work be completed by an unrealistic date. The individual or team has had to work weekends and late into the night only to find that on the appointed date the manager is not there, has taken the day off, or says they haven’t the time to read it until the following week.





 Using personal information
                The senior member of a team publicly said to another, more junior woman, ‘No wonder your boyfriend left you when you dress like that.’ In another example of using personal information, a senior manager said to a woman, ‘It was only fair we promoted a man; after all, your husband earns a good salary and you don’t need the money.’ On another occasion a male manager said to a senior female member of staff whom he had been bullying for some time: ‘I hear that your husband has retired. Don’t you want to retire and spend more time with him?’ She felt angry and humiliated in front of her staff and colleagues.
                Probably the most vindictive example of using personal information was the woman who said to her male assistant, ‘Your baby being born with a disability is an act of God. It is punishment for your having had affairs.’

Being frozen out
                One woman had her work taken away from her and the rest of the team was instructed not to speak to her. Being isolated is a powerful way in which people can be bullied. As social animals, we have a need to feel that we belong; not being invited to a social occasion, or even to lunch when everyone else in the office has been, can leave people feeling stressed, uncomfortable and isolated.

Abusive behaviour
                Bullies can show great creativity in their use of techniques. I heard an example of this when a highly experienced secretary came into my office and asked me to listen to a tape from her dictation machine. What I heard is her male boss,  a man well respected in his profession, screaming insults at her and barking instructions between telling her how stupid she is.
                If we actually think about it, this example is a form of torture; the audio headphones go directly into the person’s ears and there is no escape. One of the boss’s favourite taunt was: ‘Who else would employ you at your age?’ He had picked on this woman’s weakness, as there was a grain of truth in what he said. It is difficult for women in their late fifties to find employment. However, it does give me great pleasure to be able to write that this woman was headhunted by a competitor company and that she remains in that position, working happily in a community where she is appreciated and respected.
                Bullying rarely ends with over public remarks or outburst. The worst bullying usually goes on behind closed doors and without witnesses. I have many examples of men and women bullies behaving at their worst after most of the people in the office have gone home. An example of this was a female boss who called a junior male member of the team into her office just as he was leaving the office with his colleagues. He went in and sat down, as he was requested to do. She locked the door and stood next to it, which increased his feelings of vulnerability and, knowing that he was unable to escape, yelled and screamed insults at him about being a member of her team, as a human being and, lastly, as a man.

Sarcasm
                Sarcasm is a bullying technique which is often masked by remarks such as ‘People round here are too sensitive.’ For example, a person goes into their office not feeling at their best and returning too early after being off sick. They find a scribbled note on the sickness absence book next to her name saying  ‘Taking the p**s’ and someone in the office saying ‘You’re looking good, been on holiday?’
One woman’s treatment was described in Greenhalgh’s (1991) newspaper article. Her bullying manager e-mailed other staff members, citing her victim as an example of how not to do things.

So-called humour
                People can be bullied in many ways under the guise of ‘humour’. Jokes distributed by e-mail, cartoons on notice boards or unsuitable literature sent to people’s homes are some example. An even more extreme case was that of a funeral wreath being delivered to a person’s home with their name on it.
                Practical jokes or inappropriate humour cannot be passed off as ‘all in good fun; we were only having a laugh’ or ‘it’s him – he can’t take a joke.’ Humour at someone else’s expense is not funny – it’s bullying. In this category I include the initiation ceremonies that some cultures collude with. These are practical jokes or pranks carried out on junior or new members of staff. Examples of this can range from sending someone to another department for a long wait’ to young men being ‘debagged’ (their trousers removed) by a group of howling and laughing women. These behaviours are bullying and can leave the victim feeling distresses and humiliated. In one bank a very senior bank clerk thought it was amusing to stamp ‘turnover’ on the bare legs of junior female members of staff and encouraged the junior male employees to join in a ‘frivolity’. Offensive songs, graffiti and cartoons would also come under this category of bullying.

Ostensible accidents
                ‘Accidents’ can also be a thinly veiled excuse for bullying – e.g., bumping into someone carrying a cup or of coffee or a tray in the canteen so that they drop or spill it and end up feeling foolish. This kind of behaviour is more likely to occur in front of an audience of other team members of the team who are involved in the inappropriate treatment of another. On another occasion, one cleaning crew left a cloth that had been saturated with a toxic substance for the next shift of cleaners to move. This was the last of a long line of incidents in which a particular woman had been hurt in ‘accidents’ orchestrated by her management and colleagues.

Electronic harassment
                The age of technology that has made communication easier has also made it easier to bully and harass people. It seems to be a common misconception that e-mail communications do not have the same value (or effect) as a letter or memo. I have been shown many examples of inappropriate and even abusive e-mail.
                Voice mails and text messages can also be used in a similar way to bully and harass people. This can include the targeting of staff members who live alone to find threatening and anonymous messages, or heavy breathing, on their telephone-answering machines.

Sexual harassment
                Sexual harassment by e-mail has become popular. One woman had nearly two hundred e-mails from the same man when she went to her desk one Monday morning. They all related to asking her out and making personal remarks about her body and status.
                Harassment under the guise o f love has been brought to my attention on several occasions in the last year. Both men and women become involved with someone they work with. It usually starts as a friendship – until one falls in love with the other, who then feels harassed by the constant attention, particularly when the person responsible insists that it is out of concern, is still harassment.

Invasive behaviour
                Invading a person’s personal space and possessions is bullying. This can include standing too close to them when it is obvious they are not comfortable, standing while they are sitting, yelling t them and going into other people’s lockers, desks or bags and interfering with the contents. I have had examples of people finding out the computer passwords of their victims and going into their computers and deleting entire files – difficult to prove.
                Some bullying behaviours are so extreme that the people on the receiving end are so deeply shocked that they do not expect to be believed. Two high-powered professional women, obviously rivals, encountered each other in corridor. Ms A turned to Ms B and threatened that if she (Ms B) were to apply for a particular promotion, she would have her ‘to answer to’.

Discrediting professional reputation
                Casting aspersions on someone’s professionalism can also be a bullying tactic especially in the care professions where it is often used to encourage people to work unrealistic hours, well beyond the call of duty or what should reasonably be expected from them. For example, ‘If you cared about your patients you wouldn’t hesitate to work additional shifts.’

More subtle behaviours
                The subtler forms of bullying are by their nature more ambiguous. Here, especially, a single incident is difficult to characterise as a definite bullying behaviour and, often, a few incidents taken together indicate a pattern suggestive of bullying.

Bogus scheduling
                Arranging meetings and leaving a certain person off the invitation list, only to complain at the meeting ‘I see that X hasn’t bothered to attend.’ is one example.
                Others such as setting unrealistic deadlines, turning down someone’s request for leave or cancelling their leave at the last minute without good reason, removing responsibility from someone without grounds, ignoring them or speaking to them only through a third party or by e-mail – even if they sit on opposite sides of the room – fall into this category.
                Holiday rotas, work rotas and overtime rotas can be used to bully and manipulate people. Insisting that people cancel their pre-booked holidays and weekends away when there is not a crisis are further examples of bullying. So is giving the same person all the unsociable shift hours while everyone else on the team gets the more sought-after slots.
                Another example consists of refusing to allow some members of a team to attend training courses on the grounds of their being too busy when other members of the team have attended them.
                Making sure that work is given to an individual at a specific time when it is common knowledge that that person leaves on time on a particular day to attend to an outside activity is a definite, albeit subtle, way of bullying a subordinate.

Hiding abuse behind company policy
                Assessments, appraisals, annual reviews can also be used as instruments to bully people. People work very hard to reach deadlines, only to find they have been changed. People who actually deserve to get a pay rise are passed over whilst the ‘favourites’ who haven’t actuaaly reached their targets are awarded them.
                Another instance of cruel, abusivetreat ment by a bullying manager is to choose a Friday afternoon, or the late afternoon before someone is due to go on holiday, to give them negative feedback about their work or say ‘I’m not happy with your performance, but we will discuss it when you return from leave.’

Evasiveness
                Bullies often abuse their colleagues by not taking the responsibility for something that is officially in their remit. ‘I’m sorry you are inconvenienced. I did tell my staff to do such and such’, when they clearly did not.
                Some bullies have a tendency not to make eye contact, but when they do it can be form of a direct stare. They often make notes when they are being spoken to, instead of engaging with the person.

Controlling behaviours
                A potent example of this was a meeting between the harassment investigator and the Managing Director of the organisation he was carrying out the investigation for. He was told: ‘I’m sure you are not going to find anything untoward here. Did you hear what I said? I’m sure you won’t find anything.’ This left the investigator in no doubt as to whom one of the role models for bullying behaviour was.

Inconsistency in behaviour
                People change their behaviour not only from time to time but also with different people. The manager who is shouting at someone in their office fo rsome minor misdemeanour suddenly becomes kind and avuncular when the young secretatry appears, only to return their previously foul inappropriate behaviour as soon as she leaves.

No thanks or recognition
                Work or tasks that have been done well during a manager’s absence are not commented on. Instead, looking for the smallest detail with which to find fault, the manager then complains how ‘things don’t run smoothly when I’m away’.

Undermining someone’s authority
                One supervisor gave her team instructions not to take notice of the request made by their line manager but to come directly to her instead. The team is obviously placed in an untenable position.
Demeaning behaviour
                Inappropriate remarks in front on stafs or costumers can range from one-off ‘put down’, such as ‘If you paid attention to detail you would know the answer to that’, through to complete character assassinations. In one instance a woman was crisitcised for her appearance, on the supposed fact that no one in the office liked her and on the basis that she was thought to be an incompetent mother. This was all said in front of the team that she was supposed to manage.

Rumour mongering
                Some work cultures are such that spreading rumours is the norm and seen by most as being joke. However, jokes go too far and they hurt some people. In jokes about fictitious relationships made in front of partners can lead to disharmony at home, no matter how far from the truth they are.
                In one case the rumours were so far-fetched that the victim had to ask his councellor what they meant as he had absolutely no idea what he was being accused of. Rumours are not necessarily about an individual; they can be about friends or members of their family.

 Myths and misconcepsions about bullying and harassment at work
·         It’s in people’s imaginations.
·         It’s another excuse for a policy.
·         Policies will give people opportunities for personal vendettas and enable them to make personal complaints.
·         It’s the trendy topic.
·         People who are bullied must have asked fot it.
·         It will open the floodgates for malicious complaints.
·         All this PC stuff will make the workplace humourless and boring.
·         People can’t take a joke.
·         They don’t know what bullying is: When I was a child. . .
·         It doesn’t happen to men.
·         Only men bully.
·         It happens only to weak people. 


               
                

Wednesday, November 3, 2010

FROM A PDF

36
CHAPTER 3
RESEARCH DESIGN AND METHODOLOGY
Introduction
The review of literature has produced reoccurring themes emphasizing the
importance of technological literacy for citizens in the 21st Century (Garfinkel, 2003;
Hall, 2001; Lemke, 2003; Murray, 2003; NAE, 2002; Partnership for 21st Century Skills,
2003; Rose & Dugger, 2003; Zhao & Alexander, 2002; U.S. Department of Education,
2004; Technology Counts, 2005). Education is a critical component in preparing students
for a knowledge-based, digital society. According to Hall (2001), available technologies,
our perceptions of those technologies, and how they are used will determine the shape of
our world. Citizens of the future will face challenges that depend on the development and
application of technology. Are we preparing students, the citizens of tomorrow, for these
challenges?
Purpose of the Study
This study developed and implemented a faculty survey and a student assessment.
The purpose of the faculty survey was to determine what basic computer skills are needed
by undergraduate students for academic success in post-secondary education. This phase
of the study examined the data collected for trends and differences between the
independent variables of subject/content area, institution, gender, and years of faculty
experience. The purpose of the student assessment was to evaluate the computer
competencies of students entering a post-secondary education. This phase of the study
examined the data collected for trends and differences between the independent variables
37
of home state, number of high school computer courses taken, gender, and major field of
study? Data collection and analysis assisted in determining if students possess the
necessary computer/technology skills entering a post-secondary institution or if a need
exists for a general education course to teach computer literacy/skills to the
undergraduate student population. This study also provided valuable information in
regards to the content of such a course.
Research Questions
1. What technology skills do post-secondary faculty members deem important for all
students to possess at the college level?
2. Are there differences between the student technology skills post-secondary
faculty members deem important when grouped by subject/content area,
institution/stratum, gender, years of faculty experience?
3. What technology skills can students demonstrate proficiently upon entering a
post-secondary institution?
4. Are there differences between the proficiency level of students’ technology skills
when grouped by home state, number of high school computer courses, gender, or
major field of study?
5. Are students technologically ready entering post-secondary education or does a
need exist for a computer literacy/skills course for all undergraduate students?
Instrumentation
Two instruments were employed for data collection in this research study: a
faculty survey and a student assessment. A faculty survey was designed by the researcher
38
to help identify technology/computer skills deemed important for undergraduate students
to possess in order to be successful in their post-secondary endeavors. A survey research
design was applied to investigate the research questions.
A second instrument was developed and implemented to assess technology skills
of freshmen undergraduate students who had not yet taken a post-secondary computer
literacy/skills course. A description of the two instruments used in this study follows.
Faculty Survey
Introduction
According to Leedy and Ormrod (2001), “Research is a viable approach to a
problem only when there are data to support it” (p. 94). Nesbary (2000) defines survey
research as “the process of collecting representative sample data from a larger population
and using the sample to infer attributes of the population” (p. 10). The main purpose of a
survey is to estimate, with significant precision, the percentage of population that has a
specific attribute by collecting data from a small portion of the total population (Dillman,
2000; Wallen & Fraenkel, 2001). The researcher wanted to find out from members of the
population their view on one or more variables. As noted by Borg and Gall (1989),
studies involving surveys comprise a significant amount of the research done in the
education field. Data are ever-changing and survey research portrays a brief moment in
time to enhance our understanding of the present (Leedy & Ormrod, 2001). Educational
surveys are often used to assist in planning and decision making, as well as to evaluate
the effectiveness of an implemented program (McNamara, 1994; Borg & Gall, 1989).
39
An online faculty survey was conducted to identify computer literacy skills that
faculty members deem important for an undergraduate student to possess in order to be
academically successful at the post-secondary level.
Population and Sample
The population for this faculty survey consisted of post-secondary faculty
members at four-year public institutions in the state of Missouri. Four-year public
institutions were determined by visiting the Missouri Department of Higher Education
Web site at http://www.cbhe.state.mo.us/Institutions/pubinst.htm. Private or independent
institutions and community colleges were not included in the population. Thus the
sampling frame consisted of all faculty members at thirteen institutions in Missouri, as
summarized in Table 1.
Table 1
Summary of 4-Year Public Institutions in Missouri
Central Missouri State University
Harris-Stowe State College
Lincoln University
Missouri Southern State University
Missouri Western State College
Northwest Missouri State University
Southeast Missouri State University
Southwest Missouri State University
Truman State University
University of Missouri-Columbia
University of Missouri-Kansas City
University of Missouri-Rolla
University of Missouri-St. Louis
A sample population was drawn from the sampling frame. A sampling frame
includes the actual list of individuals included in the population (Nesbary, 2000) which
was approximately 4821 faculty members. According to Patten (2004), the quality of the
40
sample affects the quality of the research generalizations. Nesbary (2000), suggests the
larger the sample size, the greater the probability the sample will reflect the general
population. However, sample size alone does not constitute the ability to generalize.
Patten (2004), states that obtaining an unbiased sample is the main criterion when
evaluating the adequacy of a sample. Patten also identifies an unbiased sample as one in
which every member of a population has an equal opportunity of being selected in the
sample. Therefore, random sampling was used in this study to help ensure an unbiased
sample population. Because random sampling may introduce sampling errors, efforts
were made to reduce sampling errors, and thus increasing precision, by increasing the
sample size and by using stratified random sampling. To obtain a stratified random
sample, the population was divided into strata according to institutions as shown in Table
2. Typically, for stratified random sampling, the same percentage of participants, not the
same number of participants, are drawn from each stratum (Patten, 2004).
Table 2
Strata (subgroups) for Stratified Random Sampling
Instructors and professors at Central Missouri State University
Instructors and professors at Harris-Stowe State College
Instructors and professors at Lincoln University
Instructors and professors at Missouri Southern State University
Instructors and professors at Missouri Western State College
Instructors and professors at Northwest Missouri State University
Instructors and professors at Southeast Missouri State University
Instructors and professors at Southwest Missouri State University
Instructors and professors at Truman State University
Instructors and professors at University of Missouri-Columbia
Instructors and professors at University of Missouri-Kansas City
Instructors and professors at University of Missouri-Rolla
Instructors and professors at University of Missouri-St. Louis
41
Patten (2004) suggests that a researcher should first consider obtaining an
unbiased sample and then seek a relatively large number of participants. Patten (2004)
provides a table of recommended sample sizes. A table of recommended sample sizes (n)
for populations (N) with finite sizes, developed by Krejcie and Morgan and adapted by
Patten (2004), was used to determine estimated sample size. According to the table, and
for purposes of this study, the researcher used an estimated population size N = 4821 and
thus a sample size goal of n = 357.
Survey Procedures
In 1998, according to Nesbary (2000), Web surveys were almost non-existent in
the public sector. Nesbary decided to test the waters and conduct three surveys to
compare response rate and response time of Web surveys to regular mail surveys. Survey
results and respondent feedback of all three surveys indicated that Web surveys were
more cost effective, easier to use, had quicker response rates, and greater responses. One
of Nesbary’s Web surveys was distributed to selected universities. Of those surveyed,
respondents indicated a strong preference for use of technology to take advantage of
speed and convenience.
The researcher used a Web-based survey for the faculty survey portion of this
study. UNL IRB approval was obtained (Appendix A). Two approvals for change of
protocol were also obtained, one for changing the title of the study (Appendix B) and the
other for changing the survey format (Appendix C). From the original IRB request, the
survey was condensed to reduce the number of items, shortening the survey to increase
response rate.
42
Ethical Issues
McNamara (1994) identifies five ethical concerns to be considered when
conducting survey research. These guidelines deal with voluntary participation, no harm
to respondents, anonymity and confidentiality, identifying purpose and sponsor, and
analysis and reporting. Each guideline will be addressed individually with explanations to
help eliminate or control any ethical concerns.
First, researchers need to make sure that participation is completely voluntary.
However, voluntary participation can sometimes conflict with the need to have a high
response rate. Low return rates can introduce response bias (McNamara, 1994). In order
to encourage a high response rate, Dillman (2000) suggests multiple contacts. For this
study, up to five contacts were made per potential participant. The first email contact
(Appendix D) was sent a few days preceding the survey to not only verify email
addresses, but also to inform possible participants of the importance and justification for
the study. The second email contact (Appendix E) was the actual email cover letter
explaining the study objectives in more depth. This email consisted of a link to the Webbased
survey and a password to enter. By clicking on the link provided and logging into
the secure site, the participant indicated agreement to participate in the research study.
The third email contact (Appendix F) was sent a week later reminding those that had not
responded. The fourth email contact (Appendix G) was sent two weeks after the actual
survey email reemphasizing the importance of faculty expertise in providing input to the
study. The fifth and final email contact (Appendix H) was sent three weeks after the
43
actual survey email to inform faculty that the study was drawing to a close and that their
input was valuable to the results of the study.
McNamara’s (1994) second ethical guideline is to avoid possible harm to
respondents. This could include embarrassment or feeling uncomfortable about questions.
This study did not include sensitive questions that could cause embarrassment or
uncomfortable feelings. Harm could also arise in data analysis or in the survey results.
Solutions to these harms will be discussed under confidentiality and report writing
guidelines.
A third ethical guideline is to protect a respondent’s identity. This can be
accomplished by exercising anonymity and confidentiality. A survey is anonymous when
a respondent cannot be identified on the basis of a response. A survey is confidential
when a response can be identified with a subject, but the researcher promises not to
disclose the individual’s identity (McNamara, 1994). To avoid confusion, the cover email
clearly identified the survey as being confidential in regards to responses and the
reporting of results. Participant identification was kept confidential and was only used in
determining who had not responded for follow-up purposes.
McNamara’s (1994) fourth ethical guideline is to let all prospective respondents
know the purpose of the survey and the organization that is sponsoring it. The purpose of
the study was provided in the cover email indicating a need to identify technology skills
necessary for students to be successful in their academic coursework and to determine if a
general education computer literacy/skills course should be required of all undergraduate
44
students. The cover email also explained that the results of the study would be used in a
dissertation as partial fulfillment for a Doctoral degree.
The fifth ethical guideline, as described by McNamara (1994), is to accurately
report both the methods and the results of the surveys to professional colleagues in the
educational community. Because advancements in academic fields come through honesty
and openness, the researcher assumes the responsibility to report problems and
weaknesses experienced as well as the positive results of the study.
Validity and Reliability Issues
An instrument is valid if it measures what it is intended to measure and accurately
achieves the purpose for which it was designed (Patten, 2004; Wallen & Fraenkel, 2001).
Patten (2004) emphasizes that validity is a matter of degree and discussion should focus
on how valid a test is, not whether it is valid or not. According to Patten (2004), no test
instrument is perfectly valid. The researcher needs some kind of assurance that the
instrument being used will result in accurate conclusions (Wallen & Fraenkel, 2001).
Validity involves the appropriateness, meaningfulness, and usefulness of
inferences made by the researcher on the basis of the data collected (Wallen & Fraenkel,
2001). Validity can often be thought of as judgmental. According to Patten (2004),
content validity is determined by judgments on the appropriateness of the instrument’s
content. Patten (2004) identifies three principles to improve content validity: 1) use a
broad sample of content rather than a narrow one, 2) emphasize important material, and
3) write questions to measure the appropriate skill. These three principles were addressed
when writing the survey items. To provide additional content validity of the survey
45
instrument, the researcher formed a focus group of five to ten experts in the field of
computer literacy who provided input and suggestive feedback on survey items. Members
of the focus group were educators at the college and/or high school level who have taught
or are currently teaching computer literacy skills. Comments from the focus group
indicated that the skills listed in the survey were basic/intermediate skills and were
appropriate for all college students to know and be able to do. Some members of the
focus group suggested that the survey might be a bit long and that skills could be
generalized and consolidated for a more concise survey. The researcher categorized
application skills and condensed the application component items from 20 per application
to eight items per application. The computer concepts component was reduced from 22
items to eight items.
According to Patten (2004), “. . . validity is more important than reliability” (p.
71). However, reliability does need to be addressed. Reliability relates to the consistency
of the data collected (Wallen & Fraenkel, 2001). Cronbach’s coefficient alpha was used
to determine the internal reliability of the instrument. The faculty survey instrument was
tested in its entirety, and the subscales of the instrument were tested independently.
Data Collection
An informal pilot study was conducted with a small group of faculty members at
the researcher’s home institution. Conducting a local pilot study allowed the researcher to
ask participants for suggestive feedback on the survey and also helped eliminate author
bias. Once the pilot survey had been modified as per the educational expert’s feedback,
the survey was administered online to the stratified, random sample population.
46
Participants of the study were contacted by email explaining the research
objective and asking them to participate. The objective of the research was to gather
information about technology skills, in particular, what technology skills should students
possess to be successful during their post-secondary courses. The email also contained a
link to the Web-based faculty survey and a password to enter the survey. Follow-up email
contacts were sent to increase response rate. Upon completion of the survey, each
respondent was directed to a Web page thanking them for their response and offering
them a copy of the study results if they were interested. Screen shots of the Web-based
faculty survey are presented in Appendix I.
The Web-based survey was conducted using surveymonkey.com, a survey
software program offered online. For a small fee, the program offered many features
including unlimited number of survey questions, ability to add a personalized logo,
custom redirects, result filtering, and the capability to export data for statistical analysis.
The program provided a list management tool where responses could be tracked by their
email address which proved to be very useful for follow-up emails. The program also
provided security including the option to turn on SSL (Secure Sockets Layers) to utilize
data encryption and provide data protection.
Responses to the survey were recorded, exported in a spreadsheet, and transferred
to a statistical software package for in-depth analysis. Descriptive statistics were
calculated and data relationships were analyzed.
47
Variables and Measures
Variables used in the survey have been summarized in Table 3. The variables
consisted of seven independent variables that grouped respondents by common
characteristics and five dependent variables that grouped responses by content categories.
The independent variables included professional title, institution, department/content
area, school size, gender, number of years at current institution, and total number of years
in education. The dependent variables included word processing, spreadsheet,
presentation, database, and computer concepts.
Table 3
Summary of Dependent and Independent Variables in the Faculty Survey
Independent Variables (n = 7) Dependent Variables (n = 5)
Professional title Word processing
Institution Spreadsheet
Department Presentation
School size Database
Gender Computer concepts
Years as faculty member at current institution
Years in education
Data Analysis Plan
To begin the data analysis process, descriptive statistics were calculated on the
independent variables to summarize and describe the data collected. Survey results were
measured by category. There were five categories (subscales), representing the five
dependent variables. Reponses to the survey items were coded from 1 to 4 depending on
the importance of each skill. One represented ‘not important’, two represented ‘somewhat
important’, three represented ‘important’, and four represented ‘very important’. The
code for all survey items in the same category were summed together for a composite
48
score per category. This category composite score was used for statistical analysis. Item
analysis was conducted to determine the internal consistency and reliability of each
individual item as well as each subscale. Cronbach’s Alpha test was also used to test
internal reliability.
Inferential statistics were used to reach conclusions and make generalizations
about the characteristics of populations based on data collected from the sample.
Frequencies and/or percentages were used to identify computer skills that faculty
members deem important for all students to possess. Independent t-tests and/or simple
analysis of variance (ANOVA) were used to look for significant differences between the
student technology skills faculty members deem important when grouped by
department/content area, institution/stratum, gender, or years of faculty experience. The
type of tests that were used to answer specific research questions are summarized in
Table 4. A statistical software program, SPSS (Statistical Package for Social Sciences)
was used for in-depth data analyses.
Table 4
Summary of Data Sources, Types and Measures Applied by Research Question
Research
Question #
Data Source
Response Type
Data Type
Analysis Plan
1 Faculty Survey Responses Likert Scale Nominal f, %
2 Faculty Survey Responses Likert Scale Nominal t test, ANOVA
Student Assessment
Introduction
To assist in evaluating the technology skills of students, a technology assessment
was conducted to determine computer literacy and performance skills of students entering
a post-secondary institution prior to taking a computer course at the post-secondary level.
49
Population and Sample
The population for the student assessment consisted of college freshmen from a
small mid-western university enrolled in a computer literacy course.
Permission from students to use their scores in the study was requested through informed
consent forms. This resulted in a sample size of 164 students.
Student Assessment Procedures
The purpose of the student assessment was to describe specifically what a typical
student entering post-secondary education knows about computer operations and
concepts as well as the computer skills they can demonstrate proficiently.
An additional Northwest Missouri State University IRB form (Appendix J) was
obtained and student consent forms (Appendix K) were collected from participants. A
series of assessments were given to all students during the first few weeks of a computer
literacy course to determine the computer skills students possess prior to taking a
computer literacy course.
Measurement Instrument
The student assessment consisted of a few demographic questions, and two major
components: 1) computer concepts and 2) computer application skills.
The computer concepts component of the assessment covered six different
modules. Module one questions covered computer and information literacy, introduction
to application software, word processing concepts, and inside the system. Module two
questions covered understanding the Internet, email, system software, and exploring the
Web. Module three questions covered spreadsheet concepts, current issues, emerging
50
technologies, and data storage. Module four covered presentation packages, special
purpose programs, multimedia/virtual reality, and input/output. Module five questions
covered database concepts, telecommunications, and networks. Module six questions
covered creating a Web page, ethics, and security. The assessment for the computer
concepts component of the study consisted of 150 questions, 25 questions randomly
selected from each of six module test banks. The number of questions in each module test
bank ranged from 143 to 214 questions. This portion of the study was administered using
an online program called QMark (Question Mark). The assessment was automatically
graded and scores were recorded on a server at Northwest Missouri State University.
The computer application skills component was assessed using a commercial
software program called SAM (Skills Assessment Manager). SAM is a unique
performance-based testing software program that utilizes realistic, powerful simulations.
The software package works just like the actual Microsoft Word, Excel, Access, and
PowerPoint applications, but without the need for preinstalled Microsoft Office software.
Course Technology, the publisher of SAM software, provided the researcher with a site
license for use in this research. The application skills assessed for this study include word
processing skills, spreadsheet skills, presentation skills, and database skills.
Validity and Reliability Issues
Patten’s (2004) three principles to improve content validity: 1) use a broad sample
of content rather than a narrow one, 2) emphasize important material, and 3) write
questions to measure the appropriate skill, were addressed when developing assessment
items.
51
In 1998 Course Technology introduced SAM 1997 and has continued to update
the product through SAM 2000, SAM 2003, and now SAM XP. SAM is a unique
performance-based testing software program that utilizes realistic, powerful simulations.
The software package works just like the actual Microsoft Word, Excel, Access, and
PowerPoint applications, but without the need for preinstalled Office software. According
to Course Technology, SAM is the most powerful testing and reporting tool available.
According to Course Technology (2002),
SAM is becoming the provider of the most widely-used and effective technologybased
assessment product line for Microsoft Office used in educational institutions
today. SAM is used at high schools, colleges, career colleges, MBA programs, and
in the workplace as a screening tool for placing people in the right training
courses, a ‘test-out’ tool to determine students’ proficiency before they take a
course, and a seamless, in-course assessment tool to allow students to demonstrate
their proficiency as they go through a course.
Proficiency skills on the assessment matched categories on the faculty survey so results
could be compared.
Data Collection
All students enrolled in the course completed the assessments during the first few
weeks of class using the SAM and QMark software to determine their computer
literacy/skill proficiency level prior to taking the computer literacy course. The
assessments were graded online and the results were immediate. Prior to taking the
assessment, the participants provided data on a few demographic questions. A list of the
52
demographic questions can be found in Appendix L. Computer concepts questions were
randomly selected from a test bank of questions. Sample screen shots can be found in
Appendix M. A list of proficiency skills for the computer applications component of the
student assessment can be found in Appendix N. Student names were kept confidential to
ensure individual privacy.
Variables and Measures
Variables used in the student assessment have been summarized in Table 5. The
variables consisted of five independent variables that group participants by common
characteristics and five dependent variables that group participants by content categories.
The independent variables included student home state, size of high school graduating
class, number of high school computer courses taken, gender, and post-secondary major
field of study. The dependent variables included computer skills grouped in five
categories including word processing, spreadsheet, presentation, database, and computer
concepts.
Table 5
Summary of Dependent and Independent Variables in the Student Assessment
Independent Variables (n = 5) Dependent Variables (n = 5)
Home state Word processing
High school size Spreadsheet
Number of high school computer courses Presentation
Gender Database
Major Computer concepts
Data Analysis Plan
Results of the student assessment were recorded in a spreadsheet and transferred
to SPSS for statistical analysis. Descriptive statistics and data relationships were
calculated. Independent t-tests and simple analysis of variance (ANOVA) were used to
53
look for significant differences between the proficiency level of students’ technology
skills when grouped by home state, number of high school computer courses taken,
gender, and major field of study. A statistical software program, SPSS (Statistical
Package for Social Sciences) was used for in-depth data analyses. The type of tests used
to answer specific research questions are summarized in Table 6.
Table 6
Summary of Data Sources, Types and Measures Applied by Research Question
Research
Question #
Data Source Title
Response Type
Data Type
Analysis Plan
3 Student assessment score Percentage Interval f, %
4 Student assessment score Percentage Interval t test, ANOVA
5 Student assessment score Percentage Interval f, %

Elements of Research Proposal and Report

All research reports use roughly the same format. It doesn't matter whether you've done a customer satisfaction survey, an employee opinion survey, a health care survey, or a marketing research survey. All have the same basic structure and format. The rationale is that readers of research reports (i.e., decision makers, funders, etc.) will know exactly where to find the information they are looking for, regardless of the individual report.

Once you've learned the basic rules for research proposal and report writing, you can apply them to any research discipline. The same rules apply to writing a proposal, a thesis, a dissertation, or any business research report.

The Research Proposal and Report

General considerations

Research papers usually have five chapters with well-established sections in each chapter. Readers of the paper will be looking for these chapters and sections so you should not deviate from the standard format unless you are specifically requested to do so by the research sponsor.
Most research studies begin with a written proposal. Again, nearly all proposals follow the same format. In fact, the proposal is identical to the first three chapters of the final paper except that it's writtten in future tense. In the proposal, you might say something like "the researchers will secure the sample from ...", while in the final paper, it would be changed to "the researchers secured the sample from ...". Once again, with the exception of tense, the proposal becomes the first three chapters of the final research paper.
The most commonly used style for writing research reports is called "APA" and the rules are described in the Publication Manual of the American Psychological Association. Any library or bookstore will have it readily available. The style guide contains hundreds of rules for grammar, layout, and syntax. This paper will cover the most important ones.
Avoid the use of first person pronouns. Refer to yourself or the research team in third person. Instead of saying "I will ..." or "We will ...", say something like "The researcher will ..." or "The research team will ...".
A suggestion: Never present a draft (rough) copy of your proposal, thesis, dissertation, or research paper...even if asked. A paper that looks like a draft, will interpreted as such, and you can expect extensive and liberal modifications. Take the time to put your paper in perfect APA format before showing it to anyone else. The payoff will be great since it will then be perceived as a final paper, and there will be far fewer changes.

Style, layout, and page formatting

Title page

All text on the title page is centered vertically and horizontally. The title page has no page number and it is not counted in any page numbering.

Page layout

Left margin: 1½"
Right margin: 1"
Top margin: 1"
Bottom margin: 1"

Page numbering

Pages are numbered at the top right. There should be 1" of white space from the top of the page number to the top of the paper. Numeric page numbering begins with the first page of Chapter 1 (although a page number is not placed on page 1).

Spacing and justification

All pages are single sided. Text is double-spaced, except for long quotations and the bibliography (which are single-spaced). There is one blank line between a section heading and the text that follows it. Do not right-justify text. Use ragged-right.

Font face and size

Any easily readable font is acceptable. The font should be 10 points or larger. Generally, the same font must be used throughout the manuscript, except 1) tables and graphs may use a different font, and 2) chapter titles and section headings may use a different font.

References

APA format should be used to cite references within the paper. If you name the author in your sentence, then follow the authors name with the year in parentheses. For example:

Jones (2004) found that...

If you do not include the authors name as part of the text, then both the author's name and year are enclosed in parentheses. For example:

One researcher (Jones, 2004) found that...

A complete bibliography is attached at the end of the paper. It is double spaced except single-spacing is used for a multiple-line reference. The first line of each reference is indented.

Examples:

     Bradburn, N. M., & Mason, W. M. (1964). The effect of question order on response. Journal of Marketing Research 1 (4), 57-61.

     Bradburn, N. M., & Miles, C. (1979). Vague quantifiers. Public Opinion Quarterly 43 (1), 92-101.


Outline of chapters and sections

TITLE PAGE

TABLE OF CONTENTS

CHAPTER I - Introduction
     Introductory paragraphs
     Statement of the problem
     Purpose
     Significance of the study
     Research questions and/or hypotheses

CHAPTER II - Background
     Literature review
     Definition of terms

CHAPTER III - Methodology
     Restate purpose and research questions or null hypotheses
     Population and sampling
     Instrumentation (include copy in appendix)
     Procedure and time frame
     Analysis plan (state critical alpha level and type of statistical tests)
     Validity and reliability
     Assumptions
     Scope and limitations

CHAPTER IV - Results

CHAPTER V - Conclusions and recommendations
     Summary (of what you did and found)
     Discussion (explanation of findings - why do you think you found what you did?)
     Recommendations (based on your findings)

REFERENCES

APPENDIX

Chapter I - Introduction

Introductory paragraphs

Chapter I begins with a few short introductory paragraphs (a couple of pages at most). The primary goal of the introductory paragraphs is to catch the attention of the readers and to get them "turned on" about the subject. It sets the stage for the paper and puts your topic in perspective. The introduction often contains dramatic and general statements about the need for the study. It uses dramatic illustrations or quotes to set the tone. When writing the introduction, put yourself in your reader's position - would you continue reading?

Statement of the Problem

The statement of the problem is the focal point of your research. It is just one sentence (with several paragraphs of elaboration).
You are looking for something wrong.
     ....or something that needs close attention
     ....or existing methods that no longer seem to be working.

Example of a problem statement:
"The frequency of job layoffs is creating fear, anxiety, and a loss of productivity in middle management workers."
While the problem statement itself is just one sentence, it is always accompanied by several paragraphs that elaborate on the problem. Present persuasive arguments why the problem is important enough to study. Include the opinions of others (politicians, futurists, other professionals). Explain how the problem relates to business, social or political trends by presenting data that demonstrates the scope and depth of the problem. Try to give dramatic and concrete illustrations of the problem. After writing this section, make sure you can easily identify the single sentence that is the problem statement.

Purpose

The purpose is a single statement or paragraph that explains what the study intends to accomplish. A few typical statements are:

The goal of this study is to...
     ... overcome the difficulty with ...
     ... discover what ...
     ... understand the causes or effects of ...
     ... refine our current understanding of ...
     ... provide a new interpretation of ...
     ... understand what makes ___ successful or unsuccessful

Significance of the Study

This section creates a perspective for looking at the problem. It points out how your study relates to the larger issues and uses a persuasive rationale to justify the reason for your study. It makes the purpose worth pursuing. The significance of the study answers the questions:

     Why is your study important?
     To whom is it important?
     What benefit(s) will occur if your study is done?


Research Questions and/or Hypotheses and/or Null Hypotheses

Chapter I lists the research questions (although it is equally acceptable to present the hypotheses or null hypotheses). No elaboration is included in this section. An example would be:

The research questions for this study will be:

     1. What are the attitudes of...
     2. Is there a significant difference between...
     3. Is there a significant relationship between...


Chapter II - Background


Chapter II is a review of the literature. It is important because it shows what previous researchers have discovered. It is usually quite long and primarily depends upon how much research has previously been done in the area you are planning to investigate. If you are planning to explore a relatively new area, the literature review should cite similar areas of study or studies that lead up to the current research. Never say that your area is so new that no research exists. It is one of the key elements that proposal readers look at when deciding whether or not to approve a proposal.

Chapter II should also contain a definition of terms section when appropriate. Include it if your paper uses special terms that are unique to your field of inquiry or that might not be understood by the general reader. "Operational definitions" (definitions that you have formulated for the study) should also be included. An example of an operational definition is: "For the purpose of this research, improvement is operationally defined as posttest score minus pretest score".



Chapter III - Methodology

The methodology section describes your basic research plan. It usually begins with a few short introductory paragraphs that restate purpose and research questions. The phraseology should be identical to that used in Chapter I. Keep the wording of your research questions consistent throughout the document.

Population and sampling

The basic research paradigm is:
     1) Define the population
     2) Draw a representative sample from the population
     3) Do the research on the sample
     4) Infer your results from the sample back to the population

As you can see, it all begins with a precise definition of the population. The whole idea of inferential research (using a sample to represent the entire population) depends upon an accurate description of the population. When you've finished your research and you make statements based on the results, who will they apply to? Usually, just one sentence is necessary to define the population. Examples are: "The population for this study is defined as all adult customers who make a purchase in our stores during the sampling time frame", or "...all home owners in the city of Minneapolis", or "...all potential consumers of our product".

While the population can usually be defined by a single statement, the sampling procedure needs to be described in extensive detail. There are numerous sampling methods from which to choose. Describe in minute detail, how you will select the sample. Use specific names, places, times, etc. Don't omit any details. This is extremely important because the reader of the paper must decide if your sample will sufficiently represent the population.

Instrumentation

If you are using a survey that was designed by someone else, state the source of the survey. Describe the theoretical constructs that the survey is attempting to measure. Include a copy of the actual survey in the appendix and state that a copy of the survey is in the appendix.

Procedure and time frame

State exactly when the research will begin and when it will end. Describe any special procedures that will be followed (e.g., instructions that will be read to participants, presentation of an informed consent form, etc.).

Analysis plan

The analysis plan should be described in detail. Each research question will usually require its own analysis. Thus, the research questions should be addressed one at a time followed by a description of the type of statistical tests that will be performed to answer that research question. Be specific. State what variables will be included in the analyses and identify the dependent and independent variables if such a relationship exists. Decision making criteria (e.g., the critical alpha level) should also be stated, as well as the computer software that will be used.

Validity and reliability

If the survey you're using was designed by someone else, then describe the previous validity and reliability assessments. When using an existing instrument, you'll want to perform the same reliability measurement as the author of the instrument. If you've developed your own survey, then you must describe the steps you took to assess its validity and a description of how you will measure its reliability.

Validity refers to the accuracy or truthfulness of a measurement. Are we measuring what we think we are? There are no statistical tests to measure validity. All assessments of validity are subjective opinions based on the judgment of the researcher. Nevertheless, there are at least three types of validity that should be addressed and you should state what steps you took to assess validity.

Face validity refers to the likelihood that a question will be misunderstood or misinterpreted. Pretesting a survey is a good way to increase the likelihood of face validity. One method of establishing face validity is described here.

Content validity refers to whether an instrument provides adequate coverage of a topic. Expert opinions, literature searches, and pretest open-ended questions help to establish content validity.

Construct validity refers to the theoretical foundations underlying a particular scale or measurement. It looks at the underlying theories or constructs that explain a phenomena. In other words, if you are using several survey items to measure a more global construct (e.g., a subscale of a survey), then you should describe why you believe the items comprise a construct. If a construct has been identified by previous researchers, then describe the criteria they used to validate the construct. A technique known as confirmatory factor analysis is often used to explore how individual survey items contribute to an overall construct measurement.

Reliability is synonymous with repeatability or stability. A measurement that yields consistent results over time is said to be reliable. When a measurement is prone to random error, it lacks reliability.

There are three basic methods to test reliability : test-retest, equivalent form, and internal consistency. Most research uses some form of internal consistency. When there is a scale of items all attempting to measure the same construct, then we would expect a large degree of coherence in the way people answer those items. Various statistical tests can measure the degree of coherence. Another way to test reliability is to ask the same question with slightly different wording in different parts of the survey. The correlation between the items is a measure of their reliability.

Assumptions

All research studies make assumptions. The most obvious is that the sample represents the population. Another common assumptions are that an instrument has validity and is measuring the desired constructs. Still another is that respondents will answer a survey truthfully. The important point is for the researcher to state specifically what assumptions are being made.

Scope and limitations

All research studies also have limitations and a finite scope. Limitations are often imposed by time and budget constraints. Precisely list the limitations of the study. Describe the extent to which you believe the limitations degrade the quality of the research.

Chapter IV - Results

Description of the sample

Nearly all research collects various demographic information. It is important to report the descriptive statistics of the sample because it lets the reader decide if the sample is truly representative of the population.

Analyses

The analyses section is cut and dry. It precisely follows the analysis plan laid out in Chapter III. Each research question addressed individually. For each research question:

     1) Restate the research question using the exact wording as in Chapter I
     2) If the research question is testable, state the null hypothesis
     3) State the type of statistical test(s) performed
     4) Report the statistics and conclusions, followed by any appropriate table(s)

Numbers and tables are not self-evident. If you use tables or graphs, refer to them in the text and explain what they say. An example is: "Table 4 shows a strong negative relationship between delivery time and customer satisfaction (r=-.72, p=.03)". All tables and figures have a number and a descriptive heading. For example:

Table 4
The relationship between delivery time and customer satisfaction.

Avoid the use of trivial tables or graphs. If a graph or table does not add new information (i.e., information not explained in the text), then don't include it.

Simply present the results. Do not attempt to explain the results in this chapter.

Chapter V - Conclusions and recommendations

Begin the final chapter with a few paragraphs summarizing what you did and found (i.e., the conclusions from Chapter IV).

Discussion

Discuss the findings. Do your findings support existing theories? Explain why you think you found what you did. Present plausible reasons why the results might have turned out the way they did.

Recommendations

Present recommendations based on your findings. Avoid the temptation to present recommendations based on your own beliefs or biases that are not specifically supported by your data. Recommendations fall into two categories. The first is recommendations to the study sponsor. What actions do you recommend they take based upon the data. The second is recommendations to other researchers. There are almost always ways that a study could be improved or refined. What would you change if you were to do your study over again? These are the recommendations to other researchers.


References

List references in APA format alphabetically by author's last name


Appendix

Include a copy of any actual instruments. If used, include a copy of the informed consent form.

babasahin natin mamaya.

The first thing to say is that if it has to be done, it's you who's got to do it.
  • You. There's no-one else to do the project for you. Very likely, there's no-one who will give you any help. You're the one who cares, the drive has got to come from within you. You're the one who knows how to answer the question, no-one else knows as much.
    The above is not the whole truth --- as your project evolves, you'll find yourself receiving technical help, wisdom, and emotional support from a great many people --- but you need to be aware that self-reliance is vital.
    There may be other types of research, e.g., in which a neophyte is fitted into a slot in the God Professor's program. I don't have experience of them.
  • Do. It is important that you start the main activities of the research --- experimentation, or survey, or simulation, or data analysis --- sooner rather than later. A common mistake is to think and plan and read, and never get round to the main game. When you start, you may find that the real difficulties and issues are quite different from what you thought in the abstract.
Research has several components, for example: thinking of the question, answering the question, communicating the answer. It may be worth mentioning that not all of these are essential.
  • Even without much of a question, one can collect some data, summarise it, and report the results. Certainly if you're the first to think of this kind of data, that's all you need to do. Certainly if the data keeps changing from year to year (e.g., accidents, and many other society-level phenomena), this type of study is always useful to a greater or lesser extent.
  • Even without an answer, the question is sometimes such a good one that it is worth telling others about. (It might be that you don't have time to answer it, or are uninterested, or not competent in the necessary techniques.)
  • Sometimes you don't want to communicate your question and answer. They might have turned out to be less interesting than you thought when you started. Or you might want to wait until you've done something additional.
Nevertheless, most research has all three components. The answering and the communicating can be dismissed quite briefly.
  • The fact that you're doing research is strong evidence that you have the skills necessary to answer the question. Not all of them, perhaps, but enough to work out a way of attacking the research and to serve as a foundation for additional skills to be learnt along the way. Incidentally, as a rule of thumb, if there is a course on something, people think of it as a low-level skill --- it is not necessarily below your dignity as a researcher, but you don't get any merit badges for mastering it. (I am a little concerned at the increasing amount of structured studies that universities are prescribing for their first-year research students. To my mind, the overwhelming priority at that stage is to do a doable piece of research, not attend lectures on bibliographic tools and the institution's policy on intellectual property.) As to non-technical aspects of answering the question, there is some advice below.
  • There are lots of textbooks and courses to aid you in your scientific writing. Essentially, though, it is all a matter of practice.
    One of the reasons I recommend psychology to students is that you can't escape either the use of numbers or the use of words. (Economics is another good subject from this point of view, but it doesn't have much of an experimental component.)
    As soon as you've got a message to communicate, you should write a report on it, not wait until the whole project is finished. That probably means writing 1000 words per week, every week. Just occasionally, supervisors forget to tell new writers the obvious --- the easiest sections to write are the descriptions of the results, and of what you did (because these sections are highly constrained by the facts), and you should write these first; what the results mean, and why you did what you did, are altogether more problematic, so the Discussion and Introduction sections should be written later. If you're desperate, imagine you're speaking to a friend about your work, and write down what you would say. (Then, as always, revise and edit, revise and edit, revise and edit.)
Some research has to be done to a tight deadline --- for example, the final-year projects carried out by undergraduates. The big difference this makes is that you don't have time (which you do in most situations) to get things wrong and then do them again. The consequence is that there's a good deal of luck involved at this stage, so I think that if your final-year project falls flat on its face, it doesn't mean you won't be good at research in a more normal environment.
It is only natural for a student to ask "What will get me good marks?" In assessing theses, some universities have marking schemes --- u marks are allocated for the project design, v marks are allocated for the conduct of the experiment or survey, w marks are allocated for the general writing of the thesis, x marks are allocated for the review of previous literature, y marks are allocated for the presentation of the results, z marks are allocated for the interpretation of the findings, and so on. Other universities have no such marking scheme. (Even where one exists, it is very often impracticable to use it strictly.) My advice in such a situation may be different from that of others. But, for what it's worth, here it is. My impression is that students typically spend too much time reviewing past literature, and not enough time thinking about it and criticising it. The student's ideas need to be firmly grounded in established wisdom, not plucked out of cloud cuckoo land. But the thing that will most impress the marker of a thesis is a focus on the central ideas, a willingness to criticise imprecise thinking by previous authors, the dissection of what is essential from what is less relevant, and the demonstration of how the student's ideas sharpen what has previously been blunt. In other words, intellectual oomph.
An important issue is that of what to study. If you have ideas, there's no problem. But, is there anything you can do to prompt ideas to come to you? Or, in the absence of ideas, how can you do something worthwhile? I don't suppose there's a complete answer to these questions, but the following comments are intended to be helpful.
To get ideas, expose yourself to them.
  • Talk to someone about your research. Your mum or dad, for example. Or (though I've never tried this) the top stream of 13-year-olds at your local selective high school. Brainstorming with your colleagues is better than nothing, but you're not really forced to decode your talk sufficiently. Anyway, there are other uses for your colleagues, such as being rude to you.
    It is tremendously valuable if there's someone in your circle who can bring off the trick of being rude without causing offence. Suppose you are a cognitive psychologist, and in a seminar you claim that "activation flows through the levels" of your model. You benefit if someone from another tradition confronts you with the claim that "activation", "flows", and "levels" are all meaningless. It is generally up to the young to do this.
    Informal seminars, where you listen to others, and talk yourself, are an important part of this.
  • Read widely. (But remember what I said earlier: reading is not what research is about. Indeed, I would say there is a type of personality for whom too much reading is a major danger to their ever doing anything worthwhile.)
    • How to read: read the title, skim the abstract, look at the pictures and maybe the tables, and if there's anything interesting, then consult the text, looking for that specific point. (No-one starts at the beginning of a paper and works their way through to the end.)
    • What to read: many of the "prestigious" journals are best avoided, as they have got into the habit of attending to a lot of details that everyone knows are unimportant, thus diluting the real message. The papers are polished and bland, and the reader is sucked into taking them on their own terms. You need something rougher, that you can get to grips with.
      • Comments and letters in any journal. Whatever the point is, it is made quickly. Furthermore, an area of controversy may be highlighted for you.
      • Journals from outside your main discipline that are relevant to your topic. For example, if your own discipline is psychology, see what the management, sociology, medicine, and engineering journals are saying about your topic. They may very well take sufficiently different an attitude to provoke thought. And interdisciplinary journals, e.g., on the borders of music and psychology, or law and statistics.
      • For the same reason, journals from countries that Americans have never heard of (such as Germany, India, and Japan).
      • Low prestige journals sometimes publish simple data on unusual questions --- if the topic attracts you, ideas for improving the method or generalising the results will soon come.
      • For the same reason, you can often find mental stimulation in journals from 60 or 80 years ago.
    • Your attitude: sympathetic criticism. So much junk is published that a degree of sceptical hostility is appropriate when approaching most papers (unless the author is on the Approved List). But temper this with sympathy: a paper may be useless to you, and it may even be clear that by any reasonable standard the research was a waste of time, but yet the method may be useful to one reader, a detail in the results may be just what a second reader is looking for, and a third reader may find her thoughts clarified by a point in the discussion.
      If someone says your paper is trash, retort that anyone who writes a decent paper can get it published, but it takes real talent to get rubbish into print.
  • Forget harsh realities for a moment, and think seriously about what question really interests you, gets you passionate. Even though you're trying to keep your mind on the ideal, you will find practicalities obtrude themselves. Some of these practical problems may turn into research projects.
  • Look to a leisure interest of yours for inspiration. Many students are interested in at least one of music, sport, and politics; well, if their academic subject is psychology, surely this has some interesting things to say about all of these? The same goes for many other academic subjects.
  • Many of us are lucky enough to have easy access to computerised databases. Try searching them with an unusual combination of words, e.g., statistics and music, or laughter and Wagner, or measurement and theology.
  • Ask successful researchers in your department whether they have any particular methods for getting ideas to come.
  • I find making lists, and then organising them, often useful. Lists of what? Of questions, possible experiments, possible surveys, sources of data, ways of operationalising a concept, ways in which your supervisor could be more helpful,...
  • Draw an analogy. (Theoretical, or method of analysis.)
  • Thought experiments. If the results were to be ..., then we would conclude .... On the other hand, if they were to be ..., our conclusion would be ....
Suggestions for what to do if you don't have any ideas.
  • Criticise a paper by someone else.
    • Can I improve the experiment?
    • Can I improve the theory? In statistical modelling, one might replace a discrete variable with a continuous variable, or vice versa --- e.g., if "blue" and "white" types have been proposed, replace them with a trait ("blueness").
    • Can I refine the definition?
    • Can I broaden the circumstances in which the effect occurs?
    • Can I simulate this on the computer?
    • Can I improve the statistical analysis? ("Two of a trade never agree", it is said, and certainly no two statisticians have agreed with each other this century.) Improvement does not necessarily mean additional complication. Sometimes, simple descriptive results are overlooked in the rush to perform a complicated hypothesis test. If nothing better comes to mind, try confidence intervals in place of hypothesis tests, or nonparametric methods in place of parametric, or Bayesian weight of evidence in place of a p-value.
    If you do comment on a published paper, you're actually paying it quite a high compliment. (If you come across 24-carat dross, you should follow the convention of politely ignoring it.)
  • You can always write a paper on:
    • The useability of university Calendars.
    • The comprehensibility of ergonomics textbooks.
    • The rise and fall of the use of particular cliches in the titles of learned articles.
    • Do readers want more raw data in journal articles?
    • Bayesian weight of evidence in psychology journals, 1970--1996.
    • How first-year students view research participation.
    • Comparison of several fields in respect of their tolerance of low survey response rates.
    • Why do students hate statistics?
    Get the idea?
  • Think of an experiment that no-one else has done, and do it. (The emphasis here is on doing something because it can be done, not because you actually understand the implications.)
  • The different packaging of something unoriginal.
  • Go to a journal that is less quantitative than you are, and teach the readers something (be tactful, now).
General advice.
  • If you have a rigid deadline to work to, it is vital that you are not reliant on someone else for anything important. Among the possible areas of difficulty are: administrative approval from an outside body, construction of equipment, computing. You must have a workable plan that can be implemented if approval is refused/the equipment isn't delivered/the computing expertise is unavailable.
  • Most people beginning research underestimate the importance of the physical aspects --- going through 500 files a second time because you didn't extract a particular item of information the first time, coming in to the lab on Sunday afternoon to try out an idea you've just had, driving at 4am to another city because the evening news told you of a "natural experiment" there that day, taking 60 journal volumes off the shelf to skim through for the key paragraph you wish you had made a note of, and so on.
  • Write down an idea when you have it, because you may easily have forgotten it by the time you get to the office. Some people even keep an ideas book for this purpose; but the trouble with this is that it really need to be physically very small, so that it can be always with you.
  • Work around the problem, if you can't solve it.
  • Sometimes a problem is best handled by defining it away.
  • There is some research that simply isn't worth doing. It may be obvious to the drover's dog roughly what the problem is... and that the institutional barriers to solving it are insurmountable. Surveys, especially, can be a substitute for actually doing anything about the problem.
  • Importance of flexibility.
  • Luck. Have several projects going --- diversification --- they won't all be disasters.
  • Don't be afraid to admit your ignorance. The nature of academic research is that one is continually venturing into areas where one is an ignoramus. It is unpleasant to admit this. But the earlier you do, the sooner you get an explanation and the quicker you can put that little bit of extra learning to use.
  • There can be a danger in overmuch personal involvement. Some people love people with Alzheimer's disease, or autism, or depression, and want to research the condition. For a beginning researcher, the problem of separating scientific evidence about the mass of people with the condition from the loved one's personal experience is one problem too many --- it is better to select another subject for research.
  • Perfectionism and attention to detail is a virtue. Taken to excess, it is a vice.
  • Some researchers seem to suffer from a "fear of success", which I think is different from obsessive perfectionism. I think what happens is that they perceive the successful completion of their research as the end of one stage in their life and the beginning of another, and they are reluctant to move on to the new stage.
  • Universities have become very bureaucratic about documenting the progress of research students. One can appreciate the reasons for this, and yet there is a sense in which it is all a waste of time --- if the student is successful, there's no need to have a file of routine documentation; if he or she isn't, no amount of paperwork will rescue the situation.
    The supervisor may feel the student is wasting his or her time, and yet be reluctant to say this (or say it forcefully enough) because it may actually be wrong and because in any case it will be discouraging. The supervisor may proffer clear advice, and yet the student be stubborn. If I had an answer, I'd give it here...
  • All sorts of stuff has been written about the role of the supervisor. Much of it is sensible. But it's all a long-winded way of saying that it is reasonable for the student to expect the supervisor to listen to him or her for something like 20--50 hours per year. Anything more than that is taking a responsibility away from where it properly belongs (i.e., with the student).
  • Don't miss any deadlines that have been agreed with your supervisor. If you do, then (depending on the nature of your excuse) you'll go on to either the list of people who probably won't make it, or the list of people who probably won't make it but who deserve sympathy. You do not want to be on either of these lists. And don't be late for meetings with your supervisor.
  • Don't expect much praise from your supervisor. If your supervisor thought about it, he or she would realise that a few words of praise were both justified (by the brilliance of your work) and desirable (for the benefits of positive reinforcement). Sad to say, he or she doesn't think about it: from about your second week of research, you've been accepted as a grown-up and been judged by the same standards as the professors.
  • You might find it worthwhile reading How to get a PhD. Managing the peaks and troughs of research by E M Phillips and D S Pugh (Milton Keynes: Open University Press, 1987).
    Here are three points that Phillips and Pugh make, all of which I mildly disagree with. (i) Intelligence-gathering (i.e., description of the facts) is not research. (ii) Research of the testing-out type (as contrasted with exploratory research and problem-solving research) is much the most appropriate for a PhD student. (iii) It is necessary for your thesis to have a "thesis", in the sense of a connected argument and message.
    It may be that Phillips and Pugh are substantially correct in the context of relatively mature fields of study; and that my opinion that (i) intelligence-gathering may be research, (ii) exploratory research is useful and appropriate, and (iii) the research may be too messy to have much of a "thesis" to it, is formed by my experience being mostly in relatively immature fields of study.
  • Taste. Is it permissible to use phrases like "glass ceiling" or "evidence-based medicine" or "gold standard" in reporting your research? Fastidious writers avoid cliches most of the time, but not every use of them should be condemned. (In some allegedly scholarly fields of study, they seem to be compulsory.)
  • Don't worry about the possibility of someone else duplicating your research. It is rare for this to happen, and, when it does, it is stimulating to examine the similarities and the differences.
  • Don't despair if it becomes necessary to tear up two years' work. This is quite common, and often means you are so expert and can see things so clearly that you can finish the whole project in only six more months.
  • If you do happen to discover something important, be sure to do it at the right time and place. Perhaps you remember the inventor of the Infinite Improbability Drive: "just after he was awarded the Galactic Institute's Prize for Extreme Cleverness he got lynched by a rampaging mob of respectable physicists who had finally realized that the one thing they really couldn't stand was a smartass" (The Hitch Hiker's Guide to the Galaxy, Chapter 10). Read here about an earlier savant who suffered an equally dire fate.
Applying to do research.
There is quite a lot of advice available on the WWW about how to choose which departments to apply to (for a research degree), and how to increase your chances of success. I'm a bit sceptical about the need for this --- I think that by the end of their undergraduate careers, most students know in what sub-field of the subject they want to work, or what style of approach they want to use. Furthermore, they know which departments at which universities study that sub-field or take that style of approach. Consequently, their list of desirable departments is already a short one; if they have made personal contacts, the list will be very short indeed. However, if you think you need advice, here is mine.
  • You should appreciate that there are an enormous number of specialisations available in the scholarly world. Flick through one of the directories of scholarly societies if you don't believe me. For example, there is a Colour Society of Australia, an International Association for Cross-Cultural Psychology, and a Stress and Anxiety Research Society. If some specialisation appeals to you (surely you'll like something in all your undergraduate studies!), then sometime about the middle of your undergraduate career, you should stop thinking of yourself as (for example) an embryo psychologist, and start thinking of yourself as (for example) an embryo colour psychologist. You will then only be interested in departments that are strong in colour studies.
    If you are in this position, do not limit yourself to departments of psychology. You might find the most suitable place is in a group within a chemistry or physiology department.
  • Make personal contacts. If colour psychology really does fascinate you, make contact with people in the field. Academics are enthusiasts for their own little patch, and will be delighted to find someone who shares that enthusiasm. If possible, get some lowly job in their department during the vacation. But be warned --- I would say it is impossible to successfully fake this. Alleged enthusiasm without an appropriate level of specialist knowledge and native intelligence will get you nowhere.
  • If you really are unsure --- if you just feel you'd like to do research in "psychology" or "chemistry" --- you may do better to delay your entry into research until your ideas have matured a little further. (You cannot be expected to have a fully-thought-out research project at this stage; what I mean is that you should know (for example) that your interest is in cognitive psychology studied by computer-controlled experiments on normal humans.)
A little advice of a statistical or technical nature.
This is not the real theme of the present document, but a few comments of this type may be worthwhile. You might also like to read Pitfalls of data analysis by Helberg.
  • There is often tension between the respective advantages of standardising on one particular method of answering a question, and of using a variety of different methods. Think of measuring unemployment, or television viewing, for instance. On the one hand, one wants a standard, reproducible, method --- comparisons from month to month are important, and one has reason to think that whatever defects are present remain pretty constant over time. On the other hand, any single practicable method is bound to have peculiarities and biases, and the "real, true" answer should probably be synthesised from several different methods, even if each of them makes use of only a small sample size.
    When controversies break out, one factor is often the intrusion of a new player into the system: the existing interests have learned to co-exist with each other and with the defects of a question-answering method, but this doesn't suit someone new.
    The following is very much a broad-brush statement, to which there must be numerous exceptions: too often, a standard method is adopted too early in the development of a subject. In research, one usually wants the "real, true" answer, whereas for administration, one may prefer a standardised, reproducible, answer.
    "Figures often beguile me, particularly when I have the arranging of them myself" (Mark Twain). Presumably sociologists have studied the use of statistics in the wielding of power?
  • Meta-analysis: this term refers to quantitatively combining the results of previous studies of a subject, in order to arrive at the right answer. I'm not terribly keen on the idea, but this may be because I have not worked in fields where there have been lots of previous studies of the same question. There are some difficult principles and practicalities involved, and inadequate handling of these led critics to refer to meta-analysis as being "meta-silliness". I think there have been substantial improvements in the methodology, and these were winning me over, but then I read a paper by Sohn (Clinical Psychology Review, 16, 1996, 147--156), which argues that publication bias is so serious as to vitiate all conclusions from literature reviews in the field he examines, effectiveness of psychotherapy (and I think the implication of the paper is that publication bias has a more serious impact on a meta-analysis than on a review in traditional narrative format).
  • Contrast between exploring data, and using data for hypothesis testing.
  • Maybe you want to do both. One easy technical solution is to randomly split your data into two portions, explore one half and generate hypotheses from it, and then test the hypotheses using the other half.
  • Importance of a random sample, or random allocation. I have the feeling that often it is foolish to prefer a rubbishy sample of 200 to a random sample of 20, yet often this is done.
    It might be possible to argue that a rubbishy sample of 200 is the better for hypothesis generation, whereas a random sample of 20 is the better for hypothesis testing.
  • My impression is that the subject which is leading the way as regards strictness of methodology is medicine. Many people will be familiar with the basics. In order to measure the effect of something, it is necessary to compare it with something else. That is, we have a treatment group and a control group. Patients are assigned at random to the one group or the other. It is desirable that neither the patient nor the doctor who evaluates the patient's condition know whether the patient received the drug or not. (This is termed a double-blind experiment.) But did you know this idea is being taken so far that Gotzsche (Controlled Clinical Trials, 17, 1996, 285--293) seriously suggests that the data should be blinded during statistical analysis? That is, two reports are written. The first assumes A to be the treatment and B to be the control, and the second is based on the reverse assumption. Both are completed before the code is broken. This is to avoid bias creeping in during the data analysis and writing.
  • Moderate-sized effects imply large studies. What may turn out to be the most important statistical paper of 1996 was published in the Oxford Textbook of Medicine, of all places: the convincingly-written article by Collins et al (1996) will surely encourage tens of thousands of medical professionals towards good statistical practice in their research. One of the messages is that there may be a number of treatments that are only moderately beneficial, but yet would be very worthwhile because the disease is common or the treatment is cheap or both; and which, because the treatment vs. control difference is not great, require large sample sizes in order to establish statistical significance of the difference. These sample sizes are so large that they imply collaboration between many investigators. Collins et al give clear and interesting examples of both large single clinical trials (e.g., aspirin in treating heart attacks), and of meta-analyses of many small trials (e.g., adjuvant therapy for early breast cancer treated surgically). The effect sizes in these studies were large enough to be well worth having --- a reduction in 35-day mortality from 11.8% to 9.4% among patients with acute myocardial infarction who were treated with aspirin, and a reduction in 10-year mortality from 48% to 38% among women aged 50+ with stage II breast cancer who had tamoxifen included in their treatment --- yet were also small enough that clinical trials of the usual sizes were too small to detect them.
    It would not surprise me if there were to be a trend towards simple, large, collaborative studies in many areas of scientific enquiry besides medicine. Now, consider that very many students conduct some sort of small research project as part of their studies. I suggest that if different students in different universities together decided on a common topic and worked out a common methodology (and stuck to it), then (i) the results would be of much greater value than results from a single institution, and (ii) the experience of collaborative work would itself be valuable training, if indeed we are entering an era when this will be increasingly common.
    Collins R, Peto R, Gray, R, Parish S (1996). Large-scale randomized evidence: Trials and overviews. In Weatherall D J, Ledingham J G G, Warrell D A (Editors),Oxford Textbook of Medicine. 3rd Edition, pp. 21--32. Oxford: Oxford University Press.
  • Don't get carried away with the statistical analysis. Tell the story of your research using the tools from your first course in statistics, not your last --- that is, using nothing more complicated than well-chosen descriptive techniques plus the concept of standard error. As much as is practicable, present your results in such a way that the reader can see that the conclusions from your more complicated calculations are probably correct.
    Suppose your research is experimental; and involves obtaining a measurement (e.g., an accuracy score) from each subject in each of four conditions; that the four conditions consist of each combination of two levels (e.g., big and small) of one factor with two levels (e.g., light and dark) of another factor; that the subjects are in two groups (e.g., males and females); and that interest centres on whether the average interaction effect is different for males from what it is for females. Such an experimental question would be recognised by some people as being about the three way interaction in a three-factor experiment, one factor being a between-subjects factor and two of the factors being within-subjects factors. Such people would know that a concise way of analysing the data and presenting the results would be via Analysis of Variance. I recommend: (a) Yes, do that. (b) But also, present your results in a way that enables them to be roughly checked. (Otherwise, your readers will presume you've got them wrong.) In the example, the interaction can be calculated for each subject. It is the difference between two differences, [(big, light) - (big, dark)] - [(small, light) - (small, dark)]. So calculate this for each subject, and summarise the values for the males by their mean and s.d., and the values for the females likewise. It will then be clear how similar or different are the interactions for the two genders.
    It might be worth going to quite some trouble to make sure you, your supervisor, and your examiners agree on this one. Having gone to the trouble of really understanding your data and then explaining it comprehensibly, it would be vexing to come across an examiner who wants to see 20 pages of ANOVA tables. Or vice versa.
  • Just occasionally, it may happen that careful attention to notation in writing an equation will reveal that you really did not understand something you thought you did (or even that everyone else did not understand it).
    This can happen with any type of mathematical equation, not necessarily one involving random variables. But, to give a simple example of this latter type, there might be an averaging process somewhere, and one can sometimes get confused about what domain it is that one is averaging over.