Mistake when translating design into survey

[sj_tan_]

Hi all,

Apologies if this has been covered elsewhere but it's difficult to search for.

What kinds of problems occur if you make a mistake when putting a design into a survey?

A specific example: In a design with 4 blocks and 4 questions per block, block 1 question 1 is substituted for block 2 question 1 so that it appears as the first question for participants in both block 1 and block 2.

Another example: In the same design, block 1 question 1 and block 2 question 1 are swapped.

Is the data still usable in either case -- are results reliable? Can we say anything about the properties of the design?


Thanks so much.

[Michiel Bliemer]

Swapping questions or having some duplicate questions is usually no problem and in most cases you can still estimate all your models. Issues may arise if you have a small number of different questions that were asked to respondents.

You may lose some of the properties of the design. For example, if it was orthogonal, it will no longer be orthogonal. If it was attribute level balanced, it may no longer be. If it was D-optimal, it may no longer be. But none of these properties are necessary, they may be nice to have but they are not needed to estimate your model, so if you can estimate your model with the data you collected you need not worry. Even designs with random choice tasks work.

Michiel

[sj_tan_]

Thanks so much for the advice! So essentially you lose the ability to claim anything about the properties of the design (e.g. we use a d-optimal design) but there isn't a need for worry regarding reporting model results.

[Michiel Bliemer]

You can test whether the D-error of the design (divided by the number of respondents) is similar to the D-error in the data. At least you can see how much efficiency you lost, maybe it is not too much.

Michiel