Experimental design for all 2-way interactions
Moderators: Andrew Collins, Michiel Bliemer, johnr
Re: Experimental design for all 2-way interactions
Thank you very much again, also for pointing out the Asterisk (that was a careless copy-and-paste error). I'm looking forward to the new version 2.0.
Re: Experimental design for all 2-way interactions
Dear Michiel,
I used the OOD in a pretest and encountered a few problems. The preference data did not appear as expected. However, in a previous simulation using the OOD, I was able to reproduce all assumed coefficients. I took a closer look at the design and found some conspicuous patterns.
This was my syntax:
And here is an excerpt of the design sorted by column opt1.x1.
If opt1.x1 equals 0, then opt2.x1 always equals 1, opt3.x1 always equals 3 and opt4.x1 always equals 7. Similar patterns exist for other levels. Are these normal patterns of an OOD foldover or am I missing something. I'm not sure if the unexpected results of the pretest are due to uncertain and inconsistent choices made by the participants or possibly due to the design and I should replace it. Maybe I just misunderstood the concept of the foldover design.
I have imported the design into SurveyEngine. The assignment of the choice sets is randomized. It happens that participants receive very similar sets one after the other and, for example, choose the same alternative 8 times in a row because it contains a preferred level. Maybe I'm just being a little overcautious. Because I mostly used software with integrated design creation (e.g. Lighthouse Studio). I would be very grateful for your feedback before I continue with my main survey.
Thank you very much!
Andrew
I used the OOD in a pretest and encountered a few problems. The preference data did not appear as expected. However, in a previous simulation using the OOD, I was able to reproduce all assumed coefficients. I took a closer look at the design and found some conspicuous patterns.
This was my syntax:
Code: Select all
design
;alts = opt1, opt2, opt3, opt4
;orth = ood
;rows = 144
;foldover
;model:
U(opt1) = b1.dummy[0|0|0|0|0|0|0] * x1[0,1,2,3,4,5,6,7]
+ b2.dummy[0|0|0|0|0] * x2[0,1,2,3,4,5]
+ b3.dummy[0|0|0|0|0] * x3[0,1,2,3,4,5]
/
U(opt2) = b1 * x1
+ b2 * x2
+ b3 * x3
/
U(opt3) = b1 * x1
+ b2 * x2
+ b3 * x3
/
U(opt4) = b1 * x1
+ b2 * x2
+ b3 * x3
$Code: Select all
Choice situation opt1.x1 opt1.x2 opt1.x3 opt2.x1 opt2.x2 opt2.x3 opt3.x1 opt3.x2 opt3.x3 opt4.x1 opt4.x2 opt4.x3 Foldover block
3 0 5 4 1 0 5 3 1 0 7 4 3 1
11 0 1 4 1 2 5 3 3 0 7 0 3 1
19 0 2 4 1 3 5 3 4 0 7 1 3 1
29 0 1 0 1 2 1 3 3 2 7 0 5 1
35 0 0 3 1 1 4 3 2 5 7 5 2 1
41 0 3 3 1 4 4 3 5 5 7 2 2 1
43 0 5 0 1 0 1 3 1 2 7 4 5 1
45 0 0 5 1 1 0 3 2 1 7 5 4 1
67 0 1 2 1 2 3 3 3 4 7 0 1 1
76 0 4 1 1 5 2 3 0 3 7 3 0 1
I have imported the design into SurveyEngine. The assignment of the choice sets is randomized. It happens that participants receive very similar sets one after the other and, for example, choose the same alternative 8 times in a row because it contains a preferred level. Maybe I'm just being a little overcautious. Because I mostly used software with integrated design creation (e.g. Lighthouse Studio). I would be very grateful for your feedback before I continue with my main survey.
Thank you very much!
Andrew
-
Michiel Bliemer
- Posts: 2039
- Joined: Tue Mar 31, 2009 4:13 pm
Re: Experimental design for all 2-way interactions
This pattern has nothing to do with the foldover but rather with the optimal orthogonal design (OOD) procedure proposed by Street, Burgess, and Louviere. This is the same procedure as implemented in the default orthogonal design generator in SurveyEngine (without using Ngene). In their procedure, they take the level of the first alternative, and use a design generator to create the levels of the next alternatives.
So level 0 for opt1 would always appear with level 1 for opt2, and level 1 for opt1 would always appear with level 2 for opt2, etc. Street et al have shown that this results in a design with "optimal trade-offs" as it ensures that opt1 is always different to opt2 for all attributes. Please refer to their paper.
If you have a large number of levels, like you have, you can indeed observe a "conspicuous pattern". For that reason, I generally do not use OOD designs. But you should still be able to estimate all your coefficients. You could avoid such patterns by using ;orth = seq, which avoids such strict patterns, or using ;eff = (mnl) with zero or non-zero priors.
The fact that a respondent chooses the same alternative because it contains a preferred level suggests that there is a fairly dominant attribute or alternative. You could avoid such issues by generating an efficient design with information priors, which accounts for the dominance through the priors.
Michiel
So level 0 for opt1 would always appear with level 1 for opt2, and level 1 for opt1 would always appear with level 2 for opt2, etc. Street et al have shown that this results in a design with "optimal trade-offs" as it ensures that opt1 is always different to opt2 for all attributes. Please refer to their paper.
If you have a large number of levels, like you have, you can indeed observe a "conspicuous pattern". For that reason, I generally do not use OOD designs. But you should still be able to estimate all your coefficients. You could avoid such patterns by using ;orth = seq, which avoids such strict patterns, or using ;eff = (mnl) with zero or non-zero priors.
The fact that a respondent chooses the same alternative because it contains a preferred level suggests that there is a fairly dominant attribute or alternative. You could avoid such issues by generating an efficient design with information priors, which accounts for the dominance through the priors.
Michiel
Re: Experimental design for all 2-way interactions
Thank you very much! I really appreciate your help and I apologize for the constant questions. I was a bit confused because my simulation data looked good, but the pretest showed a completely different picture. (The "conspicuous pattern" was then only due to the fact that individual participants were sometimes assigned many very similar choice sets which may make individual analysis difficult and straightliners hard to identify). I will take a closer look at my sample and re-read the Street et al paper regarding the design. This forum is an incredible resource when working with Ngene - you never stop learning! Thanks again!