How many choice tasks to keep in pivot design

[neeraj85]

Hi,

I am working on developing a pivot (reference alternative) design comprising of 3 alternatives each defined by 4 attributes with 5 levels each.

I am a little bit confused on how many questions to ask a respondent.

I tried 2 cases of single block design:
1. 15 questions having a D-error = 0.2242
2. 10 questions having a D-error = 0.3364

I rolled out the design with 15 questions for initial pilot survey (since some of the relevant literature in the field of transportation also asked around the same number of questions).

However, the feedback I got from few respondents was that 15 questions was a bit too much to answer.

So, I thought of designing with 10 questions (a higher D-error was expected due to lesser questions).


I'll be glad to hear your experiences on whether it would be okay to continue with 15 questions. Or I should change to 10 questions??

I tried finding similar posts on the forum but cannot find one.

Thanks
Neeraj

[Michiel Bliemer]

Please note that this is not an Ngene question, please ask general stated choice questions in our other forum.

I would think that 15 complex choice tasks (3 alternatives and 4 attributes is quite a big table) is a bit much, we usually do not go beyond 10 unless we do face to face interviews with a personal interviewer. We have done more than 10 when the choice tasks are very simple (2 alternatives, 2 or 3 attributes). But this is just our experience and there is no general rule. If you do internet surveys, you have to limit the number of choice tasks presented to a respondent in order to avoid quick fatigue and boredom. You can always block your design to get fewer choice tasks.

[neeraj85]

My apologies for posting the question at a wrong place...

Yes. Mine is an internet based survey.

Having heard from you and the respondents who took my survey, I feel that having upto 10 questions will be ideal.

Currently, for the case of 15 scenarios, since I maintained the attribute balance property. Thus I got each attribute level repeating thrice. The D-error was equal to 0.2242

Now I changed the number of scenarios to 10 getting each attribute level to repeat twice. The D-error was equal to 0.3364

I feel that results for 10 scenarios look decent, even-though the D-error is a little bit high (this was expected).

Will there be some unforeseen issues if I proceed with 10 scenarios in my pivot design?

I tried block design too to make 2 blocks of 10 questions each. But the run is getting aborted saying the design couldn't be found. Is it advisable to use ;alg = mfederov here to make it run (as it relaxes the attribute level balance property)

P.S. The scenarios dont have any case of dominant alternative. I used asterik (*) on the alternatives.

Neeraj

[Michiel Bliemer]

It may be preferable to create 20 choice tasks and block it in two.
Are you able to post your Ngene syntax so I can have a look as to why the design cannot be found? It may indeed be because of dominance checks and then mfederov may work well. You can still impose some attribute level balance constraints on the design.

[neeraj85]

I thought of keeping a single block design for keeping it simple while designing the internet based survey. It wud become a little bit challenging to equally represent the blocks among respondents.
But I don't want to compromise on the quality of data either. Any clues on how to accomplish block design on the internet based surveys (I couldnt find some information on it in the Ngene manual).

Kindly suggest some solution between the options single block with 10 or 2 blocks with 10 each!!

Here is the script.

;alts(Cat1) = alt1*, alt2*, alt3*
;alts(Cat2) = alt1*, alt2*, alt3*
;alts(Cat3) = alt1*, alt2*, alt3*
;alts(Cat4) = alt1*, alt2*, alt3*
;alts(Cat5) = alt1*, alt2*, alt3*
;alts(Cat6) = alt1*, alt2*, alt3*

;rows = 20
;block = 2
;eff = fish(mnl,d)
;fisher(fish) = design1(Cat1[0.191], Cat2[0.163], Cat3[0.207], Cat4[0.289], Cat5[0.134], Cat6[0.016])

;model(Cat1):
U(alt1) = b2[-0.001] * tt.ref[7.5] + b3[-0.002] * tts.ref[2] + b4[-0.003] * sn.ref[5] + b5[-0.004] * vr.ref[0.85] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%]

;model(Cat2):
U(alt1) = b2[-0.001] * tt.ref[22.5] + b3[-0.002] * tts.ref[6] + b4[-0.003] * sn.ref[12] + b5[-0.004] * vr.ref[2.45] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%]

;model(Cat3):
U(alt1) = b2[-0.001] * tt.ref[45] + b3[-0.002] * tts.ref[11] + b4[-0.003] * sn.ref[20] + b5[-0.004] * vr.ref[5.3] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%]

;model(Cat4):
U(alt1) = b2[-0.001] * tt.ref[75] + b3[-0.002] * tts.ref[19] + b4[-0.003] * sn.ref[30] + b5[-0.004] * vr.ref[12.0] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%]

;model(Cat5):
U(alt1) = b2[-0.001] * tt.ref[105] + b3[-0.002] * tts.ref[26] + b4[-0.003] * sn.ref[42] + b5[-0.004] * vr.ref[19.3] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%]

;model(Cat6):
U(alt1) = b2[-0.001] * tt.ref[135] + b3[-0.002] * tts.ref[34] + b4[-0.003] * sn.ref[50] + b5[-0.004] * vr.ref[21.0] /
U(alt2) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] /
U(alt3) = b2[-0.001] * tt.piv[-20%, -10%, 0%, 10%, 20%] + b3[-0.002] * tts.piv[-50%, -25%, 0%, 25%, 50%] + b4[-0.003] * sn.piv[-50%, -25%, 0%, 25%, 50%] + b5[-0.004] * vr.piv[-25%, -12.5%, 0%, 12.5%, 25%] $

[Michiel Bliemer]

I just checked, and the modified Federov algorithm is unfortunately now compatible with pivot designs. Your design is heavily constrained due to the dominancy requirements, which makes it difficult for the swapping algorithm to find a design without any dominant alternatives.

Using 10 choice tasks should work well in your case because you have 3 alternatives, but if you would rather have 20 choice tasks you can simply generate two efficient designs with 10 rows. They will likely have different choice tasks, although a few may overlap (which is not a problem because you give them to different respondents).

Michiel

[neeraj85]

Yes. I agree that it would be better to generate 2 blocks of 10 scenarios each.
As pointed by you, the modified Federov algorithm doesn't seem to work because of heavily constrained pivot design.

When you say "but if you would rather have 20 choice tasks you can simply generate two efficient designs with 10 rows", do you mean that I generate 20 rows using the default search algorithm and then manually block these 20 rows into 2 blocks of 10 each?

Please confirm.

Thanks
Neeraj

[Michiel Bliemer]

No, Ngene is not able to find 20 rows with the default swapping algorithm. So I suggest simply running Ngene to find a design with 10 rows. And then again find a design with 10 rows. Since there are 5^8 = 390,625 possible choice tasks, it is unlikely to find exactly the same choice tasks (although a few may be the same, which is not a problem), so in the end you end up with 20 efficient choice tasks.

[neeraj85]

I generated 2 blocks of pivot design following the steps explained by you. Running the script twice with default search algorithm.

I now have two distinct blocks of 10 tasks each. Also, there is no overlap of tasks between the two blocks.

Hope this works

Thanks a lot for you valuable advice sir.