Dear all,
I am building a binary choice experiment with utilities for each alternative in expected utility form.
My model assumes that I should not have main effects with one three way interaction and one two way interaction
I have two questions:
1. When I include only the interactions NGENE crashed. I included main effects with zero priors and I can run the syntax. How can I fix this bug?
2. I choose the fedorov algorithm for 100k iteration and then swap algorithm for 10 minutes. I use the output of this run as a starting point in the next round and repeat this process several times. The D error seems to lower down every time I run the first interaction. Do you think there is a problem with this strategy?
3. Finally. In your opinion, How many rows should be the minimum in this case? All attributes have four levels and I am selecting 32 rows with 8 blocks.
4. I am selecting 100 halton draws. Do you think is enough?
Here is my code:
Design
;alts = alt1, alt2
;rows = 32
;eff = (mnl,s)
;rdraws = halton(100)
;block=8
;alg = mfederov(stop=total(100000 iterations)),swap(stop=total(10 mins))
;start = swap_4.ngd
;model:
U(alt1) = b1 * A[1,5,20,30] + b2 * B[0.005,0.01,0.02,0.04]+ b3 * C[8.5,10.8,11.9,12.4] + b4 * D[2.3,4.6,6.2,6.9] + i1[u,0.5,1.5] * A * B * C + i2[u,0,1] * A * D/
U(alt2) = b1 * A + b2 * B + b3 * C + b4 * D + i1 * A * B * C + i2 * A * D
$
three way interaction with no main effects
Moderators: Andrew Collins, Michiel Bliemer, johnr
-
Michiel Bliemer
- Posts: 2057
- Joined: Tue Mar 31, 2009 4:13 pm
Re: three way interaction with no main effects
1. I do not think there is a bug, I think your utility function or priors are just very badly specified. Your attribute levels are very high for the priors you are using, leading to numerical problems. You would not be able to estimate this model. I have changed the levels for the first attribute, but I do not know whether this makes sense. if not. You probably want to keep your levels but you have to make your priors much smaller. You can just include the interactions, this is not a problem. For example:
Design
;alts = alt1, alt2
;rows = 16
;eff = (mnl,s)
;block=2
;model:
U(alt1) = i1[1] * A[0.1,0.5,0.2,0.3] * B[0.005,0.01,0.02,0.04]* C[8.5,10.8,11.9,12.4] + i2[0.5] * A * D[2.3,4.6,6.2,6.9]/
U(alt2) = i1 * A * B * C + i2 * A * D
$
2. I am not sure why you chose the modified Federov algorithm because you do not have constraints. The default algorithm just works fine. There is no need to save initial starting points.
3. Since you are only estimating very few coefficients, i would think 16 choice tasks is enough, you could block it in 2 if you like.
4. I am not sure why you are doing 100 Halton draws because you have specified your model as an MNL model. Or do you mean you want to estimate a mixed logit model? In that case you need to change mnl to rp. Further, you would want to estimate a panel model, to mnl should then be rppanel. This also means you need to add ;rep to your commands. But perhaps you intend to include Bayesian priors. Then you need to specify this correctly in the utility function, so not u2[u,0,1] but rather u2[(u,0,1)] and changing to ;eff = (mnl,s,mean). You can use 100 Halton draws for bdraws (not rdraws), or you can use ;bdraws = gauss(5) or something like that.
As you can see, there are many issues with your current syntax. I recommend looking carefully at the Ngene manual.
Design
;alts = alt1, alt2
;rows = 16
;eff = (mnl,s)
;block=2
;model:
U(alt1) = i1[1] * A[0.1,0.5,0.2,0.3] * B[0.005,0.01,0.02,0.04]* C[8.5,10.8,11.9,12.4] + i2[0.5] * A * D[2.3,4.6,6.2,6.9]/
U(alt2) = i1 * A * B * C + i2 * A * D
$
2. I am not sure why you chose the modified Federov algorithm because you do not have constraints. The default algorithm just works fine. There is no need to save initial starting points.
3. Since you are only estimating very few coefficients, i would think 16 choice tasks is enough, you could block it in 2 if you like.
4. I am not sure why you are doing 100 Halton draws because you have specified your model as an MNL model. Or do you mean you want to estimate a mixed logit model? In that case you need to change mnl to rp. Further, you would want to estimate a panel model, to mnl should then be rppanel. This also means you need to add ;rep to your commands. But perhaps you intend to include Bayesian priors. Then you need to specify this correctly in the utility function, so not u2[u,0,1] but rather u2[(u,0,1)] and changing to ;eff = (mnl,s,mean). You can use 100 Halton draws for bdraws (not rdraws), or you can use ;bdraws = gauss(5) or something like that.
As you can see, there are many issues with your current syntax. I recommend looking carefully at the Ngene manual.
Re: three way interaction with no main effects
Hi,
I agree with everything Michiel said in his post. The only thing I would add is that you can compute the minimum number of choice tasks required quite easily. The degrees of freedom for designs estimated on logit models is such that S => k*(J-1), where S is the number of tasks required, k the number of parameters,and J the number of alternatives. In your case J = 2, k = 8 (b1, b2, b3, b4, i1(low), i1(high), i2(low), i2(high)) assuming you want to estimate a MMNL model and do as Michiel suggests with the syntax. In that case, S => 8*(2-1), suggesting that you need at minimum 8 rows to invert the Hessian. This would be the absolute minimum, and you may want higher due to other reasons such as for level balance for example. I agree with Michiel that 16 would be a good number in this case however.
John
I agree with everything Michiel said in his post. The only thing I would add is that you can compute the minimum number of choice tasks required quite easily. The degrees of freedom for designs estimated on logit models is such that S => k*(J-1), where S is the number of tasks required, k the number of parameters,and J the number of alternatives. In your case J = 2, k = 8 (b1, b2, b3, b4, i1(low), i1(high), i2(low), i2(high)) assuming you want to estimate a MMNL model and do as Michiel suggests with the syntax. In that case, S => 8*(2-1), suggesting that you need at minimum 8 rows to invert the Hessian. This would be the absolute minimum, and you may want higher due to other reasons such as for level balance for example. I agree with Michiel that 16 would be a good number in this case however.
John
Re: three way interaction with no main effects
Thank you so much to both for your response!
This is very helpful. This was my first design attempt in NGENE and I guess I did a poor job.
I have a couple of follow up question, though. I don't understand why scaling down the priors (or the levels) should improve estimation. Both utility specifications should explain the same order. Ideally, I would like to preserve the priors and levels, but I guess it does not matter if I divide one level by 10.
Here is my corrected syntax following your instructions and now it runs perfectly. I am building an mnl design to run a first pilot and get better priors; and then I thought changing the specification to rppanel. Do you think mnl would be a good start or I should jump directly to rppanel?
Also, regarding the number of choice sets, I understand John explanation, but here the three way interaction will have 48 (4*4*4) combinations do you think 16 will do the trick? I wanted to present no more than 4 choice sets to each respondent, is there a cost in doing this?
Best
Cristobal
Design
;alts = alt1, alt2
;rows = 32
;eff = (mnl,s,mean)
;bdraws = gauss(5)
;block=8
;model:
U(alt1) = i1[(u,0.5,1.5)] * A[0.1,0.5,2,3] * B[0.005,0.01,0.02,0.04]* C[8.5,10.8,11.9,12.4] + i2[(u,0,1)] * A * D[2.3,4.6,6.2,6.9]/
U(alt2) = i1 * A * B * C + i2 * A * D
$
This is very helpful. This was my first design attempt in NGENE and I guess I did a poor job.
I have a couple of follow up question, though. I don't understand why scaling down the priors (or the levels) should improve estimation. Both utility specifications should explain the same order. Ideally, I would like to preserve the priors and levels, but I guess it does not matter if I divide one level by 10.
Here is my corrected syntax following your instructions and now it runs perfectly. I am building an mnl design to run a first pilot and get better priors; and then I thought changing the specification to rppanel. Do you think mnl would be a good start or I should jump directly to rppanel?
Also, regarding the number of choice sets, I understand John explanation, but here the three way interaction will have 48 (4*4*4) combinations do you think 16 will do the trick? I wanted to present no more than 4 choice sets to each respondent, is there a cost in doing this?
Best
Cristobal
Design
;alts = alt1, alt2
;rows = 32
;eff = (mnl,s,mean)
;bdraws = gauss(5)
;block=8
;model:
U(alt1) = i1[(u,0.5,1.5)] * A[0.1,0.5,2,3] * B[0.005,0.01,0.02,0.04]* C[8.5,10.8,11.9,12.4] + i2[(u,0,1)] * A * D[2.3,4.6,6.2,6.9]/
U(alt2) = i1 * A * B * C + i2 * A * D
$
Re: three way interaction with no main effects
One last comment. When I runed this syntax I got a high correlation (0.9) for attribute A in the first alternative vs the same attribute in the second alternative.
Can this be expected?
Best
Cristobal
Can this be expected?
Best
Cristobal
Re: three way interaction with no main effects
Hi Cristobal
The magnitude of the priors matter insofar as they impact upon the choice probabilities. Assume for example you have a design with two alternatives and say two attributes, say time and cost. Let time and cost be 10 and $5 for alt 1 respectively and 12.5 and $2 for alt 2. Now assume prior parameters such that for time it is -0.6 and cost -0.1. The utility for the first alternative will be in this example 0.77 and 0.23 for the second alternative. Now multiply the parameters by 10. The utilities are now -65 and -77 and the choice probabilities 1 and 0. Another way to think about this is to look at what is happening to utility on average. Let us assume the average travel time over the design is say 11.25 and average cost $3.5. If you take the first set of parameters (-0.6 and -0.1), then the average impact upon utility for time would be -6.75 (-0.6*11.25) and the average impact for cost -0.35 (-0.1*3.5). If we use the larger parameters, these become -67.5 (11.25*-0.6) and -3.5 (3.5*-0.1). Whilst the ratio of these remains constant, the differences becomes wider the larger the parameters are. In this case, travel time is completely dominating utility and it becomes worse the larger the parameters are.
In estimation, if you scale a variable by some number, the parameters will scale by the same number in the opposite direction (i.e., if you use income in units of 10s, or 100s, the parameters simply rescale to account for this). it is actually good practice to scale your data if the units are large as the magnitude of the parameter outcomes in estimation may become less precise if they are too small (the software will report parameters up to some level of precision).
Personally, I would start with an MNL and test later how it performs for a MMNL model. MMNL designs are wonderful in theory but can be impractical in, well, practice.
In terms of the minimum number of tasks, K is the number of parameters, not attribute level combinations. If you want all combinations to appear in the design, then you should use a full factorial design, not a fractional factorial. In your first syntax, you had six parameters (b1, b2, b3, b4, i1 and i2). You have not in your syntax indicated that you want these dummy or effects coded, so the number of parameters is six! if you want to dummy or effects code, then yep, you will need many many many many more choice tasks, as you will have many many many many more parameters to estimate. In that case however, you may need even more than the equation I suggested as the data matrix will be extremely space (you have lots of zeros and very few 1s) which can cause lots of problems in inverting the Hessian (but for different reasons than suggested by the equation).
With regards to your second question about correlation, I have answered this exact same question at least a half a dozen times in the other forum section. I would urge you to read those posts. If you don't have time to go through the posts, I will give you the short answer: No.
John
The magnitude of the priors matter insofar as they impact upon the choice probabilities. Assume for example you have a design with two alternatives and say two attributes, say time and cost. Let time and cost be 10 and $5 for alt 1 respectively and 12.5 and $2 for alt 2. Now assume prior parameters such that for time it is -0.6 and cost -0.1. The utility for the first alternative will be in this example 0.77 and 0.23 for the second alternative. Now multiply the parameters by 10. The utilities are now -65 and -77 and the choice probabilities 1 and 0. Another way to think about this is to look at what is happening to utility on average. Let us assume the average travel time over the design is say 11.25 and average cost $3.5. If you take the first set of parameters (-0.6 and -0.1), then the average impact upon utility for time would be -6.75 (-0.6*11.25) and the average impact for cost -0.35 (-0.1*3.5). If we use the larger parameters, these become -67.5 (11.25*-0.6) and -3.5 (3.5*-0.1). Whilst the ratio of these remains constant, the differences becomes wider the larger the parameters are. In this case, travel time is completely dominating utility and it becomes worse the larger the parameters are.
In estimation, if you scale a variable by some number, the parameters will scale by the same number in the opposite direction (i.e., if you use income in units of 10s, or 100s, the parameters simply rescale to account for this). it is actually good practice to scale your data if the units are large as the magnitude of the parameter outcomes in estimation may become less precise if they are too small (the software will report parameters up to some level of precision).
Personally, I would start with an MNL and test later how it performs for a MMNL model. MMNL designs are wonderful in theory but can be impractical in, well, practice.
In terms of the minimum number of tasks, K is the number of parameters, not attribute level combinations. If you want all combinations to appear in the design, then you should use a full factorial design, not a fractional factorial. In your first syntax, you had six parameters (b1, b2, b3, b4, i1 and i2). You have not in your syntax indicated that you want these dummy or effects coded, so the number of parameters is six! if you want to dummy or effects code, then yep, you will need many many many many more choice tasks, as you will have many many many many more parameters to estimate. In that case however, you may need even more than the equation I suggested as the data matrix will be extremely space (you have lots of zeros and very few 1s) which can cause lots of problems in inverting the Hessian (but for different reasons than suggested by the equation).
With regards to your second question about correlation, I have answered this exact same question at least a half a dozen times in the other forum section. I would urge you to read those posts. If you don't have time to go through the posts, I will give you the short answer: No.
John
Re: three way interaction with no main effects
Thank you so much John for the clear explanation.
This forum is fantastic!
Best
Cristobal
This forum is fantastic!
Best
Cristobal