Hello, thank you for offering this forum. It has been some time since I have used Ngene and on behalf of my study team I wanted to post a question on a national survey we are working on. There are 3 attributes besides cost, the 1st is a negative on utility with 3 levels, the 2nd is a positive on utility also with 3 levels, the 3rd is also positive and is binary. From focus group pretests we have set priors on these so that their influence on utility is approximately equal. We have also set cost levels such that the middle cost level would approximately balance the maximum improvement offered. We borrowed some syntax from the stormwater demo project in Ngene.
We have 4 questions related to this design:
1) Is there any guidance on how to set cost levels other than offering a decent spread above and below the level that would exactly offset a "maximum" policy change? I have heard the lowest cost should be attractive to most respondents and the highest cost should deter almost everyone, but wondering if there are other factors practitioners use, including deciding on the total number of cost levels to offer. More levels seems like it would be easier to avoid dominance problems.
2) The below script does not run. However if I change "tax[24,56..] below to "tax[0,56,... ], in other words just replace 24 with 0, it does run. However I do not plan to offer policyA at zero cost. So I am wondering if there is another problem with my script that I am not understanding, please advise if anyone recognizes a problem here. I am trying to get the script to run without the need to add a 0 cost option for PolicyA.
3) The choice experiment is fairly sparse, and I am wondering if 32 runs is overly high (8 blocks 4 questions each). More runs means more statistical variability but also more dominance problems, are there rules of thumb here?
4) I was interested in the "s" efficiency option because it seemed it would have a more direct interpretation than "d" efficiency, such as to sample size but the "s" measure of 0.95, I am not sure if it has a direct interpretation.
Thanks much for any advice! -Matt
design
;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,s)
;alg = mfedorov(candidates = 68)
;require:
sq.HAB_area = 100, sq.OE = 0, sq.HAB_info = 0, sq.tax = 0
;model:
U(policyA) = b_HAB_area[-0.2] * HAB_area[100,50,0] ? sqmi of HAB area
+ b_OE[0.1] * OE[0,50,100] ? sqmi of high qual habitat
+ b_HAB_info[10] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.1] * tax[24,56,112,173,240] ? 0, 68, 127, 173, 246 annual tax ($)
/
U(sq) = b_sq[0]
+ b_HAB_area * HAB_area
+ b_OE * OE
+ b_HAB_info * HAB_info
+ b_tax * tax
$
Setting cost levels in Ngene
Moderators: Andrew Collins, Michiel Bliemer, johnr
-
Michiel Bliemer
- Posts: 2062
- Joined: Tue Mar 31, 2009 4:13 pm
Re: Setting cost levels in Ngene
1) There is no definite guidance on which attribute levels to select. It does not hurt to increase the number of levels of a numerical attribute, especially if the range is large. What you want to avoid is that going from one price level to the next makes one of the alternatives "too expensive" compared to the other. Since different respondents have different willingness to pay, it is therefore often a good idea to have a sufficient number of "steps" in your price attribute.
2) You are using the same attribute name "tax" for the status quo and try to set it to zero, but you defined this attribute without zero. There are two solutions: (i) you define tax[0,24,56,...etc} and then impose require: sq.tax = 0 and reject: policyA.tax = 0, (ii) you give the tax attribute a different name, so you can give it different attribute levels, so simply tax_sq[0]. The second option is of course much easier, as adopted in the script below. You can do the same for all other attributes and remove the require constraints. You would need require constraints if you have categorical attributes.
3) There is usually no issue in using a large number of choice tasks in your design. A design can be too small, but not too large. You probably do not need 32 rows, but it is fine.
4) An "S-error" of 0.95 indicates that essentially a single respondent answering 32 questions would suffice to estimate your 5 parameters. Since you have 8 blocks, you would need to multiply the number of respondents by 8, so 0.95*8 is about 8 respondents to estimate all parameters at a statistically significant level. I generally advise against using S-efficiency as an optimisation criterion because it only optimises on the most difficult to estimate parameter. The D-error optimises over all parameters equally. So I would recommend using ;eff = (mnl,d) and then simply look at the S-estimates in the Ngene output to see how many respondents you need for each parameter to be statistically signifcant (again multiplying each value with 8).
IMPORTANT: Your priors are really bad: they are far too large compared to your attribute levels. Bad priors will lead to bad (inefficient) designs. If you insist on using informative priors without conducting a pilot study, then I would recommend dividing all of them by 10. This will likely give you very different (but more realistic) sample size estimates.
Other comments:
1. You may want to use PolicyA, PolicyB, SQ. This would increase variation and information in your data. But of course it makes the choice tasks more difficult for the respondent.
2. It may be best to use the default swapping algorithm, which maintains attribute level balance. See script below.
Finally, it may be best to post questions that contain Ngene syntax into the relevant Ngene forums (in your case Ngene Online) instead of the general forum, so I know which Ngene version you are using (as the syntax varies slightly).
Michiel
2) You are using the same attribute name "tax" for the status quo and try to set it to zero, but you defined this attribute without zero. There are two solutions: (i) you define tax[0,24,56,...etc} and then impose require: sq.tax = 0 and reject: policyA.tax = 0, (ii) you give the tax attribute a different name, so you can give it different attribute levels, so simply tax_sq[0]. The second option is of course much easier, as adopted in the script below. You can do the same for all other attributes and remove the require constraints. You would need require constraints if you have categorical attributes.
3) There is usually no issue in using a large number of choice tasks in your design. A design can be too small, but not too large. You probably do not need 32 rows, but it is fine.
4) An "S-error" of 0.95 indicates that essentially a single respondent answering 32 questions would suffice to estimate your 5 parameters. Since you have 8 blocks, you would need to multiply the number of respondents by 8, so 0.95*8 is about 8 respondents to estimate all parameters at a statistically significant level. I generally advise against using S-efficiency as an optimisation criterion because it only optimises on the most difficult to estimate parameter. The D-error optimises over all parameters equally. So I would recommend using ;eff = (mnl,d) and then simply look at the S-estimates in the Ngene output to see how many respondents you need for each parameter to be statistically signifcant (again multiplying each value with 8).
IMPORTANT: Your priors are really bad: they are far too large compared to your attribute levels. Bad priors will lead to bad (inefficient) designs. If you insist on using informative priors without conducting a pilot study, then I would recommend dividing all of them by 10. This will likely give you very different (but more realistic) sample size estimates.
Other comments:
1. You may want to use PolicyA, PolicyB, SQ. This would increase variation and information in your data. But of course it makes the choice tasks more difficult for the respondent.
2. It may be best to use the default swapping algorithm, which maintains attribute level balance. See script below.
Finally, it may be best to post questions that contain Ngene syntax into the relevant Ngene forums (in your case Ngene Online) instead of the general forum, so I know which Ngene version you are using (as the syntax varies slightly).
Code: Select all
design
;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_area[-0.02] * HAB_area[100,50,0] ? sqmi of HAB area
+ b_OE[0.01] * OE[0,50,100] ? sqmi of high qual habitat
+ b_HAB_info[1] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.01] * tax[24,56,112,173,240] ? 0, 68, 127, 173, 246 annual tax ($)
/
U(sq) = b_sq[0]
+ b_HAB_area * HAB_area_sq[100]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$Re: Setting cost levels in Ngene
Thanks Michiel for the feedback, we appreciate it!
Something I am not grasping is why scaling all of the priors up or down by the same factor would matter. Conceptually I am thinking that I am setting my medium cost level offered to approximately balance out the maximum policy offered. If the prior on cost is on the one side of the equation, with a negative effect on utility, and the policy attributes are on the other side of the equation, with positive effects on utility, if they balance each other why does scaling everything up or down by the same factor matter? If my prior on cost is -1, a $100 cost will be -100, so for someone to choose the policy it would have to generate at least 100 positive points of utility. It seems I could just scale priors on both sides and would keep the same balance.
We would probably be ok doing noninformative priors and let the pilot study generate priors, but I wanted to understand what I am missing here. Is it simply desirable from an optimization point of view to have priors be small relative to attribute numbers? Sorry if this is in the manual and I missed it.
We have qualitative data from several focus groups that the three policy attributes are valued approximately equally and that the full policy is valued at approximately $100. If I believe that qualitative information, I am trying to generate a design script that would reflect that.
Also thanks for the syntax help - if I have more questions there I will post on the Ngene forum!
-Matt
Something I am not grasping is why scaling all of the priors up or down by the same factor would matter. Conceptually I am thinking that I am setting my medium cost level offered to approximately balance out the maximum policy offered. If the prior on cost is on the one side of the equation, with a negative effect on utility, and the policy attributes are on the other side of the equation, with positive effects on utility, if they balance each other why does scaling everything up or down by the same factor matter? If my prior on cost is -1, a $100 cost will be -100, so for someone to choose the policy it would have to generate at least 100 positive points of utility. It seems I could just scale priors on both sides and would keep the same balance.
We would probably be ok doing noninformative priors and let the pilot study generate priors, but I wanted to understand what I am missing here. Is it simply desirable from an optimization point of view to have priors be small relative to attribute numbers? Sorry if this is in the manual and I missed it.
We have qualitative data from several focus groups that the three policy attributes are valued approximately equally and that the full policy is valued at approximately $100. If I believe that qualitative information, I am trying to generate a design script that would reflect that.
Also thanks for the syntax help - if I have more questions there I will post on the Ngene forum!
-Matt
Re: Setting cost levels in Ngene
P.S.
As a followup to my last post, I just wanted to mention that i do understand that the prior on cost is related to the marginal utility of income, and therefore there is a "correct" value for the prior. However I was thinking that this wouldn't change an efficient design, so long as the cost prior is scaled appropriately with respect to the priors for the other CE attributes. Is that correct? In other words, if cost is twice as important, so long as the other attributes are also scaled twice as important, the relative effect on utility should be the same.
-Matt
As a followup to my last post, I just wanted to mention that i do understand that the prior on cost is related to the marginal utility of income, and therefore there is a "correct" value for the prior. However I was thinking that this wouldn't change an efficient design, so long as the cost prior is scaled appropriately with respect to the priors for the other CE attributes. Is that correct? In other words, if cost is twice as important, so long as the other attributes are also scaled twice as important, the relative effect on utility should be the same.
-Matt
Re: Setting cost levels in Ngene
Ok I think I understand now - no need to reply to my prior post.
In estimation, the model determines the numeric disutility of an extra unit of cost, because this will show up in choice probabilities, as a factor independent of the other choice experiment variables. That is how cost gets its own coefficient (scaled to the population and currency used), and that coefficient therefore has a bearing on creating the most efficient design. That is why the absolute value of the cost coefficient matters, when setting priors. Again - we appreciate all the feedback in your initial reply.
-Matt
In estimation, the model determines the numeric disutility of an extra unit of cost, because this will show up in choice probabilities, as a factor independent of the other choice experiment variables. That is how cost gets its own coefficient (scaled to the population and currency used), and that coefficient therefore has a bearing on creating the most efficient design. That is why the absolute value of the cost coefficient matters, when setting priors. Again - we appreciate all the feedback in your initial reply.
-Matt