errors when using ";eval"
Posted: Fri May 22, 2026 1:47 am
Hello,
We have done our pilot and are updating our design script with revised utility weights. What we are trying to do now is evaluate a design that is slightly modified from what ChoiceMetrics generates. I go through the choice blocks and manually adjust it to eliminate dominating alternatives within a block. In environmental applications, I know other researchers that do this also. The idea is, if I show an environmental policy option that is better and at a lower price than one already seen by the respondent (or the reverse), it can create distrust with the survey respondent and lead to scenario rejection or strategic responses.
Our process is to export the ChoiceMetrics design to Excel and then make the adjustments manually, and then import it and ask ChoiceMetrics to evaluate it with the "eval" command to see how much d-error is affected. Sometimes it runs with errors and sometimes aborts. At the end of this msg is the code we are running and some of the warning messages we have gotten.
I have a few specific questions about the modifications that I am doing and using "eval":
1) Where in the syntax should "eval" be placed? The manual doesn't specify that I can see, and maybe we are placing it too early, causing the error messages.
2) When the syntax does run, sometimes the d-error seems to actually go down after my adjustments, is this possible because the manual adjustments are not constrained to level balance?
3) Is there any kind of % rule of thumb of how much d-error change is too much efficiency loss?
4) Would ChoiceMetrics consider having a "no dominating alternatives within a block" type of feature in a future release, or maybe this is possible now?
Thanks,
-Matt
*************
design
;eval = My_Design_51_v2
;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat added
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,48,75,110,165,240] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$
03:15:52 PM
Warning: Defaulting to prior values of zero for the following parameters: 'b_hab_reduc, b_oe, b_hab_info, b_tax'
03:15:52 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'
03:15:52 PM
Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policya.tax
Aborted
Something went unexpectedly wrong. You may wish to email ChoiceMetrics at contact@choice-metrics.com for assistance. It may be something as simple as an error in the syntax that Ngene did not detect, or something a little more tricky. Either way, we might be able to provide an immediate solution, and we want to fix all reported bugs in the next point release.
We have done our pilot and are updating our design script with revised utility weights. What we are trying to do now is evaluate a design that is slightly modified from what ChoiceMetrics generates. I go through the choice blocks and manually adjust it to eliminate dominating alternatives within a block. In environmental applications, I know other researchers that do this also. The idea is, if I show an environmental policy option that is better and at a lower price than one already seen by the respondent (or the reverse), it can create distrust with the survey respondent and lead to scenario rejection or strategic responses.
Our process is to export the ChoiceMetrics design to Excel and then make the adjustments manually, and then import it and ask ChoiceMetrics to evaluate it with the "eval" command to see how much d-error is affected. Sometimes it runs with errors and sometimes aborts. At the end of this msg is the code we are running and some of the warning messages we have gotten.
I have a few specific questions about the modifications that I am doing and using "eval":
1) Where in the syntax should "eval" be placed? The manual doesn't specify that I can see, and maybe we are placing it too early, causing the error messages.
2) When the syntax does run, sometimes the d-error seems to actually go down after my adjustments, is this possible because the manual adjustments are not constrained to level balance?
3) Is there any kind of % rule of thumb of how much d-error change is too much efficiency loss?
4) Would ChoiceMetrics consider having a "no dominating alternatives within a block" type of feature in a future release, or maybe this is possible now?
Thanks,
-Matt
*************
design
;eval = My_Design_51_v2
;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat added
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,48,75,110,165,240] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$
03:15:52 PM
Warning: Defaulting to prior values of zero for the following parameters: 'b_hab_reduc, b_oe, b_hab_info, b_tax'
03:15:52 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'
03:15:52 PM
Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policya.tax
Aborted
Something went unexpectedly wrong. You may wish to email ChoiceMetrics at contact@choice-metrics.com for assistance. It may be something as simple as an error in the syntax that Ngene did not detect, or something a little more tricky. Either way, we might be able to provide an immediate solution, and we want to fix all reported bugs in the next point release.