errors when using ";eval"

[mattw]

Hello,

We have done our pilot and are updating our design script with revised utility weights. What we are trying to do now is evaluate a design that is slightly modified from what ChoiceMetrics generates. I go through the choice blocks and manually adjust it to eliminate dominating alternatives within a block. In environmental applications, I know other researchers that do this also. The idea is, if I show an environmental policy option that is better and at a lower price than one already seen by the respondent (or the reverse), it can create distrust with the survey respondent and lead to scenario rejection or strategic responses.

Our process is to export the ChoiceMetrics design to Excel and then make the adjustments manually, and then import it and ask ChoiceMetrics to evaluate it with the "eval" command to see how much d-error is affected. Sometimes it runs with errors and sometimes aborts. At the end of this msg is the code we are running and some of the warning messages we have gotten.

I have a few specific questions about the modifications that I am doing and using "eval":
1) Where in the syntax should "eval" be placed? The manual doesn't specify that I can see, and maybe we are placing it too early, causing the error messages.
2) When the syntax does run, sometimes the d-error seems to actually go down after my adjustments, is this possible because the manual adjustments are not constrained to level balance?
3) Is there any kind of % rule of thumb of how much d-error change is too much efficiency loss?
4) Would ChoiceMetrics consider having a "no dominating alternatives within a block" type of feature in a future release, or maybe this is possible now?

Thanks,
-Matt

*************
design

;eval = My_Design_51_v2

;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat added
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,48,75,110,165,240] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$

03:15:52 PM

Warning: Defaulting to prior values of zero for the following parameters: 'b_hab_reduc, b_oe, b_hab_info, b_tax'

03:15:52 PM

Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'

03:15:52 PM

Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policya.tax

Aborted

Something went unexpectedly wrong. You may wish to email ChoiceMetrics at contact@choice-metrics.com for assistance. It may be something as simple as an error in the syntax that Ngene did not detect, or something a little more tricky. Either way, we might be able to provide an immediate solution, and we want to fix all reported bugs in the next point release.

[Michiel Bliemer]

I have not heard of this "removing dominant alternatives within a block" process before. This assumes that respondents will actually remember all profiles in previous choice tasks. While I can understand that a dominant alternative within a choice tasks may create distrust, which is why Ngene automatically avoids them during the design generation phase, I do not see why it would create distrust if a better profile exists in a previous choice task. In each choice task, the respondent is asked to evaluate the profiles within the given choice task, not across choice tasks. But maybe I misunderstand what you are doing. As an analogy, suppose that you purchase an item from the supermarket that is on sale/discounted. The next day, the same item is no longer discounted, so the choice set is different, but the same item on the previous day is now dominant. Why would that be a problem?

The warning that priors are defaulted to zero does not make sense because you specified non-zero priors. It is hard to say why evaluating a specific design would not work. An aborted run usually happens when the design input contains symbols/formatting that it does not understand, or attribute levels are inconsistent with the specified levels in the script (e.g. using OE=30 in the corrected design).

Answers to your questions:
1) It does not matter. You can place them anywhere in the script. See Chapter 3: The basics of writing scripts in the manual (page 54, "The order in which these instructions appear is irrelevant").
2) Yes, this will happen because the original design was generated under level balance constraints. It is unlikely to happen when the design was generated using the modified Fedorov algorithm. Using more of the outer attribute levels is usually more efficient.
3) Not really. As long as the D-error is finite you can estimate the model.
4) It is currently not possible. To be honest, I am not convinced that it makes sense to check for dominance ACROSS choice tasks.

Michiel

[mattw]

Thanks Michiel for taking time to consider all of those questions,

I am successfully running some Excel modified designs now and I must have been doing something incorrect earlier, I am no longer getting the abort error.

Regarding dominant alternatives within a block, I just wanted to share a bit more about my thinking on that. Unfortunately I can't find a citation right now, but it is not my own idea. It is true that it would not matter if people really do treat each question independently. You could show the same profile on two sequential questions except with very different prices (this happens a few times in my computer-generated design, probably because it is fairly sparse in attributes & levels). However if you are presenting the choice options in the context of being a potential government policy, people expect that it is not profit maximizing, and you have worked hard to give them the best possible options for the public to consider. Government trust is already low with some populations, and so the same or worse option at a higher price doesn't sit well (if people are paying attention to the attributes and levels, which we hope they are). So the context can be seen as different from a profit-maximizing grocery store where I think people are more apt to understand it as a marketing study just looking for the maximum WTP so the grocery can earn the maximum amount.

-Matt

[Michiel Bliemer]

Thanks Matt for clarifying. I can see your point.

One "trick" that may work is to extend the number of alternatives during the design generation phase, since Ngene can ensure that none of the alternatives is dominant. Then afterwards you can put the profiles into separate choice tasks. This does not properly optimise the design for efficiency, but maybe it is reasonably efficient?

Michiel

[mattw]

Ah ok, I will consider that! Multiple options per choice task even though I will not actually pose them that way. That seems worth trying. I can then test the efficiency of placing the profiles as I plan to use them with eval. Thanks Michiel- Matt

[mattw]

Hi Michiel,

I did try your idea, but the program was not able to generate a design within 10 min and aborted, possibly because my attributes/levels are somewhat sparse. One solution of course would be for me to ask fewer than 4 questions per respondent, but because I have the ability to do so, it would be better to make use of that possibility somehow than not to, even if I cannot ask the most efficient questions (because of wanting to avoid dominance within a block).

However it could be a problem with my code, would you mind taking a look? Initially I posed this as 4 sequential choice tasks per respondent, always a binary choice between a policy and the status quo. But below I have written the script as though there are 4 policies to choose from and the status quo, just one question per respondent (one question per block), to avoid dominance.

Let me know if you have any further suggestions, and thank you in advance.

-Matt

design

;alts = (policyA, policyB, policyC, policyD, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA, policyB, policyC, policyD) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,46,75,110,162,210,264,325] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$

01:35:06 PM
Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policyb.hab_reduc, policyb.oe, policyc.hab_reduc, policyc.oe, policyd.hab_reduc, policyd.oe
01:35:06 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policyb', 'sq'
01:35:06 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'
01:35:06 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policyc', 'sq'
01:35:06 PM
Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policyd', 'sq'
01:35:06 PM
Note: Defaulting to assigning blocks with the 'minsum' method.
01:35:06 PM
Started, at Mon Jun 01 2026 13:35:06 GMT-0700 (Pacific Daylight Time)
01:35:16 PM
A valid initial random design could not be generated after approximately 10 seconds. In this time, of the 181299 attempts made, there were 0 row repetitions, 8159 alternative repetitions, and 173140 cases of dominance. There are a number of possible causes for this, including the specification of too many constraints, not having enough attributes or attribute levels for the number of rows required, and the use of too many scenario attributes. A design may yet be found, and the search will continue for 10 minutes. Alternatively, you can stop the run and alter the syntax.
01:45:06 PM
Error: Aborting the run. After approximately 10 minutes, an initial random design was not found.
01:45:06 PM
Finished, at Mon Jun 01 2026 13:45:06 GMT-0700 (Pacific Daylight Time)

[Michiel Bliemer]

The default swapping algorithm will struggle to find a design because of all the dominance constraints. But switching to the modified Fedorov algorithm resolves this. Simply add:

;alg = mfedorov

Note that this algorithm does not guarantee attribute level balance. So you could consider adding attribute level frequency constraints, such as below, although this will make it more difficult for Ngene to find a feasible design:

b_HAB_reduc[0.01] * HAB_reduc[0,10,20](8-14,8-14,8-14)

Michiel

[mattw]

Thanks Michiel. That did work! For anyone reading this I want to also mention I had a code problem I didn't notice at first, I was still using 32 rows, but switched that to 8. That way I end up with 32 profiles total, with 4 policies per choice task.

However in the process of inspecting the new design, I realized I have also been manually modifying options within a block to eliminate obvious price logic problems. For example, the respondent can sometimes impute the maximum price for a given attribute from prior profiles offered. Then on a successive question, it is clear that the price has changed. Again I think this is more of a problem for me due to the relatively sparse experiment. I am not sure if this consideration has a name other than illogical price combinations within a block. If this doesn't make sense I can post an example. In the end it seems the best option for me is the manual adjustment I had been doing with the original design (32 rows, 8 blocks, policy vs. status quo) that screens both for dominance and obviously illogical price combinations within a block.

Thanks for the help on this!

-Matt

[Michiel Bliemer]

It sounds like you have some specific rules in your design that are non-standard, so then making small manual modifications is likely your best way forward.

Michiel