errors when using ";eval"

[mattw]

Hello,

We have done our pilot and are updating our design script with revised utility weights. What we are trying to do now is evaluate a design that is slightly modified from what ChoiceMetrics generates. I go through the choice blocks and manually adjust it to eliminate dominating alternatives within a block. In environmental applications, I know other researchers that do this also. The idea is, if I show an environmental policy option that is better and at a lower price than one already seen by the respondent (or the reverse), it can create distrust with the survey respondent and lead to scenario rejection or strategic responses.

Our process is to export the ChoiceMetrics design to Excel and then make the adjustments manually, and then import it and ask ChoiceMetrics to evaluate it with the "eval" command to see how much d-error is affected. Sometimes it runs with errors and sometimes aborts. At the end of this msg is the code we are running and some of the warning messages we have gotten.

I have a few specific questions about the modifications that I am doing and using "eval":
1) Where in the syntax should "eval" be placed? The manual doesn't specify that I can see, and maybe we are placing it too early, causing the error messages.
2) When the syntax does run, sometimes the d-error seems to actually go down after my adjustments, is this possible because the manual adjustments are not constrained to level balance?
3) Is there any kind of % rule of thumb of how much d-error change is too much efficiency loss?
4) Would ChoiceMetrics consider having a "no dominating alternatives within a block" type of feature in a future release, or maybe this is possible now?

Thanks,
-Matt

*************
design

;eval = My_Design_51_v2

;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat added
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,48,75,110,165,240] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$

03:15:52 PM

Warning: Defaulting to prior values of zero for the following parameters: 'b_hab_reduc, b_oe, b_hab_info, b_tax'

03:15:52 PM

Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'

03:15:52 PM

Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policya.tax

Aborted

Something went unexpectedly wrong. You may wish to email ChoiceMetrics at contact@choice-metrics.com for assistance. It may be something as simple as an error in the syntax that Ngene did not detect, or something a little more tricky. Either way, we might be able to provide an immediate solution, and we want to fix all reported bugs in the next point release.

[Michiel Bliemer]

I have not heard of this "removing dominant alternatives within a block" process before. This assumes that respondents will actually remember all profiles in previous choice tasks. While I can understand that a dominant alternative within a choice tasks may create distrust, which is why Ngene automatically avoids them during the design generation phase, I do not see why it would create distrust if a better profile exists in a previous choice task. In each choice task, the respondent is asked to evaluate the profiles within the given choice task, not across choice tasks. But maybe I misunderstand what you are doing. As an analogy, suppose that you purchase an item from the supermarket that is on sale/discounted. The next day, the same item is no longer discounted, so the choice set is different, but the same item on the previous day is now dominant. Why would that be a problem?

The warning that priors are defaulted to zero does not make sense because you specified non-zero priors. It is hard to say why evaluating a specific design would not work. An aborted run usually happens when the design input contains symbols/formatting that it does not understand, or attribute levels are inconsistent with the specified levels in the script (e.g. using OE=30 in the corrected design).

Answers to your questions:
1) It does not matter. You can place them anywhere in the script. See Chapter 3: The basics of writing scripts in the manual (page 54, "The order in which these instructions appear is irrelevant").
2) Yes, this will happen because the original design was generated under level balance constraints. It is unlikely to happen when the design was generated using the modified Fedorov algorithm. Using more of the outer attribute levels is usually more efficient.
3) Not really. As long as the D-error is finite you can estimate the model.
4) It is currently not possible. To be honest, I am not convinced that it makes sense to check for dominance ACROSS choice tasks.

Michiel

[mattw]

Thanks Michiel for taking time to consider all of those questions,

I am successfully running some Excel modified designs now and I must have been doing something incorrect earlier, I am no longer getting the abort error.

Regarding dominant alternatives within a block, I just wanted to share a bit more about my thinking on that. Unfortunately I can't find a citation right now, but it is not my own idea. It is true that it would not matter if people really do treat each question independently. You could show the same profile on two sequential questions except with very different prices (this happens a few times in my computer-generated design, probably because it is fairly sparse in attributes & levels). However if you are presenting the choice options in the context of being a potential government policy, people expect that it is not profit maximizing, and you have worked hard to give them the best possible options for the public to consider. Government trust is already low with some populations, and so the same or worse option at a higher price doesn't sit well (if people are paying attention to the attributes and levels, which we hope they are). So the context can be seen as different from a profit-maximizing grocery store where I think people are more apt to understand it as a marketing study just looking for the maximum WTP so the grocery can earn the maximum amount.

-Matt