errors when using ";eval"

[mattw]

Hello,

We have done our pilot and are updating our design script with revised utility weights. What we are trying to do now is evaluate a design that is slightly modified from what ChoiceMetrics generates. I go through the choice blocks and manually adjust it to eliminate dominating alternatives within a block. In environmental applications, I know other researchers that do this also. The idea is, if I show an environmental policy option that is better and at a lower price than one already seen by the respondent (or the reverse), it can create distrust with the survey respondent and lead to scenario rejection or strategic responses.

Our process is to export the ChoiceMetrics design to Excel and then make the adjustments manually, and then import it and ask ChoiceMetrics to evaluate it with the "eval" command to see how much d-error is affected. Sometimes it runs with errors and sometimes aborts. At the end of this msg is the code we are running and some of the warning messages we have gotten.

I have a few specific questions about the modifications that I am doing and using "eval":
1) Where in the syntax should "eval" be placed? The manual doesn't specify that I can see, and maybe we are placing it too early, causing the error messages.
2) When the syntax does run, sometimes the d-error seems to actually go down after my adjustments, is this possible because the manual adjustments are not constrained to level balance?
3) Is there any kind of % rule of thumb of how much d-error change is too much efficiency loss?
4) Would ChoiceMetrics consider having a "no dominating alternatives within a block" type of feature in a future release, or maybe this is possible now?

Thanks,
-Matt

*************
design

;eval = My_Design_51_v2

;alts = (policyA, sq)
;rows = 32
;block = 8
;eff = (mnl,d)
;model:
U(policyA) = b_HAB_reduc[0.01] * HAB_reduc[0,10,20] ? sqmi of HAB area reduced
+ b_OE[0.01] * OE[0,10,20] ? sqmi of high qual habitat added
+ b_HAB_info[0.3] * HAB_info[0,1] ? 0 = low (base), 1 = high
+ b_tax[-0.0035] * tax[22,48,75,110,165,240] ? annual tax ($)
/
U(sq) = b_sq[-0.14]
+ b_HAB_reduc * HAB_reduc_sq[0]
+ b_OE * OE_sq[0]
+ b_HAB_info * HAB_info_sq[0]
+ b_tax * tax_sq[0]
$

03:15:52 PM

Warning: Defaulting to prior values of zero for the following parameters: 'b_hab_reduc, b_oe, b_hab_info, b_tax'

03:15:52 PM

Warning: Two alternatives were specified for alternative repetition checking, but do not have the same attribute names, and so will not be checked. 'policya', 'sq'

03:15:52 PM

Warning: One or more attributes will not have level balance with the number of rows specified: policya.hab_reduc, policya.oe, policya.tax

Aborted

Something went unexpectedly wrong. You may wish to email ChoiceMetrics at contact@choice-metrics.com for assistance. It may be something as simple as an error in the syntax that Ngene did not detect, or something a little more tricky. Either way, we might be able to provide an immediate solution, and we want to fix all reported bugs in the next point release.

[Michiel Bliemer]

I have not heard of this "removing dominant alternatives within a block" process before. This assumes that respondents will actually remember all profiles in previous choice tasks. While I can understand that a dominant alternative within a choice tasks may create distrust, which is why Ngene automatically avoids them during the design generation phase, I do not see why it would create distrust if a better profile exists in a previous choice task. In each choice task, the respondent is asked to evaluate the profiles within the given choice task, not across choice tasks. But maybe I misunderstand what you are doing. As an analogy, suppose that you purchase an item from the supermarket that is on sale/discounted. The next day, the same item is no longer discounted, so the choice set is different, but the same item on the previous day is now dominant. Why would that be a problem?

The warning that priors are defaulted to zero does not make sense because you specified non-zero priors. It is hard to say why evaluating a specific design would not work. An aborted run usually happens when the design input contains symbols/formatting that it does not understand, or attribute levels are inconsistent with the specified levels in the script (e.g. using OE=30 in the corrected design).

Answers to your questions:
1) It does not matter. You can place them anywhere in the script. See Chapter 3: The basics of writing scripts in the manual (page 54, "The order in which these instructions appear is irrelevant").
2) Yes, this will happen because the original design was generated under level balance constraints. It is unlikely to happen when the design was generated using the modified Fedorov algorithm. Using more of the outer attribute levels is usually more efficient.
3) Not really. As long as the D-error is finite you can estimate the model.
4) It is currently not possible. To be honest, I am not convinced that it makes sense to check for dominance ACROSS choice tasks.

Michiel