Constraints and correlation issues

[Sharon McLaughlin999]

Hi everyone,

We have an unlabelled design with 4 attributes and 3 alternatives. In our survey, participants will be split into different scenarios but the choice cards will remain the same for everyone. We want to see how being in these different scenarios affect their preferences.

We believe we may have to use a constraint as one of the combinations of levels isn’t feasible in the real world. We have a time attribute (TimeToCompletion), which is continuous with levels of 6, 9, and 12 months, and a categorical attribute (FullAnalysisofHTADomains) with three levels. Our goal is to ensure that when the time attribute equals 6 months, the third level of FullAnalysisofHTADomains is not allowed to appear in the same alternative.

When I run the syntax in Ngene, the D-error is 0.28, which is quite similar to the value we obtained without the constraint. However, as expected, the correlation between some attributes has increased. In particular, I’m concerned about some of the Pearson correlation coefficients being quite high in magnitude — for example, in alternative 2, the correlation between the time attribute and the full analysis of HTA domains is -0.883452. At what point should I consider this correlation too high and reconsider applying the constraint? We had correlations of around 0.6 before we applied it. I've provided my syntax below, if you spot any issues let me know. We did conduct a small pilot study which is why I've estimated some of the priors as negative. For example, when I used level 3 of the full analysis of HTA domains as the reference level, levels 1 and 2 were negative (I hope I've estimated the priors correctly from this).

My syntax is as follows:
Design
;alts=alt1,alt2,alt3
;rows=12
;eff = (mnl, d)
;alg=mfederov
;reject:
alt1.TimetoCompletion = 6 and alt1.FullAnalysisofHTADomains = 3, alt2.TimetoCompletion = 6 and alt2.FullAnalysisofHTADomains = 3, alt3.TimetoCompletion = 6 and alt3.FullAnalysisofHTADomains = 3;
;model:
U(alt1) = b0[0] + b1.dummy[-0.01|-0.02]*FullAnalysisofHTADomains[1,2,3] + b2.dummy[-0.01|-0.02]*SimplifiedHTADomains[1,2,3] + b3[-0.05]*TimetoCompletion[6,9,12] + b4.dummy[0|0]*StakeholderEngagement[1,2,3]/
U(alt2) = b1.dummy*FullAnalysisofHTADomains + b2.dummy*SimplifiedHTADomains + b3*TimetoCompletion + b4.dummy*StakeholderEngagement/
U(alt3) = b1.dummy*FullAnalysisofHTADomains + b2.dummy*SimplifiedHTADomains + b3*TimetoCompletion + b4.dummy*StakeholderEngagement
$


Thanks so much for your time!

[Michiel Bliemer]

There is usually only a concern with correlations if the D-error is very high (much larger than 1). When using generic attributes and coefficients, as in your case, it is only correlations WITHIN EACH ALTERNATIVE, that matters. Correlations across alternatives are not really of concern (and in optimal orthogonal designs are often even -1 or +1, indicating perfect correlations). Within each alternative, I do not see any concerning correlations, so I think that you are fine.

Regarding your priors, b1.dummy[-0.01|-0.02]*FullAnalysisofHTADomains[1,2,3] indicates that level 3 is most preferred (utility 0), then level 1 (utility -0.01), and then level 2 (utility -0.02).
Note that indicating such a preference order with near-zero priors is really only useful when checking for dominant alternatives. So only when you use ;alts = alt1*, alt2*, alt3* the preference order really matters, otherwise you can just set all priors to zero.

Note that the modified Federov algorithm does not guarantee attribute level balance, and will often exhibit low frequencies for "inner" levels of numerical attributes.
In your case, you may want to consider using b3[-0.05]*TimetoCompletion[6,9,12](4,4,4) to indicate that each level should appear 4 times across 12 rows. Or you can dummy code this attribute during the design phase (not necessarily in the estimation phase), which will automatically guarantee a high degree of attribute level balance.

Note that the semi-colon (;) at the end of your reject constraints should not appear there, you can remove it.

Michiel

[Sharon McLaughlin999]

Thanks very much for replying Michiel, it is really appreciated.

Thanks for the distinction in correlation, I understand this now. I have updated my syntax based on your feedback (added 4,4,4, reset categorical priors to 0). I've reran it and our D-error is 0.304 which is fine. Correlation matrix also looks good.

When I stopped the design I got the following error: [Modified Fedorov] ERROR: Unable to find a design that satisfies the specified attribute level frequency constraints. You may need to relax some attribute level constraints, combinatorial constraints (if specified), or dominance checks. The candidate set has 2000 rows.

Before I applied the constraint, I noticed that the categorical variables always varied across the alternatives within each choice task. For example, levels 1, 2, and 3 of the 'full analysis of HTA domains' attribute would typically all be present in a single choice task—just in a random order across the alternatives. However, the continuous variable 'time' behaved differently: not all levels appeared in each task. For instance, one choice task might include time values of 6, 9, and 6 months, while another might show 6, 9, and 12 months. Is this typical for continuous variables? I would still technically have level balance across all the tasks, but they showed up differently for the continuous variable.

Obviously it's not ideal to not have level balance, but I do think the constraint is important here. It seems that level 1 of all categorical variables (Full analysis, simplified, and stakeholder engagement) appears 15 times across all cards in alternative 2, in comparison to 9 times for level 3 for all categorical variables. Is there anything extra I can put into the syntax to level it a little more, or is this just an accepted caveat of the constraint? My syntax is:

Design
;alts=alt1,alt2,alt3
;rows=12
;eff = (mnl, d)
;alg=mfederov
reject:
alt1.TimetoCompletion = 6 and alt1.FullAnalysisofHTADomains = 3, alt2.TimetoCompletion = 6 and alt2.FullAnalysisofHTADomains = 3, alt3.TimetoCompletion = 6 and alt3.FullAnalysisofHTADomains = 3;
;model:
U(alt1) = b0[0] + b1.dummy[0|0]*FullAnalysisofHTADomains[1,2,3] + b2.dummy[0|0]*SimplifiedHTADomains[1,2,3] + b3[-0.05]*TimetoCompletion[6,9,12](4,4,4) + b4.dummy[0|0]*StakeholderEngagement[1,2,3]/
U(alt2) = b1.dummy*FullAnalysisofHTADomains + b2.dummy*SimplifiedHTADomains + b3*TimetoCompletion + b4.dummy*StakeholderEngagement/
U(alt3) = b1.dummy*FullAnalysisofHTADomains + b2.dummy*SimplifiedHTADomains + b3*TimetoCompletion + b4.dummy*StakeholderEngagement
$

Thanks again,
Sharon

[Michiel Bliemer]

The script runs for me (but needs a ";" in front of reject). You may sometimes get an error message if it cannot find a design that satisfies exactly those attribute level frequencies. In that case, you can relax it, e.g. (5-7,5-7,5-7) instead of (4,4,4).
You can do the same for the other attributes if you think that the attribute level frequency is too unbalanced. Note that attribute level balance is not required for model estimation, and minimising the D-error automatically satisfies a sufficient level of balance for categorical variables. But for numerical variables I usually recommend dummy coding all of them when using uninformative priors.

Michiel

[Sharon McLaughlin999]

Great, thanks very much for your help Michiel, it is really appreciated!