choice-metrics.com

Hi,

When I run a D-efficient design in Ngene, I often encounter situations where some attribute levels overlap between the two alternatives. For example, after running the code below and allowing it to process for some time:

? Bayesian efficient design
Design
; alts = alt1*, alt2*, alt3
; rows = 32
; eff = (mnl, d)
; block = 4
; model:
U (alt1) = b0[0] + b1.dummy[(u, -1.2, -0.7)|(u, -0.7, -0.15)|(u, -0.15, 0)]*betaone[1,2,3,4] + b2.dummy[(u, -1.7, -1)|(u, -1, -0.5)|(u, -0.6, -0.2)]*betatwo[1,2,3,4] + b3.dummy[(u, -0.5, -0.1)]*betathree[1,2] + b4.dummy[(u, -1.8, -1)|(u, -1.1, -0.5)|(u, -0.6, -0.1)]*betafour[1,2,3,4] + b5.dummy[(u, -1, -0.5) |(u, -0.5, -0.2)|(u, -0.3, -0.05)]*betafive[1,2,3,4] + b6.dummy[(u, -0.75, -0.4) |(u, -0.35, -0.01)]*betasix[1,2,3] + b_cost[(u, -0.000010, -0.000006)]*betaseven[0,5000,15000,30000,50000,100000] /

U (alt2) = b0[0] + b1.dummy*betaone + b2.dummy*betatwo + b3.dummy*betathree + b4.dummy*betafour + b5.dummy*betafive + b6.dummy*betasix + b_cost*betaseven

$

I typically select the resulting design with the lowest D-error (around 0.3). However, when I review the 32 choice tasks, I often observe overlapping levels between the two alternatives for some attributes. In my case, there are seven attributes, and the resulting design may include overlapping levels for each of them. In a few of the 32 choice tasks, this issue appears more pronounced, with, for example, three overlapping attribute levels between the two alternatives within a single task.

I would like to ask whether having several overlapping levels between the two alternatives within a choice task is a concern. Specifically, I am worried that a high degree of overlap may reduce the amount of trade-off information that can be obtained. Is there any generally accepted threshold for the number of overlapping levels?

I have tried re-running the code multiple times, and the extent of overlap varies across generated designs. In some cases, only a few attributes (e.g. 3–4) have overlapping levels in certain tasks, while in others the overlap is more substantial, with up to seven overlapping levels for one attribute across the 32 tasks. Should I prioritise selecting a design with fewer overlapping levels? My concern is that excessive overlap across many choice tasks may reduce the available trade-off information, thereby affecting the likelihood of obtaining statistically significant coefficients in subsequent mixed logit models. On the other hand, could it be argued that overlapping some attributes may actually create sharper contrasts for the non-overlapping attributes and thus improve trade-off estimation?

Would it be advisable to impose constraints to ensure that all attribute levels differ within each choice task? I understand that Ngene provides a rejection function. Would it be appropriate to include something like the following in the syntax?
; reject:
alt1.betaone = alt2.betaone
, alt1.betatwo = alt2.betatwo
, alt1.betathree = alt2.betathree
, alt1.betafour = alt2.betafour
, alt1.betafive = alt2.betafive
, alt1.betasix = alt2.betasix
, alt1.betaseven = alt2.betaseven

I found that Ngene can still generate a D-efficient design after adding these constraints, with only a slight increase in D-error (to around 0.4). However, the process becomes much slower, and it takes considerable time to generate even a few candidate designs. So I just maybe can pick the one with the lowest D-error among only around three generated candidate sets.

Thank you!

By minimising the D-error, Ngene is optimising the trade-offs across all attributes simultaneously. Therefore, it is not necessary to impose non-overlap constraints.

The reason that is happens is because your priors are quite large for some of the attributes. This indicates that some of these attributes may be quite dominant, at least for some of the Bayesian draws, and therefore it becomes more efficient to provide overlapping levels for those attributes in some choice tasks to let the respondents focus on the trade-offs of the other attributes. So it is as you say, "overlapping some attributes may actually create sharper contrasts for the non-overlapping attributes and thus improve trade-off estimation?".

If you reduce the values of the priors, for example divide all of them by 10, then you will see that the attribute level overlap disappears.

Note that for Bayesian efficient designs with many Bayesian priors it is best to use many Sobol draws, so something like:
;eff = (mnl,d,mean)
;bdraws = sobol(1000)

Michiel