Bayesian D-efficient design: attribute-level balance with continuous attributes

[JoyceL186]

Hi Michiel,

I would really appreciate your advice on the final Bayesian design for my DCE.

I have an unlabelled experiment with two alternatives, 24 choice tasks in two blocks of 12, and five attributes. Based on the pilot data, the primary main-effects MNL specification treats TST, falls, and cost as continuous, while sedation (SED) and evidence (EV) are categorical. I also estimated a categorical sensitivity model in which TST and falls were specified categorically, given qualitative pre-testing evidence suggesting potential non-linearity in these attributes. However, the continuous specification provided a better fit according to both AIC and BIC and was therefore retained as the preferred specification. I did not investigate a dummy-coded specification for cost.

I therefore used the coefficients and Rao-Cramer standard errors from the continuous MNL model as Bayesian priors:

Design

;alts = (treatA, treatB)
? unlabelled design with two generic alternatives (treatment A and treatment B)
;rows = 24
;block = 2
? Each respondent completes 12 tasks in one of 2 blocks.
;eff = (mnl,d,mean)
? minimise the mean D-error for the multinomial logit model across Bayesian prior draws
;alg = mfederov
? Design generation algorithm: Modified Fedorov algorithm (row-based), selecting from 2000 random choice tasks (default)
;bdraws = gauss(3)
? Draws for Bayesian priors: Gaussian quadrature with 3 abscissas
;reject:
treatA.TST = treatB.TST
AND treatA.FALLS = treatB.FALLS
AND treatA.SED = treatB.SED
AND treatA.EV = treatB.EV
AND treatA.COST = treatB.COST

;model:
? Bayesian D-efficient fractional factorial design under a main-effects multinomial logit (MNL) specification
? Bayesian priors (n, mean, SD): n=normally distributed; mean, pilot-only MNL coefficients; SD = 1.0 x SE.
? Coding: Continuous TST (total sleep time), FALLS and COST.
? Coding: SED=daytime sedation: 0=Uncommon, 1=Common;
? Coding: EV=certainty of evidence: 0=Low, 1=Moderate, 2=High.

U(treatA) = b_tst[(n, 0.04356633, 0.01263462)] * TST[10, 20, 40]
+ b_falls[(n, -0.2292554, 0.04290887)] * FALLS[0,11,17]
+ b_sed.dummy[(n, -1.0790061, 0.30433279)] * SED[1,0]
+ b_ev.dummy[(n, 0.928756, 0.440479297)|(n, 1.089891, 0.46989917)] * EV[1,2,0]
+ b_cost[(n, -0.021118391, 0.00965448)] * COST[10,35,60]
/

U(treatB) = b_tst * TST
+ b_falls * FALLS
+ b_sed * SED
+ b_ev * EV
+ b_cost * COST
$

The resulting design has a Bayesian mean D-error of approximately 0.01623 and a Bayesian mean S-estimate of 23.413547, corresponding to a minimum sample size of 47.

My main concern is the level distribution for the continuous attributes. In particular, across the 48 alternative profiles:

TST 10/20/40 minutes appears approximately 20 / 4 / 24 times.
Cost AUD 10/35/60 appears 22 / 1 / 25 times.
Thus, the middle cost level ($35) appears only once, and the middle TST level is also used relatively infrequently. I understand that this can occur in D-efficient designs when attributes are treated as continuous. However, I am concerned that these middle levels are almost absent from the actual questionnaire.

Could you please advise on the preferred approach in this situation?

1. Should I retain the continuous specification and accept the unequal level frequencies, given that the primary analysis assumes linear effects for TST, falls, and cost?
2. Would it be reasonable to generate the design with TST, falls, and cost dummy coded, using the coefficients and SEs from my categorical pilot sensitivity model as Bayesian priors, with the aim of improving representation of all three levels, while still estimating these attributes as continuous in the primary analysis?
3. If I use that approach, should I then evaluate the resulting 24-task design under my original continuous Bayesian MNL specification and compare its Bayesian D-error and S-estimate with those of the current design? If so, how similar would these efficiency measures need to be for it to be reasonable to prefer the alternative design on the basis of better level representation?
4. Are there any other approaches you would recommend for addressing the very limited representation of the middle levels?

A related feature of the current design is that FALLS has the same level across the two alternatives in 6 of the 24 tasks. From your previous forum responses, I understand that some overlap can be desirable when an attribute is relatively dominant (Falls), so I have not imposed a non-overlap constraint. Is my understanding appropriate in this situation?

My main concern is therefore the very limited representation of the middle TST and cost levels.

Thank you very much for your time and advice.
Joyce


I previously sought advice on this issue and would like to document/clarify the recommendations here. The advice is as follows:

1. The modified Federov algorithm causes the issue. If you would reformulate your ;reject constraint into a ;cond constraint, you can use the swapping algorithm, which produces attribute level balanced designs. Alternatively, you can impose attribute level frequency constraints for the numerical attributes, e.g. TST[10,20,40](6-10,6-10,6-10), where each attribute level appears between 6 and 10 times, noting that (8,8,8) would mean perfect level balance.
2. Yes, that is acceptable, and something that I often do.
3. You can if you want. You can formulate two models in Ngene at the same time. You optimise for the model with dummy coding, but you can evaluate directly for the model with linear coding.

[JoyceL186]

Thanks for Michiel's previous response.

• One other point that came up during the qualitative pre-testing was that some participants found the combination of uncommon sedation and a higher risk of falls unexpected. However, we would still like to retain both attributes, as they are important to the study. Now that we have pilot data (n-14), do you have any suggestions on how best to address this?
  Would it be reasonable to impose a constraint excluding implausible combinations, for example common sedation with 0 additional falls? If so, could this materially affect design efficiency or the estimation of the sedation and falls effects?
  

  Code: Select all

  Sedation    Falls    Allowed
  Uncommon    0        Yes
  Uncommon    11       Yes
  Uncommon    17       Yes
  Common      0        No
  Common      11       Yes
  Common      17       Yes
  

  I have now generated and compared three candidate Bayesian D-efficient designs under the current specification. The main design characteristics are summarised below.
  

  Code: Select all

                       DCE0                 DCE1                 DCE2
                       Balanced levels      No level-frequency    Balanced levels +
                       in numerical         constraints           sedation–falls
                       attributes                                 constraint
  
  Bayesian D-error     0.017474             0.016231             0.018740
  Bayesian S-estimate  24.479               23.414               25.378
  Approx. S × 2 blocks 49                   47                   51
  
  TST 10/20/40         16 / 12 / 20         20 / 4 / 24          16 / 12 / 20
  FALLS 0/11/17        16 / 17 / 15         13 / 21 / 14         12 / 16 / 20
  COST 10/35/60        19 / 12 / 17         22 / 1 / 25          18 / 12 / 18
  
  Common sedation
  + 0 falls profiles   12                   9                    0
  
  FALLS overlap        9/24                 6/24                 9/24
  By block             5 / 4                1 / 5                3 / 6
  SED overlap          1/24                 0/24                 7/24
  

  DCE0 uses level-frequency constraints for the numerical attributes; DCE1 does not impose these constraints; and DCE2 additionally excludes profiles combining Common daytime sedation with 0 additional falls. Which would you recommend as the most appropriate design to retain for the main DCE?
• During the pilot, 2 of 14 participants consistently chose the alternative with the lower FALLS risk in every choice task, regardless of the levels of the other attributes. This appears to reflect non-trading on FALLS. Would this raise concern that the FALLS attribute is too dominant in the design, or is some individual-level non-trading acceptable in a DCE? If so, would you recommend modifying the design/levels for the main DCE, or addressing this mainly at the analysis stage?
• For the Bayesian priors, would it be appropriate to use the robust standard errors (samller numbers), rather than the Rao–Cramer standard errors, as the prior standard deviations? That will generate smaller sample size in our case.

Thanks in advance.
Joyce

[Michiel Bliemer]

If levels are not realistic, I would remove them. Since you are only removing one combination, this should not materially affect design efficiency (as also shown in the Bayesian D-errors) or estimation.

I usually like a high degree of level balance in the design. I would probably opt for DCE2.

Non-trading could indeed mean a dominant attribute. You can usually see this also from the parameter estimates, and if the prior indeed indicates such dominance then the efficient design would automatically adapt to that (for example, by adding more choice tasks where the falls attribute is the same across alternatives). You would see this happening when you would increase the prior for b_falls. If it is "only" 2 out of 14, then I would not worry too much about it. In model estimation, you could pick up this effect for example in a latent class model, where one class would essentially only look at falls and ignore the other attributes.

You can use robust standard errors. They are often larger (not smaller) than classical standard errors, but either of them is fine. The standard errors are merely a rough indicator for the reliability of the prior. You should not interpret it such that you can simply use a smaller sample size when changing the type of standard error.

Michiel