choice-metrics.com

Dear all,
We are finalizing the design for an unlabelled Discrete Choice Experiment and would appreciate your expert opinion on interpreting the results due to a design constraint.
Attributes and Levels
Our design includes two key attributes:
• Transparency Seal:
o Level 0: No Seal (Reference Level)
o Level 1: Yellow Seal
o Level 2: Green Seal
• External Endorsement:
o Level 0: No Endorsement (Reference Level)
o Level 1: Public Body
o Level 2: Private Certifying Company
o Level 3: NGO
________________________________________
The Design Constraint
The attributes are logically dependent: an endorsement can only exist if a seal is present.
• If Transparency Seal is Level 0 (No Seal), then External Endorsement must also be Level 0, as an endorsement is "not applicable."
• If Transparency Seal is Level 1 or 2 (Yellow or Green), then External Endorsement can be Level 0 (meaning the seal is present but has "no endorsement") or Levels 1, 2, or 3.
________________________________________
Our Question
Our question relates to the correct interpretation of the parameter estimates for the External Endorsement levels.
The reference level for this attribute (Level 0) carries a dual meaning:
1. It means "Not Applicable" when no seal is present.
2. It means "No Endorsement is Present" when a seal is present.
When we estimate the model using dummy coding, how should we interpret the parameters for the other endorsement levels (Public, Private, NGO)? For example, will the parameter for "Public Body" represent the marginal utility of adding a public endorsement to a seal, compared to a seal with "no endorsement"? Or is this interpretation complicated by the "not applicable" meaning also being part of the reference level?

We'd like to double-check our understanding of this before launching the survey to ensure we interpret the results correctly.
Thank you for your time and help.
Best regards,

The short answer is: the interpretation is compared to "no endorsement".

The "not applicable" part needs to be dealt with via an interaction effect in the utility function so that the attribute "External endorsement" simply drops out. For example, in model estimation you would essentially estimate something like the following:

V = b_yellow * (Seal==1) + b_green * (Seal==2) + (Seal>0) * (b_public * (Endorsement==1) + b_PCC * (Endorsement==2) + b_NGO * (Endorsement==3))

In other words, the last part simply drops out due to the interaction with (Seal>0), which equals one if Seal is 1 or 2 and zero otherwise. By letting it drop out of the utility function, parameters b_public, p_PCC and P_BGO then become purely compared to "no endorsement". Of course this is cosmetic because simply setting Endorsement=0 when the seal is not available yields exactly the same result, but it merely demonstrates that you can interpret the parameters compared to "no endorsement" UNDER THE CONDITION THAT THERE IS A SEAL.

You could also reformulate the attributes as follows, whereby you estimate a parameter for having a seal separately from the colour:
- Seal: Yes, No
- Colour: Yellow, Green (interaction with Seal so that it only applies when Seal=Yes)
- Endorsement: None, public body, PCC, NGO (interaction with Seal so that it only applies when Seal=Yes)

Michiel

Dear Michiel,
Thank you very much again for your previous reply.
Following your earlier advice, we imposed the logical dependency between the Transparency Seal and External Endorsement in the Ngene pilot design through a reject rule. We have already collected the pilot data, and we would now like to check whether we are estimating the pilot correctly and whether the resulting coefficients can be used as priors for the final efficient design.
Attributes used in the pilot
• Transparency Seal: 0 = No Seal (reference) ; 1 = Yellow Seal ; 2 = Green Seal
• External Endorsement: 0 = No Endorsement (reference); 1 = Public Body ; 2 = Private Certifying Company ; 3 = NGO
• Commitment to the Farmer: 0 = No ; 1 = Yes
• Small and Medium Enterprise (SME): 0 = No ; 1 = Yes
• Price: 0.86 ; 1.19 ; 1.52 ; 1.85
Ngene pilot design
Design
;alts=alt1*, alt2*,SQ
;rows = 36
;block=6,minsum,noimprov(5 secs)
;eff = (mnl,d)
;bseed=12345
;alg = mfederov(stop=noimprov(30 secs))
;model:
U(alt1) = b1.dummy[0.00001|0.00001]*Transparency_Seal[2,1,0]
+ b2.dummy[0.00001|0.00001|0.00001]*Endorsement[3,2,1,0]
+ b3.dummy[0.00001]*Commitment_with_Farmer[1,0]
+ b4.dummy[0.00001]*Small_Medium_Enterprise[1,0]
+ b5[-0.00001]*PRICE[0.86, 1.19, 1.52, 1.85]/
U(alt2) = b1*Transparency_Seal + b2*Endorsement
+ b3*Commitment_with_Farmer + b4*Small_Medium_Entrepraise + b5*PRICE
/ U(SQ) = b0
;reject:
alt1.Transparency_Seal=0 and alt1.Endorsement<>0,
alt2.Transparency_Seal=0 and alt2.Endorsement<>0$
How we estimated the pilot data
Because the reject rule ensures that endorsement only appears when a seal is present, we assumed that we could estimate the pilot data without explicitly interacting Endorsement with a Seal>0 indicator. In other words, we estimated a standard conditional logit / MNL model in STATA using plain dummy coding:
• Transparency Seal: Yellow, Green (reference = No Seal)
• External Endorsement: Public, Private, NGO (reference = No Endorsement)
• Commitment to the Farmer: Yes (reference = No)
• SME: Yes (reference = No)
• Price: continuous
The STATA code was:
asclogit choice ASC_SQ green yellow public private NGO farmer SME price, ///
case(csid) ///
alternatives(alt) ///
noconstant ///
vce(cluster id) //
MNL pilot estimates
Choice Coefficient Std. err. Z P>|z|
ASC_SQ -4.345383 .5174994 -8.40 0.000
Yellow -.1577625 .4359277 -0.36 0.717
Green .0402857 .4398433 0.09 0.927
Public .467612 .3642127 1.28 0.199
Private .4817074 .3520505 1.37 0.171
NGO .6401489 .3436077 1.86 0.062
Farmer .4564314 .2126417 2.15 0.032
SME -.1215323 .2082181 -0.58 0.559
Price -1.687303 .1765331 -9.56 0.000


We would be very grateful if you could confirm whether the following understanding is correct:
1. Estimation of the pilot data
Given the reject rule, it is correct to estimate the pilot data with the plain dummy-coded specification above, without explicitly interacting endorsement with a (Seal > 0) indicator, right?
2. Interpretation of the endorsement coefficients
Under this estimation approach, the coefficients for Public, Private, and NGO can be interpreted as the utility of those endorsement types relative to “no endorsement”, conditional on a seal being present, right?
3. Use of pilot coefficients as priors
Can we use these pilot estimates as priors for the final Ngene design while keeping the same attribute structure and reject rule as in the pilot design?
4. Bayesian priors
Would it be reasonable to specify normal Bayesian priors in Ngene using the estimated coefficient as the mean and the corresponding standard error as the standard deviation, or would you recommend a different approach?
5. Preferred final specification for Ngene
For the final design, would you recommend:
o keeping the current attribute structure exactly as above, right?

As an additional concern, when we generated the pilot design in Ngene, we noticed that almost all choice tasks were extreme in terms of price. In most cases, one alternative had the lowest price, while the other had the highest. From a consumer perspective, this makes several choice sets look rather artificial and lacks realism, because the price gap is so large that respondents may feel pushed to reject the more expensive option almost automatically, especially for a relatively standard product such as milk, where they may perceive little added value in paying the highest price. We suspect that this may be one reason why most pilot coefficients are weak and statistically insignificant, while the ASC_SQ and price coefficients are large and highly significant. For the final design, we are therefore considering imposing some degree of price-level balance or restricting extreme price contrasts across alternatives (for example, making the extreme prices appear less often and increasing the frequency of middle price levels). We would be very grateful for any suggestions on how best to handle this in Ngene so that the final design remains efficient but also more realistic for respondents.

Thank you very much for your time and help.
Best regards,

1. Yes I believe so.
2. Yes, they can be interpreted as utilities relative to "no endorsement" (without the need of 'conditional on a seal' since that is the reference level).
3. Yes, although I notice large standard errors, so your priors are not very reliable. But they are still the best guess you have. If the preference order indicated by the prior makes sense, then they should be okay.
4. Yes that should be fine, although the distributions are quite wide because your standard errors are relatively large.
5. I do not see any obvious issues with it.

Regarding your concern: Efficient designs aim to maximise information and minimise standard errors. When your priors are (near) zero, then Fisher information can be maximised by comparing the highest versus lowest levels, especially when using the modified Federov algorithm. This issue is also discussed in more detail in the Ngene Online manual. This issue mostly occurs with non-zero priors. This is often not desirable, so there are a few options:
(i) use non-zero priors based on your pilot study and ensure that all price levels are more or less equally represented in the design by adding attribute level frequency constraints. So in your case, you could for example use b5[...] * PRICE[0.86, 1.19, 1.52, 1.85](6-12,6-12,6-12,6-12) to indicate that each price level should appear between 6 and 12 times across the 36 rows. If comparisons are mostly 0.86 versus 1.85, and 1.19 versus 1.52, then you can consider the next option:
(ii) dummy code the PRICE attribute when generating the experimental design. So: b5.dummy[-0.55|-1.11|-1.67] * PRICE[1.19, 1.52, 1.85, 0.86], where 0.86 is the base level. When using dummy coding (or effects coding), the efficient design will aim to optimise the efficiency for each attribute level and therefore will consider a wider variety of comparisons. During model estimation you can of course still estimate a single price parameter.

Michiel