Questions to (pilot study) choice design

[greenvanilla]

Dear all,

I am new to Choice Design and using Ngene and I have some questions that I so far was not able to answer (I have read the Ngene user manual and searched in this forum).

I am planning to conduct an unlabelled choice experiment with two alternatives and one opt-out option per choice set.
First, I would like to conduct a pilot study with a zero priors design to obtain priors and then use Baysian priors based on the estimates from the pilot study, as suggested in the Ngene user manual.

The code for the pilot study I am using is currently the following:

Code: Select all

Design
;alts = alt1*, alt2*, alt3
? alt1 and alt2 are unlabeled alternatives, alt3 is a generic no choice option
;rows = 16
;block = 2
;eff = (mnl, d)

;model:
u(alt1) = b1[0]*x1[0.79,1.49,2.49,3.99] + b2.effects[0|0|0]*x2[0,1,2,3] + b3.effects[0]*x3[0,1] + b4.effects[0]*x4[0,1] + b5[0]*x5[0.2,0.4] /
u(alt2) = b1*x1 + b2*x2 + b3*x3 + b4*x4 + b5*x5 /
u(alt3) = b0[0]

$

x1 is the product price.

Now I have a few questions, and I would appreciate your advice on those.

1. The designs resulting from this code have a d-error of around 0.28.
I feel that this is relatively high, and I am wondering whether I might have made any mistake in the design, or whether that is a realistic d-error value for this pilot study with zero priors.

2. If I replace the effects coded variables by dummy coded variables (ceteris paribus) the resulting d-error is much higher. I don't understand why this should be the case.

3. The resulting design seems to use 'complementary' levels for the two alternatives, which is particularly striking for the price attribute. If Alternative A has a price of 0.79, Alternative B always has the price of 3.99, or the other way round. In other choice sets the two medium price levels are combined. But there are no combinations with the lowest and second lowest price level, for instance.
I feel that this might be problematic because of the huge price differences (I derived prices from getting actual prices for products with the different attributes I use in the study.)
As participants are expected to be sensitive to prices, I expect that in choice sets with the lowest and the highest price, it is quite clear how they will decide, no matter how the other attributes are defined.
Designs with more similar alternatives seem to have a higher d-error.
I don't know how to resolve that issue - or whether I should just take the design as it is, although there might be 'dominant' alternatives (not in all attributes, but referring to the huge price difference).
Do you have any suggestion on that?

Furthermore, I have some questions that are not directly related to this pilot study design but more general.

4. I have read in the manual that for dummy/effects coded variables, the highest level is treated as the base level.
I am not sure about what that means for the design, i.e. if I had priors, whether I would need to insert them into the code in a certain order, or whether this does not matter?

5. I found some previous studies that used some of the attributes I use in my study as well, so I would have some coefficients I could use as priors, but I am not so sure on how exactly I could to that.
I asked a colleague and he suggested to stick with the zero priors pilot study to get the priors and then cross-check with the coefficients from the literature and maybe use some combinations.
Would you agree with that procedure, or would it be better to use some priors from the literature already for the pilot study design?
My problem would be that I would need to combine values from different studies, and that I don't have estimates for all of the attributes.
For instance another study provides estimates for two levels of my effects-coded variables b2, but not for the third level.
Also, the other study used dummy coding, while I would like to used effects coding.
I would expect that I would then need to adjust the coefficients, as they would be smaller for effects coding?
Also the study indicates standard errors for the coefficients, and I understood that if I would like to use distributions, I need to use standard deviations, not standard errors.
My colleage said that it does not matter, it would also be possible to use the standard errors. But how would Ngene now that it's standard errors and not standard deviations?
Would it be possible in any way to use priors from the literature if I include an additional level for my variable? Wouldn't this affect the coefficient values maybe?

I would appreciate any ideas and answers to my questions!

Thanks a lot in advance!

[Michiel Bliemer]

1. Why do you feel that this is high? D-errors are case specific and do not really have a meaning, sometimes 0.1 is considered high, other times 0.1 is considered low. Models with many dummy/effects coded variables generally have a higher D-error since they are more difficult to estimate. However, do I feel that your number of rows is not enough, an optimal orthogonal design with 16 rows has a D-optimality of less than 2%, while with 24 rows it goes to 90%. I think that with 16 rows there is simply not enough variation in the data. I suggest that you use 24 rows (and block it in 3).

2. Applying dummy coding results in a different model (describing the same behaviour), therefore has a different D-error. Note that the D-error will change when you use informative (non-zero) priors.

3. Using zero priors means that efficiency is maximised with maximum trade-offs between attributes, which is why you will see 0.79 versus 3.99. If you use non-zero priors that provide information that price is a dominant attribute then the prices generated within the design will be closer together. For example, if you set the prior for b1 to -1 (and remove the * after alt1 and alt2 as it is not possible to check for dominant alternatives) then you will see that prices will become closer (although the D-error increases). However, I do not recommend manipulating priors manually. I think that in this case you would be better off using an orthogonal design for your pilot study. I generated one with the syntax below, which has a D-error of 0.20.

Code: Select all

Design
;alts = alt1, alt2, alt3
? alt1 and alt2 are unlabeled alternatives, alt3 is a generic no choice option
;rows = 24
;block = 3
;orth = ood
;model:
u(alt1) = b1[0]*x1[0.79,1.49,2.49,3.99] + b2.effects[0|0|0]*x2[0,1,2,3] + b3.effects[0]*x3[0,1] + b4.effects[0]*x4[0,1] + b5[0]*x5[0.2,0.4] /
u(alt2) = b1*x1 + b2*x2 + b3*x3 + b4*x4 + b5*x5 /
u(alt3) = b0[0]
$

4) The last level (not the highest) in Ngene is the base level, so in attribute x2 the level 3 will be the base. The first prior refers to level 0, the second prior to level 1, etc., so yes the order matters. If you prefer level 0 to be the base the simply use x2[1,2,3,0].

5) You could use priors from the literature in your pilot study if you like, but I would not use them for the main study. From your pilot study you obtain priors for all parameters and I would use those for your main efficient design and I would not combine them with priors from the literature because of scale differences in data collections affecting the relative size of parameters. If you do combine them, it is best to only transfer ratios of parameters instead of absolute values (using the price parameter as the reference attribute), which avoids the scale issue. You can always rewrite effects coding into dummy coding so that is not a problem. You can add a level in dummy coding without affecting the other parameters. Note that with effects coding adding another level will change the other parameters as well, so you may want to consider using dummy coding (note that there is no benefit in using effects coding, it is merely a matter of taste which one you prefer). Regarding standard errors and standard deviations, yes they are something different but that is exactly why you can use standard errors as standard deviations for Bayesian priors. The standard error denotes the unreliability of the parameter estimate. The standard deviation of a Bayesian prior refer to the unreliability of the prior. Therefore, it is appropriate to use the standard error of the PARAMETER ESTIMATE as a standard deviation of the PRIOR DISTRIBUTION.

Michiel

[greenvanilla]

Dear Professor Bliemer,

thank you so much for your detailed answers and explanations!

With respect to question no. 1, I now understand that it does not make any sense to compare d-error values between different studies.
I will try the design with 24 rows and 3 blocks, as you suggested.
Do you maybe have any recommendation on the sample size I should try to achieve for the pilot study

2. I do not really understand why using dummy coding instead of effects coding would result in a different model, as the attributes and levels themselves do not change, and the resulting choice design also seems to be the same. But I guess this is a point I do not really need to understand in detail but rather should accept as a fact.

3. As you suggested, I will try the OOD design with zero priors. This looks indeed better in terms of price combinations.

4. Thanks for clarification! Just to be sure: the position of the base level is only relevant for dummy/effects-coded variables, right? I don't have to reverse the order of levels for x1 and x5 (attributes assuming a linear relationship)?
With using zero as the base level for the effects coded variables, the code would then look like:

Code: Select all

Design
;alts = alt1, alt2, alt3
? alt1 and alt2 are unlabeled alternatives, alt3 is a generic no choice option
;rows = 24
;block = 3
;orth = ood
;model:
u(alt1) = b1[0]*x1[0.79,1.49,2.49,3.99] + b2.effects[0|0|0]*x2[1,2,3,0] + b3.effects[0]*x3[1,0] + b4.effects[0]*x4[1,0] + b5[0]*x5[0.2,0.4] /
u(alt2) = b1*x1 + b2*x2 + b3*x3 + b4*x4 + b5*x5 /
u(alt3) = b0[0]
$

5. Thank you for your explanation on why it is okay to use the standard error of the estimates for the Baysian priors!
I am not so sure whether it really does not matter if I use effects coding or dummy coding. I have read Bech & Gyrd-Hansen (2005): "Effects coding in discrete choice experiments" and they explain why it makes sense to use effects coding if you have an opt-out option in your choice experiment and would like to be able to interprete the effect of the alternative-specific constant in the analysis. That is why I would prefer effects coding, but I would appreciate your opinion on this aspect.

Thanks a lot!

[Michiel Bliemer]

1. I cannot recommend a sample size for your pilot study since this is case specific (sometimes 5 is enough, other times you may need more than 50).

2. It is a different model because the parameters are different and therefore the standard errors are different, influencing the D-error. As I said, it describes the same behaviour (the outcome is the same), but the model specification is different.

3. Yes the order is only relevant for dummy/effects coded variables, linear coding does not have a reference level.

5. The fact that you would be able to interpret the ASC better using effects coding was indeed the reason why for many years people preferred effects coding, but others have now claimed that you can also do this with dummy coding and that there really is no difference. I refer to this paper by Hess et al. (2016):

https://www.sciencedirect.com/science/a ... 4516300781

As you may know, Hess is the editor-in-chief of the Journal of Choice Modelling and one of the most prominent choice modellers (together with his well-known co-author Daly), therefore I would trust their opinion on this.

Michiel

[greenvanilla]

Thanks a lot for your reply!
I didn't know the mentioned article by Daly et al. (2016) yet, I will read it now.

[greenvanilla]

Dear Prof. Bliemer and all,

I have implemented the pilot study now and am now concerned with the analysis of the data obtained.
I have some prior experience with Stata used for choice analysis, but my colleagues rather use Nlogit, so I am thinking about whether it makes sense to learn Nlogit or whether I should stick with Stata.
Do you maybe have a general advice on that? Is Nlogit "better" than Stata in some aspects?
For the actual choice experiment I am planning to do a Latent Class Analysis (and I haven't done this analysis in either Stata or Nlogit).

For the analysis of the pilot study, I first estimated a conditional logit model (clogit) with cluster-robust standard errors in Stata,
because I thought it might make sense to cluster according to ID (individual participants), as each participant answered several choice tasks.

However, I haven't found out how to get cluster-robust standard errors in Nlogit. I found different commands like "cluster" and "robust",
but it seems that Nlogit does not recognize the number of different individuals, so I don't get the same results as in Stata.
(If I estimat the conditional logit model without cluster-adjustment, the results are the same in both tools.)

So my questions are basically the following:
1. Is there any advantage on using Nlogit instead of Stata for the estimation, considering that I am rather used to Stata?
2. Is it correct that it would be better to estimate cluster-robust standard errors for the pilot study and use these as input for the design of the efficient model, or should I use the "normal" version, without clustering?
3. If I should use cluster-robust standard errors and Nlogit, could you please tell me the correct command in Nlogit?

I know that this is not a forum for Nlogit (sorry for that), but I could not find a similar forum for questions on the actual analysis.
If you have any advice on whereelse I should post my question, I would be grateful, too.

Thanks a lot in advance!

[Michiel Bliemer]

These are all model estimation questions unrelated to stated choice experimental design, so I am not really able to help much. Perhaps others could answer them.

1. I believe that Nlogit can estimate many advanced logit models with simple syntax while Stata may require more programming, but I have never used either of them.

2. I am not familiar with the terminology "cluster-robust", but I think this may be the same as "accounting for panel effects", i.e. accounting for the fact that a single respondent faces multiple choice tasks. I know that Nlogit can estimate panel mixed logit and panel latent class models, for panel clogit the only thing that needs to be done is "correct" the standard errors I believe, so the parameter estimates do not change, only the standard errors (and t-ratios) change.

3. I am not familiar with Nlogit, please ask your question on the Nlogit forum: http://www.limdep.com/listserver/

Michiel

[greenvanilla]

Thank you for your reply!

Yes, I meant accounting for panel effects, as you explained it.
For some reason I haven't figured out how to do this for clogit in Nlogit, but I will try to ask the Nlogit forum.

But is it correct to use these panel-adjusted standard errors than for input into Ngene for the improved choice design?
Or would you rather use unadjusted standard errors?

Best regards

[Michiel Bliemer]

The standard errors used to inform Bayesian priors for the conditional logit model (mnl) in Ngene should be unadjusted standard errors. We assume that only rppanel, ecpanel, and rpecpanel account for the panel effects.

Michiel

[greenvanilla]

Thank you for your reply!

But now I am a bit confused.
In the pilot study I used a conditional logit (MNL) model, but the data has panel structure (several choice sets for each respondent).
But finally I am planning to generate a Bayesian efficient design for an rppanel model.
As I understood from the Ngene manual, it makes sense to try it for an MNL model first due to estimation effort.
In the data analysis, I would like to be able to do a latent class analysis (if this is relevant).

So if I try now to generate the final design and start off with the MNL, would I need to use the unadjusted standard errors from the pilot study,
and then switching to the rppanel model change the standard errors to the adjusted ones?
I could not find any information on this in the Ngene manual.

Thanks a lot!