Fractional Factorial Design - Issue

[davidj]

Hi Ngene Team,

I was wondering if you can please assist in advising on the following issue experienced when developing a fractional factorial design. I am developing a fractional factorial design to test a MNL and Hybrid choice model.

When developing the following syntax, ngene is able to locate about 5 designs. However, before the evaluation is complete and when running through the ‘examining table combinations,’ the program crashes at 20/38 table combinations and 240,000 evaluations.

So my questions are:

1. Is this simply a memory limitation in Ngene
2. Is it ok to use on of the design solutions – understanding that it may not have the lowest d-error, given that all possible combinations have been evaluated?

The syntax is as follows:

Design
;alts = Base, ALT1*, ALT2*
;rows = 72
;orth = sim
;eff = (mnl,d)
;block = 16
;foldover
;model:

U(ALT1) = b1 * A[30,40,50,60] + b2 * B[12,14,16] + b3 * C[128,256,384] + b4 * D[0,18,36] + b5 * E[10,12,14] + b7*A*B + b8*C*D /
U(ALT2) = b6 + b1 * A + b2 * B + b3 * C + b4 * D + b5 * E + b7*A*B + b8*C*D $

Thank you so much for all the assistance and support.

David

[Michiel Bliemer]

This seems a bug, I am not sure how to quickly fix this, thank you for pointing this out.

In the mean time, I would suggest using the best design after 240,000 evaluations (which should be pretty hard to beat). Alternatively, since AltA and AltB are generic, it is likely better to use a sequential orthogonal design. So you could use the following code, which does not crash:

Code: Select all

Design
;alts = Base, ALT1*, ALT2*
;rows = 36
;orth = seq
;eff = (mnl,d)
;block = 16
;foldover
;model:
U(ALT1) = b1 * A[30,40,50,60] + b2 * B[12,14,16] + b3 * C[128,256,384] + b4 * D[0,18,36] + b5 * E[10,12,14] + b7*A*B + b8*C*D /
U(ALT2) = b6 + b1 * A + b2 * B + b3 * C + b4 * D + b5 * 

It also results in a smaller design with 2 x 36 rows.

[davidj]

Hi Michiel,

Thank you for confirming that, very much appreciated!

I was also hoping to get your thoughts on a few follow up questions:

I was hoping to complete an efficient design, however my research team has requested an orthogonal design due to their past experiences with this design framework and the assumptions of no information at all on priors. Given this, I was hoping to get your thoughts on a few follow up questions:

1. As I am using a fractional factorial design, my intention was to maximise the number of rows (in this case 72 rows), so as to develop a better design that is closer to the full factorial. The alternative would be say 36 rows.
Pros: Design with more rows better represents full factorial and fully orthogonal design
Cons: Increased sample size required, results may not be orthogonal over the blocks and possible thoughtless choice questions

What have you done previously or what do you recommend? Do you always maximise the number of rows in orthogonal design, pending sample size and design output?

2. My final design contains a reference alternative (with fixed attribute levels). This will incorporate different parameters in the utility function for this alternative, but should not affect the fractional factorial design. Is my understanding correct?

3. Sequential for simultaneous design: My preference in this case would be to use simultaneous orthogonal design, over the sequential approach. I understand that given the parameters are generic and alternatives have same attribute levels, this is feasible. But how would you evaluate the two designs or what would you recommend.

Thanks again for all your help and assistance. I really appreciate it.

David

[Michiel Bliemer]

These are not really Ngene questions and should therefore go in our other forum as they are general design questions.

My short answer would be that we never really use orthogonal designs since they are not very efficient, and they often contain dominant alternatives. Orthogonal designs also cannot handle reference or status quo alternative. Efficient designs aim to optimise information from each choice task and are able to remove dominant alternatives and take reference alternatives into account.

If you prefer to use a (simultaneous) orthogonal design, then you should be aware of the above limitations. Orthogonal designs work fine, they just require a larger sample size.

Researchers typically are happy with a small orthogonal design, but if you want to have a larger design, then that is fine. You can block it and send different blocks to different respondents (but note that you loose orthogonality in the data when you do not have the same number of respondents filling out the survey in each block). It is not common to maximise the number of rows, as this will give you a full factorial design, which is very inefficient. In a full factorial design, there are typically many questions that do not make many (or any) trade-offs in attribute levels, so you will to manually delete them from your design (again loosing orthogonality).

Sequential orthogonal designs are optimal for unlabelled experiments, but if you want to go for a simultaneous orthogonal design that is perfectly fine, you will just again loose a bit of efficiency. But if your sample size is large enough, efficiency is not so much an issue.

[davidj]

Hi Michiel,

Apologies for posting in the wrong forum – my mistake.

Thank you for answering the questions. I greatly appreciate the assistance.

David