Ngene won't search with mfedorov(candidates = candidate_set)

[dce_learner]

Hi Michiel,

I'm running into an issue when using a custom candidate set. Below is my code for generating an efficient experimental design with no priors. It runs fine when I use mfedorov(candidates = 3000), but when I switch to referencing an external candidate set file using mfedorov(candidates = candidate_set), Ngene doesn't initiate the search and gives the following warning message:

Warning: Defaulting to prior values of zero for the following parameters: 'b1, b2, b3, b4, consq'
Note: Defaulting to assigning blocks with the 'minsum' method.

These same warnings also appear when using candidates = 3000, but Ngene proceeds with the design search. Is there something about my model specification that’s preventing Ngene from moving forward with the search when a custom set is supplied?

Any insights or suggestions would be greatly appreciated! Thank you.

Here's my syntax:

Code: Select all

design
;alts = (alt1, alt2), sq   
;rows = 60
;block = 20
;eff = (mnl,d)
;alg = mfedorov(candidates = candidate_set)

;require:
sq.reg = 8,
sq.young = 44,
sq.mature = 20

;reject:
alt1.reg + alt1.young + alt1.mature >= 100,
alt2.reg + alt2.young + alt2.mature >= 100,

alt1.young + alt1.mature >= 67 and alt1.cost = 7 or alt1.cost = 10,
alt2.young + alt2.mature >= 67 and alt2.cost = 7 or alt2.cost = 10,

alt1.young + alt1.mature <= 54 and alt1.cost = 1 or alt1.cost = 3,
alt2.young + alt2.mature <= 54 and alt2.cost = 1 or alt2.cost = 3,

alt1.reg >= 7 and alt1.young + alt1.mature <= 49 and alt1.cost = 1,
alt2.reg >= 7 and alt2.young + alt2.mature <= 49 and alt2.cost = 1

;model:
U(alt1) = b1   * reg     [1, 4, 8, 10] 
        + b2   * young   [29, 34, 39, 44]
        + b3   * mature  [20, 24, 28, 32]
        + b4 [-] * cost   [1, 3, 5, 7, 10]
/
U(alt2) = b1 * reg
        + b2 * young
        + b3 * mature
        + b4 * cost
/
U(sq) = consq + b1 * reg
              + b2 * young
              + b3 * mature
              + b4 * cost_SQ[0]
$

I've also tried to add status quo to the candidate set and remove the check constraints. However, the search still won't start.

[Michiel Bliemer]

Warnings are fine, errors are not. In this case, Ngene merely warns you that you did not specify priors, so it defaults them to zero. This is not an issue.

If the design search works with candidates = 3000, but not with candidates = candidate_set, then there is an issue with your candidate set. Either you specified the wrong format, or you specified inconsistent attribute levels, or you created multicollinearity in the design levels, or something else.

First would be to check that you are using the correct format. You can see check for the correct format by running the script with e.g. candidates = 1000 and then choose Export to Excel on the Results tab. There you can see that the status quo alternative MUST be provided. The alternatives and attributes must be specified in exactly this order.
Second would be to check that all attribute levels in your spreadsheet coincide with levels specified in the script.
Third would be to ensure that you did not impose too many restrictions when you generated this candidate set, since it could result in an unidentified model.

Please let me know if you were able to find the issue with the candidate set.

Note that Ngene cannot perform dominance checks when you specify the preference order of only one attribute, so please remove the [-] prior from cost, you should just let it default to zero in this case.

Michiel

[dce_learner]

Hi Michiel,

This is so helpful. Thank you. It turns out that the issue was due to a mismatched variable name. In the script, the cost variable in the status quo option is named as sq.cost_sq, but I had it as sq.cost in the candidate set. Once I fixed the mismatch, the script ran without any problems.

Thanks for the note about dominance checks as well. I have now added [+] for another attribute, along with the [-] prior for cost. Really appreciate your help!

[Michiel Bliemer]

Note that only using a preference order for two attributes will severely restrict the allowable choice tasks because most will contain a dominant alternative since it is assumed that only these two attributes exist when checking for dominance. If the other attributes do not have a clear preference order, it is best to remove the + and - for the two attributes to have a larger set of allowable choice tasks.

Michiel

[dce_learner]

Got it. Thank you, Michiel.

I had to change the status quo level for attribute Reg from 8 to 12 and the upper bound for this attribute from 10 to 12. I did not change any restrictions I imposed. Below is my revised syntax (It's very similar to the original syntax). I also made sure: 1. The attribute levels are consistent between the script and the candidate set; 2. The status quo option in the candidate set has the same attribute level as specified in the script. 3. The alternatives and attributes are specified in the exact order as in the script.

The search wouldn't start, and there were no error messages telling me the reason. Did I miss something?

The candidate set generated for my study, based on a series of constraints, has about 20,000 plus rows. I understand that the maximum length Ngene allows is 10,000. Additionally, a set of 3000 should be sufficient to generate an efficient design, as you noted in your responses to other posts and also in the user manual. Therefore, I randomly sampled 4000 rows from the 20,000 rows. Is this a valid approach?

Code: Select all

design
;alts = (alt1, alt2), sq    ? Two policy alternatives + status quo
;rows = 60
;block = 20
;eff = (mnl,d)
;alg = mfedorov(candidates = candidate_set3)

;require:
sq.reg = 12,
sq.young = 44,
sq.mature = 20

;model:
U(alt1) = b1   * reg     [1, 4, 8, 12] 
        + b2   * young   [29, 34, 39, 44]
        + b3  * mature  [20, 24, 28, 33]
        + b4  * cost    [1, 3, 5, 7, 10]
/
U(alt2) = b1 * reg
        + b2 * young
        + b3 * mature
        + b4 * cost
/
U(sq)   = consq  + b1 * reg
                        + b2 * young
                        + b3 * mature
                        + b4 * cost_SQ[0]

Thank you again.

[Michiel Bliemer]

The script runs when not using your external candidate set, so the issue seems again to be in the candidate set. It is difficult to say what could be wrong, perhaps you can email it to me?

Yes, the maximum length of the candidate set that Ngene allows is 10,000, and there is never really a reason to have more than 10,000. The modified Fedorov algorithm becomes very slow with a large candidate set, so while 10,000 will work, I usually recommend 2,000 (which is the default in Ngene when using the modified Fedorov algorithm) or a bit more. Randomly sampling 4,000 from the 20,000 rows is a valid approach, yes.

Michiel

[dce_learner]

Thanks so much, Michiel! I got it working after some finessing (was really hoping not to bother you by sending over the candidate set)!

Just in case it’s helpful for anyone else who runs into a similar issue: when I exported the candidate set from R to a CSV, I originally used the following line: write.csv(design_with_sq, "candidate_set4.csv", row.names = FALSE) The "row.names = FALSE" setting omits the row numbers, which are automatically generated by R during the sampling process and not needed for Ngene.
I eventually changed it to write.csv(design_with_sq, "candidate_set4.csv", row.names = TRUE), then deleted the extra row index column manually before importing into Ngene, and the search ran successfully.

Not entirely sure why, but that adjustment did the trick!

[Michiel Bliemer]

Interesting, thank you for letting us know.
I did notice that Ngene sometimes does not accept the candidate set if the row numbers are not consecutive, so that could perhaps have been the issue.

Based on your feedback, we will implement some checks on the candidate set and also consider what you propose: automatically replacing the first column with a computer-generated row index (so that it does not matter if there are any mistakes in the numbering in the CSV file).

Michiel