This paper investigates the limits of foundation models in medical image segmentation, mainly focusing on SAM by Meta. While previous research demonstrated SAM’s potential for cost-efficient segmentation, this study explores its performance enhancement through integration with prompt enhancement optimization and genetic algorithms, aiming to minimize user input further. As a proof of concept, we apply this novel approach to lung segmentation tasks using public axial lung CT scans, frontal chest X-ray datasets, and spleen MRIs. Our findings reveal that the genetic algorithm optimization significantly improves SAM’s segmentation accuracy, bringing it closer to the state-of-the-art performance achieved by specifically trained models. In particular, when compared with our previous approach, this technique reaches a 94.85 % Jaccard Index (+3.77 delta) and a 97.17 % Dice Score (+2.50 delta) for lung CT scans, a 93.39 % Jaccard Index (+5.95 delta) and a 96.57 %Dice Score (+3.38 delta) for chest X-rays, and a 91.00 % Jaccard Index (+6.51 delta) and a 95.07 % Dice Score (+4.12 delta) for spleen MRIs. Notably, this improvement is achieved without retraining or modifying SAM’s architecture. However, our analysis also identifies an inherent limitation in this optimization approach, revealing a performance ceiling that cannot be surpassed despite further genetic algorithm iterations. The implications of these findings emphasize the potential of combining foundation models with non-intrusive optimization techniques for cost-effective and accessible medical image segmentation. While dataset-related limitations may affect generalizability, validating the approach across broader clinical scenarios remains essential. Future work should explore applications to additional organs, diverse datasets, and the integration of expert-in-the-loop strategies to enhance clinical utility.