Venetoclax and Sequential Busulfan, Cladribine, and Fludarabine Phosphate Before Donor Stem Cell Transplant in Treating Patients With Acute Myelogenous Leukemia or Myelodysplastic Syndrome

This randomized phase II trial studies how well venetoclax and sequential busulfan, cladribine, and fludarabine phosphate before donor stem cell transplant work in treating patients with acute myelogenous leukemia or myelodysplastic syndrome. Giving chemotherapy before a donor peripheral blood stem cell transplant helps kill cancer cells in the body and helps make room in the patient's bone marrow for new blood-forming cells (stem cells) to grow. When the healthy stem cells from a donor are infused into a patient, they may help the patient's bone marrow make more healthy cells and platelets and may help destroy any remaining cancer cells.

Inclusion Criteria: * Patients with biopsy-proven acute myeloid leukemia or myelodysplastic syndrome with persistent disease or in remission * Human leukocyte antigen (HLA)-identical sibling or 8/8 matched unrelated donor transplant * Patients age 18 to 70 years old; eligibility for pediatric patients will be determined in conjunction with an MD Anderson Cancer Center (MDACC) pediatrician; patients age 2-17 years may be enrolled after at least 10 adults (ages 18-70 years old) have been assessed for safety at day 30 * Direct bilirubin \< 1 mg/dl * Alanine aminotransferase (ALT) \< 3 times upper limit of normal * Creatinine clearance \> 50 ml/min (calculated creatinine clearance is permitted) * Forced expiratory volume in 1 second (FEV1) \>= 50% of expected corrected for hemoglobin and/or volume * Forced vital capacity (FVC) \>= 50% of expected corrected for hemoglobin and/or volume * Diffusing capacity of the lungs for carbon monoxide (DLCO) \>= 50% of expected corrected for hemoglobin and/or volume * Children unable to perform pulmonary function tests (e.g., less than 7 years old) pulse oximetry of \>= 92% on room air * Left ventricular ejection fraction (LVEF) \>= 40% * Patient, legally authorized representative (LAR), or parent able to sign informed consent; able to give assent for patients age 7-17 * Negative beta human chorionic gonadotropin (HCG) test in a woman with child bearing potential, defined as not post-menopausal for 12 months or no previous surgical sterilization; women of child bearing potential must be willing to use an effective contraceptive measure while on study * Performance score of \>= 70 by Karnofsky/Lansky or performance status (PS) 0 or 1 (Eastern Cooperative Oncology Group \[ECOG\] =\< 1) Exclusion Criteria: * Prior allogeneic or autologous transplantation * Uncontrolled infections * Human immunodeficiency virus (HIV) seropositivity * Hematopoietic cell transplantation (HCT) co-morbidity index score \> 3; the principal investigator is the final arbiter of eligibility for comorbidity score \> 3 * Patients with prior coronary artery disease