This phase I/II trial is designed to study the side effects, best dose and efficacy of adding hydroxychloroquine to dabrafenib and/or trametinib in children with low grade or high grade brain tumors previously treated with similar drugs that did not respond completely (progressive) or tumors that came back while receiving a similar agent (recurrent). Patients must also have specific genetic mutations including BRAF V600 mutations or BRAF fusion/duplication, with or without neurofibromatosis type 1. Neurofibromatosis type 1 is an inherited genetic condition that causes tumors to grow on nerve tissue. Hydroxychloroquine, works in different ways to stop the growth of tumor cells by killing the cells or stopping them from dividing. Trametinib and dabrafenib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving hydroxychloroquine with trametinib and/or dabrafenib may lower the chance of brain tumors growing or spreading compared to usual treatments.