OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining

We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these w...