The project was to do PDF data extraction and annotation for a LLM that was to be based on the full text of 13 books in PDF form. The PDFs had OCR'd text of middling-to-poor quality, so extracting the full text was the major challenge. Then the text needed to be cleaned up, stripped of irrelevant details, and broken into labelled chunks. Finally it was parcelled into individual chunks and, using the OpenAI API, I generate two question & answer pairs for each chunk to train the model with.
Built with
- Data Engineering
- Data Annotation
- Data Wrangling
- Data Extraction