spaCy sentencizer script

01-preprocess-sentences.py

import spacy
import srsly  # to easily read/write JSONL etc.

nlp = spacy.load("en_core_web_sm")  # or whatever you need
examples = srsly.read_jsonl("./data.jsonl")
texts = (eg["text"] for eg in examples)

new_examples = []
for doc in nlp.pipe(texts):
    for sent in doc.sents:
        new_examples.append({"text": sent.text})
        
srsly.write_jsonl("./data-with-sentences.jsonl", new_examples)

data-with-sentences.jsonl

{"text":"Prodigy is a scriptable annotation tool so efficient that data scientists can do the annotation themselves, enabling a new level of rapid iteration."}
{"text":"Today\u2019s transfer learning technologies mean you can train production-quality models with very few examples."}
{"text":"With Prodigy you can take full advantage of modern machine learning by adopting a more agile approach to data collection."}
{"text":"You'll move faster, be more independent and ship far more successful projects."}

data.jsonl

{"text": "Prodigy is a scriptable annotation tool so efficient that data scientists can do the annotation themselves, enabling a new level of rapid iteration. Today’s transfer learning technologies mean you can train production-quality models with very few examples. With Prodigy you can take full advantage of modern machine learning by adopting a more agile approach to data collection. You'll move faster, be more independent and ship far more successful projects."}