Irfan Danish irdanish11

🎯

Focusing

Senior MLOps Engineer | Architecting Agentic AI & Production-Grade ML Systems

irdanish11 / corpus_read.py

Last active October 16, 2019 08:25

Reading and splitting the data into tokens.

	def read_file(filepath):
	with open(filepath) as f:
	str_text = f.read()
	return str_text

	text = read_file('NameofYourFile.txt')
	tokens = text.split(" ")

irdanish11 / preprocessing.py

Last active October 16, 2019 08:14

Removing Redundant data, cleaning the data, and removing the sentences that are too short are too long.

	#removing the redundant lines
	start_time = time.time()
	unique_data = []
	for i in range(len(data)):
	if data['description'][i] not in unique_data:
	unique_data.append(data['description'][i])
	if i % 5000 == 0:
	print('{0}'.format(i)+' lines have been processed')
	else:
	None