Lam Nguyen malloxpb

Final report for GSoC 2018

Chau Tung Lam Nguyen

Scrapy project - Python Software Foundation

Summary

Code Repository: https://github.com/scrapy/scurl

FF delta process query improvement

As of the moment, we have this query to retrieve the zip files containing the delta FF data since the last full batch. However, not every zip file contains the lipper ids that we want to target. Therefore, we will need to create a new index that allows us to query only the files that are relevant to the lipper ids provided.

part 1: lambda function FF read process (3 point)

update the lambda function to read the incoming delta files content. The file names have this format flow_feed_004875.xml, while the content would look like

	<!DOCTYPE html>
	<html>
	<head>
	<meta charset="UTF-8">
	<title>Hello World!</title>
	</head>
	<body>
	<h1>Hello world !</h1>
	<!-- All of the Node.js APIs are available in this renderer process. -->
	We are using Node.js <script>document.write(process.versions.node)</script>,