Hoemas Ruud tomas-wood

Process Tables and Turn them Into Graphs!

We want to be able to take the data that people give us in CSV files and turn them into graphs.

Some approaches to doing this and my thoughts/feelings/opinions on each:

DON'T
Just don't do it. Let them get the data into a graph format we know and love, like a graph on GES we can access through an API. This is an extreme case, but I feel it merits thinking about. Processing data is hard to automate since data is generally noisy and we don't want to trust ourselves with being able to handle every corner case and user input problem that might arise. So we can just say

Hey! So you want to use the GES API to do gremlin queries? Well we can help you with that!

We're going to assume you have an account and have already uploaded your graph to GES. This walkthrough will help you get programmatic access to that graph from anywhere in the world.

Generate a token
Use this code and enter in your own region, username, password, and domain_name.

import urllib3

DATA INGESTION CONTINUED

I've been thinking about it, and if we're given CSV flat files, it shouldn't be that difficult to come up with an automated ingestion method that wouldn't necessarily require a schema if the CSV files have header rows. If we're going to be ingesting data from a SQL table, I can't really see a way around having a user define a schema because the columns in the table aren't necessarily ordered by a user-defined method. Pretty sure the automated way of getting the columns from a SQL table returns the column names in alphabetical order. So we would need to have users name their columns something specific, or more simply just have them provide us a schema.

My thoughts on the MLoG module are that it will serve to make it easier for people in industry to convert their tabular data into a graph format and leverage powerful Graph Machine Learning methods on their structured graph data. But it isn't just ingestion

input url: https://cdn.japantimes.2xx.jp/wp-content/uploads/2018/03/p9-masangkay-cathat-a-20180319-870x580.jpg

output error:

scene_graph_1  | Traceback (most recent call last):
scene_graph_1  |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in wsgi_app
scene_graph_1  |     response = self.full_dispatch_request()
scene_graph_1  |   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in full_dispatch_request
scene_graph_1  |     rv = self.handle_user_exception(e)

	import urllib3

	http = urllib3.PoolManager()

	with open('token', 'r') as f:
	token = f.read()

	# Headers are the way we pass through the token.
	headers = {}
	headers['Content-Type'] = 'application/json'

	2018-10-22 21:44:44.331954: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
	2018-10-22 21:44:44.399573: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
	2018-10-22 21:44:44.399625: W tensorflow/stream_executor/stream.cc:2089] attempting to perform BLAS operation using StreamExecutor without BLAS support
	E1022 21:44:44.403764 140092477048576 tf_logging.py:105] Blas GEMM launch failed : a.shape=(3, 12), b.shape=(12, 10), m=3, n=10, k=12
	[[{{node edge_block_1/mlp/linear_0/MatMul}} = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](edge_block_1/concat, edge_block/mlp/linear_0/w/read)]]

	Caused by op u'edge_block_1/mlp/linear_0/MatMul', defined at:
	File "blocks_test.py", line 1103, in <module>
	tf.test.main()
	File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/test.py", line 64, in main

	{u'adjacencies': [{u'data': {u'category': u'image-to-region',
	u'updated_on': u'13-Nov-2018 23:23:24'},
	u'nodeTo': u'https://cs.stanford.edu/people/rak248/VG_100K_2/4887.jpg'},
	{u'data': {u'category': u'region-to-label',
	u'updated_on': u'13-Nov-2018 23:23:24'},
	u'nodeTo': u'potted plant'}],
	u'data': {u'category': u'region', u'updated_on': u'13-Nov-2018 23:23:24'},
	u'id': u'1cba0c4a06bb4080a7ca06b6b2555aca',
	u'name': u'1cba0c4a06bb4080a7ca06b6b2555aca'}

	Scanning dependencies of target caffe2_hip
	[ 83%] Linking HIP shared library ../lib/libcaffe2_hip.so
	Stack dump:
	0. Program arguments: /opt/rocm/hcc/bin/llc -O2 -mtriple amdgcn--amdhsa-amdgiz -mcpu=gfx803 -filetype=obj -o /tmp/tmp.dO38OsZ6ZU/caffe2_hip_generated_ReduceOpsKernel.cu.kernel.bc-gfx803.isabin /tmp/tmp.dO38OsZ6ZU/caffe2_hip_generated_ReduceOpsKernel.cu.kernel.bc-gfx803.isabin.opt.bc
	1. Running pass 'CallGraph Pass Manager' on module '/tmp/tmp.dO38OsZ6ZU/caffe2_hip_generated_ReduceOpsKernel.cu.kernel.bc-gfx803.isabin.opt.bc'.
	2. Running pass 'Simple Register Coalescing' on function '@_ZN2at6native13reduce_kernelILi512ENS0_8ReduceOpIsZNS0_15sum_kernel_implIssEEvRNS_14TensorIteratorEEUlssE_EEEEvT0_'
	/opt/rocm/hcc/bin/llc(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x2a)[0x15554ba]
	/opt/rocm/hcc/bin/llc(_ZN4llvm3sys17RunSignalHandlersEv+0x4c)[0x15537ec]
	/opt/rocm/hcc/bin/llc[0x1553957]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fe7b73bd390]

	(2030, 'building', 'has', 'window'),
	(1703, 'man', 'has', 'hair'),
	(1437, 'man', 'has', 'hand'),
	(1334, 'man', 'has', 'head'),
	(1294, 'woman', 'has', 'hair'),
	(959, 'man', 'has', 'arm'),
	(853, 'tree', 'has', 'leaf'),
	(675, 'sky', 'has', 'cloud'),
	(656, 'cat', 'has', 'ear'),
	(653, 'man', 'has', 'leg'),