Ian Cook ianmcook

First start an HTTP server to serve Arrow IPC stream data. You can do this using one of the server examples in HTTP GET Arrow Data: Simple Examples or simply by starting a Python HTTP server in the same directory where you have an Arrow IPC stream file (named file.arrows in this example).

python -m http.server 8008

Download the attached Python script script.py. You might need to do chmod +x script.py to make it executable.

Why IbisML?

This is a simple example demonstrating why you might want to use IbisML instead of just plain Ibis in an ML preprocessing pipeline.

Scenario

You are training an ML model that gets better accuracy when the floating point number columns in the training data are normalized (by subtracting the mean and dividing by the standard deviation). Your data contains multiple floating point columns.

To demonstrate this, we can use the iris flower dataset.

This is a set of examples demonstrating whether various Python and R dataframe libraries and OLAP query engines preserve (or do not preserve) the original order of the records in the data.

Example data

The examples all use this dataset describing the 28 times when a person walked on the moon:

year	mission	name	minutes
1969	Apollo 11	Neil Armstrong	151
1969	Apollo 11	Buzz Aldrin	151

	import pandas as pd
	import pyarrow as pa

	file_path = 'fruit.arrow'
	stream_path = 'fruit.arrows'

	df = pd.DataFrame(data={'fruit': ['apple', 'apple', 'apple', 'orange', 'orange', 'orange'],
	'variety': ['gala', 'honeycrisp', 'fuji', 'navel', 'valencia', 'cara cara'],
	'weight': [134.2 , 158.6, None, 142.1, 96.7, None]})

	{
	"statement": "SELECT * FROM MYTABLE",
	"resultSetMetaData": {
	"format": "arrowv1"
	},
	"timeout": 60,
	"database": "MYDATABASE",
	"schema": "MYSCHEMA",
	"warehouse": "MYWAREHOUSE",
	"role": "MYROLE"

	import ibis
	import random

	con = ibis.connect("duckdb://penguins.ddb")
	con.create_table(
	"penguins", ibis.examples.penguins.fetch().to_pyarrow(), overwrite = True
	)

	ibis.options.interactive = True

	import pyarrow as pa
	import numpy as np
	import pandas as pd

	# Create an array of some fixed-width type containing nulls
	a = pa.array(obj=pd.Series([1, 2, 3]), type=pa.int64(), mask=np.array([1, 0, 1], dtype=bool))

	# Get the values buffer as a bytearray
	b = a.buffers()
	v = bytearray(b[1].to_pybytes())

	/*
	* Licensed to the Apache Software Foundation (ASF) under one or more
	* contributor license agreements. See the NOTICE file distributed with
	* this work for additional information regarding copyright ownership.
	* The ASF licenses this file to You under the Apache License, Version 2.0
	* (the "License"); you may not use this file except in compliance with
	* the License. You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0
	*

	/*
	* Licensed to the Apache Software Foundation (ASF) under one
	* or more contributor license agreements. See the NOTICE file
	* distributed with this work for additional information
	* regarding copyright ownership. The ASF licenses this file
	* to you under the Apache License, Version 2.0 (the
	* "License"); you may not use this file except in compliance
	* with the License. You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0