filipelenfers

# !!!THIS SCRIPT ASSUME HIVE COMPATIBLE PARTITIONS!!!
#
# When you get the following error:
# HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced.
# what happened is that some column type changed in the table, and partitions have they own version of the schema, 
# and the version that partitions have cached are not compatible with the table actual version on glue.
# We need to:
# 1. remove partitions
# 2. add partitions again
#

filipelenfers

import io
import snappy
import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter, BinaryEncoder

#Generate avro bytes as a in-memory file, with the schema.
def generate_avro_bytes(schema,avro_dicts, codec='snappy'):
    bytes_writer = io.BytesIO()
    writer = DataFileWriter(bytes_writer, DatumWriter(), schema, codec=codec)

filipelenfers

Avro serialization

There are 4 possible serialization format when using avro:

• Avro Json encoding
• Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
• Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
• Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

filipelenfers

def dict_to_item(raw:dict):
    return {
            key: _dict_to_item_recurse(value)
            for key, value in raw.items()
        }


def _dict_to_item_recurse(raw):
    if isinstance(raw, dict):
        return {

filipelenfers

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu/bionic64"
  
  config.vm.network "forwarded_port", guest: 8080, host: 8080

  config.vm.provider "virtualbox" do |vb|

filipelenfers

# Apache Airflow Scheduler Service file for Ubuntu 18
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
# 

filipelenfers

# Apache Airflow Service file for Ubuntu 18
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
# 

filipelenfers

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

filipelenfers

-*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at

filipelenfers

//I like to use zsh(with oh-my-zsh) as my default shell.

//First define your default distro for wsl
//run this on cmd.exe
//wslconfig /l
//wslconfig /s Ubuntu

//use this options on preferences of Hyper.js termina to use it as default.
{
  shell: 'C:\\Windows\\System32\\bash.exe',