saswata-dutta

Getting started with EMRFS

The EMR File System (EMRFS) is an implementation of HDFS that all Amazon EMR clusters use for reading and writing regular files from Amazon EMR directly to Amazon S3.

How to access a file from S3 using EMRFS

Using Java

Coming from HDFS it is very easy to implement EMRFS. You just need to pass URI("s3://<bucket-name>") object while getting filesystem object.

package com.joe;

saswata-dutta

original on :https://www.webslesson.info/2018/09/insert-tree-view-node-using-php-ajax.html
http://www.techsapphire.in/index/grandparent_parent_and_child_hierarchy_sql_server_or_bread_crumb/0-170
https://www.youtube.com/watch?v=RswtHsz4v-0
https://www.ruvictor.com/article/php-recursive-functions
<?php

//fetch.php

include('database_connection.php');

saswata-dutta

Gremlin Cheat Sheet in Groovy

Gremin traversal examples taken from the excellent DS330: DataStax Enterprise Graph course.

Creating Vertices and Vertex Properties

Add a Vertex

Vertex u = graph.addVertex("user");
       u.property("userId","u2016");
 u.property("age",36);

saswata-dutta

    public static ExecutorService createDefaultPool() {
        int threads = Runtime.getRuntime().availableProcessors()*2;
        int queueSize = threads * 25;
        return new ThreadPoolExecutor(threads / 2, threads, 30L, TimeUnit.SECONDS, new ArrayBlockingQueue<>(queueSize),
                new CallerBlocksPolicy());
//                new ThreadPoolExecutor.CallerRunsPolicy());
    }
    static class CallerBlocksPolicy implements RejectedExecutionHandler {
        @Override
        public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {

saswata-dutta

package com.ossAccounts.ac.server;
import java.io.FileInputStream;

import org.dbunit.DBTestCase;
import org.dbunit.IDatabaseTester;
import org.dbunit.dataset.IDataSet;
import org.dbunit.dataset.xml.FlatXmlDataSetBuilder;
import org.dbunit.ext.mssql.InsertIdentityOperation;
import org.dbunit.operation.DatabaseOperation;
import org.hibernate.Session;

saswata-dutta

package utils

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.{DataFrame, Row}
import org.specs2.matcher.{Expectable, Matcher}
import org.specs2.mutable.Specification

/**
  * Utility class to compare DataFrames and Rows inside unit tests

saswata-dutta

import com.google.common.cache.CacheBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CachingConfigurer;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.guava.GuavaCache;
import org.springframework.cache.interceptor.CacheErrorHandler;
import org.springframework.cache.interceptor.CacheResolver;
import org.springframework.cache.interceptor.KeyGenerator;

saswata-dutta

/**
  * Generate Case class from DataFrame.schema
  *
  *  val df:DataFrame = ...
  *
  *  val s2cc = new Schema2CaseClass
  *  import s2cc.implicit._
  *
  *  println(s2cc.schemaToCaseClass(df.schema, "MyClass"))
  *

saswata-dutta

introduction

ScalaCheck 1.14.0 was just released with support for deterministic testing using seeds. Some folks have asked for examples, so I wanted to produce a Gist to help people use this feature.

simple example

These examples will assume the following imports:

saswata-dutta

# Luke's config for the Zoomer Shell

# Enable colors and change prompt:
autoload -U colors && colors
PS1="%B%{$fg[red]%}[%{$fg[yellow]%}%n%{$fg[green]%}@%{$fg[blue]%}%M %{$fg[magenta]%}%~%{$fg[red]%}]%{$reset_color%}$%b "

# History in cache directory:
HISTSIZE=10000
SAVEHIST=10000
HISTFILE=~/.cache/zsh/history