Skip to content

Instantly share code, notes, and snippets.

View de-sh's full-sized avatar

Devdutt Shenoi de-sh

192 followers · 44 following
View GitHub Profile
@de-sh
de-sh / week_nine.md
Last active August 6, 2020 15:09
A report of work done during the ninth week of the Google Summer of Code 2020 program

July 26 - August 01: Most of this week was used up in fixing the build process to run the CI on the newly added features, fixing the manner in which it handled the installation of AWS SDK as well as providing a stable based for starting work on tikv/tikv. Yi also opened an issue(#8367) which I have started solving with a draft PR(#8383). The idea is to progressively add a variable to DbConfig called s3 which will itself be composed of an enabled flag to mark the use of S3 as a storage engine.

I have been facing a lot of hurdles in building without error/warning. I think there is an issue with cargo-nightly.

Update: There was infact an error with cargo-nightly! The Cargo.lock was being forced to update on every cargo build thus there was a lot of version mismatch that was not meant to be. This was spotted in 1.47.0-nightly 2020-07-26 and solved in [1.47.0-nightly 2020-08-05](https:

@de-sh
de-sh / cncf_tikv_final_gsoc_submission.md
Last active August 24, 2020 19:09
A summary of achievements made through the Google Summer of Code program with CNCF, in adding a cloud native storage engine to the TiKV project

The project proposal I had submitted aimed to take TiKV closer to full cloud native support. A vague understanding of the project allowed me to put forward the idea of supporting cloud based data stores such as AWS S3 to the TiKV storage engine, i.e. rocksdb, by making use of the rocksdb-cloud library. The ideas put forward by the team at TiKV also helped to better formulate a plan that has been worked on through various PRs to the sister projects that associate with the storage component of the Database. The main issue being worked on was initiated well before the GSoC project started and can be found at tikv/tikv#6506.

during the community boding period, Yi Wu(@yiwu-arbug) ideated further on the project with Liu Wei(@little-wallace), who compiled a [document

@de-sh
de-sh / largest_9dig_prime.py
Last active August 12, 2020 07:58
def is_prime(n):
if n == 1:
return False
i = 2
while i*i <= n:
if n % i == 0:
return False
i += 1
return True
@de-sh
de-sh / week_eleven.md
Last active August 30, 2020 17:26
A report of work done during the eleven week of the Google Summer of Code 2020 program

August 9th - 15th: While working on using our self compiled TiKV server with the TiUP utility, it was found out that the TiKV server was unable to compile properly with AWS support, as the USE_AWS=1 definition wasn't being set properly. The solution to the problem lied in moving this step down line to the librocksdb_cloud_sys/CMakeLists.txt from the associated build.rs file. It was also learnt from this that various code components of rocksdb-cloud/cloud weren't properly written and thus had to be bug fixed. Two PRs as follows were opened to do the same:

Associated fixes to rust-rocksdb are also made through tikv/rust-rocksdb#529 and have been set to WIP as they require a solution to the linking problem we are facing with the AWS-CPP-SDK. If you have a solution to the problem, or would like

@de-sh
de-sh / week_ten.md
Last active August 30, 2020 17:17
A report of work done during the tenth week of the Google Summer of Code 2020 program

August 2nd - 8th : Yi directed me to document the observable performance metrics of AWS S3 using the fio utility. I have converted the logs that were output by fio into the following gist:

Most of the rest of this week went into crafring a TiKV config file that can be used with S3, one must be able to make the convenient addition of the following configurations to their regular TiKV config to access the feature on a build that has cloud enabled for engine_rocksdb:

[rocksdb.s3]
enabled = true
src-cloud-bucket = "{aws_bucket_name}"
src-cloud-object = "{aws_object_name}"
@de-sh
de-sh / week_twelve.md
Last active August 30, 2020 17:48
A report of work done during the twelve week of the Google Summer of Code 2020 program

August 16th - 24th: The final week of coding was focused mostly on solving the issues that were found in the linking process that was blocking the test stage. Please refer to the report for week eleven for further details on what conspired in this week.

Update: the issues still persists to the day of work product submission. Yi and I have decided to submit despite a completely working product not having been created from the project due to adverse circumstances. I believe that the main achievement of the task was the formulation of a path that can be followed to fulfill the goal of linking and utilising cloud features for storage within the TiKV server at a later stage.

@de-sh
de-sh / egyptian_frac.md
Last active October 13, 2020 18:53
Converting the Regular to Egyptian fraction python program into rust

The problem stated that we need to find the Egyptian fraction sequence given the numerator and denomintor. So I started out with a simple python psuedo code implementation:

ef = []
while nr != 0:
    x = ceil(dr/nr)
    ef += [x]
    # Update remainder values of numerator and denominator
    nr *= x
    nr -= dr
    dr *= x
@de-sh
de-sh / dstore_story.md
Last active July 2, 2021 06:20
Building dstore, a partially distributed memory store

At the end of 2020 I had learnt a lot about programming in rust-lang and had worked on some large codebases, all while doing some interesting system implementations on the side. We had planned to implement our thesis project by the end of the academic year, but faced a lot of challenges in getting started, procrastination had hit us hard. By January I felt we had done enough experimentation and that we needed to do some "real" work, and hence got started on implementing a core piece of the puzzle, the partially distributed datastore, which consisted of a central server and multiple clients with the ability to cache values and check if invalidated.

To do this, I had to learn gRPC, for which I figured out Tonic would be a great place to start with, and thus, with the code that @bkp31415 had written in proto3, I got started implementing what would be [the first iteration of dstore's gRPC API](https://github.com/vyuham/dstore/commit/727d56de805

@de-sh
de-sh / async-learnings.md
Last active October 17, 2021 08:14
Async with rust, a collection of learnings

I wrote my first few lines of async rust at the end of 2020, that was the first time I was actually learning about the core principles behind async/await, even thought I have had prior experience writing code with similar keywords in javascript and python, thoug they didn't make much sense then, it was just what I had to write to get through implementing something in my code. Having been pulled into the async world while writing network facing code, I found it to be a learning that also transfered well into college as we were learning about similar stuff in a class on Programming Paradigmns.

The journey into async-rust started for me with some exposure through a well written tutorial on the tokio project's website, which introduced the why in such a way that async/await totally made sense. This piece is a recollection of material that I've used to make sense of and interpret the crazy complex world of async/await over the past year or so. Anyways, if you are new and only ge

@de-sh
de-sh / share_manager.py
Last active November 21, 2021 11:27
Extract info about shares purchased/sold/held and more from zerodha tradebook csv
import csv
shares = {} # Per share/scrip info
holding = {} # Held share
bought = {} # Purshased shares to date
sold = {} # Sold shares to date
with open('tradebook.csv') as csvfile:
for row in csv.reader(csvfile):
scrip, nos, price, date = row[0], float(row[7]), float(row[8]), row[2]