Skip to content

Instantly share code, notes, and snippets.

@adam-phillipps
Last active January 23, 2019 16:56
Show Gist options
  • Save adam-phillipps/ea8c24d19bfb62c390d778b7f9697ea7 to your computer and use it in GitHub Desktop.
Save adam-phillipps/ea8c24d19bfb62c390d778b7f9697ea7 to your computer and use it in GitHub Desktop.
a path-bug in a demo in the json_normalize function
args = (
[["State", "Name"]],
[
"InstanceId", "VpcId", "PublicIpAddress", "PrivateIpAddress",
"PublicDnsName", "PrivateDnsName", "ImageId", "SubnetId",
"KeyName", "InstanceType", "LaunchTime",
["State", "Name"]
]
)
clean = lambda data: pd.io.json.json_normalize(data, *args, errors='ignore')
ec2 = boto3.client('ec2')
i, r = "Instances", "Reservations"
instances = [reserv[i][-1] for reserv in ec2.describe_instances()[r] if i in reserv]
data = [clean(inst) for inst in instances]
results_table = pd.concat(data)
unique_records_table = results_table.drop_duplicates("InstanceId")
unique_records_table
0 InstanceId VpcId PublicIpAddress ... KeyName InstanceType LaunchTime State.Name
0 r i-08071c704a45c09af vpc-c4e93fa1 NaN ... key-name-name-of-it t2.medium 2019-01-23 12:01:21+00:00 running
0 r i-c5707433 vpc-c4e93fa1 12.23.345.567 ... key-name-name-of-it t2.small 2015-06-17 01:29:14+00:00 running
0 r i-05a817254f1f0bf3a vpc-c4e93fa1 NaN ... key-name-name-of-it t2.medium 2018-09-04 15:38:35+00:00 running
0 r i-262f46e3 vpc-c4e93fa1 12.23.345.567 ... key-name-name-of-it t2.small 2015-08-26 17:57:54+00:00 running
0 r i-04ae135070a1ae6c3 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.small 2019-01-23 12:01:21+00:00 running
0 r i-0a3ba15d3ba74903b vpc-c4e93fa1 NaN ... key-name-name-of-it t2.medium 2018-03-12 18:15:34+00:00 running
0 s i-83356d46 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.small 2015-09-11 17:30:31+00:00 stopped
0 r i-070045e7be71e5268 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.medium 2018-07-17 17:03:11+00:00 running
0 r i-0e0ce3b7ad6429af9 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.micro 2018-08-31 16:21:39+00:00 running
0 r i-0ae87df89b65c2943 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.small 2018-02-28 21:01:21+00:00 running
0 r i-0c8321aee91358c40 vpc-c4e93fa1 NaN ... key-name-name-of-it t2.medium 2018-11-11 02:03:50+00:00 running
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment