dvu4/Get latest record based on timestamp using Pyspark Window.md

Last active August 5, 2024 23:12

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/dvu4/a83c9bd817e6c8101c1c028b951cfbad.js"></script>
Save dvu4/a83c9bd817e6c8101c1c028b951cfbad to your computer and use it in GitHub Desktop.

Raw

Get latest record based on timestamp using Pyspark Window.md

 window_spec = Window.partitionBy("rxa_claim_id", "ITEM_PRODUCT_CODE").orderBy(df_idi["EVENT_TIMESTAMP"].desc())
 df_valid_claim = df_valid_claim.withColumn("row_number", F.row_number().over(window_spec))
 df_valid_claim.display()

window_spec_2 = Window.partitionBy("rxa_claim_id", "ITEM_PRODUCT_CODE")
valid_claim =  df_valid_claim.withColumn("row_number_count", F.count("row_number").over(window_spec_2))
df_valid_claim.display()