Skip to content

Instantly share code, notes, and snippets.

How to use GDELT

× This site has no affiliated with GDELT (gdeltproject.org), but does provide an interface to some of its data services - specifically its DOC, GEO and Television APIs. These services should not be confused with the GDELT Events databases to which "GDELT" is most closely associated by some, although they are related.

Using this site you can do things like:

search in English for global content in any or all languages on a particular topic search for specific content or published in the past hour from eg. Japan, or in Japanese, or referencing a Japanese location find content based on features and text in its imagery investigate and compare media trends over time

@donbr
donbr / knowledge-graphs-ontology-free.md
Last active February 20, 2025 02:32
Scaling Knowledge Graphs for Real-Time News Processing: An Ontology-Free Approach

Scaling Knowledge Graphs for Real-Time News Processing: An Ontology-Free Approach

Rob Kalk
Emergent Methods

Abstract

Real-time news processing requires scalable and efficient methodologies for ingesting and structuring large volumes of information. This paper presents the approach employed by Emergent Methods in building a large-scale news knowledge graph, processing over a million entities per day. A key innovation is the introduction of on-the-fly subgraphs, allowing targeted domain-specific explorations. Unlike traditional knowledge graphs that rely on predefined ontologies, this system employs an ontology-free approach, leveraging fine-tuned language models to extract relationships dynamically. We discuss the core architecture, including hybrid vector and graph database storage, fine-tuning techniques using Phi-3, and strategies for mi

@donbr
donbr / gdelt_app_v5.py
Last active February 10, 2025 01:28
gdelt_app_v5.py
import os
import asyncio
from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash
from prefect.blocks.system import Secret, JSON
from prefect.task_runners import ConcurrentTaskRunner
from prefect.concurrency.sync import concurrency
from pathlib import Path
import datetime
from datetime import timedelta
@donbr
donbr / hypergraph.md
Last active February 6, 2025 03:29
Hypergraph Network

Hypergraph Summary

This document provides a comprehensive overview of the Constellation Network's Hypergraph, a decentralized network protocol. The Hypergraph utilizes a directed acyclic graph (DAG) structure, enabling parallel transaction processing for superior scalability compared to traditional blockchains. Its layered architecture features a Global L0 layer for final consensus and immutable data storage, and independent, customizable subnetworks called metagraphs that handle specific functions and data types before submitting snapshots to the Global L0. Key themes include the network's innovative consensus mechanism (Proof-of-Reputable Observation), its flexible tokenomics model (Metanomics) for the native DAG token, and the powerful functionality of metagraphs as building blocks for diverse applications. The overall purpose is to detail the structure and function of this novel blockchain alternative, highlighting its advantages in scalability, security, and interoperability.

Hypergraph Network St

@donbr
donbr / gdelt-to-graph-dataflow.md
Created February 10, 2025 01:40
A Path to Insight?: End to End data flow from GDELT to Knowledge Graphs
@donbr
donbr / competency-questions-economics-finance.md
Last active February 12, 2025 01:40
Competency Questions - economics & finance

title: Competency Questions - economics & finance author:

  • Don created: 2025-02-11 tags:
  • cq
  • competency
  • ontology
  • requirements
@donbr
donbr / gdelt_2025_v1.py
Created February 11, 2025 20:59
GDELT 2025 Prefect ETL process
import os
import asyncio
from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash
from prefect.blocks.system import Secret, JSON
from prefect.task_runners import ConcurrentTaskRunner
from prefect.concurrency.sync import concurrency
from pathlib import Path
import datetime
from datetime import timedelta
@donbr
donbr / gdelt-pipeline-tied-to-usecase.md
Last active February 20, 2025 01:56
gdelt-pipeline-tied-to-usecase.md

Crisis Analysis Pipeline: From GDELT Events to Actionable Insights

Integrated Architecture

flowchart TD
    A[GDELT Raw Data] --> B[Event Extraction]
    B --> C{Temporal Graph Builder}
    C --> D[Neo4j Knowledge Graph]
    D --> E[Replay Mode Analysis]
    D --> F[Counterfactual Testing]
@donbr
donbr / gdelt-insight-mermaid-flow.md
Last active February 16, 2025 05:28
gdelt-insight-mermaid-flow.md
flowchart TD
  %% Home Page Navigation
  subgraph Home["Home Page (app.py)"]
    A[User] -->|Selects analysis mode| B[Sidebar Navigation]
    B --> C{Available Pages}
    C -->|COVID Navigator| Page1
    C -->|COVID Event Graph Explorer| Page2
    C -->|Global Network Analysis| Page3
    C -->|Feb 2025 Navigator| Page4
@donbr
donbr / prefect-colab-event-mentions.py
Last active February 21, 2025 06:18
prefect-colab-event-mentions.py
# -*- coding: utf-8 -*-
"""prefect-colab-event-mentions.ipynb
Automatically generated by Colab.
Original file is located at
https://colab.research.google.com/drive/1KZRhjRazTGl7tjyG91Y9uJNDyxxcI2CY
"""
# Commented out IPython magic to ensure Python compatibility.