Show HN: 500 Years of Joseon Court Omens as an Observability Dashboard

Five centuries of Joseon Dynasty omens are now being tracked using modern observability tools. By applying time-series database principles to historical records, researchers can visualize patterns in ancient court predictions. This approach transforms unstructured text into actionable, real-time metrics. The project demonstrates how to ingest historical text into a structured JSON format for database ingestion, enabling the creation of a functional Grafana dashboard for long-term trend analysis. By following this guide, you will learn to map ancient celestial and terrestrial events into a high-performance InfluxDB schema. You will move from raw, noisy archives to a live monitoring system capable of querying five hundred years of historical volatility.

Step 1: Ingest and Parse Historical Text Data

By the end of this step, you will have a cleaned, structured JSON file containing filtered historical omens ready for database ingestion. This file serves as the single source of truth for your dashboard.

Prerequisites

• Python 3.10 or higher installed
• pandas library (version 2.0+) installed
• Access to the raw dataset from the original project repository

Prepare the Environment

Install the necessary data processing library by running the following command:

pip install pandas

This provides the DataFrame structures needed to manipulate large text datasets.

Create the Parsing Script

Create a new Python file named parse_omens.py. Use the following code to load your raw dataset into a pandas DataFrame:

# Load the raw dataset (assuming CSV format from the source)
df = pd.read_csv('raw_joseon_records.csv')

Note: The source repository provides data in various formats. Ensure your loading function matches your specific file type, such as JSON or CSV.

Filter for Omen Events

Historical records are often noisy and contain irrelevant administrative text. You must isolate entries that describe significant events. Add this logic to your script:

# Filter rows where the content contains any keyword
filtered_df = df[df['content'].str.contains('|'.join(keywords), case=False, na=False)]

If you see a TypeError regarding string operations, ensure the 'content' column is treated as a string. Add this line before the filter:

df['content'] = df['content'].astype(str)

Standardize Timestamps

Modern time-series databases require consistent date formats. Extract the date field and convert it to the ISO 8601 standard:

python filtered_df['timestamp'] = pd.to_datetime(filtered_df['date_column']).dt.strftime('%Y-%m-%dT%H:%M:%SZ')

This ensures that the 500 years of data^[1] can be plotted accurately on a timeline.

Verify and Save the Output

Verify that the filtering worked by printing the first 10 rows of your new DataFrame. Run your script and check the console:

print(filtered_df.head(10))

You should see output containing only rows with dates and descriptions of specific events.

Finally, save the cleaned dataset to a new file:

filtered_df.to_json('omens_cleaned.json', orient='records', indent=4)

This file is now ready for the schema mapping and InfluxDB ingestion process.

Step 2: Design the Time-Series Schema

To treat historical omens as observability metrics, you must map unstructured text to a rigid time-series schema. This schema requires three core fields: timestamp, event_type, and severity_score.

Install InfluxDB

First, start the database using Docker. This ensures the environment matches the configuration used in the original project link^[1].

Run the following command:

docker run -p 8086:8086 influxdb:2.7

Wait for the container to initialize. You can verify it is running by checking your active Docker containers.

Create a Database Bucket

Next, configure a storage location for your data. Open the InfluxDB UI at http://localhost:8086.

1. Navigate to the "Buckets" section.
2. Click "Create Bucket".
3. Name the bucket joseon_omens.
4. Set the retention policy to 0.

Setting the retention to zero ensures the data is kept indefinitely. This is necessary because historical records do not expire.

Define the Schema Mapping

Before writing data, you must define how your JSON keys map to InfluxDB tags and fields. Tags are indexed and allow for fast filtering, while fields hold the actual values.

Map the event_type to a tag key. Use values such as celestial, terrestrial, or political. This allows you to group and filter your dashboard by event category.

Map the severity_score to a field value. Assign a numerical integer between 1 and 10 based on the impact described in the record. This enables you to perform mathematical functions like averages or sums over time.

Ingest the Data

Create a Python script named write_to_influx.py. You will need the influxdb-client-python library installed in your environment.

Write the script to perform the following actions:

1. Import the influxdb_client library.
2. Connect to the local InfluxDB instance using your API token.
3. Load the omens_cleaned.json file.
4. Iterate through each record in the JSON file.
5. Write each record to the joseon_omens bucket.

Run the script using:

python write_to_influx.py

You should see success messages in your terminal console.

If you see an AuthenticationError, check your API token. Ensure you are using the specific token generated during the InfluxDB setup rather than a default password.

Verify the Ingestion

Finally, confirm the data is queryable. Open the "Data Explorer" in the InfluxDB UI.

Run a simple Flux query to count records by type:

from(bucket: "joseon_omens") |> range(start: 0) |> filter(fn: (r) => r["_measurement"] == "omens") |> group(columns: ["event_type"]) |> count()

You should see a table listing the frequency of each omen type.

By structuring these historical events as time-series metrics, you transform a static archive into a dynamic dataset ready for modern observability logic.

Step 3: Visualize Trends and Configure Alerts

By the end of this step, you will have a functional Grafana dashboard that visualizes five centuries of historical omens and triggers alerts based on event frequency.

Install Grafana

To ensure version consistency with your InfluxDB container, use Docker Compose to run Grafana 10+.

Create a docker-compose.yml file in your project directory. Add the following service definition:

yaml services: grafana: image: grafana/grafana:10.0.0 ports: - "3000:3000" depends_on: - influxdb

Run the following command in your terminal:

docker-compose up -d

This starts the Grafana instance alongside your database. You can verify the service is running by navigating to http://localhost:3000 in your browser. You should see the Grafana login screen.

Connect InfluxDB Data Source

Grafana needs permission to query your historical metrics.

1. Log in to Grafana using the default credentials (admin/admin).
2. Navigate to the Connections menu and select Data Sources.
3. Click Add data source and choose InfluxDB.
4. Enter the URL: http://localhost:8086.
5. Set the Query Language to Flux.
6. Paste your InfluxDB API token into the Auth section.

Click Save & test. You should see a green message: "Data source is working."

Create the Omens Dashboard

Now you will build the visual interface to explore the 500 years of Joseon omens^[1].

1. Click the Dashboards icon and select New > Dashboard.
2. Click Add visualization and select your InfluxDB data source.
3. Name the dashboard Joseon Court Omens.

Visualize Event Frequency

Use a time-series panel to track how often different types of omens occur over the centuries.

In the Flux query editor, enter the following query:

flux from(bucket: "joseon_omens") |> range(start: 0) // Covers all historical data |> filter(fn: (r) => r["_measurement"] == "omens") |> aggregateWindow(every: 1mo, fn: count) |> group(columns: ["event_type"])

This query aggregates the count of every omen by month. The group function ensures that 'celestial' and 'terrestrial' events appear as separate lines on the chart.

Check the panel. You should see distinct lines representing different event types. Look for spikes that correspond to known periods of historical instability or major celestial events.

Map Severity with Heatmaps

To visualize the intensity of perceived cosmic unrest, add a heatmap panel.

1. Add a new panel to your dashboard.
2. Change the visualization type to Heatmap.
3. Use a Flux query to target the severity_score field:

flux from(bucket: "joseon_omens") |> range(start: 0) |> filter(fn: (r) => r["_field"] == "severity_score") |> aggregateWindow(every: 1y, fn: mean)

This highlights periods where the average severity of recorded events was significantly higher. This provides a visual heat index of historical volatility.

Configure Threshold Alerts

Finally, set up an automated alert to notify you when celestial activity becomes unusually high.

1. Open the settings for your time-series panel.
2. Navigate to the Alerts tab and click Create alert rule from this panel.
3. Set the condition to trigger when the count of event_type == "celestial" exceeds 5 within a single month.
4. Define the evaluation interval (e.g., 1m) and the pending period (e.g., 5m).

To test the alert, you can manually inject a high-frequency data point into InfluxDB using a Python script. If the condition is met, the alert state in Grafana should transition to Pending and then to Firing.

You now have a fully functional historical observability dashboard. The system allows you to query five centuries of data using modern infrastructure, effectively turning a static archive into a live monitoring tool.

Developers working with legacy or unstructured data can apply this exact pattern to any text archive. The core principle remains that structure enables scale; by defining your schema before ingestion, you ensure that even the most ancient records can be indexed and queried with modern efficiency.