chore: commit untracked UE/scaffold files (repo cleanup triage 20260707)
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Dubai Metaverse Team
234
data/OSM_DATA_RESEARCH.md
Normal file
234
data/OSM_DATA_RESEARCH.md
Normal file
@@ -0,0 +1,234 @@
|
||||
# OpenStreetMap Data Research - Dubai Marina
|
||||
|
||||
## Overview
|
||||
|
||||
Research and documentation for acquiring OpenStreetMap (OSM) data for Dubai Marina district.
|
||||
|
||||
## Target Area
|
||||
|
||||
**Location:** Dubai Marina, Dubai, UAE
|
||||
**Coordinates:** 25.0772° N, 55.1394° E
|
||||
**Bounding Box:**
|
||||
- **Min:** 25.07° N, 55.13° E
|
||||
- **Max:** 25.09° N, 55.15° E
|
||||
- **Format:** `25.07,55.13,25.09,55.15`
|
||||
|
||||
## Data Sources
|
||||
|
||||
### 1. Overpass API (Recommended)
|
||||
|
||||
**Endpoint:** https://overpass-api.de/api/interpreter
|
||||
|
||||
**Query for Buildings:**
|
||||
```
|
||||
[out:xml][bbox:25.07,55.13,25.09,55.15];
|
||||
(way["building"];);
|
||||
out meta;
|
||||
```
|
||||
|
||||
**Query for Roads:**
|
||||
```
|
||||
[out:xml][bbox:25.07,55.13,25.09,55.15];
|
||||
(way["highway"];);
|
||||
out meta;
|
||||
```
|
||||
|
||||
**Query for Complete Area Data:**
|
||||
```
|
||||
[out:xml][bbox:25.07,55.13,25.09,55.15];
|
||||
(
|
||||
way["building"];
|
||||
way["highway"];
|
||||
relation["building"];
|
||||
);
|
||||
out meta;
|
||||
```
|
||||
|
||||
### 2. Geofabrik Downloads
|
||||
|
||||
**Regional Extract:** Middle East
|
||||
- **URL:** https://download.geofabrik.de/asia/united-arab-emirates.html
|
||||
- **Format:** OSM PBF
|
||||
- **Size:** ~50-100MB for UAE
|
||||
- **Update Frequency:** Daily
|
||||
|
||||
### 3. Planet OSM
|
||||
|
||||
**Full Planet Extract:**
|
||||
- **URL:** https://planet.openstreetmap.org/
|
||||
- **Format:** OSM XML or PBF
|
||||
- **Size:** Very large (entire planet)
|
||||
- **Use:** Extract UAE region using osmium-tool
|
||||
|
||||
## Data Acquisition Methods
|
||||
|
||||
### Method 1: Overpass API (Quick, Small Areas)
|
||||
|
||||
**Using curl:**
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "data=[out:xml][bbox:25.07,55.13,25.09,55.15];(way[\"building\"];);out meta;" \
|
||||
"https://overpass-api.de/api/interpreter" \
|
||||
> dubai_marina.osm
|
||||
```
|
||||
|
||||
**Using Python (overpy library):**
|
||||
```python
|
||||
import overpy
|
||||
|
||||
api = overpy.Overpass()
|
||||
query = """
|
||||
[bbox:25.07,55.13,25.09,55.15];
|
||||
(way["building"];);
|
||||
out body;
|
||||
"""
|
||||
result = api.query(query)
|
||||
```
|
||||
|
||||
### Method 2: Geofabrik Download (Larger Areas)
|
||||
|
||||
```bash
|
||||
# Download UAE extract
|
||||
wget https://download.geofabrik.de/asia/united-arab-emirates-latest.osm.pbf
|
||||
|
||||
# Extract Dubai Marina area using osmium-tool
|
||||
osmium extract -b 55.13,25.07,55.15,25.09 \
|
||||
united-arab-emirates-latest.osm.pbf \
|
||||
-o dubai_marina.osm.pbf
|
||||
```
|
||||
|
||||
### Method 3: Using OSMnx (Python)
|
||||
|
||||
```python
|
||||
import osmnx as ox
|
||||
|
||||
# Define area
|
||||
place = "Dubai Marina, Dubai, UAE"
|
||||
bounds = (25.07, 55.13, 25.09, 55.15)
|
||||
|
||||
# Download buildings
|
||||
buildings = ox.geometries_from_bbox(
|
||||
north=25.09, south=25.07,
|
||||
east=55.15, west=55.13,
|
||||
tags={'building': True}
|
||||
)
|
||||
|
||||
# Download roads
|
||||
roads = ox.graph_from_bbox(
|
||||
north=25.09, south=25.07,
|
||||
east=55.15, west=55.13,
|
||||
network_type='drive'
|
||||
)
|
||||
```
|
||||
|
||||
## Data Processing
|
||||
|
||||
### Convert OSM to GeoJSON
|
||||
|
||||
**Using osmtogeojson:**
|
||||
```bash
|
||||
npm install -g osmtogeojson
|
||||
osmtogeojson dubai_marina.osm > dubai_marina.geojson
|
||||
```
|
||||
|
||||
**Using Python (osmium):**
|
||||
```python
|
||||
import osmium
|
||||
import json
|
||||
|
||||
class BuildingHandler(osmium.SimpleHandler):
|
||||
def __init__(self):
|
||||
osmium.SimpleHandler.__init__(self)
|
||||
self.buildings = []
|
||||
|
||||
def way(self, w):
|
||||
if 'building' in w.tags:
|
||||
# Extract building geometry
|
||||
# Convert to GeoJSON
|
||||
pass
|
||||
|
||||
handler = BuildingHandler()
|
||||
handler.apply_file("dubai_marina.osm")
|
||||
```
|
||||
|
||||
### Process with our script
|
||||
|
||||
**Using our import script:**
|
||||
```bash
|
||||
python3 scripts/data/import_osm_data.py \
|
||||
--output data/processed/dubai_marina_buildings.geojson \
|
||||
--bounds "25.07,55.13,25.09,55.15"
|
||||
```
|
||||
|
||||
## Expected Data
|
||||
|
||||
### Buildings
|
||||
- **Estimated Count:** 50-100 buildings in Dubai Marina area
|
||||
- **Types:** Residential towers, commercial buildings, hotels
|
||||
- **Key Buildings:**
|
||||
- Cayan Tower (Hero landmark)
|
||||
- Marina Towers
|
||||
- Various residential complexes
|
||||
|
||||
### Roads
|
||||
- **Main Roads:**
|
||||
- Sheikh Zayed Road (E11)
|
||||
- Dubai Marina Walk
|
||||
- Various internal roads
|
||||
|
||||
### Additional Features
|
||||
- Water features (marina)
|
||||
- Parks and green spaces
|
||||
- Parking areas
|
||||
- Sidewalks
|
||||
|
||||
## Data Quality Considerations
|
||||
|
||||
### Accuracy
|
||||
- OSM data is community-sourced
|
||||
- Building footprints may not be 100% accurate
|
||||
- Heights are often missing
|
||||
- Some buildings may be missing
|
||||
|
||||
### Validation
|
||||
- Cross-reference with satellite imagery
|
||||
- Verify building positions
|
||||
- Check for missing major buildings
|
||||
- Validate road network
|
||||
|
||||
### Enhancement
|
||||
- Add building heights from other sources
|
||||
- Refine building footprints using satellite imagery
|
||||
- Add missing buildings manually
|
||||
- Enhance road network details
|
||||
|
||||
## Recommended Approach
|
||||
|
||||
1. **Initial Acquisition:** Use Overpass API for quick download
|
||||
2. **Validation:** Compare with satellite imagery
|
||||
3. **Enhancement:** Add missing buildings and details
|
||||
4. **Processing:** Convert to GeoJSON for Unreal import
|
||||
5. **Import:** Use our import script to bring into Unreal
|
||||
|
||||
## Tools Required
|
||||
|
||||
- **curl** or **wget** - Download data
|
||||
- **Python 3.8+** - Processing scripts
|
||||
- **overpy** - Python OSM API client
|
||||
- **osmnx** - Advanced OSM data handling (optional)
|
||||
- **osmtogeojson** - OSM to GeoJSON converter (optional)
|
||||
- **osmium-tool** - Advanced OSM data processing (optional)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Run `data/acquire_osm_data.sh` to download initial data
|
||||
2. Validate data quality
|
||||
3. Process with `scripts/data/import_osm_data.py`
|
||||
4. Import to Unreal Engine after project creation
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2024-11-21
|
||||
**Status:** Research Complete, Ready for Data Acquisition
|
||||
|
||||
25
data/acquire_osm_data.sh
Executable file
25
data/acquire_osm_data.sh
Executable file
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
# Acquire OpenStreetMap data for Dubai Marina
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
echo "=== Acquiring OSM Data for Dubai Marina ==="
|
||||
echo ""
|
||||
|
||||
# Dubai Marina bounds: 25.07,55.13,25.09,55.15
|
||||
BOUNDS="25.07,55.13,25.09,55.15"
|
||||
OUTPUT="processed/dubai_marina_buildings.geojson"
|
||||
|
||||
echo "Bounds: $BOUNDS"
|
||||
echo "Output: $OUTPUT"
|
||||
echo ""
|
||||
|
||||
if [ -f "../scripts/data/import_osm_data.py" ]; then
|
||||
python3 ../scripts/data/import_osm_data.py \
|
||||
--output "$OUTPUT" \
|
||||
--bounds "$BOUNDS"
|
||||
echo "✓ OSM data acquired"
|
||||
else
|
||||
echo "⚠ OSM import script not found"
|
||||
echo "Manual download: https://www.openstreetmap.org/export"
|
||||
fi
|
||||
Reference in New Issue
Block a user