metaverseDubai/data/OSM_DATA_RESEARCH.md

# OpenStreetMap Data Research - Dubai Marina

## Overview

Research and documentation for acquiring OpenStreetMap (OSM) data for Dubai Marina district.

## Target Area

**Location:** Dubai Marina, Dubai, UAE  
**Coordinates:** 25.0772° N, 55.1394° E  
**Bounding Box:**
- **Min:** 25.07° N, 55.13° E
- **Max:** 25.09° N, 55.15° E
- **Format:** `25.07,55.13,25.09,55.15`

## Data Sources

### 1. Overpass API (Recommended)

**Endpoint:** https://overpass-api.de/api/interpreter

**Query for Buildings:**
```
[out:xml][bbox:25.07,55.13,25.09,55.15];
(way["building"];);
out meta;
```

**Query for Roads:**
```
[out:xml][bbox:25.07,55.13,25.09,55.15];
(way["highway"];);
out meta;
```

**Query for Complete Area Data:**
```
[out:xml][bbox:25.07,55.13,25.09,55.15];
(
  way["building"];
  way["highway"];
  relation["building"];
);
out meta;
```

### 2. Geofabrik Downloads

**Regional Extract:** Middle East
- **URL:** https://download.geofabrik.de/asia/united-arab-emirates.html
- **Format:** OSM PBF
- **Size:** ~50-100MB for UAE
- **Update Frequency:** Daily

### 3. Planet OSM

**Full Planet Extract:**
- **URL:** https://planet.openstreetmap.org/
- **Format:** OSM XML or PBF
- **Size:** Very large (entire planet)
- **Use:** Extract UAE region using osmium-tool

## Data Acquisition Methods

### Method 1: Overpass API (Quick, Small Areas)

**Using curl:**
```bash
curl -X POST \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "data=[out:xml][bbox:25.07,55.13,25.09,55.15];(way[\"building\"];);out meta;" \
  "https://overpass-api.de/api/interpreter" \
  > dubai_marina.osm
```

**Using Python (overpy library):**
```python
import overpy

api = overpy.Overpass()
query = """
[bbox:25.07,55.13,25.09,55.15];
(way["building"];);
out body;
"""
result = api.query(query)
```

### Method 2: Geofabrik Download (Larger Areas)

```bash
# Download UAE extract
wget https://download.geofabrik.de/asia/united-arab-emirates-latest.osm.pbf

# Extract Dubai Marina area using osmium-tool
osmium extract -b 55.13,25.07,55.15,25.09 \
  united-arab-emirates-latest.osm.pbf \
  -o dubai_marina.osm.pbf
```

### Method 3: Using OSMnx (Python)

```python
import osmnx as ox

# Define area
place = "Dubai Marina, Dubai, UAE"
bounds = (25.07, 55.13, 25.09, 55.15)

# Download buildings
buildings = ox.geometries_from_bbox(
    north=25.09, south=25.07, 
    east=55.15, west=55.13,
    tags={'building': True}
)

# Download roads
roads = ox.graph_from_bbox(
    north=25.09, south=25.07,
    east=55.15, west=55.13,
    network_type='drive'
)
```

## Data Processing

### Convert OSM to GeoJSON

**Using osmtogeojson:**
```bash
npm install -g osmtogeojson
osmtogeojson dubai_marina.osm > dubai_marina.geojson
```

**Using Python (osmium):**
```python
import osmium
import json

class BuildingHandler(osmium.SimpleHandler):
    def __init__(self):
        osmium.SimpleHandler.__init__(self)
        self.buildings = []
    
    def way(self, w):
        if 'building' in w.tags:
            # Extract building geometry
            # Convert to GeoJSON
            pass

handler = BuildingHandler()
handler.apply_file("dubai_marina.osm")
```

### Process with our script

**Using our import script:**
```bash
python3 scripts/data/import_osm_data.py \
  --output data/processed/dubai_marina_buildings.geojson \
  --bounds "25.07,55.13,25.09,55.15"
```

## Expected Data

### Buildings
- **Estimated Count:** 50-100 buildings in Dubai Marina area
- **Types:** Residential towers, commercial buildings, hotels
- **Key Buildings:**
  - Cayan Tower (Hero landmark)
  - Marina Towers
  - Various residential complexes

### Roads
- **Main Roads:**
  - Sheikh Zayed Road (E11)
  - Dubai Marina Walk
  - Various internal roads

### Additional Features
- Water features (marina)
- Parks and green spaces
- Parking areas
- Sidewalks

## Data Quality Considerations

### Accuracy
- OSM data is community-sourced
- Building footprints may not be 100% accurate
- Heights are often missing
- Some buildings may be missing

### Validation
- Cross-reference with satellite imagery
- Verify building positions
- Check for missing major buildings
- Validate road network

### Enhancement
- Add building heights from other sources
- Refine building footprints using satellite imagery
- Add missing buildings manually
- Enhance road network details

## Recommended Approach

1. **Initial Acquisition:** Use Overpass API for quick download
2. **Validation:** Compare with satellite imagery
3. **Enhancement:** Add missing buildings and details
4. **Processing:** Convert to GeoJSON for Unreal import
5. **Import:** Use our import script to bring into Unreal

## Tools Required

- **curl** or **wget** - Download data
- **Python 3.8+** - Processing scripts
- **overpy** - Python OSM API client
- **osmnx** - Advanced OSM data handling (optional)
- **osmtogeojson** - OSM to GeoJSON converter (optional)
- **osmium-tool** - Advanced OSM data processing (optional)

## Next Steps

1. Run `data/acquire_osm_data.sh` to download initial data
2. Validate data quality
3. Process with `scripts/data/import_osm_data.py`
4. Import to Unreal Engine after project creation

---

**Last Updated:** 2024-11-21  
**Status:** Research Complete, Ready for Data Acquisition