Initial commit: add .gitignore and README

data-storage/data-catalog/README.md

@@ -0,0 +1,112 @@
+# Data Catalog
+
+**Purpose**: Unified data catalog for tracking and discovering datasets
+**Status**: 🚧 Planned
+
+---
+
+## Overview
+
+The data catalog provides a centralized registry for all datasets across the workspace, enabling discovery, access control, and metadata management.
+
+---
+
+## Features
+
+- Dataset registration
+- Metadata management
+- Search and discovery
+- Access control
+- Schema tracking
+- Lineage tracking
+
+---
+
+## Schema
+
+See `metadata-schema.json` for the complete metadata schema.
+
+### Key Fields
+
+- **id**: Unique dataset identifier
+- **name**: Human-readable name
+- **source**: Source system/project
+- **storage**: Storage location details
+- **schema**: Data schema definition
+- **tags**: Categorization tags
+- **access**: Access control settings
+
+---
+
+## Implementation Options
+
+### Option 1: Custom API
+- Build custom API using shared packages
+- Use PostgreSQL for metadata storage
+- Implement search using PostgreSQL full-text search
+
+### Option 2: DataHub
+- Deploy DataHub (open-source)
+- Use existing metadata models
+- Leverage built-in features
+
+### Option 3: Amundsen
+- Deploy Amundsen (open-source)
+- Use existing metadata models
+- Leverage built-in features
+
+---
+
+## Usage
+
+### Register Dataset
+
+```json
+{
+  "id": "user-events-2025",
+  "name": "User Events 2025",
+  "description": "User interaction events for 2025",
+  "source": "analytics-service",
+  "storage": {
+    "type": "minio",
+    "bucket": "analytics",
+    "path": "events/2025/"
+  },
+  "format": "parquet",
+  "tags": ["events", "analytics", "2025"],
+  "owner": "analytics-team",
+  "access": {
+    "level": "internal",
+    "permissions": ["read"]
+  }
+}
+```
+
+### Search Datasets
+
+```bash
+# Search by tag
+GET /api/catalog/datasets?tag=analytics
+
+# Search by source
+GET /api/catalog/datasets?source=analytics-service
+
+# Full-text search
+GET /api/catalog/datasets?q=user+events
+```
+
+---
+
+## Next Steps
+
+1. Choose implementation option
+2. Set up metadata storage
+3. Implement registration API
+4. Implement search functionality
+5. Set up access control
+6. Integrate with projects
+
+---
+
+**Status**: 🚧 Planned - Schema and design complete, implementation pending
+