Add Strix camera discovery system with comprehensive database
This commit adds the complete Strix IP camera stream discovery system: - Go-based API server with SSE support for real-time updates - 3,600+ camera brand database with stream URL patterns - Intelligent fuzzy search across camera models - ONVIF discovery and stream validation - RESTful API with health check, camera search, and stream discovery - Makefile for building and deployment - Comprehensive README documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
eduard256
@@ -0,0 +1,517 @@
|
||||
# 📹 IoT2mqtt Camera Database Format Specification
|
||||
|
||||
**Version:** 1.0.0
|
||||
**Last Updated:** 2025-10-17
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
The camera database is a collection of JSON files containing URL patterns and connection details for IP cameras from various manufacturers. This format is designed to be:
|
||||
|
||||
- **Universal**: Works with any IP camera brand
|
||||
- **Extensible**: Easy to add new models and protocols
|
||||
- **Human-readable**: Simple JSON structure
|
||||
- **Parseable**: Straightforward for automated tools
|
||||
|
||||
---
|
||||
|
||||
## 📁 Directory Structure
|
||||
|
||||
```
|
||||
connectors/cameras/data/brands/
|
||||
├── index.json # Master list of all brands
|
||||
├── d-link.json # D-Link camera models
|
||||
├── hikvision.json # Hikvision camera models
|
||||
├── dahua.json # Dahua camera models
|
||||
├── axis.json # Axis camera models
|
||||
└── ... # Additional brands
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 File Formats
|
||||
|
||||
### 1. **index.json** - Brand Directory
|
||||
|
||||
Lists all available camera brands with metadata.
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"value": "d-link",
|
||||
"label": "D-Link",
|
||||
"models_count": 250,
|
||||
"entries_count": 85,
|
||||
"logo": "/assets/brands/d-link.svg"
|
||||
},
|
||||
{
|
||||
"value": "hikvision",
|
||||
"label": "Hikvision",
|
||||
"models_count": 320,
|
||||
"entries_count": 95,
|
||||
"logo": "/assets/brands/hikvision.svg"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `value` (string, required): Brand identifier (lowercase, URL-safe)
|
||||
- `label` (string, required): Display name
|
||||
- `models_count` (integer): Total number of camera models
|
||||
- `entries_count` (integer): Number of URL pattern entries
|
||||
- `logo` (string, optional): Path to brand logo
|
||||
|
||||
---
|
||||
|
||||
### 2. **{brand}.json** - Brand Camera Database
|
||||
|
||||
Contains all URL patterns and connection details for a specific brand.
|
||||
|
||||
```json
|
||||
{
|
||||
"brand": "D-Link",
|
||||
"brand_id": "d-link",
|
||||
"last_updated": "2025-10-17",
|
||||
"source": "ispyconnect.com",
|
||||
"website": "https://www.dlink.com",
|
||||
"entries": [
|
||||
{
|
||||
"models": ["DCS-930L", "DCS-930LB", "DCS-930LB1"],
|
||||
"type": "FFMPEG",
|
||||
"protocol": "rtsp",
|
||||
"port": 554,
|
||||
"url": "live3.sdp",
|
||||
"notes": "Main HD stream"
|
||||
},
|
||||
{
|
||||
"models": ["DCS-930L", "DCS-932L"],
|
||||
"type": "MJPEG",
|
||||
"protocol": "http",
|
||||
"port": 80,
|
||||
"url": "video.cgi?resolution=VGA",
|
||||
"notes": "Medium quality fallback"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Root Fields:**
|
||||
- `brand` (string, required): Brand display name
|
||||
- `brand_id` (string, required): Brand identifier (must match filename)
|
||||
- `last_updated` (string, ISO 8601 date): When database was last updated
|
||||
- `source` (string): Where the data came from (e.g., "ispyconnect.com")
|
||||
- `website` (string, optional): Manufacturer's official website
|
||||
- `entries` (array, required): List of URL pattern entries
|
||||
|
||||
---
|
||||
|
||||
### 3. **Entry Object** - URL Pattern Entry
|
||||
|
||||
Each entry represents a specific URL pattern that works for one or more camera models.
|
||||
|
||||
```json
|
||||
{
|
||||
"models": ["DCS-930L", "DCS-930LB", "DCS-930LB1"],
|
||||
"type": "FFMPEG",
|
||||
"protocol": "rtsp",
|
||||
"port": 554,
|
||||
"url": "live3.sdp",
|
||||
"auth_required": true,
|
||||
"notes": "Main HD stream with audio"
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `models` | array[string] | ✅ Yes | List of camera model names/numbers this URL works for |
|
||||
| `type` | string | ✅ Yes | Stream type: `FFMPEG`, `MJPEG`, `JPEG`, `VLC`, `H264` |
|
||||
| `protocol` | string | ✅ Yes | Protocol: `rtsp`, `http`, `https` |
|
||||
| `port` | integer | ✅ Yes | Port number (554 for RTSP, 80/443 for HTTP) |
|
||||
| `url` | string | ✅ Yes | URL path (without protocol/host/port) |
|
||||
| `auth_required` | boolean | No | Whether authentication is needed (default: true) |
|
||||
| `notes` | string | No | Human-readable description |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 URL Template Variables
|
||||
|
||||
URL paths support the following template variables:
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `{username}` | Camera username | `admin` |
|
||||
| `{password}` | Camera password | `12345` |
|
||||
| `{ip}` | Camera IP address | `192.168.1.100` |
|
||||
| `{port}` | Port number | `554` |
|
||||
| `{channel}` | Camera channel (for DVRs) | `1` |
|
||||
| `{width}` | Video width | `1920` |
|
||||
| `{height}` | Video height | `1080` |
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Template: rtsp://{username}:{password}@{ip}:{port}/live3.sdp
|
||||
Result: rtsp://admin:12345@192.168.1.100:554/live3.sdp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Stream Types
|
||||
|
||||
### FFMPEG (Recommended)
|
||||
- **Protocol**: RTSP, HTTP
|
||||
- **Format**: H.264, H.265
|
||||
- **Use case**: High-quality video with audio
|
||||
- **Priority**: 🥇 First choice
|
||||
|
||||
### MJPEG
|
||||
- **Protocol**: HTTP
|
||||
- **Format**: Motion JPEG
|
||||
- **Use case**: Medium quality, wide compatibility
|
||||
- **Priority**: 🥈 Second choice
|
||||
|
||||
### JPEG
|
||||
- **Protocol**: HTTP
|
||||
- **Format**: Still images
|
||||
- **Use case**: Snapshot-only cameras or fallback
|
||||
- **Priority**: 🥉 Last resort
|
||||
|
||||
### VLC
|
||||
- **Protocol**: RTSP, HTTP
|
||||
- **Format**: Various (VLC-specific)
|
||||
- **Use case**: Compatibility with VLC player
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Priority Order for Testing
|
||||
|
||||
When testing multiple URLs for a camera model, use this priority:
|
||||
|
||||
1. **RTSP (type="FFMPEG")** - Best quality, supports audio
|
||||
2. **HTTP MJPEG** - Good compatibility
|
||||
3. **HTTP JPEG** - Snapshot fallback
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
def get_urls_for_model(brand_data, model_name):
|
||||
entries = [e for e in brand_data["entries"] if model_name in e["models"]]
|
||||
|
||||
# Sort by priority
|
||||
priority = {"FFMPEG": 1, "MJPEG": 2, "JPEG": 3, "VLC": 4}
|
||||
entries.sort(key=lambda e: priority.get(e["type"], 99))
|
||||
|
||||
return entries
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Search and Lookup
|
||||
|
||||
### By Brand
|
||||
```python
|
||||
# Load brand file
|
||||
with open(f"data/brands/{brand_id}.json") as f:
|
||||
brand_data = json.load(f)
|
||||
```
|
||||
|
||||
### By Model
|
||||
```python
|
||||
# Find all entries for a specific model
|
||||
def find_model_entries(brand_data, model_name):
|
||||
return [
|
||||
entry for entry in brand_data["entries"]
|
||||
if model_name.upper() in [m.upper() for m in entry["models"]]
|
||||
]
|
||||
```
|
||||
|
||||
### Fuzzy Search
|
||||
```python
|
||||
# Search across all models (case-insensitive, partial match)
|
||||
def search_model(brand_data, query):
|
||||
query = query.upper()
|
||||
results = []
|
||||
for entry in brand_data["entries"]:
|
||||
if any(query in model.upper() for model in entry["models"]):
|
||||
results.append(entry)
|
||||
return results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🌐 URL Construction
|
||||
|
||||
### RTSP URL
|
||||
```python
|
||||
def build_rtsp_url(entry, ip, username, password):
|
||||
return f"rtsp://{username}:{password}@{ip}:{entry['port']}/{entry['url']}"
|
||||
|
||||
# Example:
|
||||
# rtsp://admin:12345@192.168.1.100:554/live3.sdp
|
||||
```
|
||||
|
||||
### HTTP URL
|
||||
```python
|
||||
def build_http_url(entry, ip, username, password):
|
||||
protocol = entry["protocol"] # "http" or "https"
|
||||
return f"{protocol}://{username}:{password}@{ip}:{entry['port']}/{entry['url']}"
|
||||
|
||||
# Example:
|
||||
# http://admin:12345@192.168.1.100:80/video.cgi?resolution=VGA
|
||||
```
|
||||
|
||||
### With Template Variables
|
||||
```python
|
||||
def build_url(entry, ip, username, password, **kwargs):
|
||||
url_path = entry["url"]
|
||||
|
||||
# Replace template variables
|
||||
replacements = {
|
||||
"username": username,
|
||||
"password": password,
|
||||
"ip": ip,
|
||||
"port": str(entry["port"]),
|
||||
**kwargs # Additional variables (channel, width, height, etc.)
|
||||
}
|
||||
|
||||
for key, value in replacements.items():
|
||||
url_path = url_path.replace(f"{{{key}}}", value)
|
||||
|
||||
# Build full URL
|
||||
if entry["protocol"] == "rtsp":
|
||||
return f"rtsp://{username}:{password}@{ip}:{entry['port']}/{url_path}"
|
||||
else:
|
||||
return f"{entry['protocol']}://{username}:{password}@{ip}:{entry['port']}/{url_path}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Validation Rules
|
||||
|
||||
### Entry Validation
|
||||
```python
|
||||
def validate_entry(entry):
|
||||
# Required fields
|
||||
assert "models" in entry and isinstance(entry["models"], list)
|
||||
assert len(entry["models"]) > 0
|
||||
assert "type" in entry and entry["type"] in ["FFMPEG", "MJPEG", "JPEG", "VLC", "H264"]
|
||||
assert "protocol" in entry and entry["protocol"] in ["rtsp", "http", "https"]
|
||||
assert "port" in entry and isinstance(entry["port"], int)
|
||||
assert "url" in entry and isinstance(entry["url"], str)
|
||||
|
||||
# Port ranges
|
||||
assert 1 <= entry["port"] <= 65535
|
||||
|
||||
# Common ports check
|
||||
if entry["protocol"] == "rtsp":
|
||||
assert entry["port"] in [554, 8554, 7447] # Common RTSP ports
|
||||
elif entry["protocol"] == "http":
|
||||
assert entry["port"] in [80, 8080, 8000, 8081] # Common HTTP ports
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Naming Conventions
|
||||
|
||||
### Brand IDs
|
||||
- **Format**: lowercase, kebab-case
|
||||
- **Examples**: `d-link`, `hikvision`, `tp-link`
|
||||
- **Invalid**: `D-Link`, `D_Link`, `dlink`
|
||||
|
||||
### Model Names
|
||||
- **Format**: UPPERCASE with hyphens (as manufacturer specifies)
|
||||
- **Examples**: `DCS-930L`, `DS-2CD2142FWD-I`, `IPC-HFW1230S`
|
||||
- **Keep original**: Don't normalize or change manufacturer names
|
||||
|
||||
### Protocol Values
|
||||
- `rtsp` - RTSP protocol
|
||||
- `http` - HTTP protocol
|
||||
- `https` - HTTPS protocol
|
||||
- **Invalid**: `RTSP`, `Http`, `tcp`
|
||||
|
||||
### Type Values
|
||||
- `FFMPEG` - H.264/H.265 streams (RTSP or HTTP)
|
||||
- `MJPEG` - Motion JPEG streams
|
||||
- `JPEG` - Still image snapshots
|
||||
- `VLC` - VLC-specific streams
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Versioning and Updates
|
||||
|
||||
### Version Format
|
||||
```json
|
||||
{
|
||||
"brand": "D-Link",
|
||||
"brand_id": "d-link",
|
||||
"database_version": "1.2.0",
|
||||
"last_updated": "2025-10-17T14:30:00Z",
|
||||
"entries": [...]
|
||||
}
|
||||
```
|
||||
|
||||
### Update Policy
|
||||
- **Patch** (1.0.x): Add new models to existing entries
|
||||
- **Minor** (1.x.0): Add new URL patterns/entries
|
||||
- **Major** (x.0.0): Breaking changes to structure
|
||||
|
||||
---
|
||||
|
||||
## 📚 Examples
|
||||
|
||||
### Complete Brand File Example
|
||||
|
||||
**foscam.json:**
|
||||
```json
|
||||
{
|
||||
"brand": "Foscam",
|
||||
"brand_id": "foscam",
|
||||
"last_updated": "2025-10-17",
|
||||
"source": "ispyconnect.com",
|
||||
"website": "https://www.foscam.com",
|
||||
"entries": [
|
||||
{
|
||||
"models": ["FI9821P", "FI9826P", "FI9821W"],
|
||||
"type": "FFMPEG",
|
||||
"protocol": "rtsp",
|
||||
"port": 554,
|
||||
"url": "videoMain",
|
||||
"notes": "Main stream HD"
|
||||
},
|
||||
{
|
||||
"models": ["FI9821P", "FI9826P"],
|
||||
"type": "FFMPEG",
|
||||
"protocol": "rtsp",
|
||||
"port": 554,
|
||||
"url": "videoSub",
|
||||
"notes": "Sub stream SD"
|
||||
},
|
||||
{
|
||||
"models": ["FI9821P", "FI9826P", "FI9821W", "C1"],
|
||||
"type": "MJPEG",
|
||||
"protocol": "http",
|
||||
"port": 88,
|
||||
"url": "cgi-bin/CGIStream.cgi?cmd=GetMJStream&usr={username}&pwd={password}",
|
||||
"notes": "MJPEG fallback"
|
||||
},
|
||||
{
|
||||
"models": ["FI9821P", "C1", "C2"],
|
||||
"type": "JPEG",
|
||||
"protocol": "http",
|
||||
"port": 88,
|
||||
"url": "cgi-bin/CGIProxy.fcgi?cmd=snapPicture2&usr={username}&pwd={password}",
|
||||
"notes": "Snapshot"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Tools and Scripts
|
||||
|
||||
### Parser Script (Python)
|
||||
```python
|
||||
# scripts/parse_ispyconnect.py
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
import json
|
||||
|
||||
def parse_brand_page(brand_id):
|
||||
url = f"https://www.ispyconnect.com/camera/{brand_id}"
|
||||
response = requests.get(url)
|
||||
soup = BeautifulSoup(response.text, 'html.parser')
|
||||
|
||||
table = soup.find('table', class_='table-striped')
|
||||
entries = []
|
||||
|
||||
for row in table.find_all('tr')[1:]: # Skip header
|
||||
cols = row.find_all('td')
|
||||
if len(cols) < 4:
|
||||
continue
|
||||
|
||||
models_text = cols[0].get_text()
|
||||
models = [m.strip() for m in models_text.split(',')]
|
||||
|
||||
entry = {
|
||||
"models": models,
|
||||
"type": cols[1].get_text(strip=True),
|
||||
"protocol": cols[2].get_text(strip=True).replace('://', ''),
|
||||
"port": int(row.get('data-port', 0)),
|
||||
"url": cols[3].get_text(strip=True)
|
||||
}
|
||||
|
||||
entries.append(entry)
|
||||
|
||||
return {
|
||||
"brand": brand_id.title(),
|
||||
"brand_id": brand_id,
|
||||
"last_updated": "2025-10-17",
|
||||
"source": "ispyconnect.com",
|
||||
"entries": entries
|
||||
}
|
||||
```
|
||||
|
||||
### Validator Script
|
||||
```python
|
||||
# scripts/validate_database.py
|
||||
import json
|
||||
import os
|
||||
|
||||
def validate_brand_file(filepath):
|
||||
with open(filepath) as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Check required fields
|
||||
assert "brand" in data
|
||||
assert "brand_id" in data
|
||||
assert "entries" in data
|
||||
|
||||
# Validate each entry
|
||||
for i, entry in enumerate(data["entries"]):
|
||||
assert "models" in entry, f"Entry {i} missing models"
|
||||
assert "type" in entry, f"Entry {i} missing type"
|
||||
assert "protocol" in entry, f"Entry {i} missing protocol"
|
||||
assert "port" in entry, f"Entry {i} missing port"
|
||||
assert "url" in entry, f"Entry {i} missing url"
|
||||
|
||||
print(f"✅ {filepath} is valid")
|
||||
|
||||
# Run validation
|
||||
for file in os.listdir('data/brands/'):
|
||||
if file.endswith('.json') and file != 'index.json':
|
||||
validate_brand_file(f'data/brands/{file}')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License and Attribution
|
||||
|
||||
- **Source**: ispyconnect.com camera database
|
||||
- **Usage**: Free for IoT2mqtt project
|
||||
- **Attribution**: Must credit ispyconnect.com as data source
|
||||
- **Updates**: Community-contributed updates welcome
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
To add or update camera models:
|
||||
|
||||
1. Follow the JSON format specification
|
||||
2. Validate using `scripts/validate_database.py`
|
||||
3. Test URLs with real cameras when possible
|
||||
4. Submit pull request with changes
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support
|
||||
|
||||
For questions about the database format:
|
||||
- GitHub Issues: https://github.com/your-repo/issues
|
||||
- Documentation: https://docs.your-project.com
|
||||
|
||||
---
|
||||
|
||||
**End of Specification**
|
||||
Reference in New Issue
Block a user