Visual DOM Inspector & Schema Extractor
Test live web pages or raw text payloads with point-and-click selector discovery and instant schema mapping.
Raw Text / HTML Source Payload
INPUTStructured Extracted Output
READY// Extracted JSON payload will appear here...
Multi-Modal Legal Document & Scanned OCR Studio
Extract vector text, scanned instruments, and multi-column tabular records from legal PDFs and TIFF deeds.
Document Text / OCR Transcript Stream
RAW PDF / OCR STREAMExtracted Structured Schema Fields
| Schema Field | Extracted Value | Confidence Score | Status |
|---|
Hybrid Multi-URL Batch Crawler
Crawl hundreds of government pages with range pattern expansion (e.g. ?page={1..25}), concurrency control, and proxy rotation.
Batch Crawl Extracted Dataset
| Source URL | HTTP Status | Extracted Schema Payload | Confidence | SHA-256 |
|---|
Anti-Bot Stealth & Proxy Pool Manager
Round-robin IP rotation, residential SOCKS5 proxies, randomized browser fingerprints, and human request delays.
Active Proxy Pool
5 Proxies Active| Proxy URL / Endpoint | Protocol | Geo Location | Latency (ms) | Health Status |
|---|
Automated Watchdogs & Export Pipeline
Recurring cron schedule monitoring for court registries, SHA-256 deduplication, and multi-format exports.
Scheduled Scraping Watchdogs
| Job Name | Target URL Pattern | Schema | Cron Schedule | Status | Action |
|---|
Documentation, Architecture & System FAQ
Comprehensive reference manual for all ScrapeForge Pro features, widgets, interactive workflows, and backend subsystem APIs.
⚙️ System Architecture & Data Pipeline Flow
LOCAL-FIRST WAL ARCHITECTUREScrapeForge Pro operates as a local-first desktop daemon running on http://127.0.0.1:8150. All extractions, OCR conversions, proxy rotations, and cron watchdogs execute locally on your machine with zero third-party cloud data transmission.
?page={1..50}), scanned court PDFs, or raw OCR transcripts. Routes traffic through the Stealth Proxy Pool.scrapeforge_output.db (SQLite Write-Ahead Logging).🎛️ Complete Widget, Input & Button Encyclopedia
INTERFACE REFERENCE/api/v1/scrapeforge/preview, fetches sanitized HTML using stealth headers, and automatically tags candidate CSS/XPath selectors (tables, cards, headings).?page={1..10}) and dispatches parallel HTTP workers throttled by the Concurrency Slider.proxy_pool, computes latency in milliseconds, and flags degraded endpoints.http://user:pass@host:port or socks5://host:port).0 6 * * *) and optional webhook alert endpoints.❓ Frequently Asked Questions (FAQ) & Operational Guide
16 DETAILED ARTICLES1. What makes ScrapeForge Pro different from cloud scrapers like Apify or Octoparse?
ScrapeForge Pro is designed as a local-first desktop workbench sold under a $149 Perpetual License (one-time purchase, no recurring monthly fees). Unlike cloud services that charge monthly compute subscriptions ($49–$500/mo) and upload legal court records to third-party servers, ScrapeForge executes all extractions, regex parsing, and SQLite WAL persistence directly on your local hardware.
2. How does the Local-First Daemon on Port 8150 work?
When ScrapeForge initializes, it binds a high-performance asynchronous FastAPI daemon on 127.0.0.1:8150. This daemon serves the Baseline-UI web frontend and exposes private REST API endpoints (/api/v1/scrapeforge/*). It operates in offline mode without requiring internet access for schema extraction or license validation.
3. How do I test and discover CSS/XPath selectors from a live webpage in the Visual Workbench?
Paste any target webpage URL into the Live URL Preview bar and click 'Live URL Preview & Auto-Inspect DOM'. ScrapeForge fetches the page via stealth HTTP headers, analyzes the HTML DOM tree, and displays discovered candidate selector badges (e.g. table tr, .record-card). Clicking a badge automatically loads the selector into your active extraction schema.
4. How does the Multi-Modal Document & OCR Studio parse legal instruments and PDFs?
The Document Studio accepts raw text, OCR streams, or digital PDF bytes. It routes inputs through DocumentStudio.extract_document_payload(), which executes schema-specific regex engines to extract critical case attributes (Cause No., Decedent, Executor, Parcel APN, Estate Value) and assigns an extraction confidence score from 0% to 100%.
5. How does tabular auto-detection parse multi-column ledgers?
The table auto-detector scans document text for column separators including pipes (|), tabs (\t), and multiple whitespace delimiters (2+ consecutive spaces). When repetitive rows with 3 or more aligned columns are detected, it automatically structures them into array records stored in SQLite.
6. How do I crawl multi-page pagination with range patterns like ?page={1..50}?
In the Hybrid Batch Crawler tab, enter a URL containing bracketed integer ranges: https://county.gov/records?page={1..25}. ScrapeForge expands this template into 25 discrete requests and processes them asynchronously across your configured worker concurrency pool.
7. How does the Stealth & Proxy Manager prevent IP bans and rate-limit blocks?
The Proxy Manager applies three layers of protection:
1. Round-Robin Proxy Rotation: Cycles requests across HTTP, HTTPS, and SOCKS5 residential IPs.
2. Browser Fingerprint Randomization: Randomizes modern User-Agent strings, Sec-Ch-Ua headers, and viewport metrics.
3. Human Jitter Delay: Injects random pauses (0.5s to 2.0s) between requests and executes exponential backoff on HTTP 429/403 responses.
8. What proxy formats does ScrapeForge accept?
ScrapeForge supports standard HTTP, HTTPS, and SOCKS5 connection strings:
• Standard HTTP: http://proxy.example.com:8080
• Authenticated HTTP: http://username:password@proxy.example.com:8080
• SOCKS5 Residential: socks5://username:password@res-proxy.example.com:1080
9. How do Scheduled Watchdogs execute automated sweeps?
Watchdogs allow you to set recurring cron expressions (e.g. 0 6 * * * for daily at 6:00 AM) to automatically crawl court portals and bid boards. When the schedule triggers, the engine crawls the target URL pattern, extracts records, and commits new rows to SQLite WAL.
10. How does SHA-256 content hashing guarantee zero duplicate records?
Before storing any record, ScrapeForge computes a SHA-256 cryptographic hash of the extracted JSON payload. The SQLite database enforces a UNIQUE constraint on the sha256_hash column in extracted_records, ensuring identical dockets are never inserted twice.
11. How do I configure Webhook alerts for Zapier, Make, or my internal CRM?
When creating or editing a Watchdog schedule, provide your webhook endpoint in the Webhook Push URL field. Whenever a newly discovered record is committed to the database, ScrapeForge dispatches an asynchronous HTTP POST payload containing the structured JSON record and SHA-256 hash.
12. Which export formats are available and what are their specific use cases?
ScrapeForge supports 5 publication formats:
• Microsoft Excel (.xlsx): Pre-styled workbooks with ScrapeForge Orange headers and auto-column widths.
• Apache Parquet (.parquet): Compressed columnar binary for Pandas, Polars, DuckDB, and Snowflake analytics.
• CSV (.csv): Standard comma-delimited tabular rows for spreadsheets and CRM imports.
• JSON Lines (.jsonl): Streaming line-delimited JSON objects for LLM fine-tuning and big-data pipelines.
• SQLite (.db): Direct copy of the local WAL database file with complete relational schema.
13. How does offline perpetual license key verification work?
ScrapeForge license keys use cryptographic HMAC SHA-256 signatures format: SCRAPEFORGE-<TIER>-<HASH>. The application verifies the signature mathematically on localhost without phoning home to external license servers, guaranteeing 100% offline functionality.
14. What are the limits of the Community Trial mode vs the $149 Perpetual Pro license?
• Community Trial Mode: Limits extractions to the first 3 fields per document, single-job execution preview, and max 100 stored rows.
• Perpetual Pro License ($149): Uncapped rows, unlimited schema fields, 20 parallel crawler threads, automated cron watchdogs, and Parquet/Excel master exports.
15. Where is data stored locally and how do I back up or migrate the database?
All data is stored in products/scrapeforge_desktop/scrapeforge_output.db alongside its SQLite WAL journaling files (.db-wal, .db-shm). You can back up or migrate your entire dataset by copying this single file or downloading the SQLite database from the Watchdogs & Exports tab.
16. How do the 8 turnkey vendor platform presets work?
ScrapeForge includes pre-built drivers for the 8 major government portal vendors covering 3,244 U.S. counties:
1. Tyler Technologies Odyssey / Eagle (950 counties - court dockets & probate)
2. Granicus Landmark (620 counties - recorded deeds & tax liens)
3. Kofile QuickSearch (480 counties - land & probate dockets)
4. BIS Consultants TrueRoll (350 counties - property appraisal rolls)
5. RealAuction GovEase Bid4Assets (320 counties - foreclosure auctions)
6. CivicPlus Municode (280 portals - municipal RFPs & bid solicitations)
7. State Unified Judicial Gateways (180 jurisdictions - statewide court filings)
8. Municipal ArcGIS REST GIS (64 metros - parcel boundary polygon shapes)