diff --git a/Config.md b/Config.md index 6e361af..febb838 100644 --- a/Config.md +++ b/Config.md @@ -3,54 +3,75 @@ This page documents all configuration variables available in `config.ini`, including their purpose, possible values, and default settings. ## Server Configuration -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|--------------|-------------------------------------------------|-------------------|---------------------------| -| `Port` | Port number for the application server | `5000` | Any valid port number | -| `Domain` | Domain address of the server | `localhost` | Any valid domain string | -| `LogLevel` | Logging level for the application | `1` | `0` (Silent), `1` (Errors), `2` (Warnings), `3` (Info), `4` (Debug) | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| -------------- | ---------------------------------------- | ------------------- | --------------------------------------------------------------------- | +| `Port` | Port number for the application server | `5000` | Any valid port number | +| `Domain` | Domain address of the server | `localhost` | Any valid domain name or IP address | +| `LogLevel` | Logging level for the application | `1` | `0` (Silent), `1` (Errors), `2` (Warnings), `3` (Info), `4` (Debug) | ## Peer Configuration -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|--------------|-------------------------------------------------|-------------------|---------------------------| -| `AuthCode` | Authorization code for peer connections | Auto-generated | 64-character random string | -| `PeerID` | Unique identifier for the peer | Auto-generated | Any unique string | -| `Peers` | Comma-separated list of peer addresses | Empty | List of peer URLs | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| -------------- | ----------------------------------------- | ------------------- | ---------------------------- | +| `AuthCode` | Authorization code for peer connections | Auto-generated | 64-character random string | +| `PeerID` | Unique identifier for the peer | Auto-generated | Any unique string | +| `Peers` | Comma-separated list of peer addresses | Empty | List of peer URLs | ## Feature Toggles -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|-----------------------|---------------------------------------------------------|-------------------|---------------------------| -| `NodesEnabled` | Enable or disable peer-to-peer nodes | `false` | `true` / `false` | -| `MetaSearchEnabled` | Enable metasearch crawling from other search engines | `true` | `true` / `false` | -| `IndexerEnabled` | Enable web crawling and indexing | `false` | `true` / `false` | -| `WebsiteEnabled` | Enable website hosting feature | `true` | `true` / `false` | -| `RamCacheEnabled` | Enable caching in RAM | `true` | `true` / `false` | -| `DriveCacheEnabled` | Enable caching on the disk | `false` | `true` / `false` | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| ---------------- | ------------------------------------------------------ | ------------------- | --------------------- | +| `Nodes` | Enable or disable peer-to-peer nodes | `false` | `true` / `false` | +| `MetaSearch` | Enable metasearch crawling from other search engines | `true` | `true` / `false` | +| `Indexer` | Enable web crawling and indexing | `false` | `true` / `false` | +| `Website` | Enable website hosting feature | `true` | `true` / `false` | +| `RamCache` | Enable caching in RAM | `true` | `true` / `false` | +| `DriveCache` | Enable caching on the disk | `false` | `true` / `false` | +| `MetaProxy` | Use proxies for metasearch crawling | `false` | `true` / `false` | +| `CrawlerProxy` | Use proxies for general web crawling and indexing | `false` | `true` / `false` | ## Proxy Configuration -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|-----------------------|---------------------------------------------------------|-------------------|---------------------------| -| `MetaProxyEnabled` | Use proxies for metasearch crawling | `false` | `true` / `false` | -| `CrawlerProxyEnabled` | Use proxies for general web crawling and indexing | `false` | `true` / `false` | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| ---------------------- | ------------------------------------------------------ | ------------------- | ------------------------- | +| `MetaProxyStrict` | Enforce strict behavior when MetaProxy fails | `true` | `true` / `false` | +| `MetaProxies` | List of proxies to use for metasearch crawling | Empty | Comma-separated proxies | +| `CrawlerProxyStrict` | Enforce strict behavior when CrawlerProxy fails | `true` | `true` / `false` | +| `CrawlerProxies` | List of proxies to use for web crawling and indexing | Empty | Comma-separated proxies | + +### Notes: + +- Proxies must be specified as a **comma-separated list**. +- Each proxy should be in the format: + - `ADDRESS:PORT` + - `ADDRESS:PORT:USER:PASSWORD` +- Example: + - `proxy1.example.com:8080:user1:pass1, proxy2.example.com:8080` ## Crawling and Indexing -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|----------------------------|-----------------------------------------------------|-------------------|---------------------------| -| `ConcurrentStandardCrawlers` | Number of concurrent standard crawlers | `12` | Positive integer | -| `ConcurrentChromeCrawlers` | Number of concurrent Chrome-based crawlers | `4` | Positive integer | -| `CrawlingInterval` | Interval for refreshing crawled results | `24h` | Any valid Go duration | -| `MaxPagesPerDomain` | Maximum number of pages to crawl per domain | `10` | Positive integer | -| `IndexBatchSize` | Number of pages to index in a single batch | `50` | Positive integer | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| ------------------------------ | --------------------------------------------- | ------------------- | ----------------------------------- | +| `ConcurrentStandardCrawlers` | Number of standard crawlers running at once | `12` | Positive number | +| `ConcurrentChromeCrawlers` | Number of Chrome-based crawlers running | `4` | Positive number | +| `CrawlingInterval` | Time between refreshing crawled results | `24h` | Duration like`1h`, `30m`, `2h45m` | +| `MaxPagesPerDomain` | Maximum pages to crawl per domain | `10` | Positive number | +| `IndexBatchSize` | Number of pages to index in one batch | `50` | Positive number | ## Cache Configuration + ### RAM Cache -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|----------------------|-------------------------------------|-------------------|---------------------------| -| `Duration` | Duration to store results in RAM | `6h` | Any valid Go duration | -| `MaxUsage` | Maximum usage of RAM cache | `90%` | Percentage (`%`) or size (e.g., `2GiB`) | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| -------------- | ------------------------------ | ------------------- | ----------------------------------------- | +| `Duration` | Time to store results in RAM | `6h` | Duration like`1h`, `30m`, `2h45m` | +| `MaxUsage` | Maximum RAM usage | `90%` | Percentage (`%`) or size (e.g., `2GiB`) | ### Drive Cache -| **Variable** | **Description** | **Default Value** | **Possible Values** | -|----------------------|-------------------------------------|-------------------|---------------------------| -| `Duration` | Duration to store results on disk | `48h` | Any valid Go duration | -| `MaxUsage` | Maximum usage of drive cache | `90%` | Percentage (`%`) or size (e.g., `10GiB`) | -| `Path` | Path to the drive cache directory | `./cache` | Any valid file path | + +| **Variable** | **Description** | **Default Value** | **Possible Values** | +| -------------- | ---------------------------------- | ------------------- | ------------------------------------------ | +| `Duration` | Time to store results on disk | `48h` | Duration like`1h`, `30m`, `2h45m` | +| `MaxUsage` | Maximum disk usage | `90%` | Percentage (`%`) or size (e.g., `10GiB`) | +| `Path` | Path to the disk cache directory | `./cache` | Any valid file path |