202 lines
4.5 KiB
Markdown
202 lines
4.5 KiB
Markdown
# Document Parser
|
|
|
|
**[English](README.md)** | **[简体中文](README_zh-CN.md)**
|
|
|
|
---
|
|
|
|
# Document Parser
|
|
|
|
A high-performance multi-format document parsing service supporting PDF, Word, Excel, and PowerPoint with GPU acceleration capabilities.
|
|
|
|
## Features
|
|
|
|
- 🚀 **High-Performance Parsing**: MinerU and MarkItDown dual-engine support
|
|
- 🎯 **GPU Acceleration**: CUDA/sglang support for GPU acceleration (optional)
|
|
- 🔧 **Zero-Configuration Deployment**: Automatic environment detection and dependency installation
|
|
- 📚 **Multi-Format Support**: PDF, Word, Excel, PowerPoint, Markdown, and more
|
|
- 🌐 **HTTP API**: RESTful API interface for easy integration
|
|
- 📊 **Real-time Monitoring**: Built-in performance monitoring and health checks
|
|
- ☁️ **OSS Integration**: Alibaba Cloud OSS support for cloud storage
|
|
|
|
## Quick Start
|
|
|
|
### 1. Environment Initialization
|
|
|
|
```bash
|
|
cd document-parser
|
|
|
|
# Initialize uv virtual environment and dependencies (first time)
|
|
document-parser uv-init
|
|
|
|
# Check environment status
|
|
document-parser check
|
|
```
|
|
|
|
### 2. Start Service
|
|
|
|
```bash
|
|
# Start document parsing service
|
|
document-parser server
|
|
|
|
# Or specify custom port
|
|
document-parser server --port 8088
|
|
```
|
|
|
|
The service will start at `http://localhost:8087` (default) and automatically activate the virtual environment.
|
|
|
|
## System Requirements
|
|
|
|
### Basic Requirements
|
|
- **Rust**: 1.70+
|
|
- **Python**: 3.8+
|
|
- **uv**: Python package manager
|
|
|
|
### GPU Acceleration (Optional)
|
|
- **NVIDIA GPU**: CUDA-compatible
|
|
- **CUDA Toolkit**: 11.8+
|
|
- **GPU Memory**: At least 8GB recommended
|
|
|
|
## Supported Formats
|
|
|
|
| Format | Parsing Engine | Features |
|
|
|--------|----------------|----------|
|
|
| PDF | MinerU | Professional PDF parsing, image extraction, table recognition |
|
|
| Word | MarkItDown | Document structure preservation, format conversion |
|
|
| Excel | MarkItDown | Table data extraction, format preservation |
|
|
| PowerPoint | MarkItDown | Slide content extraction, image saving |
|
|
| Markdown | Built-in | Real-time parsing, table of contents generation |
|
|
|
|
## Configuration
|
|
|
|
### Basic Configuration
|
|
|
|
```yaml
|
|
# Server configuration
|
|
server:
|
|
port: 8087
|
|
host: "0.0.0.0"
|
|
|
|
# MinerU configuration
|
|
mineru:
|
|
backend: "vlm-sglang-engine" # Enable GPU acceleration
|
|
max_concurrent: 3
|
|
quality_level: "Balanced"
|
|
```
|
|
|
|
### GPU Acceleration Configuration
|
|
|
|
```yaml
|
|
mineru:
|
|
backend: "vlm-sglang-engine" # Use sglang backend
|
|
max_concurrent: 2 # Lower concurrency for GPU
|
|
batch_size: 1
|
|
```
|
|
|
|
## Common Commands
|
|
|
|
```bash
|
|
# Environment management
|
|
document-parser check # Check environment status
|
|
document-parser uv-init # Initialize environment
|
|
document-parser troubleshoot # Troubleshooting guide
|
|
|
|
# Service management
|
|
document-parser server # Start service
|
|
document-parser server --port 8088 # Specify port
|
|
|
|
# File parsing (CLI)
|
|
document-parser parse --input file.pdf --output result.md --parser mineru
|
|
```
|
|
|
|
## API Usage
|
|
|
|
### Parse Document
|
|
|
|
```bash
|
|
curl -X POST "http://localhost:8087/api/v1/documents/parse" \
|
|
-H "Content-Type: multipart/form-data" \
|
|
-F "file=@document.pdf" \
|
|
-F "format=pdf"
|
|
```
|
|
|
|
### Get Parsing Status
|
|
|
|
```bash
|
|
curl "http://localhost:8087/api/v1/documents/{task_id}/status"
|
|
```
|
|
|
|
### API Documentation
|
|
|
|
Once the service is running, visit:
|
|
- **OpenAPI Swagger UI**: `http://localhost:8087/swagger-ui/`
|
|
- **OpenAPI JSON**: `http://localhost:8087/api-docs/openapi.json`
|
|
|
|
## Performance Optimization
|
|
|
|
### GPU Acceleration
|
|
|
|
1. Ensure `sglang[all]` is installed
|
|
2. Configure `backend: "vlm-sglang-engine"`
|
|
3. Adjust concurrency parameters based on GPU memory
|
|
4. Monitor GPU usage
|
|
|
|
### Concurrency Control
|
|
|
|
```yaml
|
|
mineru:
|
|
max_concurrent: 2 # Adjust based on system performance
|
|
batch_size: 1 # Process in small batches
|
|
queue_size: 100 # Queue buffer size
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Virtual environment not activated**: Run `source ./venv/bin/activate`
|
|
2. **Dependency installation failed**: Run `document-parser uv-init`
|
|
3. **GPU acceleration not working**: Refer to CUDA Environment Setup Guide
|
|
4. **Permission issues**: Check directory and user permissions
|
|
|
|
### Get Help
|
|
|
|
```bash
|
|
# Detailed troubleshooting guide
|
|
document-parser troubleshoot
|
|
|
|
# Environment status check
|
|
document-parser check
|
|
|
|
# View logs
|
|
tail -f logs/log.$(date +%Y-%m-%d)
|
|
```
|
|
|
|
## Development
|
|
|
|
### Build
|
|
|
|
```bash
|
|
cargo build --release
|
|
```
|
|
|
|
### Test
|
|
|
|
```bash
|
|
cargo test
|
|
```
|
|
|
|
### Code Check
|
|
|
|
```bash
|
|
cargo fmt
|
|
cargo clippy
|
|
```
|
|
|
|
## License
|
|
|
|
This project is licensed under MIT License.
|
|
|
|
## Contributing
|
|
|
|
Issues and Pull Requests are welcome!
|