0 Fork 0

T

iceBear67 455b8c0d4a refactor: optimize region file handling and add dictionary training support

Key changes:
- Renamed RegionFileReassembler to RegionParser for better semantics
- Refactored RegionParser to use ByteBufAllocator injection and improved memory management
- Added thread-local buffer reuse in BundleWriter to reduce allocations
- Implemented Zstd dictionary training functionality for region files
- Added region file statistics collection for training optimization
- Improved error handling and resource management

Detailed changes:

BundleReader/BundleWriter:
- Updated to use RegionParser instead of RegionFileReassembler
- Modified RegionParser constructor to accept ByteBufAllocator
- Changed writeReassembled() method signature to use injected allocator
- Added thread-local buffer reuse to reduce memory allocation overhead
- Improved logging with better formatted output

Main class:
- Added dictionary training support with --train, --train-sample-size, and --train-dict-size options
- Implemented region file statistics collection with --stat option
- Added proper validation for training parameters
- Enhanced help text and error messages

RegionParser:
- Improved file reading using FileChannel for better performance
- Added size validation to prevent processing overly large files
- Implemented readChunkDecompress() method for training purposes
- Better resource management with proper buffer allocation/release

New features:
- ZstdRegionTrainer class for training zstd dictionaries on region files
- Statistics collection for chunk size distribution analysis
- Dictionary training workflow integrated into main application

Performance improvements:
- Reduced memory allocations through buffer reuse
- Better resource management with proper close() implementations
- Improved error handling for corrupted region files

Testing:
- Updated test cases to reflect RegionParser rename

2025-09-28 08:51:36 +08:00

.github/workflows

Initial commit

2025-09-20 21:59:11 +08:00

bundler

refactor: optimize region file handling and add dictionary training support

2025-09-28 08:51:36 +08:00

gradle/wrapper

Initial commit

2025-09-20 21:59:11 +08:00

src/main

feat: implement region file reassembly and improve bundle format