Key changes:
- Renamed RegionFileReassembler to RegionParser for better semantics
- Refactored RegionParser to use ByteBufAllocator injection and improved memory management
- Added thread-local buffer reuse in BundleWriter to reduce allocations
- Implemented Zstd dictionary training functionality for region files
- Added region file statistics collection for training optimization
- Improved error handling and resource management
Detailed changes:
BundleReader/BundleWriter:
- Updated to use RegionParser instead of RegionFileReassembler
- Modified RegionParser constructor to accept ByteBufAllocator
- Changed writeReassembled() method signature to use injected allocator
- Added thread-local buffer reuse to reduce memory allocation overhead
- Improved logging with better formatted output
Main class:
- Added dictionary training support with --train, --train-sample-size, and --train-dict-size options
- Implemented region file statistics collection with --stat option
- Added proper validation for training parameters
- Enhanced help text and error messages
RegionParser:
- Improved file reading using FileChannel for better performance
- Added size validation to prevent processing overly large files
- Implemented readChunkDecompress() method for training purposes
- Better resource management with proper buffer allocation/release
New features:
- ZstdRegionTrainer class for training zstd dictionaries on region files
- Statistics collection for chunk size distribution analysis
- Dictionary training workflow integrated into main application
Performance improvements:
- Reduced memory allocations through buffer reuse
- Better resource management with proper close() implementations
- Improved error handling for corrupted region files
Testing:
- Updated test cases to reflect RegionParser rename
iceBear67
455b8c0d4a