Key changes:
- Renamed RegionFileReassembler to RegionParser for better semantics
- Refactored RegionParser to use ByteBufAllocator injection and improved memory management
- Added thread-local buffer reuse in BundleWriter to reduce allocations
- Implemented Zstd dictionary training functionality for region files
- Added region file statistics collection for training optimization
- Improved error handling and resource management
Detailed changes:
BundleReader/BundleWriter:
- Updated to use RegionParser instead of RegionFileReassembler
- Modified RegionParser constructor to accept ByteBufAllocator
- Changed writeReassembled() method signature to use injected allocator
- Added thread-local buffer reuse to reduce memory allocation overhead
- Improved logging with better formatted output
Main class:
- Added dictionary training support with --train, --train-sample-size, and --train-dict-size options
- Implemented region file statistics collection with --stat option
- Added proper validation for training parameters
- Enhanced help text and error messages
RegionParser:
- Improved file reading using FileChannel for better performance
- Added size validation to prevent processing overly large files
- Implemented readChunkDecompress() method for training purposes
- Better resource management with proper buffer allocation/release
New features:
- ZstdRegionTrainer class for training zstd dictionaries on region files
- Statistics collection for chunk size distribution analysis
- Dictionary training workflow integrated into main application
Performance improvements:
- Reduced memory allocations through buffer reuse
- Better resource management with proper close() implementations
- Improved error handling for corrupted region files
Testing:
- Updated test cases to reflect RegionParser rename
- Add fastutil and lz4-java dependencies for efficient data structures and compression
- Implement VarInt/VarLong serialization for more compact bundle headers
- Redesign bundle header format to include timestamps and use variable-length integers
- Replace GZIP-based compression with region file reassembly for better compression of chunk data
- Add RegionFile and RegionFileReassembler classes for handling Minecraft region files
- Implement configurable compression types (GZIP, ZLIB, LZ4, NONE) for region files
- Add timestamp filtering and duplicate handling to BundleReader
- Improve parallel processing capabilities with configurable worker threads
- Add JUnit 5 testing support and configure shadowJar minimization
- Update file extensions from .jpack to .jbp.zst for clarity
- Enhance error handling and logging throughout the codebase
- Add comprehensive region file manipulation including chunk-level recompression
- Implement memory-efficient region file parsing using Netty ByteBuf
- Add support for dictionary-based Zstd compression
- Update backup worker to use new bundle format
- Various performance optimizations and bug fixes
- Convert project to multi-module structure with 'bundler' subproject
- Extract bundler functionality into separate module with proper dependencies
- Improve IO state management with atomic reference for better synchronization
- Enhance bundler performance with parallel processing support
- Add CLI interface to bundler with various compression options
- Fix file handling issues in bundle reader/writer
- Update compression configuration and remove redundant options
- Improve region file synchronization with proper state management
- Add dictionary support for better compression ratios
- Refactor backup strategies to use new bundler API
- Update dependencies and build configuration
- Fix path resolution issues in bundle operations
`
- Remove s3-lite submodule and replace with AWS SDK and shadow plugin
- Implement comprehensive backup commands with list, delete, create, restore, and suspend functionality
- Add bundle format with compression support using Zstd and gunzip optimization
- Implement local and S3 backup strategies with configurable options
- Add backup tracking system with JSON persistence
- Improve world directory handling and file exclusion logic
- Add standalone bundler tool for bundle creation/extraction
- Update Gradle configuration with shadow jar support and dependency management
- Enhance configuration system with additional options for compression and temporary directories
- Add zstd-jni dependency for compression support
- Define Backup record with name, key, incremental, and size fields
- Implement BackupManager with backup listing, deletion, and creation methods
- Create BackupWorker to handle backup operations with server state coordination
- Refactor WorldDir to simplify dimension handling and file listing
- Introduce bundling system with SerialBundlerReader^[[1;5P/Writer for efficient storage
- Add BackupStrategy interface with LocalBackupStrategy implementation
- Support both compressed and uncompressed backup formats
- Add config option for parallel backup operations
- Remove old backup implementation from JustBackupMod
- Update backup file format with metadata footer and magic number validation
iceBear67