չ∂αяквℓσω
Member
Hi everyone, I've been working on this for a while, and as promised, I've finally managed to create the definitive system for backing up the entire forum, all threads/posts, etc.
In previous posts, I've already described the crucial importance of a backup, and I'm extremely happy with the success of this project.
Now, I'll show you an example and a video tutorial on how I performed the procedure:
The script in question is written in Python, but this time I also managed to convert it into a Linux executable, so you can directly run the interpreted Python code by installing the libraries with "pip3 install <library>" and "python3 the_script.py" It's highly recommended to limit excessive RAM usage with: "ulimit -v 1000000 && python3 the_script.py" (this prevents excessive RAM usage, setting it to a maximum of 1GB).
There are two scripts in question. The first, the main one, is used to download an entire forum section at will. There are also counters to let the user know when it's about to finish and how much is left. The second script is very interesting. It essentially allows the complete conversion of .html files into a single .txt file, removing unnecessary content and massively compressing the already optimized files, allowing for unprecedented space savings. And here's the best part:
The total forum download (CSS omitted since it's useless) currently weighs about 13GB, which is too large. With a new and advanced .zip compression system, I managed to compress the entire file, and it weighed a total of 2.7GB.
Yes, can you believe it? Almost a million messages, etc., weighing from 13GB to 2.7GB.
But it doesn't end there, because thanks to the second script, which runs a highly optimized conversion system, I was able to convert all the .html files into a few .txt files based on sections, and without compression, it didn't even give me 1GB. And most importantly, after compression (since the .zip system is highly optimized for compressing .txt files), the forum's total size came to only 266.6MB (what a coincidence, 66.6)!!!
Yes, we've gone from about 13GB at the start to 266.6MB on the way, and anyone can download it, anyone. It's so small now that it's as big as a large Android app. Even on a slow connection, a USB stick, anything, you can archive the entire ToZ forum. I'm not kidding. At first, I couldn't believe it either. I didn't even think I could do this. And in the end, I managed to make full forum backups accessible to everyone, and we'll guarantee that our content will be preserved forever!
I'm extremely happy because I managed to keep my promise!
Here are the download mirrors, which are the entire folder where I also have other projects. I've merged everything into one (if they get banned, I'll obviously try to update them as soon as possible):
https://www.mediafire.com/folder/y90mvxjz2b5vr/Satanas
https://mega.nz/folder/E4I1xKRJ#yP9Aj9DHKA1wsfbDhdsVcw
Hail Zeus!
