VortexOfCreation
Member
- Joined
- Nov 13, 2018
- Messages
- 366
Writing a guide on how to use NCollector Studio which in my opinion is much more user and performance friendly than HTTrack, however there are some problems with NCollector which I'll explain these further along in the guide.
I TRIED TO MAKE THIS GUIDE AS USER FRIENDLY AS POSSIBLE BUT IF YOU'RE HAVING TROUBLE, BE SURE TO WRITE A REPLY DETAILING WHAT YOU ARE HAVING TROUBLE WITH AND I WILL TRY TO HELP :smile:
To get started we're going to have to download NCollector Studio which can be found here:
http://www.calluna-software.com/Products/Downloads and click on NCollector Studio once the file has downloaded run the setup and install to a directory of your choosing. Once you start up the program this is what you'll see.
Click on Settings.......First page will display rather pointless information but if you're not a fan of the bright white theme then you can opt to change it by simply clicking on the Dark Theme which will simply darken the theme of the program.
In the Folders Section will show your output folder where your project will be saved. Not too sure about the "Projects" folder but the "Downloads" folder will contain everything in from links, jpeg, png all the way to gif files and much more.
(I have not experimented with the "Other" section so that will not be included in this guide)
On the final page "Connections" we want to pay attention to Maximum Simultaneous Connections.
Below are some tests I have run using joyofsatan.org running at a maximum of 5 levels(explained further in the guide)
As you can see running the download at 500 Maximum Simultaneous Connections(MSC) yielded a result of over 8 minutes and the fastest being 125 connections at over 3 minutes....However the Joy of satan site is a rather small sub-domain of satanisgod.org so the results you see here will be alot different for larger files, but in my opinion I would keep it at 250 since during my testing of 125 connections it felt a lot slower even the timing was better by almost a full minute.
Now that we're all done the settings you can simply click "Save changes" you'll be brought back to the homepage and this is where we start to download our site. There is a lot of options to choose from ranging from downloading a single web-page,music, pictures, videos or even entire sites(recommended for people with fast internet) What we're looking for is "Offline Browser Wizard" which can simply be found here.
Click on it and it will take you to the next link.
At the top you'll see "Project Name" give it name of your choice, next is "Target Folder" which is where all the contents will be saved, next is the "User Agent" by default it'll be set at NCollector Studio 1.0 which will emulate the site as if you're browsing it on Internet Explorer or Microsoft Edge there is multiple options to choose from by clicking on the dropdown arrow. Next up we have "Source Address" this is where you put the link you want to download, a login option is also included but I'm unsure how to get this working.
Next we have Domain Restrictions, here you can include links to be downloaded or exclude links to prevent them from being downloaded I'll be using ancient-forums as a base and excluding all subforums to make it easier to download the one link. Finally we have "Limits" Pages is pretty self explanatory and should be left to Unrestricted since NCollector Studio only allows for a total of 100 pages to be downloaded, then we have "Maximum Pages" which is pretty much the meat and bones of the download. Example:
My recommended level to choose is either 5 or 6, this will take a very long time but it'll download pretty much everything you need. In my opinion(this will need testing) any higher is just a waste of time cause if you have a youtube link within the site you're trying to download NCollector Studio will attempt to download contents from YouTube. but since we're using "Ancient-Forums" as a base + additional domain restrictions we'll be using Level to assure all is downloaded without having to download every link contained within a link contained within multiple different links.
After you've finished choosing your name, adding restrictions and setting Max Levels you can simply click next until you're given the option to start it. Depending on the levels you've chosen and the link you're attempting to download this can take quite some time.....So go do something while you wait for it download.
Once it has finished downloaded you'll be met with this image
In the "Images" will display all image related content, "Files" will display name of the files, what level they were downloaded at and whether they were successful in being downloaded. Next is the "Offline" which is a view-port of the site you've just downloaded, it is best to experiment with this and test which links work and which ones dont. Finally we have "Summary" this will display all the options you picked when creating the project and even additional options such as how many links were processed, how many files were downloaded and how long it took to download the site.
Now we come to the errors. NCollector unlike HTTrack which organizes the projects into one index.html file will not work with NCollector so downloading another Project and attempting to run site on a previous index.html will not work.
NCollector will not display any error messages/logs I believe this can be seen when ticking the option(Unsure so you'll need to experiment) "Enable Logging(requires restart)" found in settings.
To Merge the index.html files together to be used as one html site, we need to download SoftSnow Merger, once downloaded run the installer then open the file once the installation is complete. This is what you'll see
Click on Add and add 2 or more html files, I have downloaded ancient-forums as a base and JoS Newsletter to be used as an addition to the Base link. You will want to navigate to Documents\NCollector\%NAME OF PROJECT)\https.ancient-forums.com or where-ever you saved your project and click on index.html some projects will have multiple index files, you can simply add those ones too. Once you have added the index files from both Project. Click on "Run tidy" and download the file from here http://tidy.sourceforge.net/ and install it whether you like, download link is found at the bottom of the page. In Tools>Options>Tidy you'll need to add a path to the Tidy.exe.....This tool will I believe clean up the html file in case you run into any strange problems (Needs experimenting)However this tool is very old, 20 years old to be exact so I believe it should only used as last resort.
Once everything is setup choose the Save destination and click on merge.
Voila you're finished downloading 2 sites and merging them into one.
HAIL SATAN AND ALL THOSE ALLIED TO HIM :smile:
I TRIED TO MAKE THIS GUIDE AS USER FRIENDLY AS POSSIBLE BUT IF YOU'RE HAVING TROUBLE, BE SURE TO WRITE A REPLY DETAILING WHAT YOU ARE HAVING TROUBLE WITH AND I WILL TRY TO HELP :smile:
To get started we're going to have to download NCollector Studio which can be found here:
http://www.calluna-software.com/Products/Downloads and click on NCollector Studio once the file has downloaded run the setup and install to a directory of your choosing. Once you start up the program this is what you'll see.
Click on Settings.......First page will display rather pointless information but if you're not a fan of the bright white theme then you can opt to change it by simply clicking on the Dark Theme which will simply darken the theme of the program.
In the Folders Section will show your output folder where your project will be saved. Not too sure about the "Projects" folder but the "Downloads" folder will contain everything in from links, jpeg, png all the way to gif files and much more.
(I have not experimented with the "Other" section so that will not be included in this guide)
On the final page "Connections" we want to pay attention to Maximum Simultaneous Connections.
Below are some tests I have run using joyofsatan.org running at a maximum of 5 levels(explained further in the guide)
Code:
JoyofSatan.org - Links Processed 1239, Documents Scanned: 554, Files Downloaded: 1215 = 89.38MB
Time: 8 minutes and 6 seconds at 500 connections!
JoyofSatan.org(250 connections) - 4 minutes 22 seconds.
JoyofSatan.org(125 connections) - 3 minutes 27 seconds
JoyofSatan.org(1000 connections) - 4 minutes 28 secondsAs you can see running the download at 500 Maximum Simultaneous Connections(MSC) yielded a result of over 8 minutes and the fastest being 125 connections at over 3 minutes....However the Joy of satan site is a rather small sub-domain of satanisgod.org so the results you see here will be alot different for larger files, but in my opinion I would keep it at 250 since during my testing of 125 connections it felt a lot slower even the timing was better by almost a full minute.
Now that we're all done the settings you can simply click "Save changes" you'll be brought back to the homepage and this is where we start to download our site. There is a lot of options to choose from ranging from downloading a single web-page,music, pictures, videos or even entire sites(recommended for people with fast internet) What we're looking for is "Offline Browser Wizard" which can simply be found here.
Click on it and it will take you to the next link.
At the top you'll see "Project Name" give it name of your choice, next is "Target Folder" which is where all the contents will be saved, next is the "User Agent" by default it'll be set at NCollector Studio 1.0 which will emulate the site as if you're browsing it on Internet Explorer or Microsoft Edge there is multiple options to choose from by clicking on the dropdown arrow. Next up we have "Source Address" this is where you put the link you want to download, a login option is also included but I'm unsure how to get this working.
Next we have Domain Restrictions, here you can include links to be downloaded or exclude links to prevent them from being downloaded I'll be using ancient-forums as a base and excluding all subforums to make it easier to download the one link. Finally we have "Limits" Pages is pretty self explanatory and should be left to Unrestricted since NCollector Studio only allows for a total of 100 pages to be downloaded, then we have "Maximum Pages" which is pretty much the meat and bones of the download. Example:
Code:
Using the joyofsatan.com site it has a main domain and sub-domains ie: "Welcome" and "Satan" LEVEL 1: will download all that you see on the first page including links to the sub-domains,
LEVEL 2: Will go deeper in downloading html files and image based files On the "Satan" link there is another link contained with in called "Lower order of Demons" LEVEL 1: will not download this link where Level 2 will, LEVEL 3 will download even more Level 4 even more. etc etc.
To put it short the lower the Level is: the faster the site will download but tonnes of information will be missing from the download. Setting the level even higher will of course download more contents but it'll take even long to download.My recommended level to choose is either 5 or 6, this will take a very long time but it'll download pretty much everything you need. In my opinion(this will need testing) any higher is just a waste of time cause if you have a youtube link within the site you're trying to download NCollector Studio will attempt to download contents from YouTube. but since we're using "Ancient-Forums" as a base + additional domain restrictions we'll be using Level to assure all is downloaded without having to download every link contained within a link contained within multiple different links.
After you've finished choosing your name, adding restrictions and setting Max Levels you can simply click next until you're given the option to start it. Depending on the levels you've chosen and the link you're attempting to download this can take quite some time.....So go do something while you wait for it download.
Once it has finished downloaded you'll be met with this image
In the "Images" will display all image related content, "Files" will display name of the files, what level they were downloaded at and whether they were successful in being downloaded. Next is the "Offline" which is a view-port of the site you've just downloaded, it is best to experiment with this and test which links work and which ones dont. Finally we have "Summary" this will display all the options you picked when creating the project and even additional options such as how many links were processed, how many files were downloaded and how long it took to download the site.
Now we come to the errors. NCollector unlike HTTrack which organizes the projects into one index.html file will not work with NCollector so downloading another Project and attempting to run site on a previous index.html will not work.
NCollector will not display any error messages/logs I believe this can be seen when ticking the option(Unsure so you'll need to experiment) "Enable Logging(requires restart)" found in settings.
Code:
Note: Create a folder on the desktop or anywhere on your choice and copy the contents from the Projects you intend to merge when...Start with the base first, then small links then finally the large links....Over-write when promptedTo Merge the index.html files together to be used as one html site, we need to download SoftSnow Merger, once downloaded run the installer then open the file once the installation is complete. This is what you'll see
Click on Add and add 2 or more html files, I have downloaded ancient-forums as a base and JoS Newsletter to be used as an addition to the Base link. You will want to navigate to Documents\NCollector\%NAME OF PROJECT)\https.ancient-forums.com or where-ever you saved your project and click on index.html some projects will have multiple index files, you can simply add those ones too. Once you have added the index files from both Project. Click on "Run tidy" and download the file from here http://tidy.sourceforge.net/ and install it whether you like, download link is found at the bottom of the page. In Tools>Options>Tidy you'll need to add a path to the Tidy.exe.....This tool will I believe clean up the html file in case you run into any strange problems (Needs experimenting)However this tool is very old, 20 years old to be exact so I believe it should only used as last resort.
Once everything is setup choose the Save destination and click on merge.
Voila you're finished downloading 2 sites and merging them into one.
HAIL SATAN AND ALL THOSE ALLIED TO HIM :smile:
![The Alchemist7 [TG]](/data/avatars/m/1/1057.jpg?1704756197){kind=avatar}
![Voice of Enki [TG]](/data/avatars/s/0/84.jpg?1705131368){kind=avatar}
