Results 1 to 24 of 24
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    148
    Hi All,

    This week has been very interesting for me and very busy. I just learned how to make datafeed sites using WebMerge. That's great, but it poses some problems.

    1.)What is the best way to upload 8716 html pages? Can a .zip file be made and uploaded and then unzipped into the correct folder on my server? Or some other way?

    2.) If I'm going to be making much larger site, my current host might not be the best place. I have a virtual hosting account with unlimited domains (.50 cents per domain a month) and my storage is only at $0.02 per MB. That sounds pretty good to me, but I have nothing to compare it too. What's the best kind of hosting for large datafeed sites, and any suggestions on hosts?

    Any help with either 1 or 2 above would be appreciated.

    Ken

  2. #2
    Full Member
    Join Date
    January 18th, 2005
    Posts
    373
    Hi Ken,

    1. Assuming you're hosted on a Linux server , it's possible, but for this the server needs to have a certain module installed. And you have to have access on the server to unzip the file. Best is to ask your host about this.

    2. One of my favorite hosts for datafeed sites is Servint. Their Virtual Private Server package has a lot of space and bandwith, a reasonable price, and their support is great. However, VPS can be quite difficult in the beginning as with VPS you have a lot more control over the server than with a normal shared hosting account.

  3. #3
    ABW Ambassador Radegast's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,978
    If you don't have shell access to the server it's worth contacting your host to see if they'll unzip it for you.

  4. #4
    Full Member
    Join Date
    January 18th, 2005
    Posts
    379
    I once had a site that was 200 megs that I compresssed dpwn to a 6 meg upload.
    Support for typical zip files is limited so a better approach was to use common Unix compression formats. In my case I created a tar.gz file. It's a two step process.

    The first step is to create a tar file of the site. A .tar file is a tape archive file that keeps related files together, and thus facilitates the transfer of multiple files between computers. However a tar file is not compressed.

    The second step is to compress the tar file into a gz file.

    Now you have a nice small file to upload called something.tar.gz

    I use Cpanel on the server and you can unpack a tar.gz file from within cpanels file manager.

    To make the tar file I used UltimateZip (donationware)
    I then used 7-zip (free) to make the gz file.

    I believe that 7-zip can do both but it's still a two step process.

    Hope this helps.

  5. #5
    Newbie
    Join Date
    January 18th, 2005
    Posts
    22
    Ken,

    One way to go would be to use the "tar" and "gzip" commands to combine all 8000 files into a single compressed binary file. Cygwin is a program for PC's that has these commands available.

    Then upload that file to your server in binary mode and then do the reverse. A simple perl program can "gunzip" and "untar" the files using Linux system calls.

    I can provide the perl program for this if you get to that point.

    I most often use this method to transfer files from server to server since these commands are built into Linux. It works great.

    I could also provide you with a perl program that does the actual "Unzip" if you'd like but it isn't as simple and would charge a few bucks for it. It would require the Perl module called "Archive::Zip" to operate properly.

    Take care,
    Bo

  6. #6
    Newbie
    Join Date
    January 18th, 2005
    Posts
    5
    I've got a couple of reseller hosting accounts on separate servers. Both these support extraction of winzip files via the file manager in the cpanel.
    Upload a small test zip, go into cpanel (usually yourdomain.com:2082/), and select file manager. Can take a bit to get used to the file manager web interface, but once you have a zip file selected the option should be there.

    Hamish

  7. #7
    Member
    Join Date
    January 18th, 2005
    Posts
    148
    Hi Guys,

    Thanks for the suggestions. I don't know which I'll eventually use, but at least I have some options now.

    I was hoping that there was a Silver Bullet that was VERY SIMPLE, but I guess not. VERY SIMPLE is my favorite way of doing things.

    But again, thanks for your input.

    Ken

  8. #8
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    I am not sure I have a "Silver Bullet," but an easy way to upload. One catch though, it involves having a Linux machine on your end.

    After building pages, I store them on my little server here ($250. investment), and turn my chair around and type in a couple of commands on that (Linux) machine. In this case "NC FTP."

    The files don't need compressing or anything else. You will need a high-speed connection too (DSL or Cable).

    I know that's a lot of caveats, but it runs very smoothly once set up. My Home Linux machine serves as both a server here, and communicates with it's "brother" web server in Florida.

    I went the Linux route because it is way more robust in handling files than the Wind*ws machine I use for most everything else.

    Good luck.

    Fred

  9. #9
    affiliate emeritus missdonna's Avatar
    Join Date
    January 18th, 2005
    Location
    Torrance, CA
    Posts
    1,986
    On my dial-up connection I can upload around 3,000 pages per hour. I just start the FTP program running and forget about it.

    As long as I don't get distracted and hang up my connection it works just fine for me. If I have many hours worth I just start it up and go to bed. In the morning it has either finished or stopped, and I can pick up where I left off.
    Affiliate Marketing - The hardest easy money I ever made.

  10. #10
    Newbie
    Join Date
    January 18th, 2005
    Posts
    4
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Ken Walton:
    Hi All,

    This week has been very interesting for me and very busy. I just learned how to make datafeed sites using WebMerge. That's great, but it poses some problems.

    1.)What is the best way to upload 8716 html pages? Can a .zip file be made and uploaded and then unzipped into the correct folder on my server? Or some other way?

    2.) If I'm going to be making much larger site, my current host might not be the best place. I have a virtual hosting account with unlimited domains (.50 cents per domain a month) and my storage is only at $0.02 per MB. That sounds pretty good to me, but I have nothing to compare it too. What's the best kind of hosting for large datafeed sites, and any suggestions on hosts?

    Any help with either 1 or 2 above would be appreciated.

    Ken <HR></BLOCKQUOTE>

  11. #11
    Newbie
    Join Date
    January 18th, 2005
    Posts
    4
    Dear Ken:

    Have you tried Webmasters.com as a web host?
    I believe they offer dedicated servers for $99 a month. http://www.webmasters.com/

    Carol

  12. #12
    Newbie
    Join Date
    January 18th, 2005
    Posts
    5
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by missdonna:
    On my dial-up connection I can upload around 3,000 pages per hour. I just start the FTP program running and forget about it.

    As long as I don't get distracted and hang up my connection it works just fine for me. If I have many hours worth I just start it up and go to bed. In the morning it has either finished or stopped, and I can pick up where I left off. <HR></BLOCKQUOTE>

    I've just started with datafeeds. I've used WebMerge and I have around 11,000 html pages that work fine on my local machine. I am having trouble with my FTP. I can start it uploading, but there's always an error when I come back. It NEVER uploads everyting correctly.

    I'm currently trying Wise-FTP with a dialup from a Win XP Home box. Do I need another version of this or another FTP software to be more reliable? I have DSL available in my area so I would think seriously about upgrading my connection if I thought that it would solve the problem.

    Another question: Wise-FTP has a "syncronize folders" command. Is it supposed to be robust enough to just start it syncronizing from the root directory? That seems to be a several hours long operation on my feed and like I said, it never completes error free. (I did get the site uploade in it's entirety once).

    Any help or advice would be apprecited. There's no way I can update my site every other week like this!

  13. #13
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    How do I spam the SE's with my fractional T-1 line connection on my work system. Tips from you experts would be appreciated on what script to use to pad 200,000 SERP slurping pages with juicy keyword strings and a few popunders, autocookie setting competing merchants.

    Is there some cloaking script to hide my feed pages from the prying eyes of shoppers, but open them up for Google and other SE robots? I'm sure I can buy a script that auto re-directs the shopper from the serp or PPCSE listing, which delays the cookie stuffing pop-unders into a 1x1 pixel display. Don't want the shopper to think they came from my page as it flashes by the browser screen so it needs to auto-close after the merchnats cookie is set.

    I don't want to jump on the 180Solutions BHO gravey train as those overworked wanks want a cut and farm out my keywords to other shopping bar HO's. They do offer a URL hit list feature fro creaming some vocal ABWers and those damn SERP competitors. I've got an extra 100 disposable domains and a nifty 1 page merchant directory page in 40 different border colors ready with a 1 minute Text logo to fool 90% of all the major merchant AM's.

    I'd normally use FP 2003 for the feed integration, but I'll just use it's page building power to for giving some PR to the disposable sites from some showcase page links. I know this is probably old hat to some, but I'm new to this doorway page cookie stuffing routine. Coupons are over worked and I'd be a fool to offer some incentive as a payout liability as that would require backoffice hassles and stupid interior page spin.... as well as expose me to paying to privitize those 100 domains from Whois viewers.
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  14. #14
    Defender of Truth, Justice and the Affiliate Way
    Join Date
    January 18th, 2005
    Location
    The Swamp
    Posts
    7,503
    Mike you really need to stop broad stroking everything. The number of pages for a site DOES NOT necessarily indicate spam nor foul play.

    Webmerge is a local software. Which is what both folks posting in this thread were talking about. WM istelf does NOT do any of the types of server side scripts itself you are talking about. Nor does using WM mean that auto-redirects, forced clicks, cookie stuffing, etc are being used on the affs web site.

    What all this broad stroking does do is diminish the discussions of when REAL and VERIFIED foul play is being discussed. I don't think that is what you want.......

  15. #15
    ABW Ambassador phillyburbs's Avatar
    Join Date
    January 18th, 2005
    Location
    in the PhillyBurbs!
    Posts
    3,097
    Kellie:

    Indeed. We generated a thousand pages or so a DAY and we obviously do not spam the search engines ... or anyone else for that matter.

  16. #16
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    bweave,

    In this case, I will say Ignore Mike, this time his post doesn't apply to you.

    As for your FTP, the problem might be on your end, and might be at the server.

    On your end, make sure you are uploading in "binary" mode. As for your FTP client (program), I am not sure if that is the issue, but it might be. I use "FTP Voyager," and it seems to work okay when I use my Windows machine to upload. I haven't done that in a while, but in the past I would usually upload about nine to ten-thousand files at a time. The problem I ran into before was my operating system couldn't handle the large number of files, you shouldn't have that with Win XP though - so I would focus on your FTP client or the server.

    You might want to ask your host what they recommend to upload a large number of files. You'll actually have to ask someone, and be specific, or they will give you the standard answer (usually - use cute FTP).

    All I have at this point, sorry.

    Fred

  17. #17
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,118
    8,000 pages? Wow, what is it some kind of search engine spam? Awesome dude!

  18. #18
    Newbie
    Join Date
    January 18th, 2005
    Posts
    5
    Thanks Fred,

    I'm using the FTP client that came on the CD software package sent out by my host (1&1). The only question is about the number of files and directories I guess. I will contact their support.

    bweave

  19. #19
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    "What all this broad stroking does do is diminish the discussions of when REAL and VERIFIED foul play is being discussed."

    You have Mike's real name, is there going to be just more of this in the Inner Cirle?

  20. #20
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    I've got the the FTP tool in WSFTP-Pro and sure know how to use the manual Unix server tools. I was just inquiring about the finer points to seperate potential feeds from the 10,000 other feed affiliates by jumping to phase II before the SE's whack me. I'd like to be considered a sharper -crafter -grey area player to get some status within this community. Money talks and 6 years of pro-affiliate practice postings got me zip respect and a ignoring yawn from networks, merchants and fellow affiliates.

    I know how feeds work with 43 client's Miva carts being feed with CVS files for years. Heck I know how to download from e-catalogs into Froogle and Yahoo store formats to feed those too. So what's up with me asking for some of the experienced feedfreaks how to do some perfectly legal SE spamming and playing with the grey areas of the CoC and Addendium. Currently it is impossible to protect or grow your income, unless you have 500,000 automatic built-in monthly visitors like PhillyBurbs, without some spammy trickery.
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  21. #21
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>I've just started with datafeeds. I've used WebMerge and I have around 11,000 html pages that work fine on my local machine...

    Any help or advice would be apprecited. There's no way I can update my site every other week like this! <HR></BLOCKQUOTE>

    I would check into the concepts of zipping it at your local machine - ftp'ing it to the host and either using the file manager in cpanel or UNZIP using SSH (or shell) to unzip the files at the host. I went from a 6 hour (when everything worked)10,000 file transfer to a 23,000 file transfer in less than 30 minutes

  22. #22
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Ditto--Zip 'em and ship 'em!
    There is no knowledge that is not power. ~Hemingway

  23. #23
    Newbie
    Join Date
    January 18th, 2005
    Posts
    5
    The "solution" I've found for anyone who is interested:

    Use Dreamweaver's FTP function if you have it. It's much more robust than the other FTP clients that I have. I may upgrade to a hosting package that has SSH access so I can zip/upload/unzip, but for now this is the best way I've found to upload.

    Thanks for the suggestions.

    bweave

  24. #24
    Newbie
    Join Date
    January 18th, 2005
    Posts
    1
    An easy step by step solution (with the cgi script to unpack the uploaded zip file) is at

    http://www.in-my-opinion.org/in-my-opinion-3138.html

    That's how http://www.ebooks-download.com was uploaded

  25. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Congrats to Rexanne on 8000 posts
    By Sue in forum Virtual Family and Off-Topic
    Replies: 27
    Last Post: March 12th, 2008, 01:46 PM
  2. 8000th Post - 8000 Thanks - $8,000.00 Cash Giveaway
    By Geno Prussakov in forum AM Navigator
    Replies: 25
    Last Post: July 12th, 2007, 02:07 PM
  3. Designers: Win 8000 USD!
    By Yugo in forum Midnight Cafe'
    Replies: 0
    Last Post: April 26th, 2003, 04:14 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •