Results 1 to 13 of 13
  1. #1
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    Overkill
    If you're using datafeeds to create static pages, what would be the most you would bother uploading in the course of a few days? Is there "too much" you can do at once?
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  2. #2
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    I take it this was a dumb question? Feel free to be honest...
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  3. #3
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    Quote Originally Posted by Noth
    If you're using datafeeds to create static pages, what would be the most you would bother uploading in the course of a few days? Is there "too much" you can do at once?
    The first time that used a datafeed about three years ago, it took me more than 24 hrs from start to finish.

    I was new at datafeeds and WebMerge so it took me a while to make my first 1000 pages, and since I was on dial-up, uploading those 1000 pages was not that easy.

    Every time that I had about two or three hundred pages in the uploading process, I would get disconected and lose all that work, only to start again and get disconected when I was about done with the uploading.

    When I finally had all those 1000 pages live, I discovered that I had a broken link on all the pages because I forgot to put the ../ before some links on the template. So I had to start all over again the painfull uploading.

    But that was then, after I got more experience with WebMerge and datafeeds, I would do any new site with under 1000 pages in just a couple of hours and had everything working on the first try.

    Last year I decided to learn PHP & MySQL and I also moved up to DSL. What a difference DSL makes, those 1000 pages that took me hours to upload, now it only take minutes to have them live on the web.

    Last month I tried my hands at some PHP & MySQL and I was able to make some good test sites in just a few seconds, so the future looks bright for me at this moment.

    While I have no regrets, I wish that I would had tried DSL and started to learn PHP & MySQL three years ago, so my 2005 earnings were better.

    But, since never it's too late to learn, I am just looking foward to the next holiday seasons and all the green I can make if I get to work on my new ideas.

    Sal.
    ...

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    Since this is programming and not fourthworld, I take it you are programming them yourself? Why not just write the scripts on the server and you don't have this issue? For any process to work, you need to automate it as much as possible, from the server you can download, process and pump out pages. Once you go this route and can do everything that way, you will have a hard time putting up with the issues of processing first, uploading later.

    Chet

  5. #5
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    I take it you are programming them yourself?
    Yes, I am.
    Why not just write the scripts on the server and you don't have this issue?
    Remember, I am just starting to learn PHP, so it will take me some time to get better at it.
    For any process to work, you need to automate it as much as possible, from the server you can download, process and pump out pages. Once you go this route and can do everything that way, you will have a hard time putting up with the issues of processing first, uploading later.
    Those are my plans for later, I first have to learn to walk, before I can run.

    At least I have learned some basic functions in less than 60 days, so eventually I will get there soon, for now I am just taking one step at the time, I will see how I do by next July.

    Sal.
    ...

  6. #6
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Noth
    If you're using datafeeds to create static pages, what would be the most you would bother uploading in the course of a few days? Is there "too much" you can do at once?
    I would try to find a way to do this without uploading thousands of HTML documents. It's very cumbersome and unnecessary. I'd listen to chetf.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  7. #7
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    As for uploading pages, there isn't a set amount that'd be "too many." It's mainly a matter of your connection speed and how much patience you have.

    The main trick is to put all the pages into a zip file, and then just upload that. Then unzip it on the server-side (and make sure the permissions are right on the unzipped pages, so people can actually see them!). I've uploaded over 100,000 pages at a go, that way Took about 40 minutes to make the zip file(!), and 2 hours to upload it over cable.

    Haven't done that since I put that site over to PHP, though!

    I am just starting to learn PHP, so it will take me some time to get better at it.
    Try making a test directory on your server and practicing your dynamic PHP skills there, if you're leery about switching your main site over to dynamic processing just yet. But do aim for getting everything working on the server-side ASAP, due to what Chet said.

    You can totally delete the tediousness of updating, with server-side processing. Stick in the new feed data, presto, site(s) updated!
    There is no knowledge that is not power. ~Hemingway

  8. #8
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Mr. Sal
    The first time that used a datafeed about three years ago, it took me more than 24 hrs from start to finish.

    I was new at datafeeds and WebMerge so it took me a while to make my first 1000 pages, and since I was on dial-up, uploading those 1000 pages was not that easy.

    Every time that I had about two or three hundred pages in the uploading process, I would get disconected and lose all that work, only to start again and get disconected when I was about done with the uploading.
    Uh oh, oh no, I am still on dial-up

    Is it impossible for someone on dial-up to deal with those datafeeds?
    ~Rhia7 -- Remember the 7
    Twitter me

  9. #9
    ABW Ambassador
    Join Date
    November 26th, 2005
    Posts
    560
    Is it impossible for someone on dial-up to deal with those datafeeds?
    Of course not
    It may take some time more but you can do it certainly as Mr. Sal says above. And yes, if you are going to do this regularly, then DSL is definitely better

    Every time that I had about two or three hundred pages in the uploading process, I would get disconected and lose all that work, only to start again and get disconected when I was about done with the uploading.
    Though Leader has suggested the best way for a lot of pages, but if you are on dial-up and doing it manualy, you can try getting a ftp client which supports interrupted uploads so that even if your connection gets disconnected, you still resume it only where you left.
    The Best Forums
    Nothing in the world can take the place of persistance
    . Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistance and determination are omnipotent.
    Abestweb Store

  10. #10
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Every time that I had about two or three hundred pages in the uploading process, I would get disconected and lose all that work, only to start again and get disconected when I was about done with the uploading.
    Oh man, I remember those days *shudder*

    That happened to me a lot with dialup. I was already doing the zipping, but it still took ages to send up each zip file, and I got hung up on a lot. One night I uploaded a batch (in a zip file) over dialup...I forgot how many pages were in that file, but it was almost done after 3 hours and 50 minutes (and wasn't anywhere near 100k pages worth)...

    And then I got disconnected!!!

    That was the last straw, on an already big pile of dialup-related straws. I yanked the phone cord out of the computer right then, plugged it into the phone (had a landline then), and dialed the number that had been all over the radio: 1-800-COMCAST!

    The Comcast lady wasn't surprised a bit to get a signup for cable internet in the predawn hours, after I told her what happened. And she said she'd had more than one person sign up at that time after having similar experiences! She could have tried to avoid sounding so overjoyed about it, though...there must be a heckuva commission for a new signup
    There is no knowledge that is not power. ~Hemingway

  11. #11
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    Thanks for the replies, but I phrased my question poorly. I have an automated process, and the "time spent" isn't the question (I just have to tidy the datafeeds a little and press "Upload").

    My question revolved more around the Search Engine perspective (in which case I maybe should have posted it elsewhere). Is adding 10 or 12 thousand pages in a couple of weeks a help or a hinderance in getting spidered, or neither? In other words, is there any penalty for growing a site too quickly?

    I asked it here because I am using feeds to do it...
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  12. #12
    15 years and counting
    Join Date
    January 18th, 2005
    Posts
    6,121
    Noth - is there any penalty for growing a site too quickly?
    In a word, YES. It's documented by Google and I experienced it a few times.

  13. #13
    ABW Ambassador
    Join Date
    November 26th, 2005
    Posts
    560
    Is adding 10 or 12 thousand pages in a couple of weeks a help or a hinderance in getting spidered, or neither? In other words, is there any penalty for growing a site too quickly?
    Yes, some say it hurts.
    However, would the following help?
    Say, you upload all your files in one go BUT you block googlebot from indexing all your files through robots.txt - just keep releasing few pages in the robots.txt in batches. Has anyone tried it?
    Disclaimer : I haven't tried it myself just am thinking if this is a solution ?
    The Best Forums
    Nothing in the world can take the place of persistance
    . Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistance and determination are omnipotent.
    Abestweb Store

  14. Newsletter Signup

+ Reply to Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •