Results 1 to 18 of 18
  1. #1
    ABW Ambassador Vrindavan's Avatar
    Join Date
    February 25th, 2003
    Posts
    1,902
    Dynamic Web Source datafeed solution
    Dynamic Web Source datafeed solution

    dynamicwebsource.com

    Have you try it ?
    Any comment?

  2. #2
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .

    currently my interest is exclusively in static pages, for many reasons.

    heisje

    .

  3. #3
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    Quote Originally Posted by heisje
    .

    currently my interest is exclusively in static pages, for many reasons.

    heisje

    .
    ahhh...the beauty of mod rewrite ;-)

  4. #4
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .

    nah, Spider Ninja,

    pretending to be static and actually being static: not the same to SE spiders, for many reasons. one of them: mod_rewrite is a server re-direct, and they can see it. and they do not like it.

    real static does make a difference : ignore this at your risk . . .

    heisje



    .

  5. #5
    Member SeanW's Avatar
    Join Date
    July 28th, 2005
    Posts
    69
    > mod_rewrite is a server re-direct

    Don't confuse internal and external redirects. With an internal redirect, the url is rewritten within the web server and never passed back to the client. The client asks for /foo.html, the script that gets executed can be /app/foo.php.

    The only way I can think of to detect this would be to ask for the same page a second time and pass an If-Modified-Since. If the page comes back with 304 it's probably static, or the template engine has cached it.

    Sean

  6. #6
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    I'm going to agree here with Sean...if you call up a page:

    http://www.domain.com/cook_books.html

    and the server triggers a script passing a variable of "cook books" to the script, then returns the html it is seen for all purposes as a static page to whoever called it.

    The resulting HTTP head would be 200 and then the html...no redirect could be detected ;-)

  7. #7
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    Quote Originally Posted by SeanW
    >The only way I can think of to detect this would be to ask for the same page a second time and pass an If-Modified-Since. If the page comes back with 304 it's probably static, or the template engine has cached it.
    Hey Sean, you could pass back Last-Modified Aug 1, 2001 or whatever else you wanted...

  8. #8
    Member SeanW's Avatar
    Join Date
    July 28th, 2005
    Posts
    69
    Hey Sean, you could pass back Last-Modified Aug 1, 2001 or whatever else you wanted...
    But if it were modified before the "If-Modified-Since" then it should return 304-Content not Modified. If I'm sending the I-M-S header it says that I have a copy of the page dated YYYYMMDD and not to bother sending me the page if it's older.

    Dynamic pages rarely have this information -- each page is generated on the fly.

    This behaviour isn't unique to datafeed sites, almost anything in php will do the same (think ads, etc), so maybe this isn't a great test after all!

    Sean

  9. #9
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    Ahh...Sean, thinking about this a little more, I realize you are right, since Apache would return the HTTP 304 ;-)

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    The SE's don't know, but your server does. Doing mod-rewrite on a high traffic site is just a losing proposition. better to batch out html pages on changes. At least that is my experience..

  11. #11
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    I've seen spiders hit web service scripts 20 - 40k in a day and even some nut trying to download the whole thing hitting it like 100k before he was blocked at the firewall and the server kept ticking. But the mod rewrite was only 7-8 lines so maybe that helped?

  12. #12
    Member SeanW's Avatar
    Join Date
    July 28th, 2005
    Posts
    69
    Quote Originally Posted by chetf
    The SE's don't know, but your server does. Doing mod-rewrite on a high traffic site is just a losing proposition. better to batch out html pages on changes. At least that is my experience..
    Like anything situation in programming, it really depends on how you set it up. Regular expressions, when properly done, can be cheap in terms of CPU.

    Sean

  13. #13
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .

    funny, nobody seems to be concerned here re: 302, 304.
    the SE's, however, are . . .


    heisje


    .

  14. #14
    Member SeanW's Avatar
    Join Date
    July 28th, 2005
    Posts
    69
    Quote Originally Posted by heisje
    funny, nobody seems to be concerned here re: 302, 304.
    the SE's, however, are . . .
    How does that map to what we're doing though? If every page it requests returns a 200 OK code the SE will consider that to be a vaild page. Yes, the SEs consider status codes, but just because we don't use some of them doesn't mean something's not up. Consider any blog out there that generates php output. Every valid page will return 200 and you'll never see a 302/304.

    FWIW, one of my dynamic sites has gone through several changes and I use redirects to send spiders and visitors to the new page. Never had a problem with it.

    Sean

  15. #15
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by heisje
    .
    pretending to be static and actually being static: not the same to SE spiders, for many reasons. one of them: mod_rewrite is a server re-direct, and they can see it. and they do not like it.

    real static does make a difference : ignore this at your risk . . .
    I think the only difference is going to be on your server CPU usage. SE spiders don't need to be concerned with whether a page is static or dynamic. To them a page is a page and neither will be docked SERP-wise. Personally, I think static files are best used as cached entities generated and manipulated solely on the server. It's very time consuming and sloppy to upload static files through FTP.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  16. #16
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    Quote Originally Posted by SeanW
    Like anything situation in programming, it really depends on how you set it up. Regular expressions, when properly done, can be cheap in terms of CPU.

    Sean
    probably the simplest .htaccess file would be something like this:

    RewriteEngine on
    RewriteBase /DIR/
    RewriteRule ^aws\.css$ - [L]
    RewriteRule ^no-img-lg\.gif$ - [L]
    RewriteRule ^images/stars-.*-0\.gif$ - [L]
    RewriteRule (.*) aws.cgi [T=application/x-httpd-cgi]
    it just checks for a css file and a few image files and lets them pass through, anything else is passed through to the script which parses it via environmental variables...thus

    /DIR/books/beckett/page_13.html

    allows the script to grab the path and parse it:
    $books = 'books';
    $author = 'beckett';
    $results_page = '13';

    I don't think you can have much more lower overhead than this ;-)

  17. #17
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .


    v. nice, if you have any use for it . . .

    heisje


    .

  18. #18
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65

  19. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Replies: 6
    Last Post: December 4th, 2007, 09:11 AM
  2. Dynamic Web Source
    By bestdeals in forum Marketing Resources & Power Tools
    Replies: 14
    Last Post: October 11th, 2005, 04:58 PM
  3. Low cost web design source?
    By soundbwoy in forum Midnight Cafe'
    Replies: 4
    Last Post: December 5th, 2002, 12:35 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •