Page 1 of 2 12 LastLast
Results 1 to 25 of 27
  1. #1
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    Incremental data feeds ...how?
    Hi,

    This is my first post here. I like this site and have been reading and learning a lot.

    One tread here had a neat php script and it works great except I'm on a shared hosting and my memory limit is only 30M.

    Seems my data feed is just little to big or something because I get this error:

    Fatal error: Out of memory (allocated 25427968) (tried to allocate 12543303 bytes)

    In called my hosting company and they said I would have to upgrade to a dedicated server or optimize my script.

    I've been looking into it and it seems all I have to do is somehow...

    1. use incremental data feeds
    2. use LOAD DATA INFILE
    3. Or use some kind of parsing script

    I just don't know enough about php to get it going or if these are really what I need. Is there a way to slow down the data feed transfer so I don't exceed the 30M memory limt?

    Thank you

  2. #2
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    I assume you are using the file() function to load the datafeed in? You should have code that looks something like this....
    $fileptr = fopen("$filename", 'r');
    $thedata = file("$filename");
    foreach ( $thedata as $index => $eachtype )
    {
    $array = explode("|", $eachtype) ;
    misc lines of code
    }
    and you get the overflow error. What you have to do is read the data sequentially using the fgets() function. This is the code I use.

    $fileptr = fopen($filename, 'r'); // Open file
    $thedata = fgets($fileptr,65536); // Get some data
    while (!feof($fileptr)) { // Check if its the end of file
    while (strpos("$thedata","\n") > 0) // See if this is a full record of data
    {
    $temp = strpos("$thedata","\n"); // How many characters are in a record of data
    $eachtype = substr($thedata,0,$temp); // Get a record of data
    $thedata = substr($thedata,$temp+1,66000); // Copy whatever is left
    $array = explode("|", $eachtype) ; // convert record to fields

    misc lines of code
    }
    $thedata = $thedata . fgets($fileptr,65536); // get next grouping of data
    }
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  3. #3
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    Thanks oranges,

    No my script looks nothing like that! I got it from here: How to FTP and Import SAS datafeeds - ABestWeb Affiliate Marketing Forum

    This script you wrote here looks like exactly what I need. I'm trying to work it into the script above.

  4. #4
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    I update my Database from my workstation, is that possible for you?? for example I run upload_feed.asp on my workstation and it uploads the data to mysql database on my host.
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Reno, NV
    Posts
    856
    You can also try using a graphical MySQL interface program called Navicat. It works superbly for uploadiing feeds from your local cpu to a web server. Perhaps this might get around your memory limit.

  6. #6
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Quote Originally Posted by jimh009 View Post
    You can also try using a graphical MySQL interface program called Navicat. It works superbly for uploadiing feeds from your local cpu to a web server. Perhaps this might get around your memory limit.
    Does this program just read the txt file and uploade the data as is or can you create a script/rules to modify the data before it is uploaded??
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Reno, NV
    Posts
    856
    Quote Originally Posted by kse View Post
    Does this program just read the txt file and uploade the data as is or can you create a script/rules to modify the data before it is uploaded??
    That's a good question. All I've ever used it for is uploading files to the MySQL database. You can do a heck of a lot more with the program, but you'd need to contact the Navicat people to see if it can run the types of scripts you want to run.

    Navicat is a visual MySQL interface, so you can do everything with it (and more) that you can do with phpMyAdmin in cPanel.

    Probably the best thing to do is to download the free demo version and see if it does what you want it to do. The product is called Navicat for MySQL.

    Navicat - MySQL Overview - Download the World's Best Oracle Manager, SQL Server GUI, MySQL Front End, SQLite Tool, PostgreSQL GUI & MariaDB Frontend for Windows, Mac OS X & Linux - Download Now!

    (edit - no affiliate or promotional link in this thing, just so you know)
    Last edited by jimh009; December 8th, 2010 at 04:06 PM.

  8. #8
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    Thanks everyone,

    I'm like kse i.e. I want to manipulate the data and plus automate it.

    I've been working on the script, trying to add oranges suggestions. In the process I discovered the problem. Seems the following part of the script causes the memory limit error. When I remove it everything works fine except my "YOURUSERID" number isn't replaced. I'm not sure how to go about fixing that? Here's the part I'm talking about:


    PHP Code:
    $datafeed file_get_contents($destination_file); 
    $fp fopen($destination_file"w"); 
    $replace str_replace("YOURUSERID",$sasid,$datafeed);
    fwrite($fp$replace); 
    fclose($fp); 
    Any suggestions?

    It just erases the file after it loads it to my server.

  9. #9
    Full Member
    Join Date
    November 21st, 2010
    Posts
    230
    MOPB-39-2007:PHP str_replace() Memory Allocation Integer Overflow Vulnerability

    might be your problem based on your information and code. could be the PHP version.

    you may want to explode the contents into an array and then iterate through the array and perform your str_replace on much smaller chunks (per record, basically).

  10. #10
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    bradk, that sounds like advanced stuff for me at my level of understanding ...but fascinating.

    know where i can see a tutorial or some good keywords i could use for a google search?

    thanks

  11. #11
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Quote Originally Posted by Freejoy View Post
    Thanks everyone,

    I'm like kse i.e. I want to manipulate the data and plus automate it.

    I've been working on the script, trying to add oranges suggestions. In the process I discovered the problem. Seems the following part of the script causes the memory limit error. When I remove it everything works fine except my "YOURUSERID" number isn't replaced. I'm not sure how to go about fixing that? Here's the part I'm talking about:


    PHP Code:
    $datafeed file_get_contents($destination_file); 
    $fp fopen($destination_file"w"); 
    $replace str_replace("YOURUSERID",$sasid,$datafeed);
    fwrite($fp$replace); 
    fclose($fp); 
    Any suggestions?

    It just erases the file after it loads it to my server.
    I had the exact same problem running this script on a very large database and inquired here at the forums maybe 6 months to a year ago.

    No one at that time had any better suggestions and -- like you, Freejoy -- my understanding of php is limited enough that I'm afraid bradk's suggestions don't mean anything to me, either.

    Whatever the issue, it really isn't related to using an older version of php, because my sites are on a VPS with the latest stable version of php5 on it.

    I finally just tried eliminating a bunch of the columns in my mysql table that I wasn't going to use, hoping that would decrease the over-all load when I ran the script on that large datafeed. But I don't think it helped much.

    And, yes, the "php/mysql gurus" who responded at the time said it was an issue of having to replace "YOURUSERID" on a line-by-line basis as the script tries to do.

    The only thing I was able to do was, finally, to change the memory alloted to run the script. But I was able to do that or rather have host support people do that (I wasn't sure how myself ) because I'm on a VPS instead of shared hosting.

    I really hope someone will come along and see this thread and give some good suggestions us "php deficient" folks can understand. LOL! (Speaking just for myself there, no offense intended.)
    Generate more fake news.

  12. #12
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    a couple ideas that may or may not be any good

    - add code to your page creation script to replace the userid when the page is built

    - run the script without that line then run another script to update the db

  13. #13
    Full Member
    Join Date
    November 21st, 2010
    Posts
    230
    Quote Originally Posted by Freejoy View Post
    know where i can see a tutorial or some good keywords i could use for a google search?

    thanks
    the commands you'll want to look at are explode and foreach. if you just google either alongside php, you should find the command's entry on php.net which is really the best resource for it all.

    i can't test it, but adapting your code above, I'm thinking something along the lines of:

    Code:
    $datafeed = file_get_contents($destination_file); 
    
    $allitems=explode("\n",$datafeed);
    foreach($allitems as $item)
       $newdatafeed.= str_replace("YOURUSERID",$sasid,$item)."\n";
    
    $fp = fopen($destination_file, "w"); 
    fwrite($fp, $newdatafeed); 
    fclose($fp);
    dunno how much data that will handle and there may be (and probably are) better ways to handle it, but see if that works for you.

  14. #14
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    I'm not good with MySQL at all - so (on a few sites) I run the script, part of which you have in Post 8, right out of the box. I've never had a problem.

    Who's your host? What kind of a plan do you have?
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  15. #15
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    OK OK OK, I see your problem. Here's all the code. Once you understand this, your next step should be putting the data into a database and manipulate that.

    function readsasdatafeed($fn) { \\ $fn is the datafeed file name

    if ($fp = fopen($fn, 'r')) {
    $thedata = fgets($fp,65536);
    while (!feof($fp)) {
    while (strpos("$thedata","\n") > 0)
    {
    $temp = strpos("$thedata","\n");
    $eachtype = substr($thedata,0,$temp);
    $thedata = substr($thedata,$temp+1,66000);
    $array = explode("|", $eachtype) ; // break record into fields

    $ProdName = $array[1]; // $array[1] is the product name
    $pic = $array[6]; // $array[6] is the item graphic etc.
    $Descr = $array[11];
    $Price = $array[7];
    $URL = str_replace("YOURUSERID", $sasid, $array[4]); // $array[4] is the url
    print "$ProdName $Price $URL $pic<br>$Descr<br><br>\r\n";
    }
    $thedata = $thedata . fgets($fp,65536);
    }
    }
    fclose($fp);
    print "<p>Finished $fn</p>";
    }
    Last edited by oranges; December 8th, 2010 at 10:07 PM.
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  16. #16
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    Bill, my host is 1&1, I have the Developer plan.

    Anyway, I got something worked out. writerguy it was a simple solution!

    PHP Code:
    mysql_query("UPDATE $maintable SET Link = replace(Link,'YOURUSERID','46xxx77')"); 
    I just added this to the end of the script.

  17. #17
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    While I have never been a fan of 1&1, that plan should have all the capability you need for a couple of nice, productive sites. However, that response from their service department was poor. Upsell, rather than offer advice or hlep with a solution is not, what I call support or customer service.

    Glad to see you have it figured out. Updating one field throughout the table "after" all else is done is a lot less processor intensive.
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  18. #18
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    Quote Originally Posted by Freejoy View Post
    Bill, my host is 1&1, I have the Developer plan.

    Anyway, I got something worked out. writerguy it was a simple solution!

    PHP Code:
    mysql_query("UPDATE $maintable SET Link = replace(Link,'YOURUSERID','46xxx77')"); 
    I just added this to the end of the script.
    writerguy beat me to the post...

    But anyway, I do the replacement on the $temptable (same thing)

    mysql_query("update $temptable set Link = replace(Link,'YOURUSERID','12345')") or die(mysql_error());

  19. #19
    Full Member
    Join Date
    November 21st, 2010
    Posts
    230
    so it appears what I found was in fact the issue.

  20. #20
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Quote Originally Posted by Mr. Sal View Post
    writerguy beat me to the post...

    But anyway, I do the replacement on the $temptable (same thing)
    So, Sal, if I use your code for the temp table, would I delete this line from the script:

    Code:
    $replace = str_replace("YOURUSERID",$sasid,$datafeed);
    I think that's what I would need to do, right?
    Generate more fake news.

  21. #21
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    Quote Originally Posted by writerguy View Post
    So, Sal, if I use your code for the temp table, would I delete this line from the script:

    Code:
    $replace = str_replace("YOURUSERID",$sasid,$datafeed);
    I think that's what I would need to do, right?
    I only have it on the temp table, and if something goes wrong I want the script to or die(mysql_error()) there...

    and if everything is ok, then continue:

    mysql_query("OPTIMIZE TABLE $temptable") or die(mysql_error());
    mysql_query("drop table $maintable");
    mysql_query("ALTER TABLE $temptable RENAME $maintable") or die(mysql_error());

  22. #22
    Newbie
    Join Date
    December 7th, 2010
    Posts
    6
    Quote Originally Posted by Mr. Sal View Post
    I only have it on the temp table, and if something goes wrong I want the script to or die(mysql_error()) there...

    and if everything is ok, then continue:
    Mr Sal, what I done for "extra credit" was have an error message sent to my email if there was a problem. Like this:

    PHP Code:
    $opti mysql_query("OPTIMIZE TABLE $maintable");
    if(!
    $opti) {
      echo 
    "<span style=\"color:red\"><center>ERROR: OPTIMIZE TABLE $maintable unsuccessful.</center></span>";
      
    $errMsg =  mysql_error();
      
    mail($to_error$subj_error$errMsg$hdrs_error);  
      die ();
    }
    else {
      echo 
    "<span style=\"color:green\"><center>SUCCESS: OPTIMIZE TABLE $maintable successful.</center></span>";

    At the top of the page I added this:

    PHP Code:
    $to_error "emailaddress@hotmail.com";
    $subj_error "ERROR In Naturehills Feed";
    $hdrs_error "From: emailaddress@hotmail.com\n"
    At the bottom of the page I added this to be notified that everything went ok:

    PHP Code:
    echo "<br /><br /><center><p style=\"color:green;font-size:22px\">Naturehills Import Completed Successfully</p></center>";
    $to "emailaddress@hotmail.com";
    $subj "Naturehills Database Table Update Report";
    $mssg "This is an automated email. SUCCESS In Naturehills Feed. Naturehills Data Base Table update has completed successfully.\n\nThe total number or items loaded products was $rowNum.\n\n";
    $hdrs "From: $to\n";
    mail($to$subj$mssg$hdrs); 
    I got the idea here: Building UNIQUE autoupdating bulletproof datafeed sites that sell. - ABestWeb Affiliate Marketing Forum

    Works great!

  23. #23
    Roll Tide mobilebadboy's Avatar
    Join Date
    January 18th, 2005
    Location
    Mobile, Alabama
    Posts
    1,220
    Forgive the code, as I threw it together crudely way back then (when I was working with smaller feeds). That part chokes on large feeds because it's writing to the file and replacing every instance of your ID. It was more or less just supposed to be a quick jumpstart for those that kept asking about doing it.

    I began rewriting it shortly thereafter with mass improvements, but gave up due to lack of feedback. Figured I'd just focus on the improvements of my own datafeed processing instead. I have long since then used the query/update option, previously mentioned. The script is also nowhere near what it was 3 years ago either.

    Shawn Kerr (.com) | Disney World | SEC Football

  24. #24
    ABW Ambassador 2busy's Avatar
    Join Date
    January 17th, 2005
    Location
    Tropical Mountaintop
    Posts
    5,636
    It was more or less just supposed to be a quick jumpstart for those that kept asking about doing it.
    Based on the number of times you see it quoted, mentioned or questioned around here, looks like lots of folks got off the ground with it -or edited versions of it

  25. #25
    ABW Ambassador writerguy's Avatar
    Join Date
    January 17th, 2005
    Location
    Springfield, Missouri, USA
    Posts
    3,248
    Quote Originally Posted by superCool View Post
    a couple ideas that may or may not be any good

    - add code to your page creation script to replace the userid when the page is built

    - run the script without that line then run another script to update the db
    superCool, or anyone else reading this -- can you give me a suggestion about what I would need to put in my page creation script to replace the userid when the page is built?

    Currently, I have this call in the script to pull data out of that monster-sized (125,000+ items) footballfanatics datafeed:

    PHP Code:
    }

    $result mysql_query("SELECT Name,Description,Thumbnail,ProductID,Link,Custom2 FROM footballfan WHERE Custom2='$category' ORDER by Name ASC LIMIT $start$limit") or die (mysql_error());
    while (
    $row mysql_fetch_array($result))
    {
    include(
    "product-display.php"); 
    That code is AFTER I've used the code above to FTP and download the feed and put it into a table on my server. The code I used based on the above thread by mobilebadboy has the line which seems to eat up a ton of server resources:

    PHP Code:
    mysql_query("update $temptable set Link = replace(Link,'YOURUSERID',$sasid)") or die(mysql_error()); 
    I'm afraid I don't know enough PHP/MySQL to figure out how to simply create a table on the server keeping the "YOURUSERID" in the Link field, then change or replace "YOURUSERID" with the actual "91xx2" number which is my SAS user ID number.

    Seems there ought to be a simple way to do that when the script I mentioned at the beginning of this post calls or selects the info out of the table and before it displays the links??

    Confused again!!
    Generate more fake news.

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. HELP! Affilliate Data feeds Don't Have All Data Needed?!
    By garyaggregator in forum Programming / Datafeeds / Tools
    Replies: 6
    Last Post: July 22nd, 2011, 04:40 PM
  2. Replies: 2
    Last Post: October 9th, 2010, 08:59 PM
  3. Data Feeds / Product Feeds
    By doogie18 in forum Commission Junction - CJ
    Replies: 6
    Last Post: December 29th, 2008, 06:23 PM
  4. Data Feeds?
    By cwsteam in forum Rakuten LinkShare - LS
    Replies: 4
    Last Post: July 13th, 2005, 12:53 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •