Post Pay Counter

The best way to pay authors on WordPress

How to handle massive Analytics data imports

by Leave a Comment

Massive Analytics imports are possible with WP-CLI commands from the console, but there are still some limits even with them. Specifically, it looks like Google limits to 1 million rows the amount of results one can obtain in one request. If your site has a low of pages, and it is likely that the number of URLs for which Analytics has logged visits for the time range you requested exceeds 1 million, then the dataset Post Pay Counter will import will be incomplete. This can be noticed if the number of pending rows after the request is exactly 1 million.

To circumvent this, one can run multiple requests for shorter timeranges and let the plugin merge the data all together. The following bash script will for example import data from 2018-01-01 to 2021-12-31.

sudo -u www-data wp ppc ga clear
for year in {2018..2021}
do
  for month in {1..12}
  do
    lastday=$(date -d "$month/1 + 1 month - 1 day" "+%d") #https://stackoverflow.com/questions/12381501/how-to-use-bash-to-get-the-last-day-of-each-month-for-the-current-year-without-u
    sudo -u www-data wp ppc ga import --time-start="$year-$month-01" --time-end="$year-$month-$lastday"
    sudo -u www-data wp ppc ga process
  done
done

WP-CLI commands

by Leave a Comment

Post Pay Counter and its addons come with some WP-CLI commands that can be executed from the terminal, especially for demanding tasks. These are useful when one wants to schedule them off peak hours, or when they are too heavy to execute through the web interface without timing out.

There is sufficient documentation in each of them: this is just a list with brief descriptions. Notice that commands that need to write files might need to be run as a different user (i.e. the one owning the plugin directory, most often www-data): use sudo su -u www-data wp [COMMAND].

Post Pay Counter WP-CLI commands

  • wp ppc stats: generates stats for given time range and author, particularly helpful to cache stats. Check out PPC caching features for details and examples.
  • wp ppc ga clear: clear current Analytics data and temp files.
  • wp ppc ga import: retrieves Analytics data for a given time range and stores it in temp files for further processing. This is equivalent to using the feature Settings > Google Analytics Status > Update data. Does not clear data, so if you want to erase current data, wp ppc ga clear is also needed beforehand.
    An import run from WP-CLI expects to be processed from WP-CLI as well.
    For massive Analytics imports, see here.
  • wp ppc ga process: processes pending Analytics rows, as it would happen from regular wp-admin refreshes when there are pending rows. No batching as in wp-admin though: it will instead process them all sequentially.
  • wp ppc ga incremental-update: performs the daily incremental Analytics data update, consisting of both gathering data and processing them.

Post Pay Counter caching features

by 2 Comments

Post Pay Counter supports caching and is compatible with WordPress caching plugins.
Post Pay Counter mostly implements caching for post stats, plugin settings, user settings and active payment criteria.

Since Post Pay Counter version 2.716 (May 2017), server side caching improves stats performance significantly.

On our tests, with several payment criteria active (words, visits, Facebook shares, comments), we manage to load around 10k posts in under 5 seconds on an average VPS with memcached. Un-cached stats may take up to 15 seconds to show up though, so the performance boost is clear. Post stats caching is active by default, and its status can be tweaked in the Miscellanea >  Performance section.

The standard caching caches stats details for each post, but actually rebuilds the stats table for every visit. Websites with massive amounts of data to handle may want to also cache the whole stats page for a given view. In other words, they may take a snapshot of a particular stats page that takes long to load, and then serve it quickly multiple times after.

[Read more…]