How to handle massive Analytics data imports

April 4, 2021 by Stefano Leave a Comment

Massive Analytics imports are possible with WP-CLI commands from the console, but there are still some limits even with them. Specifically, it looks like Google limits to 1 million rows the amount of results one can obtain in one request. If your site has a low of pages, and it is likely that the number of URLs for which Analytics has logged visits for the time range you requested exceeds 1 million, then the dataset Post Pay Counter will import will be incomplete. This can be noticed if the number of pending rows after the request is exactly 1 million.

To circumvent this, one can run multiple requests for shorter timeranges and let the plugin merge the data all together. The following bash script will for example import data from 2018-01-01 to 2021-12-31.

sudo -u www-data wp ppc ga clear
for year in {2018..2021}
do
  for month in {1..12}
  do
    lastday=$(date -d "$month/1 + 1 month - 1 day" "+%d") #https://stackoverflow.com/questions/12381501/how-to-use-bash-to-get-the-last-day-of-each-month-for-the-current-year-without-u
    sudo -u www-data wp ppc ga import --time-start="$year-$month-01" --time-end="$year-$month-$lastday"
    sudo -u www-data wp ppc ga process
  done
done

sudo -u www-data wp ppc ga clear

for year in {2018..2021}

for month in {1..12}

lastday=$(date -d "$month/1 + 1 month - 1 day" "+%d") #https://stackoverflow.com/questions/12381501/how-to-use-bash-to-get-the-last-day-of-each-month-for-the-current-year-without-u

sudo -u www-data wp ppc ga import --time-start="$year-$month-01" --time-end="$year-$month-$lastday"

sudo -u www-data wp ppc ga process

done

WP-CLI commands

March 29, 2021 by Stefano Leave a Comment

Post Pay Counter and its addons come with some WP-CLI commands that can be executed from the terminal, especially for demanding tasks. These are useful when one wants to schedule them off peak hours, or when they are too heavy to execute through the web interface without timing out.

There is sufficient documentation in each of them: this is just a list with brief descriptions. Notice that commands that need to write files might need to be run as a different user (i.e. the one owning the plugin directory, most often www-data): use sudo su -u www-data wp [COMMAND].

Post Pay Counter WP-CLI commands

wp ppc stats: generates stats for given time range and author, particularly helpful to cache stats. Check out PPC caching features for details and examples.
wp ppc ga clear: clear current Analytics data and temp files.
wp ppc ga import: retrieves Analytics data for a given time range and stores it in temp files for further processing. This is equivalent to using the feature Settings > Google Analytics Status > Update data. Does not clear data, so if you want to erase current data, wp ppc ga clear is also needed beforehand.
An import run from WP-CLI expects to be processed from WP-CLI as well.
For massive Analytics imports, see here.
wp ppc ga process: processes pending Analytics rows, as it would happen from regular wp-admin refreshes when there are pending rows. No batching as in wp-admin though: it will instead process them all sequentially.
wp ppc ga incremental-update: performs the daily incremental Analytics data update, consisting of both gathering data and processing them.

Extracting selected information from stats table

May 19, 2020 by Stefano 1 Comment

Post Pay Counter already displays payment information nicely in the stats table by default, and allows it to be included in public pages through its flexible shortcode. However, there are custom cases where you might need to extract individual bits/figures from the stats table, raw as they are.

This is easily achieved through the high-level functions exposed by the plugin. The main player here is the function PPC_generate_stats::produce_stats. The following example snippet will extract the total adsense revenue for the given $author_ID:

PPC_general_functions::get_default_stats_time_range( PPC_general_functions::get_settings('general') );
$start_time = $ppc_global_settings['stats_tstart'];
$end_time = $ppc_global_settings['stats_tend'];
 
$author_ID = 1;
$stats = PPC_generate_stats::produce_stats($start_time, $end_time, array( $author_ID ));
$raw_stats = $stats['raw_stats'];
$total_adsense = $raw_stats[$author_ID]['total']['ppc_payment']['due_payment']['adsense_revenues'];

PPC_general_functions::get_default_stats_time_range( PPC_general_functions::get_settings('general') );

$start_time = $ppc_global_settings['stats_tstart'];

$end_time = $ppc_global_settings['stats_tend'];

$author_ID = 1;

$stats = PPC_generate_stats::produce_stats($start_time, $end_time, array( $author_ID ));

$raw_stats = $stats['raw_stats'];

$total_adsense = $raw_stats[$author_ID]['total']['ppc_payment']['due_payment']['adsense_revenues'];

It will work with the default time range, but it can be changed by feeding UNIX timestamps as values of $start_time and $end_time.

Such a call of produce_stats will generate stats only for the given author, and is thus efficient. Taking away the third argument is possible, and will result in posts from any authors to be queried (and is more performant than making multiple calls with different author parameters). It is technically possible to query posts for a set of authors (in fact, notice the array-like third parameter), but there is no UI that exposes that feature in the stats page.

Adding

var_dump($raw_stats[$author_ID]['total']);

will result in exposing the data structure and show all possible choices of payment criteria and information that can be extracted.

Post Pay Counter caching features

November 18, 2019 by Stefano 2 Comments

Post Pay Counter supports caching and is compatible with WordPress caching plugins.
Post Pay Counter mostly implements caching for post stats, plugin settings, user settings and active payment criteria.

Since Post Pay Counter version 2.716 (May 2017), server side caching improves stats performance significantly.

On our tests, with several payment criteria active (words, visits, Facebook shares, comments), we manage to load around 10k posts in under 5 seconds on an average VPS with memcached. Un-cached stats may take up to 15 seconds to show up though, so the performance boost is clear. Post stats caching is active by default, and its status can be tweaked in the Miscellanea > Performance section.

The standard caching caches stats details for each post, but actually rebuilds the stats table for every visit. Websites with massive amounts of data to handle may want to also cache the whole stats page for a given view. In other words, they may take a snapshot of a particular stats page that takes long to load, and then serve it quickly multiple times after.

[Read more…]

Add a custom column to the stats table

December 3, 2018 by Stefano 2 Comments

Sometimes you may want to add an additional custom column to the stats table. This cannot be achieved through native plugin features, but some light custom coding is enough.

This tutorial will provide developers a sketch code of how this can be done. It should be easily be adapted to your needs with some (PHP) coding skills. Be sure to review the code, because there are several optional checks just to showcase them (for example, permission checks).

In the example, we add a custom column to show the user email address.

[Read more…]

Exclude columns from stats in BuddyPress page

November 10, 2017 by Stefano Leave a Comment

The BuddyPress addon allows you to display Post Pay Counter author’s stats in the BuddyPress Member page. However, by default, all columns are displayed. This may include columns that you are not interested into, or that you may not want to show to authors.

Below is a PHP code that allows to exclude some columns from display.

/**
 * Exclude post status and post type from BuddyPress stats table
 */ 
function ppc_buddypress_exclude_stats_columns( $shortcode ) {
	$exclude_columns = ''; //comma separated list of columns to be excluded

	return substr( $shortcode, 0, strlen( $shortcode )-1 ) . ' exclude="'.$exclude_columns.'"]';
}
add_filter( 'ppc_buddypress_stats_shortcode', 'ppc_buddypress_exclude_stats_columns' );

/**

* Exclude post status and post type from BuddyPress stats table

function ppc_buddypress_exclude_stats_columns( $shortcode ) {

$exclude_columns = ''; //comma separated list of columns to be excluded

return substr( $shortcode, 0, strlen( $shortcode )-1 ) . ' exclude="'.$exclude_columns.'"]';

}

add_filter( 'ppc_buddypress_stats_shortcode', 'ppc_buddypress_exclude_stats_columns' );

You should change the following line with the columns IDs you would like to hide:

$exclude_columns = ''; //comma separated list of columns to be excluded

1	$exclude_columns = ''; //comma separated list of columns to be excluded

Valid columns IDs are: post_id, post_title, post_type, post_status, post_publication_date, post_words, post_visits, post_adsense_revenues, post_images, post_comments, post_bonus, post_total_payment, post_due_payment.

[Read more…]

Backup Post Pay Counter data

January 15, 2017 by Stefano Leave a Comment

There are cases in which you may want to backup data from Post Pay Counter (and maybe Post Pay Counter PRO). If you have access to PhpMyAdmin, this can be easily achieved through a handful of custom queries that can pull out all relevant data. Selected rows can then be exported in a sql file, serving as a backup. The three queries are the following:

SELECT * FROM wp_options WHERE option_name IN ("ppc_settings", "ppc_errors", "ppcp_ga_stuff", "ppc_dismissed_notifications", "ppc_addons_list", "_ppcp_payment_history", "ppcp_ga_first_request", "ppcp_ga_last_request")

SELECT * FROM wp_postmeta WHERE meta_key IN ( "_ppcp_ga_visits", "_ppcp_adsense_revenue", "_ppcp_ga_data", "_ppcp_payment_history",  "_ppcp_exclude")

SELECT * FROM wp_usermeta WHERE meta_key IN ( "_ppcp_payment_history" ) OR meta_key LIKE "%ppcp_paypal_email"

SELECT * FROM wp_options WHERE option_name IN ("ppc_settings", "ppc_errors", "ppcp_ga_stuff", "ppc_dismissed_notifications", "ppc_addons_list", "_ppcp_payment_history", "ppcp_ga_first_request", "ppcp_ga_last_request")

SELECT * FROM wp_postmeta WHERE meta_key IN ( "_ppcp_ga_visits", "_ppcp_adsense_revenue", "_ppcp_ga_data", "_ppcp_payment_history", "_ppcp_exclude")

SELECT * FROM wp_usermeta WHERE meta_key IN ( "_ppcp_payment_history" ) OR meta_key LIKE "%ppcp_paypal_email"

These queries will extract ALL Post Pay Counter and Post Pay Counter PRO data (general settings, custom user settings, Analytics authorization information and visits for each post, payment history both for posts and authors, user PayPal emails, excluded posts from stats – it should really include ALL possible database data). Notice that other addons may need additional backup to be done.

You can then use the PhpMyAdmin Export feature to download the sql file for each query and store them safely. Below is an animated example of the process (for one query):

What is the meta key for the user_meta PayPal email address?

August 15, 2016 by Stefano Leave a Comment

The user’s PayPal email address is stored as a WordPress user_meta with the meta_key wp_ppcp_paypal_email.

Note: if you changed the default WP table prefix, the meta_key should be changed to yourprefix_ppcp_paypal_email.

Add custom statuses to Post Pay Counter stats

July 22, 2016 by Stefano Leave a Comment

This is just a quick note for developers who want to allow custom statuses in Post Pay Counter stats. By default PPC allows publish, future, pending and private, but you’ll see it’s not difficult to add your own ones. I’ll assume you are a developer, so I will only provide the details on where to hook and assume you know what to do at that point.

You will need two functions hooking to two different points:

Filter settings retrieval adding your custom statuses to the array of allowed ones. The filter in this case is ppc_get_settings, line 111 of classes/ppc_general_functions_class.php. Have a look at the structure of the array and add your status to the sub-array counting_allowed_post_statuses.
Tweak the WP_Query arguments to include your custom statuses. Here the filter is ppc_get_requested_posts_args, line 100 of classes/ppc_generate_stats_class.php. You want to edit the post_status parameter (which is an array), adding your custom ones.
(Advanced note: don’t make the WP_Query settings-dependent. You can see that by default PPC includes all possible statuses in the WP_Query post_status parameter, and not only active ones. This is to make stats quicker, and only discard irrelevant results in PPC_counting_stuff::data2cash().

This is an example code to allow drafts to be included in stats:

add_filter( 'ppc_get_settings', 'ppc_add_draft_status_settings', 10 );
function ppc_add_draft_status_settings( $settings ) {
    $settings['counting_allowed_post_statuses']['draft'] = 1;
    return $settings;
}

add_filter( 'ppc_get_requested_posts_args', 'ppc_draft_status_wp_query', 10 );
function ppc_draft_status_wp_query ( $status ) {
    $status['post_status'][] = 'draft';
    return $status;
}

add_filter( 'ppc_get_settings', 'ppc_add_draft_status_settings', 10 );

function ppc_add_draft_status_settings( $settings ) {

$settings['counting_allowed_post_statuses']['draft'] = 1;

return $settings;

}

add_filter( 'ppc_get_requested_posts_args', 'ppc_draft_status_wp_query', 10 );

function ppc_draft_status_wp_query ( $status ) {

$status['post_status'][] = 'draft';

return $status;

}

Set up an effective payment flow for lifetime visits

May 27, 2016 by Stefano Leave a Comment

If you already know how to pay your writers per visits, you may want to pay them for all the visits their posts record through all time, not just for the current month. This can be easily achieved through the PRO version mark as paid feature. This tutorial will guide you through how to set up the payment flow in order to achieve that.

How to pay authors per views across multiple months

If you use Google Analytics, make sure you have selected the desired start day for visits. This can be done through the Update Analytics Data feature in the Google Analytics Status box. For example, if you want to pay writers for views since the beginning of 2016, then you need to select 2016/01/01. You may well want to start counting visits since the day you install the plugin, in which case just select that day or don’t do anything.
[Read more…]

Add custom payment types to Post Pay Counter stats

September 28, 2015 by Stefano 4 Comments

Post Pay Counter allows by default paying for words, visits, comments and images. The PRO version allows you to pay for Adsense revenues and to give a custom bonus to each post, whereas other addons allow to further expand the things that can be paid for. But what if you want to add your own custom payment type? Well, PPC has some handy functions that allow you do that with ease.

[Read more…]