Major New Official QVSource Release (1.4.2) Now Available for QlikView

We just published our latest official QVSource release (1.4.2) and it was a big one - in addition to the usual updates and improvements to the connectors we have also added a number of stability, performance and logging enhancements which should be of interest to enterprise users of QlikView.

If you are currently evaluating QVSource and especially if you are a commercial user we would strongly recommend you read this post as it almost certainly contains information which will affect you.

Much of this information was also discussed in our recent blog posts here, here, here and here, but there are also some new updates specific to this latest build.

Last Beta Version Of Certain Connectors

We are planning to take the following Connectors out of beta in the next official build of QVSource (probably in about 6 - 8 weeks time). This is particularly important to commercial users of QVSource who might currently be using these as when they are taken out of beta you will need a commercial licence to continue to use them:

Deprecated Connectors

This will be the last official build of QVSource which contains V1 of the Google Spreadsheet and Google Analytics connectors. Both of these have now been superseded by much improved 'V2' versions which we would recommend you migrate to.

Synchronous, Asynchronous and Batch Request Modes Added

This is a significant new capability in QVSource which was explained in detail in the previous blog post and so will not be covered again here. This update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed.

One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supersedes this.

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

First Version Of New Twitter Connector (V2)

This build has the first version of the new Twitter Connector V2 in which also illustrates the new UI style we are working towards as discussed here. Unfortunately Twitter will start shutting down version 1.0 of their API in early march so all users should start upgrading to this new version ASAP.

First Version of New Sentiment Analysis and Text Analytics Connector (V2)

This build has the first version of the new Sentiment Analysis and Text Analytics Connector in. This also illustrates the new UI style we are working towards as discussed here. We would also recommend users start working with this new version and reporting any issues as it will soon replace V1 of this Connector.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID= parameter to the generated QlikView load script. We recommend setting a variable at the start of your load script, e.g.:

let vQVSourceAppId = 'my_app_identifier';

And then adding this variable to all of your QVSource requests, e.g:

....&appID=$(vQVSourceAppId)&....

This provides an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that in a recent 'nightly build' we actually generated the full appID=$(vQVSourceAppId) in the QVSource generated load script. Unfortunately it was pointed out that this actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector and certain tables in other Connectors) - so now we just include an empty appID= parameter in the URL leaving it to the user to insert a value or variable in the final QlikView script as recommended above. This isn't mandatory or required and it can be left blank.

Other Connector Changes & Updates

  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Google Analytics Connector For QlikView has had a number of minor tweaks and fixes. These are only likely to affect you if you were using the MaxNumberOfPages parameter (i.e. not bringing back all the data for a particular query url).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature.
  • The General JSON/XML API Connector now has a better preview of XML and HTML and also a new checkbox to encrypt the request URL before including in your QlikView script to help secure any sensitive credentials it might contain.
  • The Facebook Fan Pages and Insights Connectors now have buttons to 'Deauthorise' the Connectors (i.e. detach them from your Facebook account).
  • The Facebook Fan Pages Connector has a new QuestionVotes table listing the users who selected each option.
  • The Klout Connector has been tweaked to work more efficiently with the new new async feature. Note that the Klout V1 Connector for QlikView has been removed from QVSource.
  • The FreeAgent Connector has two new tables related to the Trial Balance. We have also removed V1 of this connector and recommend upgrading to the latest V2 version.
  • The first version of the Google DoubleClick For Publishers (DFP) Connector is now included. We hope to add more functionality to this soon. Please get in touch if you are interested.
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.

All these recent features further enhance the speed and ease at which you can load your QlikView applications with data from a large number of web APIs.

We hope you like the updates and we are looking forward to continuing to innovate and improve QVSource - The QlikView API Connector in 2013!

New QVSource Nightly Build (1.4.0.8) Available

We have just pushed out a new nightly build of QVSource which has all of the updates noted here plus the following:

All these recent features further enhance the speed and ease at which you can load your QlikView applications with data from a large number of web APIs. We expect this will be the basis of the new official release coming in the next two weeks.

Major New QVSource Nightly Build (1.4.0.4)

Good news - We have just made a major new QVSource release available.

As there have been many significant changes and improvements this is currently only available as a 'nightly build' - this means it's not yet recommended (or supported) for critical production environments but we would recommend all QVSource users to try out this new release on their QVSource powered QlikView applications and report back any feedback or issues you are experiencing.

Where do I find this new release? If you are a QVSource customer or have requested a trial in the past you should see this new release at the personalised download link we should have sent you via email.

Note that it's possible that in the week(s) after this blog post and release is published, we will upload new nightly builds with incremental changes and fixes based on the feedback so please check the download page regularly for further updates.

Once this new release has been out for a few days to a week we will probably push out a dedicated email newsletter so that users who might not necessarily follow this blog or our Twitter, Facebook or other social media accounts, are aware of the new release.

We would then expect this to be released as the new official release some time in February.

The rest of this blog posts outlines the main changes in this latest version.

Before that - If you like what we are doing with QVSource and haven't already, please take a minute to rate us on QlikMarket or leave a recommendation on our LinkedIn product page.

Synchronous, Asynchronous and Batch Request Modes Added

This was explained in detail in the previous blog post and so will not be covered here. However this update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed. One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supercedes this.

(Note: Version 1.4.0.7 released as a nightly build 26/01/13 fixes an issue in the Async feature of this).

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID=$(vQVSourceAppId) parameter to the generated QlikView load script. We recommend setting this variable at the start of your load script (although it is not mandatory) to help provide an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that it was pointed out that appID=$(vQVSourceAppId) actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector) - so in 1.4.0.8 and later we actually just include appID= in the URL leaving it to the user to insert a value or variable in the final QlikView script).

Connector Changes & Updates

  • The Klout V1 Connector for QlikView has been removed from QVSource, please upgrade to the new version. We have also made a couple of fixes and improvements to this new Klout Connector.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.
  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have removed V1 of the FreeAgent Connector for QlikView, please upgrade to the latest V2 version.
  • Added option to encrypt the request url in the General JSON/XML Web API Connector (version 1.4.0.7 released as a nightly build 26/01/13).
  • Deauthorise buttons have been added to the Facebook Fan Pages and Insights Connectors to remove the app from your Facebook profile (version 1.4.0.8 and later).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature(version 1.4.0.8 and later).
  • The Web Preview window in the General JSON/XML Web API Connector should now format XML and HTML properly (version 1.4.0.8 and later).

Synchronous, Asynchronous and Batch Requests - New Performance Options for QVSource

This is the second of two posts outlining some significant changes and improvements we are making to QVSource. The first post discussed the front end UI improvements we are working on. This post covers some back end changes we are making.

This post is essentially a slightly higher level version of the more detailed documentation page you will find here.

QVSource related QlikView script is often characterised by for/next loops in which each iteration makes a request via QVSource, for example to run a search on Twitter or run a text message (Tweet, Facebook post etc.) through a text analytics or sentiment analysis API. The following illustrates a typical request:

LET noRows = NoOfRows('Tweets');
    
for i=0 to $(noRows)-1
     
     let id = peek('Search_id', $(i), 'Tweets');
     let textEncoded = peek('text_urlEncoded', $(i), 'Tweets');
     
     Sentiment:
     LOAD
        '$(id)' as Search_id,
        status as sentiment_status,
        score as sentiment_score
     FROM
     [http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
api=TwitterSentiment&text=$(textEncoded)&appID=$(vQVSourceAppId)format=qvx] (qvx); next

 

The new feature in QVSource allows this code to be rewritten as the following:

let vQVSourceAppId = 'test_app_';
let vWorkingFolder = 'c:\QVSource\Temp\';
let vArgsFile = '$(vWorkingFolder)$(vQVSourceAppId)' & 'SentimentArgs.txt';
 
Params:
LOAD
    Search_id as rowKey,
    Search_text as text,
    'TwitterSentiment' as api,
resident
Tweets;
 
store Params into '$(vParamsFile)' (txt);
drop table Params; // Optional.
 
Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsSync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);

 

This brings us some immediate advantages:

  • The for/next loop is completely eliminated leading, arguably perhaps, to cleaner, easier to maintain code. If the QlikView script execution was being logged to a file, for/next loops with a large number of iterations could create huge log files which is also now avoided.
  • In the above for/next script example note that the text needs to be URL encoded before being sent to the API (this is so that certain characters, for example the #, do not break the GET request to QVSource). This means either having additional complex QlikView script to URL encode the content or ensuring that it was URL encoded before loading into QlikView. This new script version makes this requirement redundant.
  • Similarly, in the above for/next example, very long messages would occasionally break the request due to some maximum URL length (which we believed QlikView to be enforcing). Again, the new functionality makes this issue redundant.
  • In the above for/next  example you are constrained to needing your list of data for the for/next loop already in the QlikView data model. With the method outlined below you can take this data from any file location or even http/https web urls.

The real power of this new approach however is that it allows us additional methods of processing the list of parameters stored in the Params table.

A bit of background on this first.

In the above for/next example, each message would be processed sequentially, meaning that the sentiment score for the next message cannot be calculated until the current request has completed. If we estimate each request takes 250ms then we could be looking at 40 minutes to score 10,000 Tweets (of Facebook posts or whatever) for sentiment.

The thing is, many APIs allow multiple requests to be made in parallel from the same client and these requests will be handled by separate processes on the server (or cloud) allowing many more requests to be processed per second. And the Asynchronous mode described below allows us to take advantage of this feature.

We have already done some testing with asynchronous/parallel requests to many of the sentiment APIs QVSource connects to and shown some huge performance increases using this technique:


Further to this, some APIs also allow multiple requests to be combined into a single API call (i.e. http request), for example sending 1000 text messages in a single request to be processed for sentiment. This is often referred to as sending requests in a batch and it is the new Batch mode, described below, which allows us to take advantage of this feature.

Asynchronous Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsAsync.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsAsync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


QVSource will then run each or the requests in parallel on a pool of threads, and the above QlikView request will only run when all requests have completed.

Please note, that some tables in some connector might explicitly disallow Asynchronous requests (due to lack of support from the underlying API). Please see the more detailed documentation to see how to know when a table supports this.

Batch Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsBatch.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsBatch=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


Please note, that only some tables in some connectors will support Batch mode. Unlike the Asynchronous method the API needs to have specific mechanisms for supporting this and the QVSource Connector needs to be specifically coded to take advantage of it. Please see the more detailed documentation to see how to know when a table supports this.

Auto Mode

See the docs for a description of this.

We hope to have a new nightly build out by the end of the week exposing the initial version of this new functionality for testing. As always we would welcome your feedback and suggestions.

Don't forget that you can get a free fully functional trial of QVSource from our website.

Script Updates In New Twitter Demo Application For QlikView

In our last post we discussed the front end changes in our template QlikView application for Twitter analysis. We have also made a number of changes and improvements to the load script which I wanted to briefly outline.

  • The sentiment analysis can now be run asynchronously which can give huge performance boosts (up to a 90% reduction in load time of the sentiment in some of our tests as discussed here). This is turned on using the vAsyncSentiment variable in the config file.
  • New variables have been added to the config file to turn off sentiment analysis (vGetSentiment) and Klout scoring (vGetKloutScores) as we have seen some use cases where the user only wishes to load matching Tweets into the application.

As usual the full list of changes to the application can be seen on the first tab of the load script:

This latest version of this template application (version 1.0.5.1) is available for download on our wiki here. We would also recommend reading the contents of config.txt which is fully documented and explains some of the other advanced options which this application provides.

You will need QVSource to be able to reload it with your own data and you can register for a free fully functional trial on our website.

New Twitter Demo For QlikView Available Online And For Download

We've just published online a new version of one of our Twitter applications, which is tracking Tweets based around 3 popular web browsers: Google Chrome, Internet Explorer and Mozilla Firefox. In terms of a data model it's very similar to the previous version but we've given it a fresh new look and made it a bit more iPad friendly i.e. not too wide and easy on the eye.

With this application we tried to look at some interesting metrics that we hadn't explored before, such as at what rate are people joining the conversation, the share of voice each day across the search terms and what are the most influencial people talking about. But also simple things such as identifying and pulling out images that are in the Tweets and the most mentioned users.

We also wanted to improve the message that QlikView and QVSource give users a unique ability to understand what's going on in this relatively new area of analytics

Chained Reloads

This application is a little different to our normal demos because its load script doesn't actually contain any QVSource based script. Instead, it relies on another QlikView application (called the Twitter Demo - Load Script & Data Model application) to do the work using the Twitter Connector and then enriching it for Klout scores and sentiment & text analytics. This 'Primary' application creates the basic data model and generates a number of QVDs that represent different parts of the data model, such as the Klout scores or Twitter accounts being tracked.

The idea we had was that we could build other applications that use this base data but then manipulate, create tables or ignore some data so that new metrics could be developed to meet the specific dashboard needs without having to make unnecessary calls to Twitter or trying to recreate the data model every time for slightly different requirements. 

Since we want the data to be up-to-date we reload the primary application every 10 minutes. But how do you know when to reload the dependant application? Every 10 minutes, but 5 minutes after the first one started? Depending on what you're trying to track this might be fine i.e. low volume Tweets. But it's also risky - what if there's a surge in data and the primary application takes 11 minutes to load? Could the dependant application start to load too soon?

Well, thankfully QlikView has a really simple function that only allows an application to start a reload once another application it depends on has successfully completed its own reload - we call this Chained Reloading.

To do this, you need to go to your QliKView Server Management Console and set up your primary application to reload whenever you need it to.

Then, select the application from the list which you want to reload when the other's finished and select from the scheduled reload options "On Event of Another Task", then "Successful", then the primary application. It's as simple as that!

Get the Application

You can view this application online here or you can download the Twitter pack, which includes the Load Script & Data Model application and instructions to get your data in to it here.

Facebook Health Checker Template Application for QlikView Updated

We have just given our template application for Facebook social media analysis in QlikView a review and overhaul, making a number of improvements and fixes.

You can see a live version of this application on our website here:
http://www.qvsource.com/QlikView-Live-Demos/Live-Facebook-Pages-Demo

Where we are running daily reloads of Facebook data for three of the top browsers as an example. But QVSource users can reload this with data from any Facebook Fan Pages or Groups.

So for example if you are a QlikView user with responsibility for your social media strategy you could use this dashboard to monitor data for your brand and those of your top competitors, comparing engagement, sentiment and large number of other metrics which the data model makes available.

It really is that easy - a small config file allows you to specify the Facebook Fan Pages you would like to load data for, as well as a number of other settings including how far you would like to go back in time, which sentiment API(s) you would like to use etc. The application is then designed to be reloaded as often as you like (we often use daily reloads), intelligently accumulating new data into QVDs and providing a rich data model from which to base your own KPIs and dashboards.

(BTW - We also have similar applications for Twitter, Google Analytics, YouTube and a number of other APIs on our wiki).

One of the main changes to this version is the ability to load sentiment and user info asynchronously which can give big performance boosts - we also plan to allow this for posts and comments in the next version as well.

The list of updates and changes are found in the first tab of the load script and also copied below:
* Option added in config to load sentiment asynchronously for better performance (vAsyncSentiment).
* Option added in config to load user info asynchronously for better performance (vLoadPostersAndCommenterInfoAsync).
* Sentiment now only loaded when status = 'OK'.
* Commenter_name not Commenter_username now used in AllUsers data set which fixes an issue where the name for some commenters was empty (when loaded from QVDs).
* AllUsers_language and AllUsers_country added (parsed from locale) (when loaded from QVDs).
* Added Page_loaddate_Month to FB_Page (when loaded from QVDs).
* Comments Per Post vs Date no longer accumulative.
* Likes Per Post vs Date no longer accumulative.
* Removed print and export to Excel icons on charts.
* Added country and language charts for All Users.
* Fixed Sentiment Vs Gender Chart.
* Fixed error in summary count of posts and comments on intro tab.
* Various UI rearrangements and improvements on all tabs and many charts.

QVSource allows QlikView users to connect to literally hundreds of APIs quickly and easily - this application uses the Facebook Fan Pages & Groups Connector and the Sentiment Analysis & Text Analytics Connector. You can grab a free fully functional evaluation of QVSource on our website.

Page List