Major New Official QVSource Release (1.4.2) Now Available for QlikView

We just published our latest official QVSource release (1.4.2) and it was a big one - in addition to the usual updates and improvements to the connectors we have also added a number of stability, performance and logging enhancements which should be of interest to enterprise users of QlikView.

If you are currently evaluating QVSource and especially if you are a commercial user we would strongly recommend you read this post as it almost certainly contains information which will affect you.

Much of this information was also discussed in our recent blog posts here, here, here and here, but there are also some new updates specific to this latest build.

Last Beta Version Of Certain Connectors

We are planning to take the following Connectors out of beta in the next official build of QVSource (probably in about 6 - 8 weeks time). This is particularly important to commercial users of QVSource who might currently be using these as when they are taken out of beta you will need a commercial licence to continue to use them:

Deprecated Connectors

This will be the last official build of QVSource which contains V1 of the Google Spreadsheet and Google Analytics connectors. Both of these have now been superseded by much improved 'V2' versions which we would recommend you migrate to.

Synchronous, Asynchronous and Batch Request Modes Added

This is a significant new capability in QVSource which was explained in detail in the previous blog post and so will not be covered again here. This update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed.

One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supersedes this.

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

First Version Of New Twitter Connector (V2)

This build has the first version of the new Twitter Connector V2 in which also illustrates the new UI style we are working towards as discussed here. Unfortunately Twitter will start shutting down version 1.0 of their API in early march so all users should start upgrading to this new version ASAP.

First Version of New Sentiment Analysis and Text Analytics Connector (V2)

This build has the first version of the new Sentiment Analysis and Text Analytics Connector in. This also illustrates the new UI style we are working towards as discussed here. We would also recommend users start working with this new version and reporting any issues as it will soon replace V1 of this Connector.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID= parameter to the generated QlikView load script. We recommend setting a variable at the start of your load script, e.g.:

let vQVSourceAppId = 'my_app_identifier';

And then adding this variable to all of your QVSource requests, e.g:

....&appID=$(vQVSourceAppId)&....

This provides an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that in a recent 'nightly build' we actually generated the full appID=$(vQVSourceAppId) in the QVSource generated load script. Unfortunately it was pointed out that this actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector and certain tables in other Connectors) - so now we just include an empty appID= parameter in the URL leaving it to the user to insert a value or variable in the final QlikView script as recommended above. This isn't mandatory or required and it can be left blank.

Other Connector Changes & Updates

  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Google Analytics Connector For QlikView has had a number of minor tweaks and fixes. These are only likely to affect you if you were using the MaxNumberOfPages parameter (i.e. not bringing back all the data for a particular query url).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature.
  • The General JSON/XML API Connector now has a better preview of XML and HTML and also a new checkbox to encrypt the request URL before including in your QlikView script to help secure any sensitive credentials it might contain.
  • The Facebook Fan Pages and Insights Connectors now have buttons to 'Deauthorise' the Connectors (i.e. detach them from your Facebook account).
  • The Facebook Fan Pages Connector has a new QuestionVotes table listing the users who selected each option.
  • The Klout Connector has been tweaked to work more efficiently with the new new async feature. Note that the Klout V1 Connector for QlikView has been removed from QVSource.
  • The FreeAgent Connector has two new tables related to the Trial Balance. We have also removed V1 of this connector and recommend upgrading to the latest V2 version.
  • The first version of the Google DoubleClick For Publishers (DFP) Connector is now included. We hope to add more functionality to this soon. Please get in touch if you are interested.
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.

All these recent features further enhance the speed and ease at which you can load your QlikView applications with data from a large number of web APIs.

We hope you like the updates and we are looking forward to continuing to innovate and improve QVSource - The QlikView API Connector in 2013!

Major New QVSource Nightly Build (1.4.0.4)

Good news - We have just made a major new QVSource release available.

As there have been many significant changes and improvements this is currently only available as a 'nightly build' - this means it's not yet recommended (or supported) for critical production environments but we would recommend all QVSource users to try out this new release on their QVSource powered QlikView applications and report back any feedback or issues you are experiencing.

Where do I find this new release? If you are a QVSource customer or have requested a trial in the past you should see this new release at the personalised download link we should have sent you via email.

Note that it's possible that in the week(s) after this blog post and release is published, we will upload new nightly builds with incremental changes and fixes based on the feedback so please check the download page regularly for further updates.

Once this new release has been out for a few days to a week we will probably push out a dedicated email newsletter so that users who might not necessarily follow this blog or our Twitter, Facebook or other social media accounts, are aware of the new release.

We would then expect this to be released as the new official release some time in February.

The rest of this blog posts outlines the main changes in this latest version.

Before that - If you like what we are doing with QVSource and haven't already, please take a minute to rate us on QlikMarket or leave a recommendation on our LinkedIn product page.

Synchronous, Asynchronous and Batch Request Modes Added

This was explained in detail in the previous blog post and so will not be covered here. However this update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed. One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supercedes this.

(Note: Version 1.4.0.7 released as a nightly build 26/01/13 fixes an issue in the Async feature of this).

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID=$(vQVSourceAppId) parameter to the generated QlikView load script. We recommend setting this variable at the start of your load script (although it is not mandatory) to help provide an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that it was pointed out that appID=$(vQVSourceAppId) actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector) - so in 1.4.0.8 and later we actually just include appID= in the URL leaving it to the user to insert a value or variable in the final QlikView script).

Connector Changes & Updates

  • The Klout V1 Connector for QlikView has been removed from QVSource, please upgrade to the new version. We have also made a couple of fixes and improvements to this new Klout Connector.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.
  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have removed V1 of the FreeAgent Connector for QlikView, please upgrade to the latest V2 version.
  • Added option to encrypt the request url in the General JSON/XML Web API Connector (version 1.4.0.7 released as a nightly build 26/01/13).
  • Deauthorise buttons have been added to the Facebook Fan Pages and Insights Connectors to remove the app from your Facebook profile (version 1.4.0.8 and later).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature(version 1.4.0.8 and later).
  • The Web Preview window in the General JSON/XML Web API Connector should now format XML and HTML properly (version 1.4.0.8 and later).

Synchronous, Asynchronous and Batch Requests - New Performance Options for QVSource

This is the second of two posts outlining some significant changes and improvements we are making to QVSource. The first post discussed the front end UI improvements we are working on. This post covers some back end changes we are making.

This post is essentially a slightly higher level version of the more detailed documentation page you will find here.

QVSource related QlikView script is often characterised by for/next loops in which each iteration makes a request via QVSource, for example to run a search on Twitter or run a text message (Tweet, Facebook post etc.) through a text analytics or sentiment analysis API. The following illustrates a typical request:

LET noRows = NoOfRows('Tweets');
    
for i=0 to $(noRows)-1
     
     let id = peek('Search_id', $(i), 'Tweets');
     let textEncoded = peek('text_urlEncoded', $(i), 'Tweets');
     
     Sentiment:
     LOAD
        '$(id)' as Search_id,
        status as sentiment_status,
        score as sentiment_score
     FROM
     [http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
api=TwitterSentiment&text=$(textEncoded)&appID=$(vQVSourceAppId)format=qvx] (qvx); next

 

The new feature in QVSource allows this code to be rewritten as the following:

let vQVSourceAppId = 'test_app_';
let vWorkingFolder = 'c:\QVSource\Temp\';
let vArgsFile = '$(vWorkingFolder)$(vQVSourceAppId)' & 'SentimentArgs.txt';
 
Params:
LOAD
    Search_id as rowKey,
    Search_text as text,
    'TwitterSentiment' as api,
resident
Tweets;
 
store Params into '$(vParamsFile)' (txt);
drop table Params; // Optional.
 
Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsSync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);

 

This brings us some immediate advantages:

  • The for/next loop is completely eliminated leading, arguably perhaps, to cleaner, easier to maintain code. If the QlikView script execution was being logged to a file, for/next loops with a large number of iterations could create huge log files which is also now avoided.
  • In the above for/next script example note that the text needs to be URL encoded before being sent to the API (this is so that certain characters, for example the #, do not break the GET request to QVSource). This means either having additional complex QlikView script to URL encode the content or ensuring that it was URL encoded before loading into QlikView. This new script version makes this requirement redundant.
  • Similarly, in the above for/next example, very long messages would occasionally break the request due to some maximum URL length (which we believed QlikView to be enforcing). Again, the new functionality makes this issue redundant.
  • In the above for/next  example you are constrained to needing your list of data for the for/next loop already in the QlikView data model. With the method outlined below you can take this data from any file location or even http/https web urls.

The real power of this new approach however is that it allows us additional methods of processing the list of parameters stored in the Params table.

A bit of background on this first.

In the above for/next example, each message would be processed sequentially, meaning that the sentiment score for the next message cannot be calculated until the current request has completed. If we estimate each request takes 250ms then we could be looking at 40 minutes to score 10,000 Tweets (of Facebook posts or whatever) for sentiment.

The thing is, many APIs allow multiple requests to be made in parallel from the same client and these requests will be handled by separate processes on the server (or cloud) allowing many more requests to be processed per second. And the Asynchronous mode described below allows us to take advantage of this feature.

We have already done some testing with asynchronous/parallel requests to many of the sentiment APIs QVSource connects to and shown some huge performance increases using this technique:


Further to this, some APIs also allow multiple requests to be combined into a single API call (i.e. http request), for example sending 1000 text messages in a single request to be processed for sentiment. This is often referred to as sending requests in a batch and it is the new Batch mode, described below, which allows us to take advantage of this feature.

Asynchronous Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsAsync.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsAsync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


QVSource will then run each or the requests in parallel on a pool of threads, and the above QlikView request will only run when all requests have completed.

Please note, that some tables in some connector might explicitly disallow Asynchronous requests (due to lack of support from the underlying API). Please see the more detailed documentation to see how to know when a table supports this.

Batch Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsBatch.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsBatch=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


Please note, that only some tables in some connectors will support Batch mode. Unlike the Asynchronous method the API needs to have specific mechanisms for supporting this and the QVSource Connector needs to be specifically coded to take advantage of it. Please see the more detailed documentation to see how to know when a table supports this.

Auto Mode

See the docs for a description of this.

We hope to have a new nightly build out by the end of the week exposing the initial version of this new functionality for testing. As always we would welcome your feedback and suggestions.

Don't forget that you can get a free fully functional trial of QVSource from our website.

Page List