New QVSource Release (1.4.4) Available

It's been six weeks since we released 1.4.3.0 and as usual we've been working hard to make QVSource even better to give QlikView users even more power and flexibility in connecting to Web APIs and other data sources.

We are pleased to announce that a new version is now available with quite a few interesting enhancements and also a couple of fixes which may have affected some users:

  • We have fixed a couple of important stability issues:
    • Some users were finding that the connectors were not loaded/showing until the 'Include Beta' checkbox had been toggled at least once.
    • The main settings file was occasionally getting reset and losing, for example, the fact that the EULA had been accepted. This was causing issues especially for server based deployments.
  • We have added a new ListFileRevisions table to the Google Drive Connector.
  • The MongoDB Connector has had some massive performance boosts and can now be queried using JSON.
  • The Twitter Connector Search is no longer limited to a maximum of 15 pages of result (i.e. 15 x 100 = 1500 Tweets).
  • Three new tables (GroupMembers, Files and Events) have been added to the Facebook Pages & Groups Connector. We will be working on support for the new comment replies soon.
  • We have updated the DoubleClick for Publishers (DFP) Connector to use the new API (v201302).
  • BREAKING CHANGE: The Accounts table has been removed from the Facebook Insights connector (this is dues to removal of this feature from the API itself).
  • A new 'Include Depreciated Connectors' checkbox has been added and deprecated connectors are no longer shown by default.

If you are a customer or trialling QVSource and would like to get these enhancements as well as some other general bug fixes and performance improvements, simply go to the download link that you would have received in your QVSource download email.

Or, if you've never tried QVSource and would like to, then you can get a free trial by completing this form.

If you like what we do with QVSource then we'd really appreciate it if you could take a few moments to either rate us on QlikMarket or recommend us on our LinkedIn page. If you've already done this, then thank you!

Don't forget you can always keep up to date with what we're doing by following us on Twitter, Facebook and Google+.

Major New Official QVSource Release (1.4.2) Now Available for QlikView

We just published our latest official QVSource release (1.4.2) and it was a big one - in addition to the usual updates and improvements to the connectors we have also added a number of stability, performance and logging enhancements which should be of interest to enterprise users of QlikView.

If you are currently evaluating QVSource and especially if you are a commercial user we would strongly recommend you read this post as it almost certainly contains information which will affect you.

Much of this information was also discussed in our recent blog posts here, here, here and here, but there are also some new updates specific to this latest build.

Last Beta Version Of Certain Connectors

We are planning to take the following Connectors out of beta in the next official build of QVSource (probably in about 6 - 8 weeks time). This is particularly important to commercial users of QVSource who might currently be using these as when they are taken out of beta you will need a commercial licence to continue to use them:

Deprecated Connectors

This will be the last official build of QVSource which contains V1 of the Google Spreadsheet and Google Analytics connectors. Both of these have now been superseded by much improved 'V2' versions which we would recommend you migrate to.

Synchronous, Asynchronous and Batch Request Modes Added

This is a significant new capability in QVSource which was explained in detail in the previous blog post and so will not be covered again here. This update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed.

One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supersedes this.

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

First Version Of New Twitter Connector (V2)

This build has the first version of the new Twitter Connector V2 in which also illustrates the new UI style we are working towards as discussed here. Unfortunately Twitter will start shutting down version 1.0 of their API in early march so all users should start upgrading to this new version ASAP.

First Version of New Sentiment Analysis and Text Analytics Connector (V2)

This build has the first version of the new Sentiment Analysis and Text Analytics Connector in. This also illustrates the new UI style we are working towards as discussed here. We would also recommend users start working with this new version and reporting any issues as it will soon replace V1 of this Connector.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID= parameter to the generated QlikView load script. We recommend setting a variable at the start of your load script, e.g.:

let vQVSourceAppId = 'my_app_identifier';

And then adding this variable to all of your QVSource requests, e.g:

....&appID=$(vQVSourceAppId)&....

This provides an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that in a recent 'nightly build' we actually generated the full appID=$(vQVSourceAppId) in the QVSource generated load script. Unfortunately it was pointed out that this actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector and certain tables in other Connectors) - so now we just include an empty appID= parameter in the URL leaving it to the user to insert a value or variable in the final QlikView script as recommended above. This isn't mandatory or required and it can be left blank.

Other Connector Changes & Updates

  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Google Analytics Connector For QlikView has had a number of minor tweaks and fixes. These are only likely to affect you if you were using the MaxNumberOfPages parameter (i.e. not bringing back all the data for a particular query url).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature.
  • The General JSON/XML API Connector now has a better preview of XML and HTML and also a new checkbox to encrypt the request URL before including in your QlikView script to help secure any sensitive credentials it might contain.
  • The Facebook Fan Pages and Insights Connectors now have buttons to 'Deauthorise' the Connectors (i.e. detach them from your Facebook account).
  • The Facebook Fan Pages Connector has a new QuestionVotes table listing the users who selected each option.
  • The Klout Connector has been tweaked to work more efficiently with the new new async feature. Note that the Klout V1 Connector for QlikView has been removed from QVSource.
  • The FreeAgent Connector has two new tables related to the Trial Balance. We have also removed V1 of this connector and recommend upgrading to the latest V2 version.
  • The first version of the Google DoubleClick For Publishers (DFP) Connector is now included. We hope to add more functionality to this soon. Please get in touch if you are interested.
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.

All these recent features further enhance the speed and ease at which you can load your QlikView applications with data from a large number of web APIs.

We hope you like the updates and we are looking forward to continuing to innovate and improve QVSource - The QlikView API Connector in 2013!

Major New QVSource Nightly Build (1.4.0.4)

Good news - We have just made a major new QVSource release available.

As there have been many significant changes and improvements this is currently only available as a 'nightly build' - this means it's not yet recommended (or supported) for critical production environments but we would recommend all QVSource users to try out this new release on their QVSource powered QlikView applications and report back any feedback or issues you are experiencing.

Where do I find this new release? If you are a QVSource customer or have requested a trial in the past you should see this new release at the personalised download link we should have sent you via email.

Note that it's possible that in the week(s) after this blog post and release is published, we will upload new nightly builds with incremental changes and fixes based on the feedback so please check the download page regularly for further updates.

Once this new release has been out for a few days to a week we will probably push out a dedicated email newsletter so that users who might not necessarily follow this blog or our Twitter, Facebook or other social media accounts, are aware of the new release.

We would then expect this to be released as the new official release some time in February.

The rest of this blog posts outlines the main changes in this latest version.

Before that - If you like what we are doing with QVSource and haven't already, please take a minute to rate us on QlikMarket or leave a recommendation on our LinkedIn product page.

Synchronous, Asynchronous and Batch Request Modes Added

This was explained in detail in the previous blog post and so will not be covered here. However this update has the potential to give major performance improvements and also change and improve the way QVSource related QlikView script is constructed. One thing to note is that we have now removed the previous mechanism for running asynchronous requests as the new functionality improves and supercedes this.

(Note: Version 1.4.0.7 released as a nightly build 26/01/13 fixes an issue in the Async feature of this).

QVSource Windows Service

QVSource now has a dedicated Windows Service available and documented here. We have always had a well documented and stable way of configuring QVSource to run on a server environment however the new Windows Service represents an improvement over this technique and in time will likely become the recommended and only supported method.

Related to this, we have also now made it so if QVSource is running in server mode you can run QVSource desktop in order to manage connector and other settings without having to then restart the QVSource server instance for changes to be picked up.

Checking the Status of QVSource

It has always been possible to browse to http://localhost:5555/QVSource to get a html page with various pieces of information about QVSource (e.g. the licence state, version, which Connectors were running). We have now tidied this up and added additional information. More importantly, we have added a new XML option at http://localhost:5555/QVSource?format=xml which allows you to check the exact QVSource status from your QlikView load script as explained here.

Log File & Error Page Updates

QVSource now has a much more structured XML file format with additional information added. As before, logging needs to be enabled for these to be generated as explained here. It should now be possible to load these files into QlikView directly for structured reporting.

Related to this, the HTML error page which QVSource sends back has also been enhanced and tidied.

appID Load Script Parameter

QVSource now adds an appID=$(vQVSourceAppId) parameter to the generated QlikView load script. We recommend setting this variable at the start of your load script (although it is not mandatory) to help provide an identifier to QVSource to be used in error logging and tracing. This is explained in more detail here.

Note that it was pointed out that appID=$(vQVSourceAppId) actually breaks the QlikView 'Web File' Wizard (used for example in the General XML/JSON Connector) - so in 1.4.0.8 and later we actually just include appID= in the URL leaving it to the user to insert a value or variable in the final QlikView script).

Connector Changes & Updates

  • The Klout V1 Connector for QlikView has been removed from QVSource, please upgrade to the new version. We have also made a couple of fixes and improvements to this new Klout Connector.
  • We have upgraded the MongoDB Connector for QlikView to use the latest c# driver and also fixed a couple of minor issues.
  • The Google AdSense Connector for QlikView has been upgraded to use the latest API (version 1.2).
  • The Semiocast Connector for QlikView now supports new languages and has some performance improvements.
  • We have removed V1 of the FreeAgent Connector for QlikView, please upgrade to the latest V2 version.
  • Added option to encrypt the request url in the General JSON/XML Web API Connector (version 1.4.0.7 released as a nightly build 26/01/13).
  • Deauthorise buttons have been added to the Facebook Fan Pages and Insights Connectors to remove the app from your Facebook profile (version 1.4.0.8 and later).
  • Empty \Temp folder has been added to the distribution - this is currently the recommended location to store the parameters CSV files used by the new processParams[Sync|Async|Batch] feature(version 1.4.0.8 and later).
  • The Web Preview window in the General JSON/XML Web API Connector should now format XML and HTML properly (version 1.4.0.8 and later).

Synchronous, Asynchronous and Batch Requests - New Performance Options for QVSource

This is the second of two posts outlining some significant changes and improvements we are making to QVSource. The first post discussed the front end UI improvements we are working on. This post covers some back end changes we are making.

This post is essentially a slightly higher level version of the more detailed documentation page you will find here.

QVSource related QlikView script is often characterised by for/next loops in which each iteration makes a request via QVSource, for example to run a search on Twitter or run a text message (Tweet, Facebook post etc.) through a text analytics or sentiment analysis API. The following illustrates a typical request:

LET noRows = NoOfRows('Tweets');
    
for i=0 to $(noRows)-1
     
     let id = peek('Search_id', $(i), 'Tweets');
     let textEncoded = peek('text_urlEncoded', $(i), 'Tweets');
     
     Sentiment:
     LOAD
        '$(id)' as Search_id,
        status as sentiment_status,
        score as sentiment_score
     FROM
     [http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
api=TwitterSentiment&text=$(textEncoded)&appID=$(vQVSourceAppId)format=qvx] (qvx); next

 

The new feature in QVSource allows this code to be rewritten as the following:

let vQVSourceAppId = 'test_app_';
let vWorkingFolder = 'c:\QVSource\Temp\';
let vArgsFile = '$(vWorkingFolder)$(vQVSourceAppId)' & 'SentimentArgs.txt';
 
Params:
LOAD
    Search_id as rowKey,
    Search_text as text,
    'TwitterSentiment' as api,
resident
Tweets;
 
store Params into '$(vParamsFile)' (txt);
drop table Params; // Optional.
 
Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsSync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);

 

This brings us some immediate advantages:

  • The for/next loop is completely eliminated leading, arguably perhaps, to cleaner, easier to maintain code. If the QlikView script execution was being logged to a file, for/next loops with a large number of iterations could create huge log files which is also now avoided.
  • In the above for/next script example note that the text needs to be URL encoded before being sent to the API (this is so that certain characters, for example the #, do not break the GET request to QVSource). This means either having additional complex QlikView script to URL encode the content or ensuring that it was URL encoded before loading into QlikView. This new script version makes this requirement redundant.
  • Similarly, in the above for/next example, very long messages would occasionally break the request due to some maximum URL length (which we believed QlikView to be enforcing). Again, the new functionality makes this issue redundant.
  • In the above for/next  example you are constrained to needing your list of data for the for/next loop already in the QlikView data model. With the method outlined below you can take this data from any file location or even http/https web urls.

The real power of this new approach however is that it allows us additional methods of processing the list of parameters stored in the Params table.

A bit of background on this first.

In the above for/next example, each message would be processed sequentially, meaning that the sentiment score for the next message cannot be calculated until the current request has completed. If we estimate each request takes 250ms then we could be looking at 40 minutes to score 10,000 Tweets (of Facebook posts or whatever) for sentiment.

The thing is, many APIs allow multiple requests to be made in parallel from the same client and these requests will be handled by separate processes on the server (or cloud) allowing many more requests to be processed per second. And the Asynchronous mode described below allows us to take advantage of this feature.

We have already done some testing with asynchronous/parallel requests to many of the sentiment APIs QVSource connects to and shown some huge performance increases using this technique:


Further to this, some APIs also allow multiple requests to be combined into a single API call (i.e. http request), for example sending 1000 text messages in a single request to be processed for sentiment. This is often referred to as sending requests in a batch and it is the new Batch mode, described below, which allows us to take advantage of this feature.

Asynchronous Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsAsync.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsAsync=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


QVSource will then run each or the requests in parallel on a pool of threads, and the above QlikView request will only run when all requests have completed.

Please note, that some tables in some connector might explicitly disallow Asynchronous requests (due to lack of support from the underlying API). Please see the more detailed documentation to see how to know when a table supports this.

Batch Mode

To convert the above script to process the requests asynchronously simply change the parameter name processParamsSync into processParamsBatch.

Sentiment:
LOAD
    rowKey as Search_id,
    status as Sentiment_status,
    score as Sentiment_score,
    language as Sentiment_language
FROM
[http://localhost:5555/QVSource/TextAnalyserV2/?table=Sentiment&
processParamsBatch=$(vParamsFile)&appID=$(vQVSourceAppId)&format=qvx] (qvx);


Please note, that only some tables in some connectors will support Batch mode. Unlike the Asynchronous method the API needs to have specific mechanisms for supporting this and the QVSource Connector needs to be specifically coded to take advantage of it. Please see the more detailed documentation to see how to know when a table supports this.

Auto Mode

See the docs for a description of this.

We hope to have a new nightly build out by the end of the week exposing the initial version of this new functionality for testing. As always we would welcome your feedback and suggestions.

Don't forget that you can get a free fully functional trial of QVSource from our website.

A major milestone but no time to stand still

We achieved a major milestone this month which marked the end of our first year of providing QVSource commercially.

We are considering the first year a great success and are delighted that already all of our earliest customers (excluding one desktop only user) have renewed for the second year - thank you!

We have also maintained the highest rated position on QlikMarket and currently have 18 5-star reviews - if you are a user and like the product please take a minute to rate us.

But it's no time to get complacent - we have been connecting QlikView to APIs since the end of 2010 when QVSource development began, and with all the experience we have gained and lessons we have learned we know we can still do many things better. Year one was about getting the basics right, next we will be focusing on usability and performance.

So this is half of a two part post on some of the improvements we are working on. In this post I'll look at our front end (user interface) and how it's changing and in the next post I'll look at our 'back end' QlikView scripting and how we are planning to simultaneously reduce complexity and improve performance.

Our current connector config UI is perfectly functional and gets the job done but there are a few challenges it still presents:

  • It shows all the input fields all at once, even though some may not be relevant to all the tables you are interested in.
  • It lacks supportive information about the fields and tables.
  • Each connector's UI is manually built which leads to some inconsistencies between connectors.
  • It presents all the tables all of the time, even though 80% of users will typically only use a common subset of tables.

A case in point is the Twitter Connector - here's a screen shot of the current version:

 

Current users will no doubt recognise this and some of the issues outlined above which it presents. I am not picking on this connector, I think all but the simplest of connectors have a similar issue.

So we are now re-developing the connector config user interfaces to have a layout which looks more like this (note this is still in development so the soon to be released version might differ slightly): 

 

A few things to point out.

At the top left a new drop down now allows the user to see different groups of tables. This is because for many connectors, 80% of users will typically only use a common subset of tables so now the first time they use the connector they are presented only the most common or popular ones to avoid information overload. The drop down allows you to select different grouping and of course 'All Tables' will be an option for power users: 

 

Below the table list we now have a new area where additional information about the selected table which could be relevant to the user is shown - here we see an example for the Twitter search table:

 

This will often essentially be us mapping relevant documentation from the underlying API to the QVSource user. Hyperlinks can also be included here to link to our docs or other relevant web pages.

The final big change is the input parameter section - this should now only show input parameters for the currently selected table:

 

Which should make the connector far more intuitive to use. This interface is also now 'auto generated' which means that there will be a much greater level of visual consistency between the connectors and also makes our SDK easier to use as you simply need to define which parameters each table needs and let the framework take care of the rest.

You should also notice that we now have tool tips on some of the fields - where necessary this allows additional helpful information about the fields to be show.

Finally, when the table's data is requested we now have much improved parameter validation before the request even gets to the connector. Here we see what happens when the Search table is requested without entering a query.

 

We hope you like the new changes, as they are still being worked on please let us know any further suggestions or concerns you might have and we will try to address them in the coming weeks.

In the next blog post I'll be discussing some of the changes we are making to simplify and improve the performance of the QlikView scripting for QVSource.

New Twitter Demo For QlikView Available Online And For Download

We've just published online a new version of one of our Twitter applications, which is tracking Tweets based around 3 popular web browsers: Google Chrome, Internet Explorer and Mozilla Firefox. In terms of a data model it's very similar to the previous version but we've given it a fresh new look and made it a bit more iPad friendly i.e. not too wide and easy on the eye.

With this application we tried to look at some interesting metrics that we hadn't explored before, such as at what rate are people joining the conversation, the share of voice each day across the search terms and what are the most influencial people talking about. But also simple things such as identifying and pulling out images that are in the Tweets and the most mentioned users.

We also wanted to improve the message that QlikView and QVSource give users a unique ability to understand what's going on in this relatively new area of analytics

Chained Reloads

This application is a little different to our normal demos because its load script doesn't actually contain any QVSource based script. Instead, it relies on another QlikView application (called the Twitter Demo - Load Script & Data Model application) to do the work using the Twitter Connector and then enriching it for Klout scores and sentiment & text analytics. This 'Primary' application creates the basic data model and generates a number of QVDs that represent different parts of the data model, such as the Klout scores or Twitter accounts being tracked.

The idea we had was that we could build other applications that use this base data but then manipulate, create tables or ignore some data so that new metrics could be developed to meet the specific dashboard needs without having to make unnecessary calls to Twitter or trying to recreate the data model every time for slightly different requirements. 

Since we want the data to be up-to-date we reload the primary application every 10 minutes. But how do you know when to reload the dependant application? Every 10 minutes, but 5 minutes after the first one started? Depending on what you're trying to track this might be fine i.e. low volume Tweets. But it's also risky - what if there's a surge in data and the primary application takes 11 minutes to load? Could the dependant application start to load too soon?

Well, thankfully QlikView has a really simple function that only allows an application to start a reload once another application it depends on has successfully completed its own reload - we call this Chained Reloading.

To do this, you need to go to your QliKView Server Management Console and set up your primary application to reload whenever you need it to.

Then, select the application from the list which you want to reload when the other's finished and select from the scheduled reload options "On Event of Another Task", then "Successful", then the primary application. It's as simple as that!

Get the Application

You can view this application online here or you can download the Twitter pack, which includes the Load Script & Data Model application and instructions to get your data in to it here.

New Blog, Same Great Product

QVSource has a new blog - this blog to be precise!

We have been publishing QVSource related posts over at Qlikster since launching the product and we will continue to echo posts over there for the foreseeable future.

However, this will be the new home of QVSource focused posts going forward and we would encourage any interested users to subscribe to the RSS feed from their favourite feed reader (personally, I love Reeder on my iPad).

Alternatively, you can sign up for email updates (to the right in the current theme although we will hope to bring the style in line with our main website in due course) or just check back here once in a while.

The plan is of course in creating a new blog is to start making more posts for the QVSource user community - let us know if there is anything you would like to see and please check back soon!

In the meantime if you haven't already seen our most recent post about the awesome US Presidential Election demo app built by QlikTech which utilises QVSource you can read it here.

Page List