The “The question is dumb” checkbox is missing

 

√ The question is dumb
What a poll should look like

Normally I do not answer telephone based marketing or marketing research, I try to answer polls though, since I think that spreading my opinion even if it ends up as one hundredth of one percent in a final statistic is slightly useful. What is critical for my participation in such polls though, is that I feel that my opinion is actually represented. Lately I have participated in several polls where the poll is fundamentally flawed and would lead to misleading statistics.

Last night I was asked if I wanted to participate in a poll on politics. Since there was no marketing questions mixed in I said ok. The initial questions were quite normal ones, like “which party did you vote for …”, “which party would you vote for …” and so on.

After a while the questions deteriorated, I do not remember all of them, but one that had me quire frustrated was this:

“Do you think the norwegian government should hire private companies for public tasks?”

To me this question is a a case by case question, there is no single right answer for the different activities a government might undertake. Now the poll does not allow this answer though, it only allows “I do not know” or strong or weak versions of “agree” or “disagree”.

In another poll on an earlier occasion, I was asked:


“Do you think syntethic estrogen is more damaging for aquatic life compared to natural estrogen?”

This question was extremely confusing to me. First of all I do not know if there is a difference between syntethic and natural estrogen (or what is meant by syntethic or natural in the question), secondly the only honest and respectable answer I could give is: “Whatever is the current scientific consensus”. The poll instead has me rating how much I agree or disagree.

I really wonder what the statistics derived from these questions were used for. Which dimwit made a decision based on the “knowlege” gained from it.

I could rant about this all day, but In my opinion rants alone are not good, it is better to propose a solution. What I think would make these polls much better is if there always was a “The question is dumb” checkbox, which had to be included by law, and which also had to be included in the final statistic by law. This way no ones voice is completely misrepresented as mine probably was in these cases.

My experience scraping Disqus

To acquire data for my project mentioned here, I wanted to scrape the comments section of a norwegian newspaper.

Disqus comments example
Comments at a norwegian newspaper based on Disqus

Initially I thought it would be as simple as getting the source and applying whatever regular expressions that would give me what I wanted. Sadly it was not that simple.

My chosen norwegian newspaper use Disqus as comment system. Disqus uses javascript quite heavily to generate HTML, this means that the comments are not part of the original source but just exist in the browser DOM. This problem can be solved by using “view generated source” in firefox developer plugin, or by making a plugin for a browser and manipulate the DOM directly. For my problem viewing the generated source was the simpler answer. If I wanted to scrape Disqus often I would make a plugin to dump the generated source.

Firefox generated source
Showing generated source using firefox developer plugin.

Sadly this was not the only problem with scraping Disqus. Disqus is configured to only shows a small amount of comments at a time, and to show more a javascript routine must be invoked several times. I just did it a lot of times manually, though again if I really wanted to scrape Disqus based systems a lot, I would automate this step.

Once the I got the data I load it to a java routine which remove unwanted data based on regular expressions (it is faster to access the DOM, I would do that if I had lots of data). Then I generate descriptive parameters (word count, punctuation amount , etc) based on post content. The resulting parameters are stored as JSON. Then the resulting JSON is used as basis for the scatterplots here.

Adding javascript functionality to WordPress posts

Adding your own javascript implementation to posts in your wordpress blog is slightly more difficult then one might think. The difficulties stem from two things.

  • WordPress editors format html and javascript in posts.
  • The process to use a javascript file in a post is not straightforward.

Avoiding automatic formatting

There are probably several ways to accomplish this, but if found that a plugin called Text Control gives you lots of options as to how your posts are formatted, allowing javascript to be written in posts without a lot of difficulty.

Using javascript in a post

Since I want to use plots often, I choose to add the javascript files to the theme header.php file. The files can be included before the title tag in the header.php. The php command to include javascript files are as follows:

<?php
   wp_enqueue_script('data',plugins_url('/pathtojs/plots.js'));
?> 

This registers the javascript file with WordPress so that it knows where to look when a file referenced in a post. You need to include every javascript file you are using, not just the one you will be using in an eventual post.

Inside posts you only need to add files which contain methods that you use. These files are declared like this:

<script type="text/javascript" src="/pathtoplot/plots.js">
// <![CDATA[// ]]></script>

The javascript code using the declared files go in similar tags:

<script type="text/javascript">// <![CDATA[
//lots of great javascript stuff calling plots.js
// ]]></script>

An example of this in action can be seen in my previous blogpost concerning javascript scatterplots.

Adding interactive scatterplots to your website

To use the scatterplots presented in my previous post, this is what you need.

  • My scatterplot javascript code (LGPL) available for download here.
  • A dataset stored in JSON, with the structure explained below
  • Add .js files to HTML header
  • HTML canvas elements in your html
  • Assignment of data to axes and plots to canvas in javascript

Dataset structure

Datasets for use in my scatterplot code is structured as a JSON array named “dataArray” containing n maps. Each map contains n “name”:value pairs which each is considered a single datapoint. The values to be used in a scatterplot of course need to be numerical. An example of a valid dataset structure is shown below. This is all put in a .js file, and returned by a function named datasetForVisualization() as shown below.

//of course more data in a real dataset
datasetForVisualization() {
   return {"dataArray":
      [{"punctuationprword":2.0,"ekstrem":0.0"},
      {"punctuationprword":6.0,"ekstrem":0.0"},
      {"punctuationprword":0.1111111111111111,"ekstrem":0.0}]
}

Add .js files to head

To add the .js files to your page, add these tags inside the head tag of your website:

<script src="data.js"></script>
<script src="geometry.js"></script>
<script src="guielements.js"></script>
<script src="selection.js"></script>
<script src="plots.js"></script>
<script src="dataset.js"></script>

HTML canvas elements

HTML5 canvas elements are created in the html parts of you website, and are created like this:

<canvas id="plot" width="640" height="480"><canvas>

The id parameter is needed since this is what you need to refer to when drawing to the canvas using javascript.

Assignment to canvas

The following javascript code will assign datasets parts to axes in the scatterplots, as well as tie the plot to a specific canvas and link with another scatterplot

<script type="text/javascript">
   //Select data for plot
   var data = create2DDataFromJson("punctuationprword","ekstrem");

   //Reference to canvas element created above
   var canvas = document.getElementById("plot");

   //Create plot
   var plot = new Scatterplot(canvas,data,540,400,
   "Punctuation pr word","Instances of ekstrem",false);

   //Link plot to other scatterplots
   plot.addLinked(plot2);
   plot2.addLinked(plot);
</script>

Hopefully this explains how to use the scatterplot as part of a webpage. To successfully use it within a WordPress blog is slightly more difficult. One of my next posts will go more into detail about that specifically. If you have any questions about usage please post a comment.

Comments and HTML5 scatterplots

To try and learn some Javascript and use of the HTML5 canvas, I decided to make a visualization tool. After some googling I decided on scatterplots that allow linking and brushing. My idea was to allow such plots to easily be added to any webpage or blog. At the moment they only work well in Webkit based browsers (Chrome, Safari) though.

If you are not familiar with visualization lingo, brushing basically means that you can select datapoints in some fashion. Linking means selections are propagated across two linked views. So if I select some datapoints in one plot the same datapoints are selected in a linked plot. Below is an example of the canvas based scatterplot implementation I ended up with. To try the linking and brushing scroll down to here.

canvas

As you might have understood from the axis names, instead of generating a dataset, I wanted to apply this to some real world data. The idea was inspired by "kommentarfeltdugnad", which aim to not let extremism run unchallengde in public newspaper discussions. I have a hard time finding anything I want to do less then argue in a newspaper comment section, instead i figured studying the people doing this would be more interesting. Using visualization for this was inspired by Correll et al's excellent paper on text visualization. If possible I would like to implement several more of their approaches later on.

Kommentarfeltdugnad also got me thinking about alternative approaches to discussion and forum moderation, especially wondering if it is possible to profile writers over time based on their writing, and ban or even hellban users which pass a certain threshold of badly argued or impossible to decipher posts.

The dataset I'm using in this post was derived from this article due to the amount of comments. All HTML is stripped away and descriptive parameters derived from the remaining comment data.

The aim of the initial exploration of this dataset is to see if it is possible to connect some derived parameters, and possibly tie these to certain usage of some words. You are of course compelled to explore the datasets yourself using the interactive scatterplots below.

I quickly figured out that a large percentage of comments were to short to make out any meaningful statistic. I decided that comments below 50 words were not to be included. Even though 50 is quite arbitrary and decimal oriented.

From the remaining posts I derived several parameters describing each post:

  • Percent of capitals.
  • Percent of punctuation.
  • Amount of words.
  • Percentage of occurrences of specific words (islam, muslim, sosialism, …).
  • Letters in post.

Initially I thought I would start with something obvious as a test of my data data. I think it reasonable to assume a very strong correlation between the amount of letters and words in a post. This can be seen in this plot.

Less obvious is whether there is a correlation between the percent of capitals and an excessive amount of punctuation in a post. Is it so that excessive capitals typers are more likely to either forgo or misuse punctuation?

Even more hypothetical is whether there is a correlation between mentions of islam or muslims and excessive use of capitals?

Below three scatterplots are linked together, with the first showing percentage of capitals and punctuation, the second showing word count and letters in post, the third shows percentage of capitals and percentage of words that are (islam/muslim). This allows for exploration of some of the questions above, as well as some other questions.

canvas2 canvas3 canvas4

There seems to be a small correlation between the amount of capitals and punctuation in a post. It would be very interesting to compare this dataset to a dataset with properly typed text. I also think that punctuation is very broad, and just focusing on exclamation and question marks might also be interesting.

The percentage of capitals and occurrences of islam seems not really related, though curiously, very long posts with a high amount of capitals do not seem to mention islam for some reason.

I guess that concludes my very superficial analysis of this dataset. Feel free to do whatever with the dataset, and once cleaned up I'll link a zip with the JS source for convenience. My next posts will detail how to scrape the data, and how to add scatterplots to your own webpage or wordpress blog.

Finding matrix position from vector address

I have a symmetric matrix, which is stored as a vector containing only the needed elements. Is there an easy way to go from vector address to positions in the matrix?

The same operation on a square matrix stored in this way is done by:

(1)   \begin{equation*} x = v \mod{n} \end{equation*}

and

(2)   \begin{equation*} y = \frac{v}{m} \end{equation*}

Where v is the position in the vector, and m and n matrix columns and rows. This also conveniently reduces to bit shifts and masking if m and n are powers of 2. Operations are of course Integer operations.

My question is whether there exists a similar simple operation on a vector containing just one of the parts of a symmetric matrix.

 

Example of problem
Symmetric matrix stored in vector, not necessarily the best way to store it but one of many possibilities.

One opportunity is to use that the amount of elements needed to store a symmetric matrix is given by:

(3)   \begin{equation*} \frac{n(n+1)}{2} \end{equation*}

Using this one can iteratively search over n until the vector address is contained in the current block of inspected data. Then the remaining elements along with the amount of removed elements give the position. The problem is that this is n operations in a linear search, or log(n) operations if binary search is used.

so I’m wondering if there is an easier way to do this. Basically what I want is a very quick algorithm from vector position to matrix position.

Possibly useful observations are that the result from. n(n+1)/2 can always be divided by the odd component part of n and n+1, as well as the even component divided by two.

Also the resulting triangle can be divided into a structured amount s of square areas of values w:

(4)   \begin{equation*} s = 2^i \end{equation*}

(5)   \begin{equation*} w = \lfloor\frac{(n+2^i)}{2^{(i+1)}}^2\rfloor \end{equation*}

Where i is the squares where 0 is the largest square, and larger i gives the next smaller one. n is from the original matrix m and n. When the result is zero no more squares are needed to describe the area.

How this can be turned into a storage and addressing scheme, I do not know, and I wonder if there is no faster way (the binary search) to recover a position (without storing them beforehand) when something is stored in this manner?

Note that I’m not discussing packing and unpacking the entire matrix here (which would be O(n) using the linear search method from above), but finding the position of a single element from its position in the vector.

The never ending feature and bug fixing phase

At the moment I’m facing what might be the hardest part of game development. I have to make the decision: “The game is finished”. That is with the exception of game breaking bugs that might need fixing later.

This decision is hard, since I at the moment have a quite good and fun game ready, but there is a continuous stream of small changes I could do to make functionality better. None which are really needed for gameplay, but that make the game slightly more enjoyable. It is often really hard to evaluate which of these are important for player, or if it is a strive for perfection that no one will ever notice. Weeding out what is important with a small beta tester staff is hard.

On another front we have decided that the game will be harsh and hard. To use a term often used by Sirlin, it is quite slippery slope. I decided along with my main gameplay tester that it actually does not take that much away from the fun. This was also part of the decision to have an easier difficulty available, which really makes it easy to use the powerful tools you are given. What makes me very happy is that the tools really feel powerful, and can be used for nice tricks that are not all that apparent in the beginning. It is probably the part I’m the most happy with along with orb design.

That’ll be all the game design/development ranting for today

Game and application progress

The last weeks I have been very busy trying to finish my game.

The experience so far has not been great. I’m using a Java library called Slick for rendering and basic container for my game. While I was developing my game I have figured out that a lot of the stuff I have made is already present, but often not supporting the exact function I need.

Some of my code is also quite old and naive, and it does have some bad quirks that I have now hopefully fixed.

Anyway I have now added sound effects for many of the mechanics in the game, and it became even more fun. Sound is actually very hard to get perfect, and I have in some areas settled for not perfect since it would take me way to long to find something better.

As far as my other summer programming is coming along, I have got QT and OpenCL cooperating nicely and my prototype program seems to send the stipples to their correct position, what is left of the main tasks is rendering the stipples. Once that is done I will focus on making a GUI for specifying options for the stippling. This includes.

– Scaling up image to have more weights for the stipples.
– Selecting stipple size.
– Specifying threshold for weight value in initial random distribution of stipples.
– Specifying iteration number.

Game update

Since the semester at school ended I have been focusing on finishing my game, today I got rid of at least 5 issues which has been irritating me:

– Figured out what was causing staircase artifacts in scaled down game windows.

– Added some graphics to be shown while loading.

– Realized that a a screen in the game might need a complete makeover, it currently does not fit the rest of the interface.

– Added text which shows when the game is paused

First Post!

It is alive!

My blog has its first post, and all is well.

As a first post I’d like to share the current status of my game as well as some interesting CG topics I would like to look into during summer.

Artifact

My game is called Artifact, and it is an arcade style game. For those who have played Ambrosia Software’s Barrack, my game is very similar, and it was major inspiration for creating this game. The game itself is pretty similar, but introduces some different challenges and a system for better resource management, making the player more in control of the resources he or she has access to.

At this moment the game is fully playable and quite fun. What is left is balancing, and adding sound. The big issue is sound since this is a topic where I have no experience. Hopefully there will be some progress in that domain during summer, so that I can release the game itself this fall or winter.

Summer CG

During this summer I also have some projects that I’d like to finish. During the last semester I implemented a weighted k-means algorithm for creating stipple images. The algorithm was made using GPU through OpenCL for computation. The algorithm worked nicely in Mac OS X, but when transfered to Windows(the assignment used a windows only application) it stopped working properly and instead started crashing when I used above 200 points, though I had tested it on the same graphics card in Mac OS X with 5000 points. Since I had no time to fix it during the semester and I really want to make it work, this summer I will try and create an application to turn any image or animation into a stippled video or animation. I have decided to make it in C++ and QT since I feel I need to be more comfortable with C++.

I also want to test the image space ambient occlusion on some models, and see how well it would function for different types of games.