Hello avid readers!  It’s been over a week since I blogged, I know, but I’ve been cramming for exams.  The CMPT 376 assignment I mentioned a while ago has finished, so I’m no longer required to blog for grades.  Having said that, after exams, I plan to keep blogging, so keep watching the feed :).

Just to tease you a bit: I’ve got a cool little ASP.Net MVC side-project well underway that I’m just dying to show off on the blog, so stick around :P

Also, to the Reports Module users out there, 5.1 is very close to being finished.  There may be a surprise new feature in there too.  Details to follow later :D.

Microsoft is really taking major strides in terms of improving the programming experience for developers on all current and future computing platforms.  That’s why, earlier today, Microsoft released a beta of the new “System.QuantumEntanglement” library.  This library, being considered for integration with .Net 4.1 (which, my sources say, is codenamed “Looflirpa”), provides features for developers working on quantum computers.  I won’t go in to too much detail, but this essentially means working in an environment where objects can be in many different states (possibly an infinite number) and where observation may change the state of an object.  Microsoft has proposed a few options, but I have some additional ideas.

Microsoft’s Proposal

Eilon Lipton, of the ASP.Net MVC team, is proposing a number of new additions to the .Net BCL in 4.1.  The first, is a class called “StringOr<T>” which has the following API:

namespace System.QuantumEntanglement {
    public class StringOr<TOther> {
        public StringOr(string stringValue, TOther otherValue);

        public string StringValue { get; }
        public TOther OtherValue { get; }
    }
}

This class is used to encapsulate a value which may be a string, but may be another value.  This is a common user-input scenario, since almost all user input arrives as strings, but will usually be converted to another data type.  He also proposes a more general class called “SchrodingOr<TDead, TAlive>”

namespace System.QuantumEntanglement {
    public class SchrodingOr<TDead, TAlive> {
        public SchrodingOr(TDead dead, TAlive alive);

        public TAlive Alive { get; }
        public TDead Dead { get; }
    }
}

This generalizes StringOr<T> to support any two types. 

Eilon’s not the only one talking about this, Scott Hanselman (well known Microsoft blogger), and Rob Conery (Author of the .Net Object-Relational Mapper: SubSonic) have also posted on this topic.  So I figured I’d add my comments to the blogosphere.

StringOr and SchrodingOr are great starts, but what if we need to represent objects which may be in 2 states? What about 3 states? Infinite states?  That’s where my proposals come in.  I propose the following additions to Eilon’s library:

Generalizing SchrodingOr

The first is SchrodingOr<T1, T2, T3, T4, T5, … , Tn> (for infinite n).  This allows the developer to represent objects in as many states as they want. 

C# 5.0 Compiler Support

I’m also proposing the following language syntax to help developers work with these types.  Similar to the way the C# compiler converts “int?” to “Nullable<int>” (which would now be replaced with “SchrodingOr<int, Void>” as it more accurately represents the concept of a type which may or may not have a value), the new syntax takes the following:

int?string?bool?DateTime? foo = GetUserInput(...);

And produces output code which looks like this:

SchrodingOr<int, DateTime, bool, string> foo = GetUserInput(...);

Then, we can defer observation until a later time, using the AsA<T>/AsAn<T> method (since Eilon's Alive and Dead properties no longer work in an infinite state environment). If we just want to observe the type and then make a decision, we can use IsA<T>/IsAn<T> methods (of course, both are required, just in case the type starts with a vowel), which returns a boolean indicating if the object is of the type T. For example:

if(foo.IsA<string>()) { return foo.AsA<string;>; }

Probabilistic Observation

Sometimes, we may want to observe an object only if there is a high probability of it being in the right state, to avoid additional quantum variations. To do this, we can use the CouldBeA<T>/CouldBeA<T> methods. These can, optionally, accept a probability threshold beyond which a object is considered in the right state.

Quantum Snapshots

Unfortunately, in between a call to IsA/An/CouldBeA/An and AsA/An, the state of the object may change.  So, we may wish to observe an object and record the state of the object at the time of observation.  Since in observing the object we may change its state, our observations must be recorded in “Snapshots”.  The snapshot is an instance of QuantumSnapshotOf<TSchrodingOr> (where TSchrodingOr must be one of the SchrodingOr<T1, ..., Tn> types), and has a property "Taken", which is a DateTimeOffset (after all, we need precise timing here) containing the exact time that the snapshot was taken.

Here’s an example combining Probabilistic Observation and Snapshots:

if(foo.CouldBeA<string>()) {
	QuantumSnapshotOf<SchrodingOr<int, string>> snapshot = foo.Observe();
	if(snapshot.Taken.Day == 1 && snapshot.Taken.Month == 4) {
		throw new UnreliableObservationException(snapshot, "The data is unreliable");
	}
	return snapshot.AsA<string>();
}

Summary

I think this is really interesting, forward-thinking, stuff from Microsoft. Eilon has done some fantastic work getting the ball rolling, and its up to us to give our feedback! Microsoft has created a section on MSDN for discussion on these new features: http://msdn.microsoft.com/en-US/QuantumEntanglement/Default.aspx

Back in January, I started blogging more frequently due to a course I was taking.  As part of that, I started using Windows Live Writer to write all my blog posts.  I use the DasBlog engine to power my blog, and it supports one of the many blogging APIs that Windows Live Writer uses (I believe it’s MetaWeblog).  So, I hooked it up to my blog, and everything just worked!

My favorite feature is the ability to create and save drafts and then publish them on demand.  I have inspiration at the strangest times, and often in large batches, so I end up with a couple ideas of things to blog in a day, and no ideas the next day.  So, I just create a bunch of drafts, get them ready to publish and then publish them over a longer period of time (like say… a day :P).  For example, I wrote this post 5 minutes after the previous post, but I thought it might overload my readers to post it immediately :).  So, I save a draft and post it a bit later (when I don’t have anything to blog about :D).

If you have a blog, and it supports one of the blogging APIs, and you’re running on Windows, you really should try Windows Live Writer!

So, as an MSDN subscriber (no, I’m not made of money, Microsoft Interns get a free year-long subscription to MSDN for personal use :P), I had access to the Windows 7 public beta a day early.  I decided to go crazy, since I’ve been hearing its really stable, and put the latest OS on both my laptop and my desktop.  (Well, I actually put Windows 7 Server, aka Windows Server 2008 R2 on my desktop).  So, I figured I’d post my first impressions.

Installation

There’s not much to say here, Installation is exactly like Windows Vista, only a little faster.  The only new feature is that Windows 7 Setup prompts you to create a HomeGroup, if you want.  HomeGroups are the new networking construct introduced in Windows 7 designed to make it easier to share files and devices between networked computers.  I haven’t had a change to check that out yet, so I’ll come back to it later.

Initial Impression

Besides a stylish new boot screen, in which four coloured dots dance around before combining to form the Windows logo, the boot process is also identical to Windows Vista.  I did find that it booted up much faster than Vista (though I can’t make an accurate comparison, since my laptop was getting a bit overloaded).  The new taskbar is very cool, and while it is a bit of a knock-off of the OSX Dock, I think Microsoft has (in typical Microsoft fashion) gone above and beyond the OSX experience.  For example, by hovering the mouse over an icon, a list of all the windows belonging to that application appears.  Even better, applications which directly support Windows 7 can add their own “windows” to this list.  For example, even though I only have one IE8 window open, each tab in that window appears as a separate item in the windows list.

By hovering over each thumbnail, that window is brought to focus on the screen, and the rest of the windows become “glass”.

(And yes, I did blank out my Windows Messenger buddies list :P).

Jump lists are another cool feature, but I haven’t had a chance to explore it much.  Essentially, when you right click, or click and drag up on one of these taskbar icons, a jump list appears.

In this case (Internet Explorer 8), my history is displayed.  Applications designed for Windows 7, get a lot of control over this list, but applications which are not designed to support it (PowerShell 2.0 for example) just get a simple default list

I haven’t had much of a chance to explore the rest of the new stuff, so I’ll post more later, but my initial impression is that Windows 7 is just plain awesome :).

One of the courses I took last semester (Fall 2008), was “Software Engineering II”.  In this class, we were required to work in groups to implement a project that the professor specified.  We had to go through the whole process, design, implementation, testing (though we could choose any software process model we wanted: Waterfall, XP, Agile, etc.).  Our group’s project was an application called “Mappr” that would allow users to browse a map.  Well, it was a little more than that, but that’s all the background required by this post, I’ll post more background in future posts.

One of the necessary components in mapping software is called a “Projection”.  The Earth is round, and Latitude and Longitude co-ordinates are spherical measurements representing points on the Earth.  In order to convert those co-ordinates to (x, y) co-ordinates for displaying on a (flat) computer screen, you must project the geographical co-ordinates into screen co-ordinates.  One well-known technique for doing this is called the Mercator Projection.

A quick aside: The Mercator Projection is widely known (in geography circles) for being highly inaccurate.  However, it is the projection used by most road maps, atlases, etc., both physical and digital.

In Mappr, projection is handled by a component called a “Projection Strategy”.  A Projection Strategy is a C# class (Mappr was written in C#) with two methods: GeoToScreen and ScreenToGeo.   Here are the signatures of those methods:

public interface IProjectionStrategy {
    Point GeoToScreen(Point geographicalPoint, int zoomLevel, int tileSize);
    Point ScreenToGeo(Point screenPoint, int zoomLevel, int tileSize);
}

The purpose of each method is straight forward: To take in either Geographical (Latitude, Longitude) co-ordinates or Screen (X, Y) co-ordinates, and convert them to the other.  In order to do this, we must know the Zoom Level, which is an integer N indicating that there are 2^N tiles on the screen.  We also need the size, in pixels, of each map tile image.  This means that the size of the map, in pixels, is given by: 2^zoomLevel * tileSize.

The code to project a geographical point on to the screen is shown below:

public Point GeoToScreen(Point geographicalPoint, int zoomLevel, int tileSize) {
    // Convert to normalized mercator
    double lon = geographicalPoint.X;
    double lat = geographicalPoint.Y;

    if (lon > 180) {
        lon -= 360;
    }

    lon /= 360;
    lon += 0.5;

    lat = 0.5 - ((Math.Log(Math.Tan((Math.PI / 4) + 
                 ((0.5 * Math.PI * lat) / 180))) / Math.PI) / 2.0);

    double scale = (1 << zoomLevel) * tileSize;
    return new Point(lon * scale, lat * scale);
}

This code first normalizes the longitude (X direction) so that it is in the range 0.0 to 1.0 (where 0.0 is the left of the map and 1.0 is the right).  Then it does what I like to call “mathy stuff” (the calculations are taken from similar code written in Java) with the latitude to put it in the same range (0.0 is the top, 1.0 is the bottom).  Finally, we calculate the scale of the map (height/width in pixels, since the map is technically a square) and then we can use the normalized longitude and latitude as ratios of that scale.

The ScreenToGeo method is similar, the code is below.  I won’t describe this, but just provide it for reference.

public Point ScreenToGeo(Point screenPoint, int zoomLevel, int tileSize) {
    int pixelSpan = (1 << zoomLevel) * tileSize;
    double lngWidth = 360.0 / pixelSpan; // width in degrees longitude
    double lng = -180 + (screenPoint.X * lngWidth); // left edge in degrees longitude

    double latHeightMerc = 1.0 / pixelSpan; // height in "normalized" mercator 0,0 top left
    double latMerc = screenPoint.Y * latHeightMerc; // top edge in "normalized" mercator 0,0 top left
    
    // convert top and bottom lat in mercator to degrees
    // note that in fact the coordinates go from about -85 to +85 not -90 to 90!
    double lat = (180 / Math.PI) * ((2 * Math.Atan(Math.Exp(Math.PI * (1 - (2 * latMerc)))))
                       - (Math.PI / 2));

    return new Point(lng, lat);
}

By the way, feel free to use any of the code in this post in your own application. Consider it "Public Domain". However, I would appreciate (but not require) if you would place a comment near it indicating that this blog is the source of the original code.

Hopefully this helps those of you writing mapping applications in C#!  Please post any questions or comments in the comments section!

This is part 2 of my VidSpeak series, where I show off an app I wrote for my multimedia course.  Check out part 1.

To recap from last time, the problem this assignment is trying to solve is to convert video frames to some kind of audio representation.  There are the steps required:

1. Extract the next frame from the video
2. Scale the frame down to 64x64 pixels
3. Make the frame a grayscale image
4. "Quantize" the grayscale frame into 4-bit colour
5. Convert the frame to sound

I covered extracting video frames in Part 1, so now we have to manipulate the image to prepare it for rendering as sound.

2. Scale the frame down to 64x64 pixels

All the transformation steps (2-4) are performed by "filters" which recieve a System.Drawing.Bitmap and return a transformed one (a filter could also modify the incoming Bitmap and then just return the same object).  Scaling is trivial with the "GetThumbnailImage" method on System.Drawing.Image:

public Bitmap ProcessImage(Bitmap input) {
    return new Bitmap(input.GetThumbnailImage(TargetWidth, TargetHeight, null, IntPtr.Zero));
}

3. Make the frame a grayscale image

There may be a function in .Net to do this, but as part of the assignment we were provided with a formula to convert RGB images to 255-shade grayscale images so I decided to do this.  Now, the Bitmap class provides GetPixel and SetPixel methods, but they aren't really very performant when you need to touch every pixel.  Fortunately, the Bitmap class provides a method called "LockPixels" which prevents the Garbage Collector from moving the Bitmap data around in memory.  With that, I made a base class for filters that process a Bitmap on a Pixel-by-Pixel basis:

BitmapData data = input.LockBits(new Rectangle(0, 0, input.Width, input.Height), 
                                 ImageLockMode.ReadWrite, 
                                 PixelFormat.Format24bppRgb);
unsafe {
    for (int col = 0; col < data.Width; col++) {
        for (int row = 0; row < data.Height; row++) {
            PixelData* pixel = (PixelData*)data.Scan0 + col + row * data.Width;
            ProcessPixel(pixel);
        }
    }
}
input.UnlockBits(data);

PixelData is a simple 3-byte struct:

[StructLayout(LayoutKind.Sequential)]
public struct PixelData {
    public byte Blue;
    public byte Green;
    public byte Red;
}

Then, my grayscale image filter just has to implement ProcessPixel

byte convertedValue = (byte) Math.Floor((0.299 * pixel->Red) + 
                                        (0.587 * pixel->Green) + 
                                        (0.114 * pixel->Blue));
pixel->Red = convertedValue;
pixel->Green = convertedValue;
pixel->Blue = convertedValue;

Kinda strange to see the "->" operator in C#, eh?

4. "Quantize" the image to 4-bit colour

Quantization is the process of "compressing a range of values to a single quantum value" (http://en.wikipedia.org/wiki/Quantization_(image_processing)).  In this case, we are quantizing an 8-bit Grayscale image to a 4-bit Grayscale image.  To do this, we take each pixel value, and assign it one of 16 partitions (4-bits can hold values 0 through 15).  The following code (in FixedPartitionQuantizationStrategy) does this (note _partitionSize is 16 in this case).

public byte Quantize(byte input) {
    return (byte)Math.Floor((double)input / _partitionSize);
}

Quantization is done by the QuantizeImageFilter, which uses a similar base class to the Grayscale conversion above.  Then, an IQuantizationStrategy is used to perform the Quantization itself:

protected override unsafe void ProcessPixel(PixelData* pixel) {
    pixel->Red = _strategy.Quantize(pixel->Red);
    pixel->Green = _strategy.Quantize(pixel->Green);
    pixel->Blue = _strategy.Quantize(pixel->Blue);
}

And thats it! We now have an image that is 64x64 pixels in 4-bit Grayscale.  Next we have to render it out to sound.  Stay tuned for Part 3 for that!

I know, its been too long since I blogged, but its pretty busy at school right now :). Anyway, I'm taking a course in Multimedia this semester, and as part of that course, I have to write a program to convert frames in a Video to short Audio clips. I thought it might be interesting to examine how that is done, in C#.  So, over the course of about 3-4 posts, I'll go over the code that I wrote.  I've attached the full project to this post, so you can take a look at it right now.  The GUI app should work, though I can't guarantee it. All I can give it is "Works on My Machine" seal of approval :) 

Here are the steps involved:

1. Extract the next frame from the video
2. Scale the frame down to 64x64 pixels
3. Make the frame a grayscale image
4. "Quantize" the grayscale frame into 4-bit colour
5. Convert the frame to sound

The Code

The Code is a Visual Studio 2008 solution, written for .Net 3.5.  It uses unsafe code for image processing and sound generation, so you can't run it without full trust (i.e. you can't run it off of a network share).

1. Extract the next frame from the video

I used a "Pipeline" (http://en.wikipedia.org/wiki/Pipeline_(software)) architecture, so this phase is handled by a component I call a "Frame Source" which is expected to return a new frame when asked (or return null to signal the end of the input). I used the DirectShow COM library "DexterLib" to do the extraction. DexterLib contains a class called MediaDet (for MediaDetector) which does most of the work. Here's the code for the function which retrieves a frame at a specified timecode (in seconds). 

FYI: "_detector" is an instance of DexterLib.MediaDetClass() ("_detector" is of type IMediaDet), "_streamLength" is the length of of the video stream in seconds, "_bufferHandle" is an IntPtr referring to an unmanaged buffer (allocated with Marshal.AllocHGlobal) to hold the bitmap, and "_bufferSize"/"_frameSize" are the size of the buffer and the size of each video frame (respectively)

// WARNING: This method will destroy the bitmap retrieved in a previous call to this method
public Bitmap GetFrameAtTime(double timeCode) {
    // Get the bitmap at this time
    Bitmap frame = null;
    unsafe {
        byte* bufferPointer = (byte*)_bufferHandle;
        _detector.GetBitmapBits(timeCode, 
                                ref _bufferSize, 
                                ref *bufferPointer, 
                                _frameSize.Width, 
                                _frameSize.Height);
        frame = new Bitmap(_frameSize.Width, // Width
                           _frameSize.Height, // Heigth
                           _frameSize.Width * 3, // Stride
                           PixelFormat.Format24bppRgb, // Pixel Format
                           new IntPtr(bufferPointer + 
                                      Marshal.SizeOf(typeof(BITMAPINFOHEADER)))); // Start of Buffer
    }

    return frame;
}

(Note: If you look at the actualy code, you will notice I snipped out some stuff from the beginning of this function to display it on the blog. The missing code just handles an (experimental) feature I added to allow me to start at any location in the video, rather than always starting at the beginning)

After loading the frame, I have to flip it, because Dexter loads the frame upside-down, fortunately the System.Drawing.Image class provides a RotateFlip method to do just that! I also rotate it 90 degrees clock-wise, so that each row of the transformed image maps to a column of the frame. This makes step 5 easier, since Bitmaps are stored in "row-major" order (http://en.wikipedia.org/wiki/Row-major_order).

To use the Frame Source, all my program has to do is call GetFrameAtTime method passing in a timecode (in seconds).  This is handled in the FrameProcessor by the GetNextFrame method

_source.GetFrameAtTime((DateTime.Now - _startTime).TotalSeconds)

Rather than going frame-by-frame, I'm extracting the next frame by time.  So, if it takes 4 seconds to process a frame, the next frame I take is approximately 4 seconds after the frame I just processed.

Here's the code: VidSpeak.zip (267.21 KB)

Just writing a test post from Windows Live Writer. Look for a future, more detailed post about it!

Here's a map!

ooo...fancy...